Pages

Wednesday, December 11, 2013

VxVM vxassist ERROR V-5-1-15304 Cannot allocate space for dco volume

That one gave me a headache and I couldn't find any solution on the internet. So maybe this article might help some one some day.
Story so far: a customer has a RAC envinment based on Veritas CVM. One day a disk was in state FAILING like this:

# vxprint -g oraracdg
Disk group: oraracdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg oraracdg     oraracdg     -        -        -        -        -      -

dm stor01_lun01       emc_clariion1_18 - 629059328 -    FAILING  -      -
dm stor01_lun02       emc_clariion1_43 - 209639920 -    NOHOTUSE -      -
dm stor02_lun01       emc_clariion0_18 - 629059328 -    NOHOTUSE -      -
dm stor02_lun02       emc_clariion0_43 - 209639920 -    NOHOTUSE -      -

v  oraracvol01 fsgen        ENABLED  694371840 -        ACTIVE   -      -
pl oraracvol01-01 oraracvol01 ENABLED 694371840 -       RELOCATE -      -
sd stor01_lun01-01 oraracvol01-01 ENABLED 629059328 0   -        -      -
sd stor01_lun02-01 oraracvol01-01 ENABLED 65312512 629059328 -   -      -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 -       ACTIVE   -      -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0   -        -      -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 -   -      -
dc oraracvol01_dco oraracvol01   -    -        -        -        -      -
v  oraracvol01_dcl1 gen     ENABLED  47872    -         ACTIVE   -      -
pl oraracvol01_dcl1-01 oraracvol01_dcl1 ENABLED 47872 - ACTIVE   -      -
sd stor01_lun02-01 oraracvol01_dcl1-01 ENABLED 47872 0  -        -      -
pl oraracvol01_dcl1-02 oraracvol01_dcl1 ENABLED 47872 - ACTIVE   -      -
sd stor02_lun02-01 oraracvol01_dcl1-02 ENABLED 47872 0  -        -      -


Everytime when I used vxreattach, vxrecover etc. the mirror rebuilds and immediatly the disk stor01_lun01 was in state FAILING again. So I deceided to remove the entire plex and to setup up again. Removing the plex was pretty easy:

# vxplex -g oraacdg -o rm dis oraracvol01-01
# vxprint -g oraracdg
Disk group: oraracdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg oraracdg     oraracdg     -        -        -        -        -      -

dm stor01_lun01       emc_clariion1_18 - 629059328 -    FAILING  -      -
dm stor01_lun02       emc_clariion1_43 - 209639920 -    NOHOTUSE -      -
dm stor02_lun01       emc_clariion0_18 - 629059328 -    NOHOTUSE -      -
dm stor02_lun02       emc_clariion0_43 - 209639920 -    NOHOTUSE -      -

v  oraracvol01 fsgen        ENABLED  694371840 -       ACTIVE    -      -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 -      ACTIVE    -      -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0  -         -      -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 -   -      -
dc oraracvol01_dco oraracvol01   -    -        -        -        -      -
v  oraracvol01_dcl1 gen     ENABLED  47872    -         ACTIVE   -      -
pl oraracvol01_dcl1-01 oraracvol01_dcl1 ENABLED 47872 - ACTIVE   -      -
sd stor01_lun02-01 oraracvol01_dcl1-01 ENABLED 47872 0  -        -      -
pl oraracvol01_dcl1-02 oraracvol01_dcl1 ENABLED 47872 - ACTIVE   -      -
sd stor02_lun02-01 oraracvol01_dcl1-02 ENABLED 47872 0  -        -      -


Removing the FAILING flag for stor01_lun01 was pretty easy too:

# vxedit -g oraracdg set failing=off stor01_lun01

And now it began, setting up the new plex:

# vxassist -g oraracdg -f mirror oraracvol01 stor01_lun01 stor02_lun02
VxVM vxassist ERROR V-5-1-15304 Cannot allocate space for dco volume: Not enough devices for allocation
VxVM vxassist ERROR V-5-1-10127 changing volume oraracvol01:
        Record not in disk group


It took me a long to figure out this issue. After all I found out that the DCO log volume oraracvol01_dco uses space from the original luns (stor01_lun02 and stor02_lun02). The space on the luns was exhausted and so the plex couldn't be attached. To attach the the new (old) plex the DCO volume has to be removed first:

# vxsnap -g oraracdg unprepare oraracvol01
VxVM vxassist ERROR V-5-1-6169  Volume oraracvol01 has drl attach to it, use -f option to remove drl


In case that you get the above error message you can use the -f option as mentioned in the error message:

# vxsnap -f -g oraracdg unprepare oraracvol01

Next take a look at the volume again via vxprint:

# vxprint -g oraracdg
Disk group: oraracdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg oraracdg     oraracdg     -        -        -        -        -      -

dm stor01_lun01       emc_clariion1_18 - 629059328 -    NOHOTUSE -      -
dm stor01_lun02       emc_clariion1_43 - 209639920 -    NOHOTUSE -      -
dm stor02_lun01       emc_clariion0_18 - 629059328 -    NOHOTUSE -      -
dm stor02_lun02       emc_clariion0_43 - 209639920 -    NOHOTUSE -      -

v  oraracvol01 fsgen        ENABLED  694371840 -        ACTIVE   -      -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 -       ACTIVE   -      -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0   -        -      -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 -   -      -


It is just a simple volume with one plex. Now the new (old) plex can be attached again:

# vxassist -g oraracdg -f mirror oraracvol01 stor01_lun01 stor02_lun02

A new task will attach the plex and mirror the volume:

# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
   331           ATCOPY/R 29.40% 0/694371840/204161024 PLXATT oraracvol01 oraracvol01-01 oraracdg smartmove


At the end I new why I couldn't reattach and recover the lun back into the volume at first: the space on all luns were exhausted due to the DCO log volume. After removing the DCO log volume the plex could be reattached and the volume was mirrored again (ut without DCO log volume).