That one gave me a headache and I couldn't find any solution on the internet. So maybe this article might help some one some day.
Story so far: a customer has a RAC envinment based on Veritas CVM. One day a disk was in state FAILING like this:
# vxprint -g oraracdg
Disk group: oraracdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg oraracdg oraracdg - - - - - -
dm stor01_lun01 emc_clariion1_18 - 629059328 - FAILING - -
dm stor01_lun02 emc_clariion1_43 - 209639920 - NOHOTUSE - -
dm stor02_lun01 emc_clariion0_18 - 629059328 - NOHOTUSE - -
dm stor02_lun02 emc_clariion0_43 - 209639920 - NOHOTUSE - -
v oraracvol01 fsgen ENABLED 694371840 - ACTIVE - -
pl oraracvol01-01 oraracvol01 ENABLED 694371840 - RELOCATE - -
sd stor01_lun01-01 oraracvol01-01 ENABLED 629059328 0 - - -
sd stor01_lun02-01 oraracvol01-01 ENABLED 65312512 629059328 - - -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 - ACTIVE - -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0 - - -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 - - -
dc oraracvol01_dco oraracvol01 - - - - - -
v oraracvol01_dcl1 gen ENABLED 47872 - ACTIVE - -
pl oraracvol01_dcl1-01 oraracvol01_dcl1 ENABLED 47872 - ACTIVE - -
sd stor01_lun02-01 oraracvol01_dcl1-01 ENABLED 47872 0 - - -
pl oraracvol01_dcl1-02 oraracvol01_dcl1 ENABLED 47872 - ACTIVE - -
sd stor02_lun02-01 oraracvol01_dcl1-02 ENABLED 47872 0 - - -
Everytime when I used vxreattach, vxrecover etc. the mirror rebuilds and immediatly the disk stor01_lun01 was in state FAILING again. So I deceided to remove the entire plex and to setup up again. Removing the plex was pretty easy:
# vxplex -g oraacdg -o rm dis oraracvol01-01
# vxprint -g oraracdg
Disk group: oraracdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg oraracdg oraracdg - - - - - -
dm stor01_lun01 emc_clariion1_18 - 629059328 - FAILING - -
dm stor01_lun02 emc_clariion1_43 - 209639920 - NOHOTUSE - -
dm stor02_lun01 emc_clariion0_18 - 629059328 - NOHOTUSE - -
dm stor02_lun02 emc_clariion0_43 - 209639920 - NOHOTUSE - -
v oraracvol01 fsgen ENABLED 694371840 - ACTIVE - -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 - ACTIVE - -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0 - - -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 - - -
dc oraracvol01_dco oraracvol01 - - - - - -
v oraracvol01_dcl1 gen ENABLED 47872 - ACTIVE - -
pl oraracvol01_dcl1-01 oraracvol01_dcl1 ENABLED 47872 - ACTIVE - -
sd stor01_lun02-01 oraracvol01_dcl1-01 ENABLED 47872 0 - - -
pl oraracvol01_dcl1-02 oraracvol01_dcl1 ENABLED 47872 - ACTIVE - -
sd stor02_lun02-01 oraracvol01_dcl1-02 ENABLED 47872 0 - - -
Removing the FAILING flag for stor01_lun01 was pretty easy too:
# vxedit -g oraracdg set failing=off stor01_lun01
And now it began, setting up the new plex:
# vxassist -g oraracdg -f mirror oraracvol01 stor01_lun01 stor02_lun02
VxVM vxassist ERROR V-5-1-15304 Cannot allocate space for dco volume: Not enough devices for allocation
VxVM vxassist ERROR V-5-1-10127 changing volume oraracvol01:
Record not in disk group
It took me a long to figure out this issue. After all I found out that the DCO log volume oraracvol01_dco uses space from the original luns (stor01_lun02 and stor02_lun02). The space on the luns was exhausted and so the plex couldn't be attached. To attach the the new (old) plex the DCO volume has to be removed first:
# vxsnap -g oraracdg unprepare oraracvol01
VxVM vxassist ERROR V-5-1-6169 Volume oraracvol01 has drl attach to it, use -f option to remove drl
In case that you get the above error message you can use the -f option as mentioned in the error message:
# vxsnap -f -g oraracdg unprepare oraracvol01
Next take a look at the volume again via vxprint:
# vxprint -g oraracdg
Disk group: oraracdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg oraracdg oraracdg - - - - - -
dm stor01_lun01 emc_clariion1_18 - 629059328 - NOHOTUSE - -
dm stor01_lun02 emc_clariion1_43 - 209639920 - NOHOTUSE - -
dm stor02_lun01 emc_clariion0_18 - 629059328 - NOHOTUSE - -
dm stor02_lun02 emc_clariion0_43 - 209639920 - NOHOTUSE - -
v oraracvol01 fsgen ENABLED 694371840 - ACTIVE - -
pl oraracvol01-02 oraracvol01 ENABLED 694371840 - ACTIVE - -
sd stor02_lun01-01 oraracvol01-03 ENABLED 629059328 0 - - -
sd stor02_lun02-01 oraracvol01-03 ENABLED 65312512 629059328 - - -
It is just a simple volume with one plex. Now the new (old) plex can be attached again:
# vxassist -g oraracdg -f mirror oraracvol01 stor01_lun01 stor02_lun02
A new task will attach the plex and mirror the volume:
# vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
331 ATCOPY/R 29.40% 0/694371840/204161024 PLXATT oraracvol01 oraracvol01-01 oraracdg smartmove
At the end I new why I couldn't reattach and recover the lun back into the volume at first: the space on all luns were exhausted due to the DCO log volume. After removing the DCO log volume the plex could be reattached and the volume was mirrored again (ut without DCO log volume).
No comments:
Post a Comment