Monday, June 24, 2013

VxVM vxvol ERROR V-5-1-607 Diskgroup oracledg not found

Last night a customer had a blackout in his small datacenter. Luckily all machines came up again without any problem. The only problem he figured was his veritas environment. Volumes seemed to be mounted without content, disk groups could not be found etc.

The first thing I deceided to do was to unmount all volumes. First I listed all volumes in the Oracle disk group: 

# mount | grep oracledg
/u04 on /dev/vx/dsk/oracledg/oraclevol01 ioerror=mwdisable,largefiles,delaylog,dev=2005dc0 on Wed Sep 26 07:38:46 2012

Then I unmounted the above volume (the disk group had only one volume which made my life easier):

# umount /u04
# mount | grep oracledg

After the volume was unmounted I needed to stop the volume:

# vxvol -g oracledg stopall
VxVM vxvol ERROR V-5-1-607 Diskgroup oracledg not found

And that was the funny part: the volume was mounted (the mountpoint was empty) and the diskgroup was not available. I couldn't look anymore but I'm pretty sure that /etc/mnttab (HP-UX) had an 'ghost'-entry that /dev/vx/dsk/oracledg/oraclevol01 was still mounted on /u04. Anyway, I deceided to continue with my stopping-and-restarting-everything-manually-action-plan. The next logical step was to deport the disk group:

# vxdg -g oracledg deport

To restart everything I imported the disk group again:

# vxdg -g oracledg import

And started all volumes:

# vxvol -g oracledg startall

After that the volume was up and running again and I was able to mount /u04 again (after a filesystem check):

# mount -F vxfs /dev/vx/dsk/oracledg/oraclevol01 /u04
UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/oracledg/oraclevol01 is corrupted. needs checking
# fsck /dev/vx/dsk/oracledg/oraclevol01
file system is clean - log replay is not required
# mount -F vxfs /dev/vx/dsk/oracledg/oraclevol01 /u04