snapmirror job "hung" and abort won't

[R.P. Aditya]

I have a filer pair where the source filer shows this

Snapmirror is on
Source Destination State Lag Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:04:43 Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 01:04:45 Transferring (1352 MB done

and the destination filer, after various attempts to abort a job on th
source/destination shows

Snapmirror is on
Source Destination State Lag Statu
boxcar-vif0:ctfs2008 flatcar:ctfs2008 Snapmirrored 00:04:29 Idl
boxcar-vif0:orabackup flatcar:orabackup Broken-off 150:34:31 Idl
boxcar-vif0:watsadmin flatcar:watsadmin Snapmirrored 01:04:31 Transferrin
(1566 MB done

the boxcar:orabackup to flatcar:orabackup job is in a bad state and sendin

snapmirror abort -h flatcar:orabackup

on the source filer doesn't do anything -- the CLI is unresponsive since tha
command is issued and I have to send commands via ssh and there are snapshot
from the snapmirror that are still busy

Volume orabacku
working..

%/used %/total date nam
---------- ---------- ------------ -------
40% (40%) 28% (28%) Jan 29 15:34 flatcar(0101184681)_orabackup.3664 (busy
40% ( 1%) 28% ( 0%) Jan 29 14:34 flatcar(0101184681)_orabackup.3663 (busy

Netapp support recommended rebooting the source, which seems a bit drasti
(and hard to do midweek) esp. since there are two other snapmirror job
working fine, and in other respects everything is well

The immediate problem is that those snapshots are eating a lot of space and
get

snap delete -a -f orabackup

snap delete -a: Remaining snapshots are currentl
in use by dump
snap restore, SnapMirror, a CIFS share, RAID mirroring, LUNs o
retained by SnapLock
Please try to delete remaining snapshots later

if I try to delete them manually..

This problem started when the destination filer suffered a power outage
presumably in the middle of a snapmirror transfer on the orabackup volume

any ideas short of a reboot

Thanks
Ad

[R.P. Aditya]

based on off-list suggestions, I've tried turning off snapmirror an
unlicensing snapmirror, unfortunately it did not help
Snapmirror is off
Source Destination State Lag Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:11:26 Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 00:11:26 Idl

same for unlicensing and licensing -- a quiesce on the destination file
shows

flatcar> snapmirror quiesce flatcar:orabacku
snapmirror quiesce: in progres
This can be a long-running operation. Use Control - C (^C) to interrupt
snapmirror quiesce: orabackup : destination is not in snapmirrored stat

Regarding another query, this job has been running for months, so it was no
interrupted during a baseline transfer

Thanks
Ad

Post by R.P. Aditya
I have a filer pair where the source filer shows this
Snapmirror is on
Source Destination State Lag Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:04:43 Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 01:04:45 Transferring (1352 MB done
and the destination filer, after various attempts to abort a job on th
source/destination shows
Snapmirror is on
Source Destination State Lag Statu
boxcar-vif0:ctfs2008 flatcar:ctfs2008 Snapmirrored 00:04:29 Idl
boxcar-vif0:orabackup flatcar:orabackup Broken-off 150:34:31 Idl
boxcar-vif0:watsadmin flatcar:watsadmin Snapmirrored 01:04:31 Transferrin
(1566 MB done
the boxcar:orabackup to flatcar:orabackup job is in a bad state and sendin
snapmirror abort -h flatcar:orabackup
on the source filer doesn't do anything -- the CLI is unresponsive since tha
command is issued and I have to send commands via ssh and there are snapshot
from the snapmirror that are still busy
Volume orabacku
working..
%/used %/total date nam
---------- ---------- ------------ -------
40% (40%) 28% (28%) Jan 29 15:34 flatcar(0101184681)_orabackup.3664 (busy
40% ( 1%) 28% ( 0%) Jan 29 14:34 flatcar(0101184681)_orabackup.3663 (busy
Netapp support recommended rebooting the source, which seems a bit drasti
(and hard to do midweek) esp. since there are two other snapmirror job
working fine, and in other respects everything is well
The immediate problem is that those snapshots are eating a lot of space and
get

snap delete -a -f orabackup

snap delete -a: Remaining snapshots are currentl
in use by dump
snap restore, SnapMirror, a CIFS share, RAID mirroring, LUNs o
retained by SnapLock
Please try to delete remaining snapshots later
if I try to delete them manually..
This problem started when the destination filer suffered a power outage
presumably in the middle of a snapmirror transfer on the orabackup volume
any ideas short of a reboot
Thanks
Ad

[Michail Michalakis]

I'd just check that the snapmirror.conf file is correctly configured,
normally if there a problem with snapmirror it could be because of the
snapmirror.con
----- Original Message -----
From: "R.P. Aditya" <***@grot.org
To: <***@mathworks.com
Sent: Tuesday, February 05, 2008 7:36 P
Subject: Re: snapmirror job "hung" and abort won'

Post by R.P. Aditya
based on off-list suggestions, I've tried turning off snapmirror an
unlicensing snapmirror, unfortunately it did not help
Snapmirror is off
Source Destination State Lag
Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:11:26 Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 00:11:26 Idl
same for unlicensing and licensing -- a quiesce on the destination file
shows
flatcar> snapmirror quiesce flatcar:orabacku
snapmirror quiesce: in progres
This can be a long-running operation. Use Control - C (^C) to interrupt
snapmirror quiesce: orabackup : destination is not in snapmirrored stat
Regarding another query, this job has been running for months, so it was no
interrupted during a baseline transfer
Thanks
Ad

Post by R.P. Aditya
I have a filer pair where the source filer shows this
Snapmirror is on
Source Destination State Lag
Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:04:43
Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 01:04:45 Transferring (1352 MB done
and the destination filer, after various attempts to abort a job on th
source/destination shows
Snapmirror is on
Source Destination State Lag
Statu
boxcar-vif0:ctfs2008 flatcar:ctfs2008 Snapmirrored 00:04:29
Idl
boxcar-vif0:orabackup flatcar:orabackup Broken-off 150:34:31
Idl
boxcar-vif0:watsadmin flatcar:watsadmin Snapmirrored 01:04:31
Transferrin
(1566 MB done
the boxcar:orabackup to flatcar:orabackup job is in a bad state and sendin
snapmirror abort -h flatcar:orabacku
on the source filer doesn't do anything -- the CLI is unresponsive since tha
command is issued and I have to send commands via ssh and there are snapshot
from the snapmirror that are still busy
Volume orabacku
working..
%/used %/total date nam
---------- ---------- ------------ -------
40% (40%) 28% (28%) Jan 29 15:34 flatcar(0101184681)_orabackup.3664 (busy
40% ( 1%) 28% ( 0%) Jan 29 14:34 flatcar(0101184681)_orabackup.3663 (busy
Netapp support recommended rebooting the source, which seems a bit drasti
(and hard to do midweek) esp. since there are two other snapmirror job
working fine, and in other respects everything is well
The immediate problem is that those snapshots are eating a lot of space and
get

snap delete -a -f orabacku

snap delete -a: Remaining snapshots are currentl
in use by dump
snap restore, SnapMirror, a CIFS share, RAID mirroring, LUNs o
retained by SnapLock
Please try to delete remaining snapshots later
if I try to delete them manually..
This problem started when the destination filer suffered a power outage
presumably in the middle of a snapmirror transfer on the orabackup volume
any ideas short of a reboot
Thanks
Adi

[De Wit Tom (Consultant)]

Did you already try to put the destination volume offline ? Normall
this also breaks all running transfers (or aborting ones) to tha
volume

This helped me a few times with hanging transfers ..

Grtz
To

-----Original Message----
From: owner-***@mathworks.com [mailto:owner-***@mathworks.com
On Behalf Of R.P. Adity
Sent: dinsdag 5 februari 2008 18:3
To: ***@mathworks.co
Subject: Re: snapmirror job "hung" and abort won'

based on off-list suggestions, I've tried turning off snapmirror an
unlicensing snapmirror, unfortunately it did not help
Snapmirror is off
Source Destination State La
Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:11:2
Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 00:11:2
Idl

same for unlicensing and licensing -- a quiesce on the destination file
shows

flatcar> snapmirror quiesce flatcar:orabacku
snapmirror quiesce: in progres
This can be a long-running operation. Use Control - C (^C) t
interrupt
snapmirror quiesce: orabackup : destination is not in snapmirrored stat

Regarding another query, this job has been running for months, so it wa
no
interrupted during a baseline transfer

Thanks
Ad

Post by R.P. Aditya
I have a filer pair where the source filer shows this
Snapmirror is on
Source Destination State La

Statu

Post by R.P. Aditya
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:04:4

Idl

Post by R.P. Aditya
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 01:04:45 Transferrin

(1352 MB done

Post by R.P. Aditya
and the destination filer, after various attempts to abort a job o

th

Post by R.P. Aditya
source/destination shows
Snapmirror is on
Source Destination State La

Statu

Post by R.P. Aditya
boxcar-vif0:ctfs2008 flatcar:ctfs2008 Snapmirrored 00:04:2

Idl

Post by R.P. Aditya
boxcar-vif0:orabackup flatcar:orabackup Broken-off 150:34:3

Idl

Post by R.P. Aditya
boxcar-vif0:watsadmin flatcar:watsadmin Snapmirrored 01:04:3

Transferrin

Post by R.P. Aditya
(1566 MB done
the boxcar:orabackup to flatcar:orabackup job is in a bad state an

sendin

Post by R.P. Aditya
snapmirror abort -h flatcar:orabackup
on the source filer doesn't do anything -- the CLI is unresponsiv

since tha

Post by R.P. Aditya
command is issued and I have to send commands via ssh and there ar

snapshot

Post by R.P. Aditya
from the snapmirror that are still busy
Volume orabacku
working..
%/used %/total date nam
---------- ---------- ------------ -------
40% (40%) 28% (28%) Jan 29 15:34 flatcar(0101184681)_orabackup.366

(busy

Post by R.P. Aditya
40% ( 1%) 28% ( 0%) Jan 29 14:34 flatcar(0101184681)_orabackup.366

(busy

Post by R.P. Aditya
Netapp support recommended rebooting the source, which seems a bi

drasti

Post by R.P. Aditya
(and hard to do midweek) esp. since there are two other snapmirro

job

Post by R.P. Aditya
working fine, and in other respects everything is well
The immediate problem is that those snapshots are eating a lot o

space and

Post by R.P. Aditya
get

snap delete -a -f orabackup

snap delete -a: Remaining snapshots are currentl
in use by dump
snap restore, SnapMirror, a CIFS share, RAID mirroring, LUNs o
retained by SnapLock
Please try to delete remaining snapshots later
if I try to delete them manually..
This problem started when the destination filer suffered a powe

outage

Post by R.P. Aditya
presumably in the middle of a snapmirror transfer on the orabacku

volume

Post by R.P. Aditya
any ideas short of a reboot
Thanks
Ad

[R.P. Aditya]

Post by De Wit Tom (Consultant)
Did you already try to put the destination volume offline ? Normall
this also breaks all running transfers (or aborting ones) to tha
volume

thanks, tried that, didn't help -- I suspect the problem is with the sourc
filer and probably a lock that isn't getting cleaned up since the snapshot
for the transfer are locked to

Thanks
Ad

Post by De Wit Tom (Consultant)
This helped me a few times with hanging transfers ..
Grtz
To
-----Original Message----
On Behalf Of R.P. Adity
Sent: dinsdag 5 februari 2008 18:3
Subject: Re: snapmirror job "hung" and abort won'
based on off-list suggestions, I've tried turning off snapmirror an
unlicensing snapmirror, unfortunately it did not help
Snapmirror is off
Source Destination State La
Statu
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:11:2
Idl
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 00:11:2
Idl
same for unlicensing and licensing -- a quiesce on the destination file
shows
flatcar> snapmirror quiesce flatcar:orabacku
snapmirror quiesce: in progres
This can be a long-running operation. Use Control - C (^C) t
interrupt
snapmirror quiesce: orabackup : destination is not in snapmirrored stat
Regarding another query, this job has been running for months, so it wa
no
interrupted during a baseline transfer
Thanks
Ad

Post by R.P. Aditya
I have a filer pair where the source filer shows this
Snapmirror is on
Source Destination State La

Statu

Post by R.P. Aditya
boxcar:ctfs2008 flatcar:ctfs2008 Source 00:04:4

Idl

Post by R.P. Aditya
boxcar:orabackup flatcar:orabackup Source - Abortin
boxcar:watsadmin flatcar:watsadmin Source 01:04:45 Transferrin

(1352 MB done

Post by R.P. Aditya
and the destination filer, after various attempts to abort a job o

th

Post by R.P. Aditya
source/destination shows
Snapmirror is on
Source Destination State La

Statu

Post by R.P. Aditya
boxcar-vif0:ctfs2008 flatcar:ctfs2008 Snapmirrored 00:04:2

Idl

Post by R.P. Aditya
boxcar-vif0:orabackup flatcar:orabackup Broken-off 150:34:3

Idl

Post by R.P. Aditya
boxcar-vif0:watsadmin flatcar:watsadmin Snapmirrored 01:04:3

Transferrin

Post by R.P. Aditya
(1566 MB done
the boxcar:orabackup to flatcar:orabackup job is in a bad state an

sendin

Post by R.P. Aditya
snapmirror abort -h flatcar:orabackup
on the source filer doesn't do anything -- the CLI is unresponsiv

since tha

Post by R.P. Aditya
command is issued and I have to send commands via ssh and there ar

snapshot

Post by R.P. Aditya
from the snapmirror that are still busy
Volume orabacku
working..
%/used %/total date nam
---------- ---------- ------------ -------
40% (40%) 28% (28%) Jan 29 15:34 flatcar(0101184681)_orabackup.366

(busy

Post by R.P. Aditya
40% ( 1%) 28% ( 0%) Jan 29 14:34 flatcar(0101184681)_orabackup.366

(busy

Post by R.P. Aditya
Netapp support recommended rebooting the source, which seems a bi

drasti

Post by R.P. Aditya
(and hard to do midweek) esp. since there are two other snapmirro

job

Post by R.P. Aditya
working fine, and in other respects everything is well
The immediate problem is that those snapshots are eating a lot o

space and

Post by R.P. Aditya
get

snap delete -a -f orabackup

snap delete -a: Remaining snapshots are currentl
in use by dump
snap restore, SnapMirror, a CIFS share, RAID mirroring, LUNs o
retained by SnapLock
Please try to delete remaining snapshots later
if I try to delete them manually..
This problem started when the destination filer suffered a powe

outage

Post by R.P. Aditya
presumably in the middle of a snapmirror transfer on the orabacku

volume.

Post by R.P. Aditya
any ideas short of a reboot?
Thanks,
Adi