Failed verify for the label standard: expected=3 actual=0

dmitrylitvintsev · 20 March 2023 15:17

Hello,

I am working with a test instance of CTA at Fermilab.

I am extremely new to CTA so my apologies if there is something very obvious I am not seeing.

I see cta-taped spewing this error in perpetuity :

Mar 20 10:13:26 gmv18018 cta-taped: LVL="ERROR" PID="203853" TID="203961" MSG="Failed to start tape write session" thread="TapeWrite" tapeDrive="LTO8D0" tapeVid="VR5771"
 mountId="3463" vo="vo" mediaType="LTO8" tapePool="ctasystest" logicalLibrary="TS4500G1" mountType="ArchiveForUser" vendor="ibm" capacityInBytes="12000000000000" lastFse
q="0" compression="1" useLbp="1" exceptionMessage="Failed verify for the label standard: expected=3 actual=0 /lib64/libctacommon.so.0(cta::exception::Backtrace::Backtrac
e(bool)+0x69) [0x7fbddbbb13d9] /lib64/libctacommon.so.0(cta::exception::Exception::Exception(std::string const&, bool)+0x89) [0x7fbddbbb2c3d] /usr/bin/cta-taped() [0x576
00a] /usr/bin/cta-taped() [0x57d318] /usr/bin/cta-taped() [0x5083d5] /usr/bin/cta-taped() [0x501aa5] /usr/bin/cta-taped() [0x503e48] /lib64/libctacommon.so.0(cta::thread
ing::Thread::pthread_runner(void*)+0xef) [0x7fbddbbd8b4d] /lib64/libpthread.so.0(+0x7ea5) [0x7fbddb88bea5] /lib64/libc.so.6(clone+0x6d) [0x7fbdd5796b0d]"

So the issue is with Failed verify for the label standard: expected=3 actual=0

How can I remedy this?

(The tape gets mounted, this error occurs, tape gets dismounted and then it repeats).

Thank you in advance for your help.
Dmitry

vlado · 20 March 2023 15:35

Dmitry, is the tape labeled?

In other words, if the tape is empty - did you run cta-tape-label on it to create VOL1 label on it? WARNING: cta-tape-label is a destructive command, if the tape is not empty all data will became unreadable. Be careful.

We will discuss about the perpetual error (mount + error + dismount).

Vladimir

dmitrylitvintsev · 20 March 2023 15:39

Hi Vladimir,

This is what I see in history for that tape:

 6218  2023-03-17 14:29:02  cta-tape-label --vid VR5771 -f

Let me know if some crucial option was missing.

Dmitry

vlado · 20 March 2023 15:50

The label command looks good. Can you please re-try it cta-tape-label again (will delete any data on tape) and send here the output?

Then try again to write onto that tape.

V.

dmitrylitvintsev · 20 March 2023 15:58

Do I need to do something with the drives like setting it to down? The current state:

cta-admin  dr ls 
 library  drive     host desired        request status  since    vid   tapepool vo files data MB/s session priority activity age reason                                     
TS4500G1 LTO8D0 gmv18018      Up ArchiveForUser Unload     14 VR5771 ctasystest vo     -    -    -    3498        0        -  14 -

I am a bit worried of side effects of trying to relabel tape while it is being attempted to be mounted/dismounted

vlado · 20 March 2023 16:00

Yes, set the drive down, that should stop mount-error-dismount sequence.

Then the tape should be in the home slot in the library.

Then try label (I do not remember if you can do that with drive down).

I hope this is all just a test, no data loss.

dmitrylitvintsev · 20 March 2023 16:02

yes, this is test. The tape is empty.

dmitrylitvintsev · 20 March 2023 19:31

well, I ended up with this :

 library  drive     host desired request status since vid tapepool vo files data MB/s session priority activity age reason                                             
TS4500G1 LTO8D0 gmv18018    Down       -   Down  6622   -        -  -     -    -    -       -        0        -   2 [cta-taped] ERROR A tape was detected in the [...] 
TS4500G1 LTO8D1 gmv18018      Up       -   Free  7038   -        -  -     -    -    -       -        0        -   0 -

(my label command was interrupted). Not sure how I can clear this.

vlado · 20 March 2023 19:50

If the tape is stuck inside of the drive (check with mt status) you need to dismount it first (mt eject ; cta-smc -d -D <your drive number>).

Then systemctl restart cta-taped. Then the drive should be properly down. Then try the cta-tape-label.

“Simple” as that Good night.

dmitrylitvintsev · 21 March 2023 17:08

yeah, simple enough.
So, the label command has completed:

Mar 20 15:10:44.100130 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session started" userName="root" tapeVid="VR5771" tapeOldLabel="" force="true" tapeLoadTimeout="7200" 
Mar 20 15:10:44.101791 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session read configuration" catalogueDbType="DBTYPE_POSTGRESQL" catalogueDatabase="postgresql://localhost:5438/cta_dev" catalogueUsername="" devFilename="/dev/nst0" rawLibrarySlot="smc0" logicalLibrary="TS4500G1" unitName="LTO8D0" 
Mar 20 15:10:44.101947 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session set process capabilities" capabilities="cap_sys_rawio+ep" 
Mar 20 15:10:44.148311 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session mounting tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true" 
Mar 20 15:10:50.230900 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session mounted tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true" 
Mar 20 15:10:50.231155 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session loading tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:35.631954 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session loaded tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:35.632257 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session rewinding tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:37.899961 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session successfully rewound tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:40.915243 gmv18018.fnal.gov cta-tape-label: LVL="WARN" PID="2531" TID="2531" MSG="Label a non-empty tape with force option" userName="root" tapeVid="VR5771" tapeOldLabel="" force="true" tapeLoadTimeout="7200" 
Mar 20 15:38:40.918065 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session enabling LBP on drive" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:40.918353 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session is writing label with LBP to tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:45.368127 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session has written label with LBP to tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" force="true" 
Mar 20 15:38:45.368481 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session unloading tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true" 
Mar 20 15:39:13.977873 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session unloaded tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true" 
Mar 20 15:39:13.978222 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session dismounting tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true" 
Mar 20 15:39:22.727328 gmv18018.fnal.gov cta-tape-label: LVL="INFO" PID="2531" TID="2531" MSG="Label session dismounted tape" userName="root" tapeVid="VR5771" tapeOldLabel="" tapeDrive="LTO8D0" logicalLibrary="TS4500G1" useLbp="true" driveSupportLbp="true" librarySlot="smc0" force="true"```

After that I have been experiencing these weird issues that have been traced to device handles re-assigned on reboot. That's why I was seeing tape in "wrong" drives. Ren Bauer fixed it here and I have been able to write and read files to/from this tape. 

Thank you for your help! Apart from device problems running `cta-tape-label` should have fixed my initial issue.

vlado · 21 March 2023 20:44

Glad to hear that you managed to sort out the issues and move on with CTA.

It is a great product (OK, I am biased)

Vladimir