Input/output error from tape drive device /dev/nst0

george_patargias · 22 May 2024 15:44

Hello,

We see this error on a Rocky 9 tapeserver running 5.10.10.1-1.el9

{“epoch_time”:1716391385.320501824,“local_time”:“2024-05-22T16:23:05+0100”,“hostname”:“getafix-ts03”,“program”:“cta-taped”,“log_level”:“INFO”,“pid”:“184016”,“tid”:“184016”,“message”:“In Scheduler::setDesiredDriveState(): success.”,“drive_name”:“obelix_lto8_04”,“instance”:“antaresdev”,“sched_backend”:“cephUser”,“thread”:“MainThread”,“tapeDrive”:“obelix_lto8_04”,“devFilename”:“/dev/lto_0”,“errorMessage”:“EmptyDriveProbe: Could not read drive status in hasTapeInPlace: /dev/nst0 Errno=5: Input/output error”,“drive”:“obelix_lto8_04”,“up”:“down”,“force”:“no”,“reason”:“[cta-taped] ERROR EmptyDriveProbe: Could not read drive status in hasTapeInPlace: /dev/nst0 Errno=5: Input/output error”,“comment”:“”,“schedulerDbTime”:0.000723}
{“epoch_time”:1716391385.320644829,“local_time”:“2024-05-22T16:23:05+0100”,“hostname”:“getafix-ts03”,“program”:“cta-taped”,“log_level”:“ERROR”,“pid”:“184016”,“tid”:“184016”,“message”:“Notified client of end session with error”,“drive_name”:“obelix_lto8_04”,“instance”:“antaresdev”,“sched_backend”:“cephUser”,“thread”:“MainThread”,“tapeDrive”:“obelix_lto8_04”,“devFilename”:“/dev/lto_0”,“errorMessage”:“EmptyDriveProbe: Could not read drive status in hasTapeInPlace: /dev/nst0 Errno=5: Input/output error”}

The lowest level SCSI devices - /dev/sg* - cannot be “trasnlated” to the higher level devices /dev/nst*

If we try to enquire the status of an nst device, we get

sg_inq -v /dev/nst0
inquiry cdb: [12 00 00 00 24 00]
inquiry: pass-through os error: Input/output error
inquiry: failed requesting 36 byte response: OS error: Input/output error
close error: Input/output error

whereas

[root@getafix-ts03 cta]# sg_inq -v /dev/sg3
inquiry cdb: [12 00 00 00 24 00]
inquiry cdb: [12 00 00 00 46 00]
inquiry cdb: [12 01 00 00 fc 00]
inquiry cdb: [12 01 80 00 fc 00]
standard INQUIRY:
PQual=0 PDT=1 RMB=1 LU_CONG=0 hot_pluggable=0 version=0x06 [SPC-4]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=1 [BQue=0]
EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=1
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=70 (0x46) Peripheral device type: tape
Vendor identification: IBM
Product identification: ULTRIUM-TD8
Product revision level: Q384
Unit serial number: 10210067D7
[root@getafix-ts03 cta]#

udevadm info -a /dev/nst0 also works.

Can you please give us a clue?

Thanks,

George

vlado · 22 May 2024 18:02

Dear George,

there is clearly something fishy with /dev/nst0 device.

Normally, when you want to put the drive up, there is a probe in the cta-taped that checks whether the tape drive is free. If there is a tape cartridge in the drive or that probe fails, the drive will not be put up. This is a simple but effective protection against putting tape drives into production which are not ready.

First of all, to get the mapping between sg and st devices, you can either use lsscsi -g or sg_map. Example:

[root@tpsrv458 ~]# lsscsi -g | grep -v disk
[15:0:0:0]   tape    IBM      ULT3580-TD9      Q9E0  /dev/st0   /dev/sg0
[15:0:0:1]   mediumx IBM      03584L32         1901  /dev/sch0  /dev/sg1

[root@tpsrv458 ~]# sg_map
/dev/sg0  /dev/nst0
/dev/sg1

Could you please share the output of mt -f /dev/nst0 status? Example:

[root@tpsrv458 ~]# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):
 DR_OPEN IM_REP_EN

mt command comes from this package:

[root@tpsrv458 ~]# rpm -qf /usr/bin/mt
mt-st-1.4-7.el9.x86_64

Only once you get the clean output of the mt command with DR_OPEN is the drive ready.

Is this your first tape server running Rocky Linux 9? Do the others work?

Best regards,

Vladimir

george_patargias · 23 May 2024 13:35

Hi Vlado,

Thanks for your reply.

Yes, this is the very first CTA tape server running Rocky Linux 9. It is a slightly old machine: it is a Dell R740 server purchased in 2019.

lssci lists a mapping between sg and st devices

[root@getafix-ts03 ~]# lsscsi -g | grep tape
[17:0:1:0] tape IBM ULTRIUM-TD8 Q384 /dev/st0 /dev/sg3
[17:0:2:0] tape IBM ULTRIUM-TD8 Q384 /dev/st1 /dev/sg4
[18:0:0:0] tape IBM ULTRIUM-TD8 Q384 /dev/st2 /dev/sg5
[18:0:1:0] tape IBM ULTRIUM-TD8 Q384 /dev/st3 /dev/sg6

but sg_map gives an error

[root@getafix-ts03 ~]# sg_map
device /dev/nst0 failed on scsi ioctl(idlun), skip: Input/output error
device /dev/nst1 failed on scsi ioctl(idlun), skip: Input/output error
device /dev/nst2 failed on scsi ioctl(idlun), skip: Input/output error
device /dev/nst3 failed on scsi ioctl(idlun), skip: Input/output error
/dev/sg0 /dev/sda
/dev/sg1
/dev/sg2
/dev/sg3
/dev/sg4
/dev/sg5
/dev/sg6

mt, unfortunatelly, gives an error

[root@getafix-ts03 ~]# mt -f /dev/nst0 status
/dev/nst0: Input/output error
[root@getafix-ts03 ~]# mt -f /dev/st0 status
/dev/st0: Input/output error

Best,

George

vlado · 23 May 2024 13:43

Hi George,

Thank you for more information.

My reply will not be very helpful as we also see problems with Alma Linux 9 and multiple tape drives connected to a server. We have not resolved them yet (but everything works fine with CentOS 7).

I only have one comment and one suggestion:

What is the HBA card / kernel module that you are using? In our case it is:

[root@tpsrv620 ~]# lspci -v | grep Fibre
81:00.0 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter (rev 01)
        Subsystem: Emulex Corporation LPe31002-M6 2-Port 16Gb Fibre Channel Adapter
81:00.1 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter (rev 01)
        Subsystem: Emulex Corporation LPe31002-M6 2-Port 16Gb Fibre Channel Adapter

[root@tpsrv620 ~]# modinfo lpfc | head -5
filename:       /lib/modules/5.14.0-362.24.2.el9_3.x86_64/kernel/drivers/scsi/lpfc/lpfc.ko.xz
version:        0:14.2.0.12
author:         Broadcom
description:    Emulex LightPulse Fibre Channel SCSI driver 14.2.0.12
license:        GPL

In our case, the tape server works fine with one tape drive, but not with two. Can you please try to disconnect the other three on your side and re-try (reboot and/or reinstall)?

Let me know how it goes.

In the next few days I will not have time to work on this. More likely only in June.

Best regards,

Vladimir

george_patargias · 23 May 2024 15:31

Hi Vlado,

It looks like, in our case, the kernel module is not the same with the FC card…

[root@getafix-ts03 ~]# lspci -v | grep Fibre
86:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01)
86:00.1 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01)
af:00.0 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01)
af:00.1 Fibre Channel: QLogic Corp. ISP2722-based 16/32Gb Fibre Channel to PCIe Adapter (rev 01)
[root@getafix-ts03 ~]#
[root@getafix-ts03 ~]# modinfo lpfc | head -5
filename: /lib/modules/5.14.0-427.16.1.el9_4.x86_64/kernel/drivers/scsi/lpfc/lpfc.ko.xz
version: 0:14.2.0.16
author: Broadcom
description: Emulex LightPulse Fibre Channel SCSI driver 14.2.0.16
license: GPL
[root@getafix-ts03 ~]#

I disconnected 3 out of 4 drives and rebooted the tape server but I still get the errors
when I issue sg commands and mt.

[root@getafix-ts03 ~]# mt -f /dev/st0 status
/dev/st0: Input/output error
[root@getafix-ts03 ~]# sg_inq -v /dev/nst0
inquiry cdb: [12 00 00 00 24 00]
inquiry: pass-through os error: Input/output error
inquiry: failed requesting 36 byte response: OS error: Input/output error
close error: Input/output error
[root@getafix-ts03 ~]# sg_map
device /dev/nst0 failed on scsi ioctl(idlun), skip: Input/output error
/dev/sg0 /dev/sda
/dev/sg1
/dev/sg2
/dev/sg3
[root@getafix-ts03 ~]#

Maybe, I need to re-install again…?

George

vlado · 24 May 2024 10:41

Dear George,

Thank you for those details.

Obviously, if you have a QLogic fibre channel HBA card, then there is no point looking at lpfc kernel module.

Instead you need to look for something like qla2xxx. On some of our (older) tape servers we have these cards:

[root@tpsrv039 modprobe.d]# lspci -v | grep Fibre
02:00.0 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)
02:00.1 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)

For which this kernel module is used:

[root@tpsrv039 modprobe.d]# lsmod | grep qla2xxx
qla2xxx              1204224  2
nvme_fc                61440  1 qla2xxx
scsi_transport_fc     110592  1 qla2xxx

Right now, we are using the qla2xxx module provided directly by the Alma Linux 9 kernel and everything seems to work for now (with one tape drive):

[root@tpsrv039 modprobe.d]# uname -a
Linux tpsrv039.cern.ch 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 9 18:15:59 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

[root@tpsrv039 modprobe.d]# modinfo qla2xxx | grep version
rhelversion:    9.4
srcversion:     6F590C71524081C515F1D5F
vermagic:       5.14.0-427.16.1.el9_4.x86_64 SMP preempt mod_unload modversions

Unfortunately, as you can see above, there is only very long srcversion so one would need to check the QLogic website whether they have a new kernel module.

In any case, please enable extended logging for that module so that you see as much details as possible (normally this is a good practice on all nodes, but do it at least on the one where you are currently debugging):

[root@tpsrv039 modprobe.d]# grep qla /etc/modprobe.d/cta-tape-modules.conf
# To enable maximum debug logging from the QLogic qla2xxx kernel module use this option:
# options qla2xxx ql2xextended_error_logging=0x7fffffff
options qla2xxx ql2xextended_error_logging=1

In conclusion, here are my current suggestions:

You are using QLogic card which is different HBA from what we are using on our most modern tape servers (Emulex).
We have not yet tried to connect multiple drives on our old QLogic HBAs, but with one tape drive, everything works as expected.
Try to enable more debugging to see why things do not work.
Check that the topology of the card is matching what the tape drive is expecting (loop vs. point-to-point fibre channel topology).
Check whether there is not a newer Rocky Linux / RHEL 9 kernel module on the QLogic website, try to compile it and see whether it helps or not.
However, while re-reading again and double checking I am now not convinced it is the low lever fibre driver that is the problem. Given that lsscsi -g works (i.e. the sg* commands can see the drive), the problem might be something completely different. Try to install bare minimum Rocky Linux 9, no fancy (kernel) add-ons. Check whether the mt -f /dev/nst0 status works there

Please report how it goes. Good luck.

Vladimir

george_patargias · 24 May 2024 11:30

Hi Vlado,

Sorry, I was silly enough to copy/paste your exact modinfo command. We do indeed run the correct qla2xx kernel module

filename: /lib/modules/5.14.0-427.16.1.el9_4.x86_64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko.xz

Just before I read your latest reply, I tried installing the latest long-term kernel: kernel-lt-6.1.91-1.el9.elrepo.x86_64 (basically, your suggestion 5.)

which loads the following qla2xx module

filename: /lib/modules/6.1.91-1.el9.elrepo.x86_64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko.xz

and, in this, case the mappring works!!

[root@getafix-ts03 ~]# sg_map
/dev/sg0 /dev/sda
/dev/sg1
/dev/sg2
/dev/sg3 /dev/nst0
[root@getafix-ts03 ~]# sg_inq /dev/nst0
standard INQUIRY:
PQual=0 PDT=1 RMB=1 LU_CONG=0 hot_pluggable=0 version=0x06 [SPC-4]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=1 [BQue=0]
EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=1
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=70 (0x46) Peripheral device type: tape
Vendor identification: IBM
Product identification: ULTRIUM-TD8
Product revision level: Q384
Unit serial number: 10230067D7
[root@getafix-ts03 ~]#
[root@getafix-ts03 ~]# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):
DR_OPEN IM_REP_EN
[root@getafix-ts03 ~]#

and cta-taped for the first drive is running without an error this time!

George

vlado · 24 May 2024 11:37

Great George,

now try to connect all 4 drives and let me know if all 4 work.

mt -f /dev/nstX status on all of them is enough.

Still, could you please run modinfo qla2xxx | grep version to see which source version is in that new kernel?

Please report. Thanks.

Vladimir

george_patargias · 24 May 2024 12:54

Hi Vlado,

I can see ll drives now by running mt

[root@getafix-ts03 ~]# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):
DR_OPEN IM_REP_EN
[root@getafix-ts03 ~]# mt -f /dev/nst1 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):
DR_OPEN IM_REP_EN
[root@getafix-ts03 ~]# mt -f /dev/nst2 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):
DR_OPEN IM_REP_EN
[root@getafix-ts03 ~]# mt -f /dev/nst3 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (50000):

Also sg_map ang sg_inq work that didn’t work with the previous kernel.

The new kernel module

[root@getafix-ts03 ~]# modinfo qla2xxx | grep version
srcversion: FCEF05C9656E907EA39DC47
vermagic: 6.1.91-1.el9.elrepo.x86_64 SMP preempt mod_unload modversions

George

vlado · 24 May 2024 13:46

Thanks George.

I will check next week if similar trick works on our end (the newest tape servers with multiple tape drives) with the lpfc kernel module.

Vladimir

tmesser · 5 June 2024 20:51

Hi CTA team,

We also see this problem in our Fermilab environment on AlmaLinux 9 systems. In working with our vendor (ATTO), we are seeing that the SCSI error corresponds to “medium not found.”

After loading a tape, then unloading it, the I/O errors disappear, even without a tape in the drive.

This is a strange issue for us and we are looking into it further, including working with ATTO. We do not yet have a QLogic CTA system tested with AlmaLinux 9, so we have not yet seen this problem with a QLogic card.

Rafael_Rocha · 10 September 2024 20:18

Hi CTA Team,

The root cause of this problem lies in the Linux ST driver. A change request has already been submitted to the kernel to correct this behavior.

You can review the details here:
https://lore.kernel.org/lkml/20240905173921.10944-1-rrochavi@fnal.gov/T/

Rafael_Rocha · 4 October 2024 17:47

@vlado @george_patargias This change is already present in the 6.11, 6.10 and 6.6 stable tree, and 6.12-rc. However, some considerations still need to be addressed. The change resolves the issue only when the tape drive is empty. Once a tape is loaded, the problem can reoccur if the drive previously experienced an event where the drive, host, or bus was reset and the sense buffer wasn’t cleared (ASC 0x29).

I recently encountered this issue with CTA v5.10.10.1-1 after a node reboot and loading a tape into the drive. The drive once again encountered an Input/Output error, and CTA was unable to handle the situation. Since the change that triggered this issue aims to prevent data corruption when performing operations from an unknown position, my recommendation is to clear the sense buffer (if you are certain of the tape’s position) or perform an action that removes the pos_unknown flag, such as rewinding the tape, when this situation is detected by the application layer, in this case CTA. This approach will allow CTA to handle the scenario and prevent it from hanging, as it does currently.

mdavis · 5 October 2024 15:08

Thanks for the analysis, Rafael. We will discuss this at our CTA operations meeting next week.