Cannot recall with CTA 5.10.8-1

Troubleshooting
1

Hi,

I upgraded our dev eoscta instance to CTA 5.10.8-1. Archival worked fine as expected but recall did not. The retrieve request is queued by the frontend and an object is created but the tape is never mounted because of the following errors

2024-03-26T12:27:59.543081+00:00 getafix-ts09 cta-taped: LVL=“ERROR” PID=“19256” TID=“19256” MSG=“In OStoreDB::RetrieveMount::testReserveDiskSpace(): unable to request EOS free space for disk system, putting queue to sleep” thread=“MainThread” tapeDrive=“obelix_lto8_16” mountId=“5174” vo=“dteam” tapePool=“dteam” diskSystemName=“antaresdev_retrieve” failureReason=“In DiskSystemFreeSpaceList::fetchFreeDiskSpace(), failed to call ‘eos root://antares-eos14 space ls -m’ Errno=25: Inappropriate ioctl for device instanceAddress: antares-eos14 stderr: /usr/bin/eos: error while loading shared libraries: libprotobuf.so.3: cannot open shared object file: No such file or directory”
2024-03-26T12:27:59.557527+00:00 getafix-ts09 cta-taped: LVL=“INFO” PID=“19256” TID=“19256” MSG=“In Helpers::getLockedAndFetchedQueue(): Successfully found and locked a retrieve queue.” thread=“MainThread” tapeDrive=“obelix_lto8_16” mountId=“5174” vo=“dteam” tapePool=“dteam” diskSystemName=“antaresdev_retrieve” failureReason=“In DiskSystemFreeSpaceList::fetchFreeDiskSpace(), failed to call ‘eos root://antares-eos14 space ls -m’ Errno=25: Inappropriate ioctl for device instanceAddress: antares-eos14 stderr: /usr/bin/eos: error while loading shared libraries: libprotobuf.so.3: cannot open shared object file: No such file or directory” attemptNb=“1” queueName=“JL5378” queueType=“jobsToTranfer” queueObject=“RetrieveQueueToTransferForUser-JL5378-DriveProcess-obelix_lto8_16-getafix-ts09.scd.rl.ac.uk-19256-20240326-12:26:17-0-0” rootFetchNoLockTime=“0.000492” rootRelockExclusiveTime=“0.006105” rootRefetchTime=“0.000388” addOrGetQueueandCommitTime=“0.004852” rootUnlockExclusiveTime=“0.000665” queueLockTime=“0.000818” queueFetchTime=“0.000930”
2024-03-26T12:27:59.559275+00:00 getafix-ts09 cta-taped: LVL=“INFO” PID=“19256” TID=“19256” MSG=“Disk space reservation test failed, will not mount tape” thread=“MainThread” tapeDrive=“obelix_lto8_16” mountId=“5174” vo=“dteam” tapePool=“dteam”
2024-03-26T12:27:59.572490+00:00 getafix-ts09 cta-taped: LVL=“INFO” PID=“19256” TID=“19256” MSG=“Selected vid to be queued for retrieve request.” thread=“MainThread” tapeDrive=“obelix_lto8_16” mountId=“5174” vo=“dteam” tapePool=“dteam” fileId=“4294985985” copyNb=“1” tapeVid=“JL5378” fSeq=“12”

Can you please help? I am running EOS 5.1.28

Thanks,

George

2

Apologies, I had forgoten to link the library

[root@getafix-ts09 ~]# ln -s /opt/eos/grpc/lib64/libprotobuf.so.3.17.3 /opt/eos/lib64/protobuf3/libprotobuf.so.3

where

[root@getafix-ts09 ~]# rpm -qf /opt/eos/grpc/lib64/libprotobuf.so.3.17.3
eos-grpc-1.41.0-2.el7.x86_64

Once this was done, the tape was mounted for read as expected. Hopefully this was the right thing to do.

Do I need to this also on the frontend? It doesnt look like I do

George

3

Hi George,

From your error logs above, I can see that CTA itself was working fine.

The linking problem was with the EOS client, which is called by the CTA tapeserver method DiskSystemFreeSpaceList::fetchFreeDiskSpace() to determine if there is sufficient free space to commence the data transfer to disk.

If EOS client can now find the protobuf shared library you should be good, there is nothing else to do.

4

Hi Michael,

Thanks for this - indeed after creating a link for libprotobuf.so.3 the recall worked as expected.

George

1 Like