CTA mounting tape but not writing the data (CTA 5 + dCache)

General Discussion
1

Hello,

We know that maybe this is not the best place to ask for this since it is related to CTA+dCache, but maybe some of the answers can be of help for someone else.

First of all, thank you for making CTA repos publicly available. Related to that, we have already tried to install version 5, tied to our dCache instance. In our dCache instance we have right now the plugin version 0.7.0.

We have some problems writing data form dCache to CTA. The drive loads the tape, but then it doesn’t do anything. It doesn’t write data and after some time it releases the tape after showing an error [1]. For extra info, in the database the write mounts gets updated, but the bytes written are always zero.

The config files we use are the same we were using in our previous installation (4.7.8), which except for the EOSReporter warning, everything was working.

We have tried to go back to the version 4.7.12 (latest public v4 rpms) and the errors are the same with the addition of an extra error [2].

Our questions are:

  1. Could this be a bad compatibility between the dCache plugin and the newer versions of CTA? (this one is more for dCache developers)
  2. If you don’t think this is dCache related, do you have any possible idea of what could be failing?

If you need any extra information, config files or whatever, don’t hesitate to ask, we’ll be awaiting for it.

Cheers,

Jordi

[1]

Nov 10 13:24:02.474560 ctatps001 cta-taped: LVL="ERROR" PID="12203" TID="13254" MSG="Exception while reading a file" thread="DiskRead" tapeDrive="257" tapeVid="V03650" mountId="10" threadID="0" blockID="1" exceptionMessage="In XrootReadFile::XrootReadFile failed XrdCl::File::Open() on root://dc111.pic.es:36981/000098D619DE28E541788BAF2655399F3D5F [FATAL] Connection error code:108 errNo:0 status:3" fileSize="45" checksumType="ADLER32" checksumValue="67340ff4"

[2]

221111 10:38:11 8997 ssi_Pb::ServiceClientSide: pid:8997 tid:140565651937536 ServiceClientSide object was destroyed before shutting down the Service, possible memory leak
2

hi @jcasals

We’ve observed this as well with pools that pick ipv6 as default. If you could ensure ipv4 on the pool, it should go through. Let’ us know if you observe any other issues. A fix on the dCache driver is on the way.

cheers,
mwai

3

Hi Jordi,

the issue is coming from unnecessary binding of in dcache-cta driver xroot IO server to a specific interface on the host. This is fixed now and we will release a new driver version after local testing at DESY. For now, as workaround, you can specify io-endpoint to an interface that acan be accessed by tape movers, for example:

hsm create osm cta dcache-cta -cta-user=userA \
     -cta-group=groupA -cta-instance-name=instanceA \
     -cta-frontend-addr=cta-forntend-host:17017 \
     -io-endpoint=a.b.c.d -io-port=11094

Best regards,
Tigran

4

Hi @jcasals,

we just have release dcache-cta-0.8.0 driver with desired fix.

Best regards,
Tigran.

5

Hi Tigran,

Thank you so much! We’ll try to test it as soon as possible, hopefully this week since we have some things in the middle of being changed.

We’ll let you know when we do it.

Cheers,

Jordi

6

Hi Tigran,

Just so you know, we installed the new plugin version and set it up as you mentioned to us previously (don’t recall exasctly where right now) and it works, with dCache 8.2, plugin dcache-cta 0.8.0 and CTA 5.7.12-2. We still have some issues with failed requests after transferring files that @eli_carr is going to update in another thread.

Thanks again,

Jordi