Cta-change-storage-class-in-catalogue

General Discussion
1

Hi,

I am trying to use this tool (on our preprod instance for the time being).

For setting up gRPC access from the CTA admin to EOS, I have done the following

  • eos vid set map -grpc key:XXXXXX vuid:2 vgid:2
  • eos -r 0 0 vid add gateway cta-adm-preprodfac grpc
  • Created /etc/cta/eos.grpc.keytab which includes the following

eosantarespreprodfac antares-eos97.scd.rl.ac.uk:50051 XXXXXX

I have put this keytab file in /etc/cta/eos.grpc.keytab on the Frontend (cta:tape ownership) and specified

cta.ns.config /etc/cta/eos.grpc.keytab

in /etc/cta/cta-frontend-xrootd.conf

and also in /etc/cta/eos.grpc.keytab on the CTA admin node where I am trying to use the tool.

When I try to run the tool, I get an auth error

[root@cta-adm-preprodfac ~]# cta-change-storage-class-in-catalogue -j ./files_to_change.json --storageclassname factest_repack -t 1

User: eosantarespreprodfac on host: cta-front05 is not authorized to execute CTA admin commands

Is there something obviously wrong in what I did?

Thanks,

George

2

As per cmdline/standalone_cli_tools/change_storage_class/JsonFileData.cpp · v5.10.10.1-1.el9 · cta / CTA · GitLab

the files_to_change.json files looks like this

{“archiveFileId”: ‘4295371177’, “fid”: ‘200404262’, “instance”: ‘eosantarespreprodfac’, “storageclass”: ‘factest_tape2’}

3

Hi George,

The first thing to say about this tool is a warning that it has never been tested. Please test thoroughly in a pre-production environment and use at your own risk.

The tool cta-change-storage-class-in-catalogue was created to address the problem of splitting files on the same tape into multiple tapes, which I think is the use case that you have. The idea is to change the storage class of the files to be separated and then repack the tape.

Note that this tool does not change the storage class in the EOS namespace. That has to be done as a separate (manual) step. In other words, using this tool will introduce an inconsistency between the storage class of the file as reported by eos attr ls and the storage class as reported by cta-admin tf ls.

The person who wrote this tool left the team before we could test and evaluate it. It may work but we can give no guarantees. Our plan is to do an evaluation and revision of the tool before we use it ourselves. This is on the list of work to be done in 2024/25, but there are several other things which are higher priority, so I don’t expect we will get to it until next year.

Besides those caveats, the tool probably works, please let us know your experiences.

gRPC authentication to EOS

Do not configure gRPC access to EOS. See explanation below.

There are two authentication schemes, Kerberos authentication for the CTA Frontend and gRPC authentication for EOS. Setting up gRPC authentication would give root access to the EOS filesystem. This is a security risk: EOS files can be accessed or deleted by anyone who can read the gRPC keytab. This is why we reduced the scope of the tool so it only updates the CTA Catalogue and not EOS.

However, as the tool does not currently update EOS, this is not an issue. Just don’t configure gRPC.

Authentication to CTA Frontend

Authentication to the CTA Frontend uses Kerberos. The Kerberos identity must be included in the list of administrators. Probably you need to type:

cta-admin admin add --username eosantarespreprodfac --comment "User to change storage classes"

and then it should work (maybe).

4

Hi Michael,

Than you very much for you detailed and clear reply.

As per your suggestion, I removed all gRPC access setup.

I have tried to change the storage class of a single tape file. After a few tries, I figured out that only syntax that works is the following

[root@cta-adm-preprodfac ~]# cta-change-storage-class-in-catalogue -j ./files_to_change.json
Sep 18 14:25:12.808308969 cta-adm-preprodfac.scd.rl.ac.uk cta-change-storage-class: LVL=“INFO” PID=“413148” TID=“413148” MSG=“Storage class found” storageClass=“factest_repack”
[root@cta-adm-preprodfac ~]#

in which the new, i.e target class, is defined in the json file

[root@cta-adm-preprodfac ~]# cat ./files_to_change.json
{“archiveFileId”: ‘4295371177’, “fid”: ‘200404262’, “instance”: ‘eosantarespreprodfac’, “storageclass”: ‘factest_repack’}

So, it looks like the storage class has changed in the Catalogue

[root@cta-adm-preprodfac ~]# cta-admin tf ls --id 4295371177
archive id copy no vid fseq block id instance disk fxid size checksum type checksum value storage class owner group creation time
4295371177 1 JT1999 1 0 eosantarespreprodfac bf1ed26 104.9M ADLER32 ae4d3cd3 factest_repack 1200 1200 2024-07-19 11:19
[root@cta-adm-preprodfac ~]#

I will try to run the tool for all 20 files on this tap and then attempt a repack; I will let you know how this goes.

From what I understand though, using the tool and updating the storage class directly in the DB (modify archive_file table) have comparable risk,

George

5

OK, good, looks like it is working!