Cannot delete tape files

General Discussion
1

Hi,

I have a set of tape files that I want to delete (so that I can reclaim the tapes).

[root@cta-adm1 georgep]# head lhcbchall22_tapefiles
archive id copy no vid fseq block id instance disk fxid size checksum type checksum value storage class owner group creation time
4308411357 1 CT4063 37 737605 eosantares 31615cce 5.4G ADLER32 f901298b lhcbchall22 36250 24289 2022-12-15 15:08
4308411030 1 CT4063 40 799072 eosantares 31615b85 5.4G ADLER32 602e4d76 lhcbchall22 36250 24289 2022-12-15 15:08
…

These files don’t seem to have any EOS namespace entry

[root@cta-adm1 georgep]# eos fileinfo fxid:31615cce
error: cannot retrieve file meta data - Error while fetching FileMD #828464334 protobuf from QDB: Empty response (errc=2) (No such file or directory)
[root@cta-adm1 georgep]# eos fileinfo fxid:31615b85
error: cannot retrieve file meta data - Error while fetching FileMD #828464005 protobuf from QDB: Empty response (errc=2) (No such file or directory)
[root@cta-adm1 georgep]#

However, when I try to do something like

cta-admin tapefile rm --vid CT4063 --id 4308411357 --reason “test file”

I get the following

Cannot delete a copy of the file with archiveFileId 4308411357 because it is the only copy

I am not sure why CTA doesn’t let me delete the file…Hopefuly, it is not another file. Do you have any suggestions?

I am running CTA 5.10.10.1-1.el9 and EOS 5.2.23

Thanks,

George

2

Hi George,

The normal way to delete a file in CTA is using eos rm. This is a “safe” delete, as it puts the tape file in the recycle bin, along with a copy of its disk metadata.

cta-admin tapefile rm, by design, will only remove the second tape copy of a file. If there is only one copy, it gives the error that you see, in order to prevent accidental and permanent deletion of data.

Perhaps we should add a --force option, but for the moment this does not exist. I’ll raise it in our ops meeting.

Michael

3

Hi Michael,

Thank you for the reply.

I am aware that the normal way to delete a file is by doing “eos rm”. However, I cannot see them in the EOS namespace because if I try to run “eos fileinfo” on the fxid/fid of any these files, I get the message

error: cannot retrieve file meta data - Error while fetching FileMD #828404632 protobuf from QDB: Empty response (errc=2) (No such file or directory)

Also, running an “eos find -f” on the EOS area where these files are supposed to be from this tape pool (judging from the paths of deleted files from the same pool) yields nothing.

George

4

Hi Michael,

Sorry for the hassle. Since I cannot see any of these files in the EOS namespace, is it safe to delete them directly from the DB? From the tape_file and archive_file tables?

Best,

George

5

Hi George,

We don’t have a tool that allows that, and the main reason for that is the risk of data loss. We can look at adding a --force option to cta-admin tapefile rm for this situation.

The safest way to delete it manually would be to send a DELETE event to the CTA Frontend, containing the tape file metadata. I think you could do this by creating an empty file (in a tape-backed directory) and manually adding the extended attribute sys.archive.file_id="<id of the file you want to delete>". Then delete that file.

Caveat emptor: I haven’t tried this, attempt at your own risk.

6

Hi Michael,

Thanks for the suggestion as well as for the word of caution!

It looks like what you suggested did work

Creating the stub file generated an error

[root@cta-adm1 georgep]# eos touch /eos/antares/prod/lhcb/dataChallenge/stub.4308411357
error: unable to touch ‘/eos/antares/prod/lhcb/dataChallenge/stub.4308411357’ (errc=71) (Protocol error)

but the Frontend received the CREATE event and the file was created

Sep 11 15:40:32.581964274 cta-front04 cta-frontend: LVL=“INFO” PID=“1275296” TID=“95302” MSG=“In WorkflowEvent::WorkflowEvent(): received event.” user=“eosantares@cta-front04” eventType=“CREATE” eosInstance=“eosantares” diskFilePath=“/eos/antares/prod/lhcb/dataChallenge/stub.4308411357” diskFileId=“879599649”

Sep 11 15:40:32.581964274 cta-front04 cta-frontend: LVL=“INFO” PID=“1275296” TID=“95302” MSG=“In WorkflowEvent::WorkflowEvent(): received event.” user=“eosantares@cta-front04” eventType=“CREATE” eosInstance=“eosantares” diskFilePath=“/eos/antares/prod/lhcb/dataChallenge/stub.4308411357” diskFileId=“879599649”

Then, I assigned the archive file id as an extended EOS attibute

[root@cta-adm1 georgep]# eos attr set sys.archive.file_id=4308411357 /eos/antares/prod/lhcb/dataChallenge/stub.4308411357

[root@cta-adm1 georgep]# eos attr ls /eos/antares/prod/lhcb/dataChallenge/stub.4308411357
sys.archive.file_id=“4308411357”
sys.eos.btime=“1726065632.580272011”
sys.fs.tracking=“+100”
sys.utrace=“cc64f23e-704b-11ef-a7a4-1c34da4b345c”
sys.vtrace=“[Wed Sep 11 15:40:32 2024] uid:0[root] gid:0[root] tident:root.4040157:521@cta-adm1 name:daemon dn: prot:sss app: host:cta-adm1.scd.rl.ac.uk domain:scd.rl.ac.uk geo: sudo:1”

and, finally, doing an “eos rm” on this file did send the DELETE event to the Frontend and the file was removed from the tape

Sep 11 15:46:04.253622697 cta-front04 cta-frontend: LVL=“INFO” PID=“1275296” TID=“165677” MSG=“In OracleCatalogue::copyArchiveFileToRecycleBinAndDelete: ArchiveFile moved to the recycle-bin.” user=“eosantares@cta-front04” archiveFileId=“4308411357” diskFileId=“879599649” diskFilePath=“/eos/antares/prod/lhcb/dataChallenge/stub.4308411357” diskInstance=“eosantares” insertToRecycleBinTime=“0.003639” setTapeDirtyTime=“0.000916” deleteTapeFilesTime=“0.002752” deleteArchiveFileTime=“0.005183”
Sep 11 15:46:04.254170812 cta-front04 cta-frontend: LVL=“INFO” PID=“1275296” TID=“165677” MSG=“In RdbmsCatalogue::moveArchiveFileToRecycleLog(): ArchiveFile moved to the file-recycle-log.” user=“eosantares@cta-front04” fileId=“4308411357” diskInstance=“eosantares” requestDiskInstance=“eosantares” diskFileId=“828464334” diskFileInfo.owner_uid=“36250” diskFileInfo.gid=“24289” fileSize=“5368709120” creationTime=“1671116899” reconciliationTime=“1671116899” storageClass=“lhcbchall22” checksumType=“ADLER32” checksumValue=“f901298b” TAPE_FILE=“copy number: 1 vid: CT4063 fSeq: 37 blockId: 737605 creationTime: 1671116899 fileSize: 5368709120” checkDeleteRequestConsistency=“3.5e-05” copyArchiveFileToFileRecyleLogAndDeleteTime=“0.013002” totalTime=“0.013042”
Sep 11 15:46:04.255578240 cta-front04 cta-frontend: LVL=“INFO” PID=“1275296” TID=“165677” MSG=“In WorkflowEvent::processDELETE(): archive file deleted.” user=“eosantares@cta-front04” fileId=“4308411357” address=“null” filePath=“/eos/antares/prod/lhcb/dataChallenge/stub.4308411357” catalogueGetArchiveFileByIdTime=“0.001877” schedulerTime=“0.01467”

[root@cta-adm1 georgep]# cta-admin tf ls --id 4308411357
Archive file with ID 4308411357 does not exist

I will go ahead and apply this to the rest of the tape files/

Many thanks again,

George

7

Thanks George, good to know this worked.

We will be revising the command line tools next year and will look at adding an option to cover this case.