Orphaned disk buffer files

Site Administrators
1

Hi Team,

We have 4 files which is in disk buffer and couldn’t transfer to tape

Error message observed is

  • “[ERROR] Server responded with an error: [3011] Unable to open file /eos/axxxxxs/prod/servicexxx/xxx/axxxxdxxxxtape/dxxxxxxxxV/Rxx/oxxxxx/dxxxxxx.00xxxx9.physics_Lxxx.mxx.RAW/daxxxxxx.005xxxxx9.pxxxx_Lxxx.mxxxx.Rxx._lbxxx._SFO-ALL._0xx1.x; No such file or directory”

and

  • [ERROR] Server responded with an error: [3007] Unable to open file /eos/file-path; Input/output error

One of host filesystem became “readonly” and after reboot the host was fine but since then these 4 files are unable to move from disk buffer to tape.

(1) 1 file present in disk path location is /eos/data-sdo/.eosorphans/

Here in (1) scenario “ eos fileinfo “couldnt get the details like (XStype: none XS: Layout: plain )

(2) 3 files is present in disk path location is /eos/data-sdx/000166d5/ .

Here in the 2nd scenario “eos fileinfo can get details like (XStype: adler XS: 6x 9x ax 9x Layout: replica )

We found a valid duplicate copies of these 4 files with timestamp suffixed and transferred to tape.

Could you please guide us how to transfer the original files to tapes because service user can see only original files and has not reported about the valid duplicate files.

Thanks and Regards,

Maha

2

Dear Maha,

in order to cover this case transfers to CTA are valid once the file lands on tape.

Checking file integrity is a full class of problem for WLCG file transfers, for this reason we use FTS to transfer files to CTA.

Users must specify the file checksum along with their transfer and use `–archive-timeout` option set to at least 86400 seconds. The file transfer is then only successful when the file is on tape with the user provided checksum.

This also protects against other potential problems where partially transferred files could land on tape: Tape service thinks that the file is on tape but only a truncated version of the user file is available from the service.

If you managed to get some file content, you can copy these on behalf of user along the initially transferred files: you can for example happend `.recovered` suffix to the original file name and let it land on tape.

Then you need to contact the owner to check his file content.

It is important to think about how you can add file integrity verification in your existing data flows.

Best regards,

Julien Leduc