I have run cta-ops-verify-tape using –random 1000 assuming that it will verify 1000 randomly chosen files from the middle of a tape (that contains a total of 6,967 files) plus 10 from the beginning and 10 from the end of the tape (total 1020). However the verifcation completed after verifying 950 files. Is this expected? The tape session did end with a failure but not because a file failed to be verified?
Thanks,
George
Dear George,
I believe this can happen because the random selection process can select the same file multiple times hence the numbers do not match.
Can you please re-try again and then look immediately into what is queued?
Example:
tape-local@ctaproductionfrontend14(ctaproduction, cephProductionRepack) ~]$ cta-ops-verify-tape --vid L94718 --first 10 --last 10 --random 1000
2026-05-13 17:29:29 [INFO] [verify_tape] Running verify-tape for tape with vid L94718, read speed of 300 MB/s, and data size: 0B
2026-05-13 17:29:30 [INFO] [partial_tape_scan] Performing partial tape scan.
2026-05-13 17:29:31 [INFO] [verify_tape] Verifying 912 files and 3.3T from tape L94718
2026-05-13 17:29:31 [INFO] [verify_files] ArchiveId of files to verify: 4600832117,4600832092,4600831746,4600832094,4600831356,4600831615,4600831157,...
2026-05-13 17:30:53 [INFO] [verify_files] All file verifications queued for tape L94718
[tape-local@ctaproductionfrontend14(ctaproduction, cephProductionRepack) ~]$ cta-admin sq
type instance scheduler tapepool vo library vid files queued data queued oldest youngest priority min age read max drives write max drives cur. mounts cur. files cur. data tapes capacity files on tapes data on tapes full tapes writable tapes
Retrieve ctaproduction cephProductionRepack vo_CMS_2023 CMS SPECTRALIB2-LTO9 L94718 912 3.3T 154 72 50 600 10 50 0 0 0 18.0T 5112 19.0T 1 0
912 matches 912, but it is less than 1000.
Do you observe the same behavior?
Vladimir Bahyl
CERN
The problem seems to come from this line 292:
# Terribly simple/stupid, but it will have to do.
random_pick_fseqs = random.choices(range(0, max_fseq + 1), k=args.random)
in this file: /opt/cta-ops/ops-venv/lib/python3.9/site-packages/tapeverify/cta_verify_tape.py.
Please just change random.choices() function to random.sample().
That should fix it. Please let me know how it goes.
Vladimir Bahyl
CERN
Thanks for the suggestion Vlado - we will definitely try this change.
We have been busy intergrating the verification tools with our infrastructure and have made good progress although there a few loose ends to pick up. We have set up a cron running the feeder that submits a random choice of tapes for verification.
I am trying to run cta-ops-verify-tape in a wrapper script which processes a list of an operator-provided list of tapes.
In both cases, however, the submitted verifications are vrtually indistinguishable from usual user retrieves. For example, for a tape that is currently being verified the json request looks like this
[root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "CT2602")' {
"mountType": "RETRIEVE",
"tapepool": "dirac",
"logicalLibrary": "asterix_ts",
"vid": "CT2602",
"queuedFiles": "0",
"queuedBytes": "0",
"oldestAge": "0",
"priority": "0",
"minAge": "0",
"curMounts": "1",
"curFiles": "29",
"curBytes": "3959704090696",
"tapesCapacity": "20000000000000",
"tapesFiles": "1243",
"tapesBytes": "23107487512573",
"fullTapes": "1",
"writableTapes": "0",
"sleepingForSpace": false,
"sleepStartTime": "0",
"diskSystemSleptFor": "",
"vo": "dirac",
"readMaxDrives": "1",
"writeMaxDrives": "4",
"mountPolicies": [],
"youngestAge": "0",
"highestPriorityMountPolicy": "",
"lowestRequestAgeMountPolicy": "",
"schedulerBackendName": "cephUser",
"instanceName": "antares"
}
As you see “mountPolicies” points to an empty array although I was under the impression that this is not the case, i.e. that is “mountPolicies”: [verification]. How can we make sure this does not happen? In our config we do have: verification_mount_policy: verification
Best,
George
Dear George,
Unfortunately, this verification process is not yet really ready for production, but we are working on it.
I don’t know exactly why you have nothing in mountPolicies, but here are some observations:
verification option in verification_mount_policy is never used and is completely useless.cta-ops-verify-tape uses the CTA binary cta-verify-file, but this command does not allow you to pass any sort of mount policy.cta-frontend.conf file from which the verification requests are performed. (cta.verification.mount_policy verification).cta-admin.If you have configured all of the above correctly, the mount policies entry should have the corresponding verification mount policy; otherwise, there is an issue in the frontend code.
Best,
Idriss
Hi Idriss,
Thanks for the reply.
I do have
cta.verification.mount_policy verification
in the /etc/cta/cta-frontend-xrootd.conf (rather than /etc/cta/cta-frontend.conf). In fact, there was an extra space between cta.verification.mount_policy and verifcation which I deleted.
I submitted another tape for verification and it looked like this time that there is mountPolicy value
[root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "TD4006")'
{
"mountType": "RETRIEVE",
"tapepool": "diamond_m07",
"logicalLibrary": "obelix_ts1170",
"vid": "TD4006",
"queuedFiles": "16",
"queuedBytes": "81331126969",
"oldestAge": "71",
"priority": "5",
"minAge": "300",
"curMounts": "1",
"curFiles": "1",
"curBytes": "5091158016",
"tapesCapacity": "50000000000000",
"tapesFiles": "12428",
"tapesBytes": "63259891623572",
"fullTapes": "1",
"writableTapes": "0",
"sleepingForSpace": false,
"sleepStartTime": "0",
"diskSystemSleptFor": "",
"vo": "storaged_dls",
"readMaxDrives": "10",
"writeMaxDrives": "10",
"mountPolicies": [
"verification"
],
"youngestAge": "69",
"highestPriorityMountPolicy": "verification",
"lowestRequestAgeMountPolicy": "verification",
"schedulerBackendName": "cephUser",
"instanceName": "antares"
}
but after a few mins - when files started to get vertifed - it was gone…!
[root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "TD4006")'
{
"mountType": "RETRIEVE",
"tapepool": "diamond_m07",
"logicalLibrary": "obelix_ts1170",
"vid": "TD4006",
"queuedFiles": "0",
"queuedBytes": "0",
"oldestAge": "0",
"priority": "0",
"minAge": "0",
"curMounts": "1",
"curFiles": "16",
"curBytes": "81293554789",
"tapesCapacity": "50000000000000",
"tapesFiles": "12428",
"tapesBytes": "63259891623572",
"fullTapes": "1",
"writableTapes": "0",
"sleepingForSpace": false,
"sleepStartTime": "0",
"diskSystemSleptFor": "",
"vo": "storaged_dls",
"readMaxDrives": "10",
"writeMaxDrives": "10",
"mountPolicies": [],
"youngestAge": "0",
"highestPriorityMountPolicy": "",
"lowestRequestAgeMountPolicy": "",
"schedulerBackendName": "cephUser",
"instanceName": "antares"
}
Since I dont want to re-invent the wheel, I will collect the list of curently verified tapes like this
[root@cta-adm-fac1 georgep]# XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab cta-ops-verification-feeder --list_current_verifications -C /etc/cta-ops/cta-ops-config.yaml
2026-05-26 17:16:16 [INFO] [main] Executed as: /root/.local/bin/cta-ops-verification-feeder --list_current_verifications -C /etc/cta-ops/cta-ops-config.yaml
2026-05-26 17:16:24 [INFO] [main] Currently running verification on tapes: TD0012, TD4006, TD4007
Hello @vlado
Just to let you know that the source code you suggested above worked. Many thanks!
George