# Cta-ops-verify-tape usage check **URL:** \ **Category:** General Discussion\ **Created:** [13 May 2026 14:02 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433 "2026-05-13T14:02:08Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [13 May 2026 14:02 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/1 "2026-05-13T14:02:09Z")
I have run cta-ops-verify-tape using –random 1000 assuming that it will verify 1000 randomly chosen files from the middle of a tape (that contains a total of 6,967 files) plus 10 from the beginning and 10 from the end of the tape (total 1020). However the verifcation completed after verifying 950 files. Is this expected? The tape session did end with a failure but not because a file failed to be verified? Thanks, George ---
**Author:** ![vlado](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/v/779978/32.png) [@vlado](https://cta-community.web.cern.ch/u/vlado)\ **Post date:** [13 May 2026 15:35 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/2 "2026-05-13T15:35:17Z")
Dear George, I believe this can happen because the random selection process can select the same file multiple times hence the numbers do not match. Can you please re-try again and then look immediately into what is queued? Example: ```auto tape-local@ctaproductionfrontend14(ctaproduction, cephProductionRepack) ~]$ cta-ops-verify-tape --vid L94718 --first 10 --last 10 --random 1000 2026-05-13 17:29:29 [INFO] [verify_tape] Running verify-tape for tape with vid L94718, read speed of 300 MB/s, and data size: 0B 2026-05-13 17:29:30 [INFO] [partial_tape_scan] Performing partial tape scan. 2026-05-13 17:29:31 [INFO] [verify_tape] Verifying 912 files and 3.3T from tape L94718 2026-05-13 17:29:31 [INFO] [verify_files] ArchiveId of files to verify: 4600832117,4600832092,4600831746,4600832094,4600831356,4600831615,4600831157,... 2026-05-13 17:30:53 [INFO] [verify_files] All file verifications queued for tape L94718 [tape-local@ctaproductionfrontend14(ctaproduction, cephProductionRepack) ~]$ cta-admin sq type instance scheduler tapepool vo library vid files queued data queued oldest youngest priority min age read max drives write max drives cur. mounts cur. files cur. data tapes capacity files on tapes data on tapes full tapes writable tapes Retrieve ctaproduction cephProductionRepack vo_CMS_2023 CMS SPECTRALIB2-LTO9 L94718 912 3.3T 154 72 50 600 10 50 0 0 0 18.0T 5112 19.0T 1 0 ``` 912 matches 912, but it is less than 1000. Do you observe the same behavior? Vladimir Bahyl CERN ---
**Author:** ![vlado](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/v/779978/32.png) [@vlado](https://cta-community.web.cern.ch/u/vlado)\ **Post date:** [13 May 2026 15:55 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/3 "2026-05-13T15:55:18Z")
The problem seems to come from this line 292: ```auto # Terribly simple/stupid, but it will have to do. random_pick_fseqs = random.choices(range(0, max_fseq + 1), k=args.random) ``` in this file: `/opt/cta-ops/ops-venv/lib/python3.9/site-packages/tapeverify/cta_verify_tape.py`. Please just change `random.choices()` function to `random.sample()`. That should fix it. Please let me know how it goes. Vladimir Bahyl CERN ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [23 May 2026 11:05 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/4 "2026-05-23T11:05:13Z")
Thanks for the suggestion Vlado - we will definitely try this change. We have been busy intergrating the verification tools with our infrastructure and have made good progress although there a few loose ends to pick up. We have set up a cron running the feeder that submits a random choice of tapes for verification. I am trying to run cta-ops-verify-tape in a wrapper script which processes a list of an operator-provided list of tapes. In both cases, however, the submitted verifications are vrtually indistinguishable from usual user retrieves. For example, for a tape that is currently being verified the json request looks like this ```auto [root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "CT2602")' { "mountType": "RETRIEVE", "tapepool": "dirac", "logicalLibrary": "asterix_ts", "vid": "CT2602", "queuedFiles": "0", "queuedBytes": "0", "oldestAge": "0", "priority": "0", "minAge": "0", "curMounts": "1", "curFiles": "29", "curBytes": "3959704090696", "tapesCapacity": "20000000000000", "tapesFiles": "1243", "tapesBytes": "23107487512573", "fullTapes": "1", "writableTapes": "0", "sleepingForSpace": false, "sleepStartTime": "0", "diskSystemSleptFor": "", "vo": "dirac", "readMaxDrives": "1", "writeMaxDrives": "4", "mountPolicies": [], "youngestAge": "0", "highestPriorityMountPolicy": "", "lowestRequestAgeMountPolicy": "", "schedulerBackendName": "cephUser", "instanceName": "antares" } ``` As you see “mountPolicies” points to an empty array although I was under the impression that this is not the case, i.e. that is “mountPolicies”: [verification]. How can we make sure this does not happen? In our config we do have: `verification_mount_policy: verification` Best, George ---
**Author:** ![idlarbi](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/i/ebca7d/32.png) [@idlarbi](https://cta-community.web.cern.ch/u/idlarbi)\ **Post date:** [25 May 2026 21:37 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/5 "2026-05-25T21:37:37Z")
Dear George, Unfortunately, this verification process is not yet really ready for production, but we are working on it. I don’t know exactly why you have nothing in mountPolicies, but here are some observations: - In fact, the `verification` option in `verification_mount_policy` is never used and is completely useless. - `cta-ops-verify-tape` uses the CTA binary `cta-verify-file`, but this command does not allow you to pass any sort of mount policy. - The mount policy configuration for verification requests must be entered directly into the `cta-frontend.conf` file from which the verification requests are performed. (`cta.verification.mount_policy verification`). - Furthermore, of course, the corresponding mount policy needs to be defined in the catalogue, which you can create with `cta-admin`. If you have configured all of the above correctly, the mount policies entry should have the corresponding verification mount policy; otherwise, there is an issue in the frontend code. Best, Idriss ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [26 May 2026 11:26 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/6 "2026-05-26T11:26:12Z")
Hi Idriss, Thanks for the reply. I do have ```auto cta.verification.mount_policy verification ``` in the /etc/cta/cta-frontend-xrootd.conf (rather than /etc/cta/cta-frontend.conf). In fact, there was an extra space between `cta.verification.mount_policy` and `verifcation` which I deleted. I submitted another tape for verification and it looked like this time that there is mountPolicy value ```auto [root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "TD4006")' { "mountType": "RETRIEVE", "tapepool": "diamond_m07", "logicalLibrary": "obelix_ts1170", "vid": "TD4006", "queuedFiles": "16", "queuedBytes": "81331126969", "oldestAge": "71", "priority": "5", "minAge": "300", "curMounts": "1", "curFiles": "1", "curBytes": "5091158016", "tapesCapacity": "50000000000000", "tapesFiles": "12428", "tapesBytes": "63259891623572", "fullTapes": "1", "writableTapes": "0", "sleepingForSpace": false, "sleepStartTime": "0", "diskSystemSleptFor": "", "vo": "storaged_dls", "readMaxDrives": "10", "writeMaxDrives": "10", "mountPolicies": [ "verification" ], "youngestAge": "69", "highestPriorityMountPolicy": "verification", "lowestRequestAgeMountPolicy": "verification", "schedulerBackendName": "cephUser", "instanceName": "antares" } ``` but after a few mins - when files started to get vertifed - it was gone…! ```auto [root@cta-adm-fac1 georgep]# cta-admin --json sq | jq -r '.[] | select(.vid == "TD4006")' { "mountType": "RETRIEVE", "tapepool": "diamond_m07", "logicalLibrary": "obelix_ts1170", "vid": "TD4006", "queuedFiles": "0", "queuedBytes": "0", "oldestAge": "0", "priority": "0", "minAge": "0", "curMounts": "1", "curFiles": "16", "curBytes": "81293554789", "tapesCapacity": "50000000000000", "tapesFiles": "12428", "tapesBytes": "63259891623572", "fullTapes": "1", "writableTapes": "0", "sleepingForSpace": false, "sleepStartTime": "0", "diskSystemSleptFor": "", "vo": "storaged_dls", "readMaxDrives": "10", "writeMaxDrives": "10", "mountPolicies": [], "youngestAge": "0", "highestPriorityMountPolicy": "", "lowestRequestAgeMountPolicy": "", "schedulerBackendName": "cephUser", "instanceName": "antares" } ``` ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [26 May 2026 16:20 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/7 "2026-05-26T16:20:29Z")
Since I dont want to re-invent the wheel, I will collect the list of curently verified tapes like this ```auto [root@cta-adm-fac1 georgep]# XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab cta-ops-verification-feeder --list_current_verifications -C /etc/cta-ops/cta-ops-config.yaml 2026-05-26 17:16:16 [INFO] [main] Executed as: /root/.local/bin/cta-ops-verification-feeder --list_current_verifications -C /etc/cta-ops/cta-ops-config.yaml 2026-05-26 17:16:24 [INFO] [main] Currently running verification on tapes: TD0012, TD4006, TD4007 ``` ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [28 May 2026 11:28 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/8 "2026-05-28T11:28:48Z")
Hello @vlado Just to let you know that the source code you suggested above worked. Many thanks! George ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [16 September 2026 07:38 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/9 "2026-09-16T07:38:03Z")
Hi @vlado , Sorry to come back to this…For some reason when using the --all flag ```auto XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab /root/.local/bin/cta-ops-verify-tape -v TD4345 --all -C /etc/cta-ops/cta-ops-config.yaml ``` the verification is not submitted in the CTA queue although the message of the tool is ```auto 2026-09-16 08:33:18 [INFO] [verify_files] All file verifications queued for tape TD4345 ``` Do you have any clues why this can happen? Thanks, George ---
**Author:** ![idlarbi](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/i/ebca7d/32.png) [@idlarbi](https://cta-community.web.cern.ch/u/idlarbi)\ **Post date:** [16 September 2026 08:05 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/10 "2026-09-16T08:05:55Z")
Hi @george_patargias, Can you give us a full log output in debug mode please. I 'll check in the code if there is any logic issue. Thank you, Idriss ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [16 September 2026 10:21 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/11 "2026-09-16T10:21:29Z")
Thanks for the reply. This is is the tool’s output when using the `--verbose` flag (I omit the archive ids 🙂 ) ```auto 2026-09-16 11:16:02 [DEBUG] [configure] Configuring TapeAdmin commands and settings 2026-09-16 11:16:03 [INFO] [verify_tape] Running verify-tape for tape with vid TD4524, read speed of 300 MB/s, and data size: 0B 2026-09-16 11:16:03 [DEBUG] [get_tape_occupancy] Retrieving occupancy of the tape TD4524 2026-09-16 11:16:03 [DEBUG] [run_cmd] Executing shell command: XrdSecSSSKT=/etc/cta/cta-cli.keytab /usr/bin/cta-admin --json --config /etc/cta/cta-cli.conf tape ls --vid TD4524 2026-09-16 11:16:03 [DEBUG] [safe_get_tape_files] Retrieving tape file data for tape TD4524 2026-09-16 11:16:03 [DEBUG] [write_cmd_output] Executing shell command: XrdSecSSSKT=/etc/cta/cta-cli.keytab /usr/bin/cta-admin --json --config /etc/cta/cta-cli.conf tf ls -v TD4524 2026-09-16 11:16:07 [INFO] [whole_tape_scan] Performing whole tape scan. 2026-09-16 11:16:07 [INFO] [verify_tape] Verifying 18409 files and 66.2T from tape TD4524 .... .... 2026-09-16 11:16:07 [INFO] [verify_files] All file verifications queued for tape TD4524 2026-09-16 11:16:07 [DEBUG] [run_cmd] Executing shell command: XrdSecSSSKT=/etc/cta/cta-cli.keytab /usr/bin/cta-admin --json --config /etc/cta/cta-cli.conf tape ch --vid TD4524 --verificationstatus "{\"date\": 1789553767, \"status\": \"ongoing\", \"files_submitted\": 18409, \"files_verified\": 0, \"files_failed\": 0}" ``` Just to say that as a kind of a workaround, I submitted all files from a tape by running `cta-verify-file` in a for loop which works. Best, George ---
**Author:** ![idlarbi](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/i/ebca7d/32.png) [@idlarbi](https://cta-community.web.cern.ch/u/idlarbi)\ **Post date:** [18 September 2026 08:32 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/12 "2026-09-18T08:32:58Z")
Hi, This issue was already mentioned in a previous post: [Use of cta-ops-verify-tape - #41 by george\_patargias](https://cta-community.web.cern.ch/t/use-of-cta-ops-verify-tape/394/41) . Which version are you using, normally since version 3.7.5 the bug is fixed. Best, Idriss ---
**Author:** ![george\_patargias](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/g/b9bd4f/32.png) [@george\_patargias](https://cta-community.web.cern.ch/u/george_patargias)\ **Post date:** [29 September 2026 15:59 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/13 "2026-09-29T15:59:47Z")
Hi Idriss, We are using variable CTA\_TAPEVERIFY\_PACKAGE\_INDEX ?= " [Index of /cta-operations/pip/simple](https://cta-public-repo.web.cern.ch/cta-operations/pip/simple/) "; variable CTA\_TAPEVERIFY\_VERSION ?= “3.7.4.post3”; where these variables are substituted in ```auto #!/bin/bash index="%1$s" ver="%2$s" pipx install --pip-args="--extra-index-url ${index}" tapeverify==${ver} --force pipx ensurepath ``` I can see that the latest version in [Index of /cta-operations/pip/simple/tapeverify](https://cta-public-repo.web.cern.ch/cta-operations/pip/simple/tapeverify/) is 3.8.5 - should we go for that? ---
**Author:** ![idlarbi](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/i/ebca7d/32.png) [@idlarbi](https://cta-community.web.cern.ch/u/idlarbi)\ **Post date:** [30 September 2026 10:10 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/14 "2026-09-30T10:10:58Z")
Hi George, Thank you for that info. Indeed the fix is not present in your version. I encourage you to upgrade to `3.8.5`. If you have any issue on this new version, don’t hesitate to open a new discussion. Best, Idriss ---
**Author:** ![idlarbi](https://cta-community.web.cern.ch/letter_avatar_proxy/v4/letter/i/ebca7d/32.png) [@idlarbi](https://cta-community.web.cern.ch/u/idlarbi)\ **Post date:** [30 September 2026 10:11 UTC](https://cta-community.web.cern.ch/t/cta-ops-verify-tape-usage-check/433/15 "2026-09-30T10:11:17Z")