Cta-ops-verify-tape ignoring --random limit i(v3.8.2)

General Discussion
1

Hello,

We have detected an unexpected behavior in the verification module of the CTA OPS tools after updating from version 3.7.3 to 3.8.2.

The main issue is that the verify-tape process is processing significantly more files than the ones sent by the verification-feeder.

Expected behavior (Based on v3.7.3)

In the previous version, when the verify-tape was triggered with a configuration to read 30 files (first 10, random 10 and last 10), the system read exactly those 30 files.

Log file verification-feeder (v3.7.3):

2026-07-08 16:23:25 [INFO] [main] Submitting verification for tape S00024 (media type: LTO10, logical library: IBMLIB2-LTO10, tape pool: vo-cms.t1d0_CTA_data_Run2025E, total files: 6828, total bytes: 30241898664372) using command: cta-ops-verify-tape --vid S00024 --first 10 --last 10 --read_time 30
2026-07-08 16:23:31 [INFO] [main] Tape S00024 successfully submitted for verification.
2026-07-08 16:23:31 [INFO] [main] All verification jobs submitted successfully

Log verify-tape (v3.7.3):

2026-07-08 16:23:25 [INFO] [verify_tape] Running verify-tape for tape with vid S00024, read speed of 300 MB/s, and data size: 566.2G
2026-07-08 16:23:26 [INFO] [partial_tape_scan] Performing partial tape scan.
2026-07-08 16:23:28 [INFO] [verify_tape] Verifying 30 files and 141.9G from tape S00024
2026-07-08 16:23:28 [INFO] [verify_files] ArchiveId of files to verify: 4297160241,4297160301,4297159913,4297160277,4297160357,4297160071,4297160102,4297159266,4297160186,4297159973,4297150564,4297155807,4297265333,4297275475,4297315236,4297319260,4297320269,4297322066,4297309133,4297315092,4297327144,4297324268,4297323073,4297331639,4297333916,4297320252,4297328183,4297322108,4297311735,4297331241

Current behavior (Version 3.8.2)

With the new version, we can see that the verification-feeder process correctly launches the 30 files set in our configuration file, but internally verify-tape ends up selecting and verifying many more (347 files in this example):

Log file verification-feeder (v3.8.2):

2026-08-06 11:11:14 [INFO] [main] Submitting verification for tape S00036 (media type: LTO10, logical library: IBMLIB2-LTO10, tape pool: vo-atlas.data24_13p6TeV, total files: 18306, total bytes: 30634297766689) using command: cta-ops-verify-tape --vid S00036 --first 10 --last 10 --read_time 30
2026-08-06 11:11:58 [INFO] [main] Tape S00036 successfully submitted for verification.

Log verify-tape (v3.8.2):

2026-08-06 11:11:14 [INFO] [verify_tape] Running verify-tape for tape with vid S00036, read speed of 300 MB/s, and data size: 566.2G
2026-08-06 11:11:16 [INFO] [partial_tape_scan] Performing partial tape scan.
2026-08-06 11:11:20 [INFO] [partial_tape_scan] Found 18286 existing valid fseqs on tape.
2026-08-06 11:11:20 [INFO] [partial_tape_scan] Selected 327 random files (total size: 570630084429 bytes)
2026-08-06 11:11:22 [INFO] [verify_tape] Verifying 347 files and 570.6G from tape S00036
2026-08-06 11:11:22 [INFO] [verify_files] ArchiveId of files to verify: 
...
2026-08-06 11:11:58 [INFO] [verify_files] All file verifications queued for tape S00036

(I have shortened the list of ArchiveIds for readability).

We noticed that it correctly selects and verifies the first 10 and last 10 files on the tape, but instead of taking the default 10 random files, it selects 327. If we try explicitly passing the --random 10 option, it still continues to select more files than requested.

For reference, here is our current configuration:

cta-ops-tape-verify:
    debug: false
    logger:
      log_dir: "/var/log/cta-ops/verification/"
    cta-ops-verify-tape:
      default_read_data_size: '0B'
      default_read_time: 0
      default_first: 10
      default_random: 10
      default_last: 10
    cta-ops-verification-feeder:
      verification_mount_policy: 'verification'
      default_min_age: 0
      default_max_verify: 10
      default_min_data_on_tape: 0
      default_min_relative_capacity: 0
      default_verify_options: '--first 10 --last 10 --read_time 30'
      default_verify_policy: 'last_verified'
      default_tape_verify_path: 'cta-ops-verify-tape'
      ts_format: '%Y-%m-%d %H:%M:%S'
      sleep_time: 120

We are currently unsure if this change is caused by an issue in our configuration (e.g., a new parameter required for version 3.8.2) or if it is a bug introduced in the latest releases.

Thank you very much for your help.

Best regards,

Esther

2

Esther,

Thank you for your message and apologies for this new surprised behavior.

I have recently fixed a problem when the cta-ops-verify-tape would ignore the --read_time 30 option and always only verify 10+10+10 files.

What you are seeing is correct behavior:

2026-08-06 11:11:14 [INFO] [main] Submitting verification for tape S00036 (media type: LTO10, logical library: IBMLIB2-LTO10, tape pool: vo-atlas.data24_13p6TeV, total files: 18306, total bytes: 30634297766689) using command: cta-ops-verify-tape --vid S00036 --first 10 --last 10 --read_time 30

2026-08-06 11:11:14 [INFO] [verify_tape] Running verify-tape for tape with vid S00036, read speed of 300 MB/s, and data size: 566.2G
2026-08-06 11:11:22 [INFO] [verify_tape] Verifying 347 files and 570.6G from tape S00036

From the example above, you try to read first 10, last 10 and then keep the tape mounted for 30 minutes. The verification script calculates how much data it needs to read (expecting that the drive can read at 300 MB/s). For 30 minutes, that is 566 GB.
Depending on the file sizes (which are different on every tape), it then select appropriate number of files (347 in your example).

I am now going to check if I introduced another error elsewhere when only using --random 10.

Please wait …

Vladimir

3

Dear Esther,

Here is the first test I did: reading first 10, last 10 and random 10 files:

[tape-local@ctaproductionfrontend11(ctaproduction, cephUser) cta-ops]$ cta-ops-verify-tape --vid L48179 --first 10 --last 10 --random 10
2026-08-07 12:07:44 [INFO] [verify_tape] Running verify-tape for tape with vid L48179, read speed of 300 MB/s, and data size: 0B
2026-08-07 12:07:45 [INFO] [partial_tape_scan] Performing partial tape scan.
2026-08-07 12:07:45 [INFO] [partial_tape_scan] Found 3456 existing valid fseqs on tape.
2026-08-07 12:07:45 [INFO] [partial_tape_scan] Selected 10 random files (total size: 44018373192 bytes)
2026-08-07 12:07:45 [INFO] [verify_tape] Verifying 30 files and 44.0G from tape L48179
2026-08-07 12:07:45 [INFO] [verify_files] ArchiveId of files to verify: 4993599230,4993599212,4993599071,4993599224,4993599202,4993598791,4993599047,4993599055,4993599049,4993599183,4993629342,4993646471,4993649512,4993661500,4993663314,4993680004,4993729140,4993750831,4993785372,4993951711,4993975152,4993974616,4993975668,4993975149,4993975240,4993975674,4993975158,4993975206,4993975689,4993973971
2026-08-07 12:07:48 [INFO] [verify_files] All file verifications queued for tape L48179

30 files were submitted. This is correct behavior, but it only reads 44 GB from tape. This will be done very quickly so you risk shoe-shining the tapes and this puts too much stress on the whole tape infrastructure (robotics + drives) for very little (gain).

Second test was: reading first 10, last 10 and keeping the tape in the drive for 10 minutes (counting on drive reading at 300 MB/s):

[tape-local@ctaproductionfrontend11(ctaproduction, cephUser) cta-ops]$ cta-ops-verify-tape --vid L48179 --first 10 --last 10 --read_time 10
2026-08-07 12:08:03 [INFO] [verify_tape] Running verify-tape for tape with vid L48179, read speed of 300 MB/s, and data size: 188.7G
2026-08-07 12:08:04 [INFO] [partial_tape_scan] Performing partial tape scan.
2026-08-07 12:08:04 [INFO] [partial_tape_scan] Found 3456 existing valid fseqs on tape.
2026-08-07 12:08:04 [INFO] [partial_tape_scan] Selected 33 random files (total size: 194532434267 bytes)
2026-08-07 12:08:04 [INFO] [verify_tape] Verifying 53 files and 194.5G from tape L48179
2026-08-07 12:08:04 [INFO] [verify_files] ArchiveId of files to verify: 4993599230,4993599212,4993599071,4993599224,4993599202,4993598791,4993599047,4993599055,4993599049,4993599183,4993598778,4993606660,4993617565,4993635977,4993651224,4993679542,4993679994,4993689735,4993694601,4993697038,4993717087,4993720463,4993732423,4993737905,4993737882,4993756337,4993780606,4993781334,4993797080,4993803056,4993804013,4993839159,4993854158,4993861923,4993866952,4993874984,4993884223,4993884267,4993889825,4993891678,4993898466,4993936943,4993968645,4993975152,4993974616,4993975668,4993975149,4993975240,4993975674,4993975158,4993975206,4993975689,4993973971
2026-08-07 12:08:09 [INFO] [verify_files] All file verifications queued for tape L48179

53 files were submitted.

Please let me know if you see similar behavior and/or if I answered your questions.

Best regards,

Vladimir

4

Dear Vlado,

Thank you very much for the detailed explanation and the quick reply. Since we hadn’t changed anything in our configuration, I didn’t understand the change in the tape verification behavior.

Based on your comments, I understand that previous versions were ignoring the read_time option. With the current release, it makes no sense to use both read_time and random together. We will just use read_time, as you suggested, it’s a much better approach.

Many thanks,

Esther