Cta-ops-verification-feeder error

General Discussion
1

Hello,

We get the following error when we try ro run cta-ops-verification-feeder (by selecting a random tape for verification as a test)

(venv) [root@cta-adm-fac1 georgep]# XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab cta-ops-verification-feeder -C cta-operations-utilities/cta-ops-config.yaml --maxverify 1
2026-05-13 15:52:00 [INFO] [main] Executed as: /root/georgep/venv/bin/cta-ops-verification-feeder -C cta-operations-utilities/cta-ops-config.yaml --maxverify 1
2026-05-13 15:52:13 [INFO] [mark_verified_tape_as_done] Verification for tape TD4536 has finished since the last cta-ops-verification-feeder run
2026-05-13 15:52:13 [INFO] [main] Currently running verification on tapes:
2026-05-13 15:52:13 [INFO] [main] 0 tapes are currently being verified, target is 1
2026-05-13 15:52:13 [INFO] [main] 16579 tapes are eligible for verification
2026-05-13 15:52:13 [INFO] [main] Based on policy: last_verified, following new tapes have been selected for verification: CL0001
2026-05-13 15:52:13 [INFO] [main] Submitting verification for tape CL0001 (media type: LTO9, logical library: asterix_lto, tape pool: atl23raw, total files: 5480, total bytes: 17915391255300) using command: cta-verify-tape --vid CL0001 --first 10 --last 10 --read_time 30
2026-05-13 15:52:13 [WARNING] [main] Verification for tape CL0001 could not be submitted. (STDERR: /bin/bash: line 1: cta-verify-tape: command not found)
(venv) [root@cta-adm-fac1 georgep]#

I am not sure why the tool cannot find cta-verify-tape command as we are already running cta-ops-verify-tape without this error.

Thanks,

George

2

Dear George,

In your configuration file cta-operations-utilities/cta-ops-config.yaml how does this section look like?

  # -------------------------------
  # CTA Tape Verification
  # -------------------------------
  cta-ops-tape-verify:
    debug: false
    logger:
      log_dir: "/var/log/cta-ops/verification/"
    cta-ops-verify-tape:
      default_read_data_size: '0B'
      default_read_time: 0
      default_first: 10
      default_random: 10
      default_last: 10
    cta-ops-verification-feeder:
      verification_mount_policy: 'verification'
      default_min_age: 0
      default_max_verify: 10
      default_min_data_on_tape: 0
      default_min_relative_capacity: 0
      default_verify_options: '--first 10 --last 10 --read_time 30'
      default_verify_policy: 'random'
      default_tape_verify_path: 'cta-ops-verify-tape'
      default_feeder_log_path: '/var/log/cta/verification/cta-verification-feeder.log'
      ts_format: '%Y-%m-%d %H:%M:%S'
      sleep_time: 120  # 2* 60

On our frontend:

[tape-local@ctaproductionfrontend11(ctaproduction, cephUser) ~]$ type cta-ops-verify-tape
cta-ops-verify-tape is /opt/cta-ops/ops-venv/bin/cta-ops-verify-tape

Vladimir Bahyl
CERN

3

Hi Vlado,

In my config , the same line pointed to cta-verify-tape instead of cta-ops-verify-tape

default_tape_verify_path: 'cta-verify-tape'

I replaced it with cta-ops-verify-tape and the above error is gone now but the verifcation still fails to be submited

(venv) [root@cta-adm-fac1 georgep]# XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab cta-ops-verification-feeder -C cta-operations-utilities/cta-ops-config.yaml --maxverify 1
2026-05-13 16:56:35 [INFO] [main] Executed as: /root/georgep/venv/bin/cta-ops-verification-feeder -C cta-operations-utilities/cta-ops-config.yaml --maxverify 1
2026-05-13 16:56:43 [INFO] [main] Currently running verification on tapes:
2026-05-13 16:56:43 [INFO] [main] 0 tapes are currently being verified, target is 1
2026-05-13 16:56:43 [INFO] [main] 16581 tapes are eligible for verification
2026-05-13 16:56:43 [INFO] [main] Based on policy: last_verified, following new tapes have been selected for verification: CL0001
2026-05-13 16:56:43 [INFO] [main] Submitting verification for tape CL0001 (media type: LTO9, logical library: asterix_lto, tape pool: atl23raw, total files: 5480, total bytes: 17915391255300) using command: cta-ops-verify-tape --vid CL0001 --first 10 --last 10 --read_time 30
2026-05-13 16:56:43 [WARNING] [main] Verification for tape CL0001 could not be submitted. (STDERR: )
(venv) [root@cta-adm-fac1 georgep]#

I noticed that I have

default_verify_policy: ‘last_verified’

but I tried random and got the same result.

Thanks,

George

4

Please add --verbose option to see if you get more output.

5

Thanks Vlado. I added –verbose in the cta-ops-verification-feeder command line and it looks like cta-ops-verify-tape fails to run

2026-05-13 17:23:36 [DEBUG] [run_cmd] Executing shell command: cta-ops-verify-tape --vid CL0001 --first 10 --last 10 --read_time 30
2026-05-13 17:23:36 [DEBUG] [run_cmd] Command failed: cta-ops-verify-tape --vid CL0001 --first 10 --last 10 --read_time 30 (STDERR: ).

It is strange because I can run cta-ops-verify-tape outside the feeder.

Best,

George

6

Dear George,
My info may be a little out of date, but it might be worth a shot:

Is there a difference in how you run the cta-ops-verify-tape and cta-ops-verification-feeder, in terms of user account or environment?
I ask because when using just the executable name in the default_tape_verify_path of the config files above, it should be picked up based on the environment’s $PATH variable, which might differ.
You could try specifying it absolutely, with something like /usr/bin/…/cta-ops-verify-tape (or the venv equivalent) to see if it makes a difference.

Note it should also be possible to overwrite the value interactively using cta-ops-verification-feeder --verify-path <path-to-cta-ops-verify-tape> , as it takes precedence over the config file.

7

Thanks Richard.

Indeed, we had not specified XrdSecSSSKT in a couple of places in the config.

cta-admin: XrdSecSSSKT=/etc/cta/cta-cli.keytab /usr/bin/cta-admin
...
cta_verify_file: XrdSecSSSKT=/etc/cta/tape-verification.keytab /usr/bin/cta-verify-file
      --instance tape-verification --id

Once this was done, the command ran as expected.

1 Like