Questions on verification queuing

Hello,

We are submitting multiple files for verification via look like the following

vid=TD4615; for i in $(cat ${vid}_archiveids_minusone); do XrdSecPROTOCOL=sss XrdSecSSSKT=/etc/cta/tape-verification.keytab cta-verify-file --id $i --vid $vid; done

In this case, 9,638 files were submitted and an equal number of job objects were created and added to the object store. Looking immediately at the cta-admin sq output for this tape, I see number much smaller (>7,000)

Looking at the queue objects

cat /var/log/cta/cta-frontend.log | grep "added job to queue" | grep tapeVid\=\"TD4615\"  | awk -F "queueObject=" '{print $2}' | awk '{print $1}'  | tr -d '"'  | sort | uniq -c
   1450 RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-150148
    725 RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-151796
   7463 RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-152525

These numbers do add up to 9,638 but the first two queue objects

RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-150148
RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-151796

have been deleted from the object store

[root@cta-front01 ~]# cta-objectstore-dump-object RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-150148                                 Object not found in the object store
[root@cta-front01 ~]# cta-objectstore-dump-object RetrieveQueueToTransferForUser-TD4615-Frontend-cta-front01.scd.rl.ac.uk-704523-20260924-13:02:50-0-151796                                 Object not found in the object store

whereas the third is still there and it points to 6,674 job objects which is very close to the number I saw when I inspected the queue via cta-admin sq | grep TD4615

Can you please explain?

It has happened that for this and other tapes and not all files that were submitted via cta-verify-file were actually verified…

Is it again a question of upgrading the version of ops tools (we have now 3.7.4.post3)

Many thanks,

George

Hi George,

The deleted queue objects do not necessarily indicate lost jobs: CTA removes empty queues and can recreate them as submissions continue. The queue count shows pending jobs, so it could be that processing already started and therefore shows a number that is lower than the initial amount of submitted jobs.

What CTA version are you running? Did you see any activity in the logs for the relevant tapes? It might help if you could share the logs for a few affected ArchiveI IDs. That may help trace where the problem is.

From what I can tell, it’s unlikely that updating the version of the ops tools is the reason for this.

Cheers,
Niels