Kerberos keytab issue after updating frontend to 5.11.18.0-2

Troubleshooting
1

Summary

After updating a test frontend from 5.11.11.0-1 to 5.11.18.0-2, cta-frontend-grpc generates continuous errors about being unable to find a Kerberos principal.

Details

CTA version: 5.11.18.0-2
Operating System and version: AlmaLinux 9.8
Xrootd version: xrootd-server-5.9.2-1.el9.x86_64
Objectstore backend: CEPH

Steps to reproduce

Confirm principal is present

[root@ctaitb01 cta]# klist -k cta-frontend.keytab
Keytab name: FILE:cta-frontend.keytab
KVNO Principal
---- --------------------------------------------------------------------------
2 cta/ctaitb01.fnal.gov@FNAL.GOV
2 cta/ctaitb01.fnal.gov@FNAL.GOV

Restart cta-frontend-grpc
Errors continue

What is the current bug behaviour?

cta-frontend-grpc fails to negotiate Kerberos authentication.

What is the expected correct behaviour?

cta-frontend-grpc should be able to negotiate Kerberos authentication. These errors were not seen prior to this upgrade.

Relevant logs and/or screenshots

{“epoch_time”:1780512157.226145139,“local_time”:“2026-06-03T13:42:37-0500”,“cta_version”:“5.11.18.0-2”,“log_schema_version”:“0.1.0”,“hostname”:“ctaitb01”,“program”:“cta-frontend”,“source_location”:“ServerNegotiationRequestHandler.cpp:74”,“log_level”:“ERROR”,“pid”:3847640,“tid”:3847640,“message”:“In grpc::server::NegotiationRequestHandler::acquireCreds(): gss_acquire_cred() major status.”,“instance”:“itb”,“sched_backend”:“cephUser”,“GSS-API-ERROR:0”:“No credentials were supplied, or the credentials were unavailable or inaccessible”,“tag”:“0x2758700”}
{“epoch_time”:1780512157.226207486,“local_time”:“2026-06-03T13:42:37-0500”,“cta_version”:“5.11.18.0-2”,“log_schema_version”:“0.1.0”,“hostname”:“ctaitb01”,“program”:“cta-frontend”,“source_location”:“ServerNegotiationRequestHandler.cpp:74”,“log_level”:“ERROR”,“pid”:3847640,“tid”:3847640,“message”:“In grpc::server::NegotiationRequestHandler::acquireCreds(): gss_acquire_cred() minor status.”,“instance”:“itb”,“sched_backend”:“cephUser”,“GSS-API-ERROR:0”:“No key table entry found for cta/ctaitb01@FNAL.GOV”,“tag”:“0x2758700”}
{“epoch_time”:1780512157.226950060,“local_time”:“2026-06-03T13:42:37-0500”,“cta_version”:“5.11.18.0-2”,“log_schema_version”:“0.1.0”,“hostname”:“ctaitb01”,“program”:“cta-frontend”,“source_location”:“NegotiationService.cpp:112”,“log_level”:“ERROR”,“pid”:3847640,“tid”:3847640,“message”:“Failed to initialize negotiation handler”,“instance”:“itb”,“sched_backend”:“cephUser”,“error”:“In grpc::server::NegotiationRequestHandler::acquireCreds(): Failed to get Kerberos credentials.”,“keytab”:“/etc/cta/cta-frontend.keytab”,“servicePrincipal”:“cta/ctaitb01”}

Principal is owned by cta:tape. This is what we had to do to get this working in the past.

-rw------- 1 cta tape 164 Jun 3 12:05 cta-frontend.keytab

Possible causes

I am not sure why cta-frontend-grpc is insisting to look for cta/ctaitb01 rather than cta/ctaitb01.fnal.gov which is configured in /etc/cta/cta-frontend-grpc.conf.

[root@ctaitb01 cta]# grep krb5 /etc/cta/cta-frontend-grpc.conf 
sec.protocol krb5 /etc/cta/cta-frontend.keytab cta/ctaitb01.fnal.gov@FNAL.GOV
sec.protbind * only krb5 sss
2

Hi Tim,

These configurations that you mention here are specific to the SSI (XRootD) Frontend:

They are not being read in case of the gRPC Frontend, so what you see in the logs are actually defaults.

If you want to test the gRPC Frontend, please try setting up these two values instead in the configuration and let me know if it works:

grpc.keytab /etc/cta/cta-frontend.keytab
grpc.service_principal cta/ctaitb01.fnal.gov@FNAL.GOV

Best,
Joao

3

Thank you! I had looked at those files and thought something seemed off. We will reconfigure and try further testing.