Problem about cta-frontend-grpc, cta-taped, cta-rmcd

Patrick · 12 May 2025 08:23

Summary

I downloaded it according to the official installation instructions of CTA. When I checked the status of the three CTA services, I found various errors, as follows:

Details

– cta-frontend-grpc

cta-frontend-grpc.conf cannot be read successfully
[root@tapesrv01 tmp]# ls -l /usr/lib64/libpq.so.5
lrwxrwxrwx 1 root root 13 May 5 13:51 /usr/lib64/libpq.so.5 → libpq.so.5.17

-log
May 12 14:58:12 tapesrv01 systemd[1]: Starting gRPC frontend for CERN Tape Archive (CTA)…
May 12 14:58:12 tapesrv01 env[3323156]: /usr/bin/cta-frontend-grpc: /usr/lib64/libpq.so.5: no version information available (required by /usr/lib64/libctardbmspostgres.so.0)
May 12 14:58:12 tapesrv01 env[3323156]: cta-frontend-grpc version: 5.11.2.0-1
May 12 14:58:12 tapesrv01 systemd[1]: Started gRPC frontend for CERN Tape Archive (CTA).
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc: /usr/lib64/libpq.so.5: no version information available (required by /usr/lib64/libctardbmspostgres.so.0)
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: terminate called after throwing an instance of ‘cta::exception::Exception’
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: what(): Failed to instantiate object representing CTA logging system: Unknown log URL: syslog:
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/lib64/libctacommon.so.0(cta::exception::Backtrace::Backtrace(bool)+0x6b) [0x7f21705bf4a1]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/lib64/libctacommon.so.0(cta::exception::Exception::Exception(std::basic_string_view<char, std::char_traits >, bool)+0x91) [0x7f21705c04b9]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc() [0x49d76c]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc() [0x47ab38]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc() [0x478e57]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc() [0x46b36a]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/lib64/libc.so.6(+0x29590) [0x7f216aa29590]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/lib64/libc.so.6(__libc_start_main+0x80) [0x7f216aa29640]
May 12 14:58:12 tapesrv01 cta-frontend-grpc[3323157]: /usr/bin/cta-frontend-grpc() [0x46afa5]
May 12 14:58:12 tapesrv01 systemd-coredump[3323159]: [🡕] Process 3323157 (cta-frontend-gr) of user 1001 dumped core.
May 12 14:58:12 tapesrv01 systemd[1]: cta-frontend-grpc.service: Main process exited, code=dumped, status=6/ABRT
May 12 14:58:12 tapesrv01 systemd[1]: cta-frontend-grpc.service: Failed with result ‘core-dump’.
May 12 14:58:12 tapesrv01 systemd[1]: cta-frontend-grpc.service: Scheduled restart job, restart counter is at 5.
May 12 14:58:12 tapesrv01 systemd[1]: Stopped gRPC frontend for CERN Tape Archive (CTA).
May 12 14:58:12 tapesrv01 systemd[1]: cta-frontend-grpc.service: Start request repeated too quickly.
May 12 14:58:12 tapesrv01 systemd[1]: cta-frontend-grpc.service: Failed with result ‘core-dump’.
May 12 14:58:12 tapesrv01 systemd[1]: Failed to start gRPC frontend for CERN Tape Archive (CTA).

================================================================

– cta-taped(The function will run smoothly for the first few seconds, but will fail in the end.)

[root@tapesrv01 tmp]# sudo systemctl status cta-taped
○ cta-taped.service - CERN Tape Archive (CTA) tape server daemon
Loaded: loaded (/etc/systemd/system/cta-taped.service; enabled; preset: disabled)
Active: inactive (dead) since Tue 2025-05-06 15:35:27 CST; 6 days ago
Duration: 59ms
Main PID: 2046209 (code=exited, status=0/SUCCESS)
CPU: 66ms

log
May 06 15:35:27 tapesrv01 cta-taped[2046209]: May 6 15:35:27.592170545 tapesrv01 cta-taped: LVL=“INFO” PID=“2046209” TID=“2046209” MSG=“Configuration entry” category=“taped” key=“TapeCacheMaxAgeSecs” value=“600” source=“Compile time default”
May 06 15:35:27 tapesrv01 cta-taped[2046209]: May 6 15:35:27.592196723 tapesrv01 cta-taped: LVL=“INFO” PID=“2046209” TID=“2046209” MSG=“Configuration entry” category=“taped” key=“RetrieveQueueCacheMaxAgeSecs” value=“10” source=“Compile time default”
May 06 15:35:27 tapesrv01 cta-taped[2046209]: May 6 15:35:27.592247435 tapesrv01 cta-taped: LVL=“INFO” PID=“2046209” TID=“2046209” MSG=“Set log mask” drive_name=“drive1” instance=“production” sched_backend=“cephUser” logMask=“INFO”
May 06 15:35:27 tapesrv01 systemd[1]: cta-taped.service: Deactivated successfully.
May 12 13:52:54 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:10: Failed to parse service type, ignoring: simple #forking
May 12 13:52:54 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:11: Failed to parse service restart specifier, ignoring: on-failure #no
May 12 14:03:51 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:10: Failed to parse service type, ignoring: simple #forking
May 12 14:03:51 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:11: Failed to parse service restart specifier, ignoring: on-failure #no
May 12 14:55:38 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:10: Failed to parse service type, ignoring: simple #forking
May 12 14:55:38 tapesrv01 systemd[1]: /etc/systemd/system/cta-taped.service:11: Failed to parse service restart specifier, ignoring: on-failure #no
================================================================
–cta-rmcd
[root@tapesrv01 tmp]# sudo systemctl status cta-rmcd
× cta-rmcd.service - CERN Tape Archive (CTA) rmcd daemon
Loaded: loaded (/etc/systemd/system/cta-rmcd.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Fri 2025-04-25 16:05:27 CST; 2 weeks 3 days ago
Duration: 822us
Main PID: 3892251 (code=exited, status=1/FAILURE)
CPU: 3ms

Apr 25 16:05:27 tapesrv01 systemd[1]: Starting CERN Tape Archive (CTA) rmcd daemon…
Apr 25 16:05:27 tapesrv01 systemd[1]: Started CERN Tape Archive (CTA) rmcd daemon.
Apr 25 16:05:27 tapesrv01 systemd[1]: cta-rmcd.service: Main process exited, code=exited, status=1/FAILURE
Apr 25 16:05:27 tapesrv01 systemd[1]: cta-rmcd.service: Failed with result ‘exit-code’.

mdavis · 13 May 2025 13:40

Hi @Patrick

cta-frontend-grpc is still under development and there have been some changes to the configuration file in recent versions. Please can you report the contents of /etc/cta/cta-frontend-grpc.conf on your install?

Specifically, the Frontend is failing because the logging URL is set to syslog:. Try setting cta.log.url to file://location/of/your/log/file.

On the tape server, looks like there is a problem with the configuration file /etc/systemd/system/cta-taped.service. Please include the contents of that file as well.

For the mediachanger, take a look at /var/log/cta/cta-rmcd.log to see what the problem is.

Patrick · 14 May 2025 05:26

Hi @mdavis

The following is the content of /etc/cta/cta-frontend-grpc.conf on my install, but it doesn’t seem to work successfully(including reading the setting of the config document).

frontendGrpc = {
  port = 50051
  catalogueConnectionString = "postgresql://cta:password@localhost/ctacatalogue"
  logUrl = "file:///var/log/cta/cta-frontend-grpc.log"
  logLevel = "INFO"
  agentName = "frontend-grpc01"
}

The following is the content of /etc/systemd/system/cta-taped.service on my install.

[Unit]
Description=CERN Tape Archive (CTA) tape server daemon
After=syslog.target network-online.target

[Service]
EnvironmentFile=-/etc/sysconfig/cta-taped
ExecStart=/usr/bin/cta-taped --config /etc/cta/cta-taped.conf --log-dir /var/log/cta
LimitCORE=infinity
OOMScoreAdjust=-1000
Type=simple  #forking
Restart=on-failure #no

[Install]
WantedBy=default.target

/var/log/cta/cta-rmcd.log has no content to display.

Thank you for your reply.
Patrick

mdavis · 14 May 2025 07:21

The format of /etc/cta/cta-frontend-grpc.conf is wrong. Please look at cta-frontend-grpc.conf.example and create your configuration from there.

I think the configuration file you have may be the one from the proof-of-concept gRPC Frontend. The code has since been refactored and integrated with the common Frontend code, which already has a configuration file parser (incompatible with the one used in the POC code).

In cta-taped.service, as it says in the log messages, there is a problem with lines 10 and 11. Remove those lines:

Type=simple  #forking
Restart=on-failure #no

Instead add the line:

RemainAfterExit=yes

For rmcd, without a log message I have no idea what error is causing it to exit.

Patrick · 14 May 2025 08:12

Thank you for your reply, the service of cta-taped work successfully.
But I am confused about the document of cta-frontend-grpc.conf.example, I don’t know how to modify it.

# @project      The CERN Tape Archive (CTA)
# @copyright    Copyright © 2024 CERN
# @license      This program is free software, distributed under the terms of the GNU General Public
               Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". You can
               redistribute it and/or modify it under the terms of the GPL Version 3, or (at your
               option) any later version.
               This program is distributed in the hope that it will be useful, but WITHOUT ANY
               WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
               PARTICULAR PURPOSE. See the GNU General Public License for more details.
               In applying this licence, CERN does not waive the privileges and immunities
               granted to it by virtue of its status as an Intergovernmental Organization or
               submit itself to any jurisdiction.
####################################
# GENERAL OPTIONS (required)
###########

For rmcd, The following is the content of /etc/cta/cta-rmcd.conf on my install.

[root@tapesrv01 tmp]# cat /etc/cta/cta-rmcd.conf
# @project      The CERN Tape Archive (CTA)
# @copyright    Copyright © 2015-2022 CERN
# @license      This program is free software, distributed under the terms of the GNU General Public
#               Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". You can
#               redistribute it and/or modify it under the terms of the GPL Version 3, or (at your
#               option) any later version.
#
#               This program is distributed in the hope that it will be useful, but WITHOUT ANY
#               WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
#               PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
#               In applying this licence, CERN does not waive the privileges and immunities
#               granted to it by virtue of its status as an Intergovernmental Organization or
#               submit itself to any jurisdiction.

################################################################################
#
# CTA Remote Media Changer Daemon Example Configuration File
#
################################################################################
[RMCD]
logLevel = DEBUG

[ACS]
vendor = SPECTRA
devname = /dev/sg4

The following is the content of /etc/systemd/system/cta-rmcd.service on my install.


[Unit]
Description=CERN Tape Archive (CTA) rmcd daemon
After=syslog.target network-online.target

[Service]
User=cta
EnvironmentFile=-/etc/sysconfig/cta-rmcd
ExecStart=/usr/bin/cta-rmcd --config /etc/cta/cta-rcmd.conf 
LimitCORE=infinity
Type=forking
Restart=no

[Install]
WantedBy=default.target

Thank you very much.
Patrick

mdavis · 14 May 2025 09:45

As I mentioned, the gRPC Frontend is still under development and the configuration file has changed with recent CTA versions. I am not sure which version you have installed.

If you are running in containers with EOS, we recommend using the XRootD Frontend for the time being. If you are running with dCache, then you will need to use the gRPC Frontend, but we do not yet have this setup working in our Continuous Integration system. If you can explain a little more about your setup I will be better able to help you.

The /etc/systemd/system/cta-rmcd.service file looks fine to me.

Patrick · 15 May 2025 02:49

Hi, @mdavis
OK, I try to use cta-frontend-xrootd.conf. The following is the content of the service.

[root@tapesrv01 tmp]# systemctl status cta-frontend
● cta-frontend.service - CTA frontend plugin for the XRoot deamon
     Loaded: loaded (/etc/systemd/system/cta-frontend.service; enabled; preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2025-05-15 10:43:00 CST; 6s ago
    Process: 3918465 ExecStart=/usr/bin/xrootd -l /var/log/cta-frontend-xrootd.log -c /etc/cta/cta-frontend-xrootd.conf -k fifo -n cta (code=exited, status=1/FAILURE)
   Main PID: 3918465 (code=exited, status=1/FAILURE)
        CPU: 9ms
================================================================
[root@tapesrv01 tmp]# cat /etc/systemd/system/cta-frontend.service
[Unit]
Description=CTA frontend plugin for the XRoot deamon 
After=network-online.target

[Service]
EnvironmentFile=-/etc/sysconfig/cta-frontend
ExecStart=/usr/bin/xrootd -l /var/log/cta-frontend-xrootd.log -c /etc/cta/cta-frontend-xrootd.conf -k fifo -n cta
User=cta
Type=simple
Restart=always
RestartSec=10
StartLimitInterval=0
StartLimitBurst=0
OOMScoreAdjust=-1000
LimitCORE=infinity
LimitNOFILE=65536

[Install]
WantedBy=default.target

And the setting of /etc/cta/cta-frontend-xrootd.conf:

cta.instance_name CI_local

# Unique string to identify the backend scheduler resources. As an exmple, it can be structured as:
# "[ceph|postgres|vfs][User|Repack]".
cta.schedulerdb.scheduler_backend_name vfsCI

# CTA ObjectStore options
#cta.objectstore.backendpath /tmp/jobStoreXXXXXXX

# CTA Scheduler DB - thread options
cta.schedulerdb.numberofthreads 500
cta.schedulerdb.threadstacksize_mb 1

# CTA Scheduler DB - enable requests for user or repack
cta.schedulerdb.enable_repack_requests on
cta.schedulerdb.enable_user_requests on

# CTA Scheduler DB - cache timeout options
# cta.schedulerdb.tape_cache_max_age_secs 600
# cta.schedulerdb.retrieve_queue_cache_max_age_secs 10

# CTA Catalogue options
cta.catalogue.numberofconnections 10

# Maximum file size (in GB) that the CTA Frontend will accept for archiving
cta.archivefile.max_size_gb 1000

# Disallow 0-length files
cta.archivefile.zero_length_files_forbidden on
# Tapepools exempted from this check (if enabled)
cta.archivefile.zero_length_files_forbidden_vo_exception_list vo1 vo2 vo3

# CTA Frontend log URL
cta.log.url file:/var/log/cta/cta-frontend.log

# CTA Logger log level
# Valid log levels are EMERG, ALERT, CRIT, ERR, WARNING, NOTICE (==USERERR), INFO, DEBUG
# cta.log.level DEBUG

# CTA Logger log format
# Valid formats are default, json
# cta.log.format json

# CTA XRootD SSI Protobuf log level
# cta.log.ssi debug protobuf
cta.log.ssi warning

# CTA Repack buffer URL
# cta.repack.repack_buffer_url root://ctaeos//eos/ctaeos/repack

# CTA Verification Mount Policy
cta.verification.mount_policy verification

# Keytab containing gRPC endpoints and tokens for each disk instance
#cta.ns.config /etc/cta/eos.grpc.keytab

#
# XRootD/SSI options
#

# Enable SSI debug logging
#ssi.trace debug

# Load the CTA SSI plugin
xrootd.fslib -2 libXrdSsi.so

# Load the SSI module
ssi.svclib libXrdSsiCta.so

# Use the security module
xrootd.seclib libXrdSec.so

# Security protocol specification
#
# Note: the XRootD process needs to be able to read the keytab file
sec.protocol krb5 /etc/cta/cta-frontend.krb5.keytab cta/cta-frontend@TEST.CTA
sec.protocol sss -s /etc/cta/eos.sss.keytab

# Only Kerberos 5 and sss are allowed
sec.protbind * only sss krb5

# Turn off asynchronous I/O
xrootd.async off

# Use a port other than 1094, already used by EOS xroot server
xrd.port 10955

For rmcd, which section maybe what I can check, please give me some advice.

Thank you very much.
Patrick

mdavis · 15 May 2025 07:46

Hi Patrick, I am not able to advise you on your rmcd problem as I don’t have enough information about what the problem actually is. Can you explain in more detail and provide log messages?

Patrick · 15 May 2025 08:51

Hi, @mdavis
I think the error of rmcd may be the setting of /etc/cta/cta-rmcd.conf.

[root@tapesrv01 tmp]# cat /etc/cta/cta-rmcd.conf
[RMCD]
logLevel = DEBUG

[ACS]
vendor = SPECTRA
devname = /dev/sg4
===============================================================
[root@tapesrv01 tmp]# sudo -u cta /usr/bin/cta-rmcd --config /etc/cta/cta-rmcd.conf
Unknown option
===============================================================
[root@tapesrv01 tmp]# systemctl status cta-rmcd
× cta-rmcd.service - CERN Tape Archive (CTA) rmcd daemon
     Loaded: loaded (/etc/systemd/system/cta-rmcd.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Thu 2025-05-15 16:55:21 CST; 5s ago
   Duration: 1ms
    Process: 3973436 ExecStart=/usr/bin/cta-rmcd --config /etc/cta/cta-rmcd.conf (code=exited, status=1/FAILURE)
   Main PID: 3973436 (code=exited, status=1/FAILURE)
        CPU: 1ms

May 15 16:55:21 tapesrv01 systemd[1]: Started CERN Tape Archive (CTA) rmcd daemon.
May 15 16:55:21 tapesrv01 cta-rmcd[3973436]: Unknown option
May 15 16:55:21 tapesrv01 systemd[1]: cta-rmcd.service: Main process exited, code=exited, status=1/FAILURE
May 15 16:55:21 tapesrv01 systemd[1]: cta-rmcd.service: Failed with result 'exit-code'.

Thank you.
Patrick

mdavis · 15 May 2025 11:56

In our production setup, cta-rmcd.conf is empty, everything set to default.

While we can provide answers to specific questions on this forum, we do not have resources to debug your setup for you. If you need further help, the first thing you should do is get a log for rmcd.

If you can’t get to that stage, try starting rmcd directly from the command line and check the output and return code.

Patrick · 16 May 2025 09:04

OK, I’ll try my best and if there is any new progress on rmcd, I’ll update the status here.

Thank you.
Patrick