Multi drive support with CTA 5.10.10.1-1

General Discussion
1

Hi,

Trying to implement the new type of cta-taped config that supports multiple drives (as per https://indico.cern.ch/event/1353243/contributions/5847456/attachments/2820733/4925645/DriveAndTapeDaemonConfiguration-CTAWorkshop-2024.pdf).

I have created the seperate cta-taped-DriveName.conf files, one for each of the four drives controlled by the tapeserver. Apart from DRIVE OPTIONS section, these files contain duplicate directives such as

ObjectStore BackendPath
general FileCatalogConfigFile
taped BufferCount 5120
taped BufferSizeBytes 5242880
…
My question: the seperate cta-taped-DriveName.conf files need to contain
only the DRIVE OPTIONS or also the duplicate directives?

I have also created /etc/systemd/system/cta-taped.service and /etc/systemd/system/cta-taped@.service as per slides 19 and 20

I cannot start cta-taped though, because a config file is not found…

2024-05-17T16:53:07.248484+01:00 getafix-ts03 cta-taped[15003]: LVL=“ERROR” PID=“15003” TID=“15003” MSG=“Caught an unexpected CTA exception, cta-taped cannot start” exceptionMessage=“ConfigurationFile failed: Failed to open configuration file: m_fileName=”

I dont have an /etc/cta/cta-taped.conf any more. Can this be file that is missing and must and the duplicate cta-taped directives??

Thanks,

George

2

If I try to provide a /etc/cta/cta-taped.conf containing the duplicate directives, then cta-taped complains that

2024-05-17T17:08:47.822746+01:00 getafix-ts03 cta-taped[16886]: LVL=“ERROR” PID=“16886” TID=“16886” MSG=“Caught an unexpected CTA exception, cta-taped cannot start” exceptionMessage=“In SourcedParameter::setFromConfigurationFile: mandatory parameter not found: category=taped key=DriveName configFilePath=/etc/cta/cta-taped.conf”

3

Dear George,
the cta-taped config file is intended to be per cta-taped process. So for each process you should have one separate config file.
Each config file should have the full list of config options present. You may point each cta-taped service to its own config file by having a start command such as the following in the service config:

ExecStart=/usr/bin/cta-taped --log-format=json --log-to-file=/var/log/cta/cta-taped-%i.log --config=/etc/cta/cta-taped-%i.conf ${CTA_TAPED_OPTIONS}

Based on the above error message it looks like your config is missing the taped DriveName <name> option, which is needed since new versions of CTA will get this information from cta-taped.conf instead of /etc/TPCONFIG/.

Here is a reference config file from one of our production servers:

4

Hi @rbachman

Many thanks for your. We have managed to get the multi-drive cta-taped running and working.

I noticed that after a while, the logging from the individual cta-taped processes stops, i.e there are log files but they are empty. It is on EL9 that this happenning. Have you see it too?

Best,

George

5

Hi George,
great!
We have an issue open on the log writing situation: taped writing to log file continues to write to rotated log (#570) · Issues · cta / CTA · GitLab
cta-taped is likely still writing to the rotated log file on your end.

For the time being we recommend to use the copytruncate directive in your logrotate config. Here is a sample of what we run on one of our multi-drive machines:

/var/log/cta/cta-taped-IBMLIB4-TS1160-F02C4R2.log {
  compress
  copytruncate
  create 0644 cta tape
  dateext
  dateformat -%Y%m%d-%s
  hourly
  maxage 2
  minsize 1G
  missingok
  nodelaycompress
  olddir /var/log/cta/old
  rotate 60
  su cta tape
  postrotate
    /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
  endscript
}
6

Hi Richard,

Thanks for this! We see a similar issue: cta-taped continues to write to the the rotated log file.

We added the copytruncate directive to /etc/logrotate.d/cta-taped on all tapeservers of our tapeservers and changed the ownership of all cta-taped logs from root:root to cta:tape.

What I mentioned above, is that after a while (at least on our non prod tapeservers, the logging stops.

Would it be possible to paste the rsyslog config you use for cta-taped (EL9 please) in /etc/rsyslog.d/

Best,

George

7

Hi George,
It just so happens that I put some example rsyslog config files into the Alma9 branch of the operations utilities repo: monitoring/rsyslog · alma9 · cta / CTA Operations utilities · GitLab
These are pretty much exactly what we use for Alma9, you will just have to fill in the changeme fields (target and port).
Note that this config is intended for the running cta-taped/frontend in the json output mode, with --log-format=json.

As of CTA 5.10.10.1 we no longer write the logs locally using rsyslog: We let cta-taped and cta-frontend write to the file directly and have rsyslog re-read the file. The purpose of this is basically just for the purpose of collecting the logs elsewhere and making secure copies.
Direct file output is enabled using the --log-to-file=/var/log/cta/cta-taped.log flag on cta-taped.

Be sure to clean up any lingering rsyslog configs that cause writing to the cta-taped log files if/when making the switch. We had some log file confusion in our qa environment due to this.