Hi @vlado,
How is CTA deployed at CERN to use multiple libraries?
Thanks,
Mwai
Dear Mwai,
thanks for asking I will try my best to answer this question here so the wider CTA community can benefit as well. At the same time, this answer could perhaps later be turned into a proper documentation by someone who tried to follow it (as I simply have too much implicit knowledge in my head).
Here is how to configure multiple tape libraries in CTA:
-
On each library, make sure all drives are allowed to request cartridge mounts/dismounts. This is called in IBM terminology control path while on Spectra Logic, such drives are called library exporters.
Example from IBM tape library - see Drive Use column, all drives are Control Path Drives:
-
I do now know how many tape drive per tape server you plan to use (connect). We at CERN run with just one tape drive connected to one tape server (which keeps the operations simple). Anyway, on each tape server you need to make sure the
cta-rmcddaemon is properly configured and running. Since we have only one drive per server, each server/drive is requesting mounts just for itself. If you have more than one drive per server, onecta-rmcddaemon should be enough to serve all (I would guess up to 4) drives connected to that server.
Example ofcta-rmcdrelated configuration from one of our servers:
[root@tpsrv301 ~]# lsscsi -g
[0:0:0:0] disk SEAGATE ST3000NM0023 0003 /dev/sda /dev/sg0
[0:0:1:0] disk SEAGATE ST3000NM0023 0003 /dev/sdb /dev/sg1
[2:0:0:0] tape IBM ULT3580-TD9 NCA0 /dev/st0 /dev/sg2
[2:0:0:1] mediumx IBM 03584L32 1802 /dev/sch0 /dev/sg3
[root@tpsrv301 ~]# ls -la /dev/smc
lrwxrwxrwx. 1 root root 3 Jul 18 15:13 /dev/smc -> sg3
[root@tpsrv301 ~]# sg_inq /dev/smc
standard INQUIRY:
PQual=0 Device_type=8 RMB=1 version=0x03 [SPC]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=0]
EncServ=0 MultiP=1 (VS=1) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=58 (0x3a) Peripheral device type: medium changer
Vendor identification: IBM
Product identification: 03584L32
Product revision level: 1802
Unit serial number: 0000078BB2840401
[root@tpsrv301 ~]# cat /etc/sysconfig/cta-rmcd
DAEMON_COREFILE_LIMIT=unlimited
CTA_RMCD_OPTIONS=/dev/smc
[root@tpsrv301 ~]# ps axwu | grep rmcd | grep -v grep
cta 4490 0.0 0.0 13340 808 ? Ss Oct19 0:07 /usr/bin/cta-rmcd /dev/smc
- You should now be able to see all tape drives configured in that physical library. Check the library configuration first to see if you can communicate via the SCSI media changer device, then check the list of drives.
Example:
[root@tpsrv301 ~]# cta-smc -q L
Vendor/Product/Revision = <IBM 03584L32 1802>
Transport Count = 2, Start = 1
Slot Count = 22356, Start = 1025
Port Count = 255, Start = 769
Device Count = 48, Start = 257
[root@tpsrv301 ~]# cta-smc -q D | head -10
Drive Ordinal Element Addr. Status Vid
0 257 free
1 258 free
2 259 free
3 260 free
4 261 free
5 262 free
6 263 free
7 264 free
8 265 free
IMPORTANT - the element address number in the output above corresponds with the value of element address from the library GUI. However, in CTA we use drive ordinal value to identify which drive we are talking about.
- Configure logical libraries as you like. We split them by tape drive types.
Example:
[root@tpsrv301 ~]# cta-admin logicallibrary ls
library disabled reason c.user c.host c.time m.user m.host m.time comment
IBM1L9 false - CASTOR CASTOR 2020-11-30 16:14 vlado ctaproductionfrontend02 2022-06-29 17:01 IBMLIB1 (IBM TS4500 tape library in B513) - LTO-9 tape drives
IBM360 false - CASTOR CASTOR 2020-06-23 12:39 vlado ctaproductionfrontend02 2022-08-23 12:39 IBMLIB3 (IBM TS4500 tape library in B513) - TS1160F tape drives
IBM460 false - vlado localhost 2021-11-02 20:25 vlado ctaproductionfrontend02 2022-10-10 20:06 IBMLIB4 (IBM TS4500 tape library in B513) - TS1160F tape drives
SPC1L9 false - CASTOR CASTOR 2020-11-30 16:14 vlado ctaproductionfrontend02 2022-09-23 12:50 SPECTRALIB1 (Spectra Logic TFinity tape library in B513) - LTO-9 tape drives
SPC2L9 false - vlado localhost 2021-10-14 16:50 vlado ctaproductionfrontend02 2022-08-11 12:28 SPECTRALIB2 (Spectra Logic TFinity tape library in B613) - LTO-9 tape drives
- Setup properly the
/etc/cta/TPCONFIGfile.
Define the tape drive as you like, make sure the logical library corresponds to where the drive is / what you configured in the above point 4. For the tape drive name, we use a mix of characters (for legacy (CASTOR) reasons we used to be limited to 10, but CTA allows for longer texts now):
VendorName-LibraryNumber-DriveType-Frame-Row-Column= (example below =)IBM-1-LTO9-06-4-2= (short version of the example)I1L90642.
Feel free to invent your own naming, we have been using the above convention for many years as it helps to identify where the tape drive physically is which is helpful for interventions.
Example of our TPCONFIG file:
[root@tpsrv301 ~]# cat /etc/cta/TPCONFIG
#######################################################################
#
# CTA Tape Server Configuration file
#
# tape library
# drive logical system control
# name library device path
I1L90642 IBM1L9 /dev/nst0 smc13
# Tape Drive Comment:
# Tape Service Comment: CTA
# Modified by: vlado
# Modify date: 26-NOV-2021 15:20
#
#######################################################################
#
# Generated on Sun Nov 28 09:17:43 2021 by /usr/local/bin/tape-config-generate
IMPORTANT - the system device is the path to the tape drive device name on the server, while the control path starts always with smcXXX. The XXX = number after smc is the drive ordinal value as explained in the above step 3. Do not mix that with the SCSI media changer device that cta-rmcd is using.
In this example, the tape server is connected to the drive 13 which has element address of 270:
[root@tpsrv301 ~]# cta-smc -q D | grep 13
13 270 free
Repeat that for all tape drives in all libraries. As cta-taped starts, it will check the drive using the configured path and if it is available (and has no tape stuck inside) it will register it in the CTA so you can see it in the cta-admin drive ls output.
Give it a try and let me know if you need more information.
Best regards,
Vladimir
1 Like
Another important note: if you are planning to use multiple tape drives per tape server all the tape drives connected on one tape server must belong to the same logical library.
1 Like