Read mount failure ('volume in use') when requested just after a write session

Hi,

We have encountered a read mount error for a tape because the volume was already in use for writing on another drive:

{
  "epoch_time": 1786633343.156102670,
  "local_time": "2026-08-13T17:02:23+0200",
  "cta_version": "5.11.18.0-2",
  "log_schema_version": "0.1.0",
  "hostname": "ctatps010",
  "program": "cta-taped",
  "source_location": "TapeSingleThreadInterface.hpp:99",
  "log_level": "ERROR",
  "pid": 3875248,
  "tid": 3876809,
  "message": "Failed to mount the tape for read-only access",
  "drive_name": "IBMLIB2-LTO10-F04C3R3",
  "instance": "prod",
  "sched_backend": "cephprod",
  "bytesQueued": 2405022,
  "capacityInBytes": 30000000000000,
  "drive_Slot": "smc7",
  "exceptionMessage": "Failed to mount tape for read-only access: vid=S00217 slot=smc7: Failed to mount tape in SCSI tape-library for read/write access: vid=S00217 librarySlot=smc7: Received error from rmcd: rmcRc=2204 rmcErrorStream=smc_mount: Asked for S00217, got reply for S00217\n smc_mount: Location: data transfer element (0x4)\n smc_mount: SR018 - mount of S00217 on drive 7 failed : volume in use\n",
  "labelFormat": "0000",
  "logicalLibrary": "IBMLIB2-LTO10",
  "mediaType": "LTO10",
  "mountId": "7341",
  "mountType": "Retrieve",
  "tapeDrive": "IBMLIB2-LTO10-F04C3R3",
  "tapePool": "vo-atlas.atlas",
  "tapeVid": "S00217",
  "thread": "TapeRead",
  "vendor": "Fujifilm",
  "vo": "vo-atlas"
}

Investigating the issue, we noticed the following:

  • The tape was mounted for writting on drive F2C4R3 at 12:07 and was not unmounted until 17:05. Meanwhile, drive F4C3R3 attempted to mount it for reading at 17:02:

Log rmcd - drive F2C4R3:

Log rmcd drive F2C4R3 (ctatps004):
08/13 12:07:23 2009323 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost
08/13 12:07:23 2009323 rmc_srv_mount: RMC98 - mount S00217/0 on drive 20
08/13 12:07:37 2009323 rmc_srv_mount: returns 0
08/13 17:05:20 2009323 rmc_srv_unmount: RMC92 - unmount request by 1000,33 from localhost
08/13 17:05:20 2009323 rmc_srv_unmount: RMC98 - unmount S00217 20 0

Log rmcd - drive F4C3R3:

8/13 17:02:22 2230139 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost
08/13 17:02:22 2230139 rmc_srv_mount: RMC98 - mount S00217/0 on drive 7
08/13 17:02:23 2230139 rmc_sendrep: smc_mount: Asked for S00217, got reply for S00217
08/13 17:02:23 2230139 rmc_sendrep: smc_mount: Location: data transfer element (0x4)
08/13 17:02:23 2230139 rmc_sendrep: smc_mount: SR018 - mount of S00217 on drive 7 failed : volume in use
08/13 17:02:23 2230139 rmc_srv_mount: returns 2204
  • In the library logs, we can also see that tape S00217 was mounted on drive F2, C4, R3 during that interval:
Demount	Drive F2, C4, R2	S00217LA	8/13/26, 5:31 PM
Mount	Drive F2, C4, R2	S00217LA	8/13/26, 5:26 PM
Demount	Drive F2, C4, R3	S00217LA	8/13/26, 5:05 PM
Mount	Drive F2, C4, R3	S00217LA	8/13/26, 12:07 PM
  • We observe that drive F2C4R3, which is writing to the tape, finishes its write operations at 17:02:21 because the tape became full:
{
  "epoch_time": 1786633341.130451377,
  "local_time": "2026-08-13T17:02:21+0200",
  "cta_version": "5.11.18.0-2",
  "log_schema_version": "0.1.0",
  "hostname": "ctatps004",
  "program": "cta-taped",
  "source_location": "MigrationTaskInjector.cpp:171",
  "log_level": "INFO",
  "pid": 2579625,
  "tid": 3623654,
  "message": "No more files to migrate: triggering the end of session.",
  "drive_name": "IBMLIB2-LTO10-F02C4R3",
  "instance": "prod",
  "sched_backend": "cephprod",
  "mountId": "7317",
  "tapeDrive": "IBMLIB2-LTO10-F02C4R3",
  "tapePool": "vo-atlas.atlas",
  "tapeVid": "S00217",
  "thread": "MigrationTaskInjector",
  "vo": "vo-atlas"
}
  • However, after finishing these writes, the drive keeps the tape mounted while logging statistics, and the tape unload completes at 17:05:

"local_time": "2026-08-13T17:02:21+0200","message": "Logging mount general statistics"

"local_time": "2026-08-13T17:02:21+0200","message": "Logging drive statistics" "local_time": "2026-08-13T17:02:21+0200","message": "Logging volume statistics",

{"epoch_time":1786633520.877569239,"local_time":"2026-08-13T17:05:20+0200","cta_version":"5.11.18.0-2","log_schema_version":"0.1.0","hostname":"ctatps004","program":"cta-taped","source_location":"TapeWriteSingleThread.cpp:121","log_level":"INFO","pid":2579625,"tid":3623652,"message":"TapeWriteSingleThread: Tape unloaded","drive_name":"IBMLIB2-LTO10-F02C4R3","instance":"prod","sched_backend":"cephprod","mountId":"7317","tapeDrive":"IBMLIB2-LTO10-F02C4R3","tapePool":"vo-atlas.atlas","tapeVid":"S00217","thread":"TapeWrite","vo":"vo-atlas"}

  • Meanwhile, drive F04C3R3 attempts to mount the tape for reading at 17:02:23. This is just 2 seconds after the write session ended on drive F02C4R3, which is not enough time for the tape to be unmounted.

Could you confirm if this is a known behavior? Are there perhaps any configuration parameters or timeouts we could tune to prevent this overlap?

Thanks in advance,

Esther

Hi Esther,

Thank you for reporting this and for the detailed investigation.

As far as I am aware, we have not observed this before (though it is possible that it happened and we did not notice).

Indeed a drive should not schedule a mount of a tape that is in use. In your case, the write session had completed but the cartridge had not yet been unmounted, and in rare cases the behaviour that you observed is possible.

We are planning some architectural changes to the mount scheduling logic, so we will bear this in mind for the new design. The proposed new design for the scheduler will be presented at the CTA workshop, I hope you will be able to attend.

Michael

I am seeing this as well and was about to report it in my previous thread on the topic:

08/26 14:20:05 67585 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost.localdomain
08/26 14:20:05 67585 rmc_srv_mount: RMC98 - mount FB7933/0 on drive 33
08/26 14:20:06 67585 rmc_sendrep: smc_mount: Asked for FB7933, got reply for FB7933
08/26 14:20:06 67585 rmc_sendrep: smc_mount: Location: data transfer element (0x4)
08/26 14:20:06 67585 rmc_sendrep: smc_mount: SR018 - mount of FB7933 on drive 33 failed : volume in use
08/26 14:20:06 67585 rmc_srv_mount: returns 2204
08/26 15:11:14 67585 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost.localdomain
08/26 15:11:14 67585 rmc_srv_mount: RMC98 - mount FB7933/0 on drive 33
08/26 15:11:14 67585 rmc_sendrep: smc_mount: Asked for FB7933, got reply for FB7933
08/26 15:11:14 67585 rmc_sendrep: smc_mount: Location: data transfer element (0x4)
08/26 15:11:14 67585 rmc_sendrep: smc_mount: SR018 - mount of FB7933 on drive 33 failed : volume in use
08/26 15:11:14 67585 rmc_srv_mount: returns 2204
08/26 15:15:19 67585 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost.localdomain
08/26 15:15:19 67585 rmc_srv_mount: RMC98 - mount FB7933/0 on drive 32
08/26 15:15:20 67585 lasterror: Function entered: asc=78 ascq=0 save_errno=5 rc=-4 sensekey=5 skvalid=1
08/26 15:15:20 67585 lasterror: Entry found in scsierr_acttbl: action_str=RBT_NORETRY
08/26 15:15:20 67585 rmc_sendrep: smc_mount: SR018 - mount of FB7933 on drive 32 failed : Overlapped Commands Attempted
08/26 15:15:20 67585 rmc_srv_mount: returns 2203
08/26 15:15:20 67585 rmc_sendrep: RMC03 - illegal function 4
08/26 15:20:24 67585 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost.localdomain
08/26 15:20:24 67585 rmc_srv_mount: RMC98 - mount FB7933/0 on drive 32
08/26 15:20:25 67585 rmc_sendrep: smc_mount: Asked for FB7933, got reply for FB7933
08/26 15:20:25 67585 rmc_sendrep: smc_mount: Location: data transfer element (0x4)
08/26 15:20:25 67585 rmc_sendrep: smc_mount: SR018 - mount of FB7933 on drive 32 failed : volume in use
08/26 15:20:25 67585 rmc_srv_mount: returns 2204

Hi Tim and Esther,

Thanks for the detailed report. This seems to be a bug in the code. When the transfer session finishes, cta-taped immediately marks the drive state as CleaningUp. However, when the scheduler determines which tapes to exclude, it does not consider drives in this CleaningUp state to be holding an active mount. As a result, it does not add these tapes to the exclude list, allowing them to be scheduled again for another drive before they are properly unmounted.

This situation is more likely to occur when there is a longer delay between cta-taped finishing its writes and unloading the tape. From the initial logs Esther provided, there was about a 3-minute delay there, so that might explain why you are seeing this (as opposed to a situation where it only spends a few seconds in CleaningUp).

I have created this ticket to investigate a solution and get a fix in one of the future releases: Drives in CleaningUp state are not considered when fetching existing mounts (#1731) · Issues · CTA / CTA · GitLab

Cheers,
Niels