Files with “undeleted” replicas

Site Administrators
1

Hi Team,

We are running EOS and CTA in our site. We have a couple of hosts were many files have been marked as undeleted from EOS which results in the ghost replica and not being deleted.

It looks like evict command couldn’t remove the disk-replica from only a couple of hosts and we are trying to understand what could be the reason that disk-replicas couldn’t get deleted from the disk file-system of these hosts.

Sharing few fxid output results of eos fileinfo fxid:xxxxx

  • Sample output
 File: '/eos/file/path/xxxxx/xxxxx'  Flags: 0644  Clock: 185xxxxf491c1eb
  Size: 7271xx0785
Status: locations::incomplete
Modify: Wed Jul 16 08:44:18 2025 Timestamp: 17xxx1858.007838000
Change: Wed Jul 16 08:43:08 2025 Timestamp: 17xxx51788.495389197
Access: Wed Jul 16 08:43:08 2025 Timestamp: 17xxxx51788.490845350
 Birth: Wed Jul 16 08:43:08 2025 Timestamp: 17xxx51788.490844862
  CUid: 3xxx0 CGid: xxxxx Fxid: xxxxx Fid: xxxxxxxx Pid: 801189442 Pxid: 2xxxxx42
XStype: adler    XS: 6x x8 x3 x0    ETAGs: "24593xxx169922560:6xx8x3x0"
Layout: replica Stripes: 1 Blocksize: 4k LayoutId: 00xxxx12 Redundancy: d0::t1
  #Rep: 1
TapeID: 4xxxxxx367 StorageClass: atl24
(undeleted) $ 210
  • error:couldn’t retrieve file meta data :
    No information about( file details,Tape ID, pid, Status: locations::uncommitted,Pid: 0 Pxid: 00000000, Redundancy:d0:t0)
    (undeleted) $ 210

Could you please let us know how we can clean-up the “ghost” disk replicas?

Thanks and Regards,
Maha-RAL SCD Tape team

2

Hello Maha,

The problem you are seeing happens if the MGM does not receive the deletion validation back from the FST, for example, this can happen if there is a problem with the disk where the files are located.

To better understand the problem you could check the logs for the EOS MGM and FST nodes for errors to better pinpoint where the problem lies.

Best regards,
Pablo Oliver Cortés.

3

First, you should check if the affected file systems are online. The example you show could mean that the filesystem 210 is offline. You should understand first what went wrong before attempting the following step to force the removal of the replica.

For the removal of the ghost replicas. As you mention evict, I assume this is a retrieval workflow. IF this problem has happened during a retrieval workflow and you ARE SURE that all the affected files are safely archived on tape you can use the following command: eos file drop fxid:<val> <fsid_val> -f for each affected file.

For archival workflows, as the EOS buffer only operates in 1 replica mode, issuing the file drop command before ensuring that the file is safely on tape, could result in data loss.

4

Hi Pablo,

Thank you!
We tried to restart fst service placing the filesystem in “ro” and the ghost replicas got cleaned from the disk.

Thanks and Regards,
Maha