I’m wondering how to determine when I should repack a tape. Data being deleted from tape media is something that we will permit over a longer time frame, due to retention requirements, and this would lead to logical holes in the tape media. Obviously enough data deleted, it’d be worth repacking a set of tapes.
Is there a method for detecting when we should do this?
Additionally, what is the preferred method for verifying a tape is readable? It’s certainly not something I’m going to let a user perform, but as a compliance step, we need to be able to illustrate a random tape and tape drive will work on demand, and maybe even verify tapes on a schedule. Is there a defined method for this?
Regarding the repacking of a tape with logical holes after data deletion, you have to find the right balance between the need to recover the wasted tape space and the ability to recover data deleted accidently.
Obviously, once the tape is repacked and labeled again, the possibility to undelete data is basically gone unless you want to involve vendors (usually a payable service).
At CERN, we only have very few tape pools (so called backup pools) where the data regularly ages out and where we periodically perform automatic repack when the occupancy falls bellow certian threshold (<50%).
For most of the tape pools with physics data, we do not do any automatic repacking. Instead we wait for being notified by the experiment that a deletion campaign has happened and that they would like to recover the space in order to fit to their respective pledges.
Completely orthogonal to this however is a need to migrate data from one set of tapes to another because of density changes. This bulk repack is done as the technology evolves and as needed for CERN and is transparent to the users.
Regarding the tape verification, over the years we identified that is seems to be enough to read just small set of files to check if a tape is in a reasonable shape. That means that we have a verification process that reads 10 files at the beginning of a tape, 10 at the end and 10 random files somewhere in the middle.
This is regular opportunistic process running with low priority, using resources when the system is not busy.
This may not be ideal, but we found it to be a good balance between being able to identify issues and not overload the system with reading everything.
In case if any problems a ticket is opened and the tape is repacked as part of the standard problematic tape workflow.
Regarding the verification, is this using a CTA/CASTOR recovery process, or simply just a mt/dd style seek and read?
I’m not sure how in CTA to detect a tape with purged data. Experimenting just now, I can’t seem to figure out how to find that information. As best I can tell, I need to match up the output of tf ls -v <vid> against the output of ta ls -v <vid>. Is there a nicer way to do this that I’ve missed?
Note that as of CTA release 3.1-4, cta-admin tapefile ls only lists active files on a tape by default. You can provide the --show-superseded option to see all files including purged files.