TS1170 drives and humidity

General Discussion
1

Hello,

We have been having serious and persistent humidity issues affecting the operation of TS1170 tape drives in both of our Spectra TFinity libraries.

For one, the Recommended and Allowed Relative Humidity (RH) range for TS1170 drives is 20% to 50%, i.e much narrower than TS1160 and LTO9. It is only relatively recently that we fully realised this in the midst of successive heatwaves that resulted in elevated outdoor and machine room RH values. In turn, the increased RH in the machine room brought about an increase in the “Chassis Top” RH in both tape libraries to values up to 56-57% and in the tape drives up to 70% (and even higher).

These RH values caused multiple I/O errors on all TS1170 drives resulting in the damage of TS1170 media and, in some cases, to data loss. At the moment, we are left with no choice but to keep all of our TS1170 drives down in CTA which is significantly disruptive for archives (mitigated to some extent by the temporary use of TS1160 media across pools) and, mainly, recalls.

Have you encountered these issues and how are you dealing with them? Are you actively controlling your machine room RH levels through the operation of de-humidifiers or similar devices?

Thanks,

George

Tape read error when using cta-readtp
2

Hello George,

I am very sorry to hear that you have such serious issues with IBM TS1170 tape drives related to humidity.

What I do not understand though is how could CTA continue to write if humidity was out of range? Normally, there should have been a TapeAlert triggered in which case CTA refuses to ARCHIVE files (RETRIEVEs will still be allowed).

At CERN, we are running IBM TS1170 tape drives in a fully climate controlled room (the tape vault in building 513 which you saw multiple times) so we are protected there.

However, in building 613 (which you also visited I believe) we had issues like this. The reason was that in that building we are using outside air to cool down the room. This can increase the humidity during the hot summer months.

Modifying the room to be fully climate controlled was financially not possible so we had to find alternative solution.

In the end what we did was just to stop taking in the outside air if it is humid. This was possible because the room itself does not produce lot of heat internally (few servers there). Like this we normally use the outside air for cooling, but if it too humid, the air intake is closed until the outside humidity does not drop below certain value.

Last, but not least I would like to point you to a simple script which is part of the CTA operations utilities that extracts the humidity and temperature values from the sensors in the tape drives:

[vlado@tpsrv436 /]$ /opt/cta-ops/ops-venv/bin/cta-ops-drive-environmentals
{"timestamp": 1787324446, "command": "cta-ops-drive-environmentals", "date": 1787324446, "library": "IBMLIB4-TS1160", "tapeserver": "tpsrv436", "tapedrive": "IBMLIB4-TS1160-F14C4R1", "temperature": 27, "humidity": 40}
{"timestamp": 1787324446, "command": "cta-ops-drive-environmentals", "date": 1787324446, "library": "IBMLIB4-TS1160", "tapeserver": "tpsrv436", "tapedrive": "IBMLIB4-TS1160-F14C2R4", "temperature": 25, "humidity": 47}
{"timestamp": 1787324446, "command": "cta-ops-drive-environmentals", "date": 1787324446, "library": "IBMLIB4-TS1160", "tapeserver": "tpsrv436", "tapedrive": "IBMLIB4-TS1160-F14C1R1", "temperature": 28, "humidity": 39}
{"timestamp": 1787324446, "command": "cta-ops-drive-environmentals", "date": 1787324446, "library": "IBMLIB4-TS1160", "tapeserver": "tpsrv436", "tapedrive": "IBMLIB4-TS1160-F14C2R1", "temperature": 28, "humidity": 38}

Hope this helps. Best regards,

Vladimir

3

Hello,

Here in Barcelona, we also experience high humidity during the summer months, and we have also had issues.

We use LTO-8, LTO-9, and LTO-10 drives, which share the same humidity range you mentioned for the TS1170: 20% to 50%. Although the documentation states that the allowed limit is up to 80%, it clarifies further down that this is only for short periods of time (such as a cooling system failure). https://www.ibm.com/docs/en/ts4500-tape-library?topic=planning-environmental-specifications

The new 40TB LTO-10 media is more tolerant, expanding its recommended limit to 80%, but the LTO-10 drive itself still only allows up to 50% according to specifications. So, the problem persists.

Humidity degrades the drive heads, which negatively impacts write quality. This results in data being recorded with a poor signal, making it potentially unreadable over time.

We experienced a major incident two years ago. After detecting repeated read errors in a short period of time, we identified that the affected files had all been written by two LTO-9 drives. We preemptively migrated all the tapes containing files written by these drives and, in parallel, shipped the hardware to IBM for analysis. Their evaluation revealed oxidation on the drive heads.

Since it was not possible to have humidity control in our current room at PIC, we decided to build a dedicated room for the tape libraries with proper humidity control. To manage the humidity while this room was being built, we agreed with IBM on a periodic inspection of all drives, especially during the summer months. Other measures adopted included limiting writes to tape during July and August.

Best regards,

Esther

4

Hi Vladimir,

from what we have been told, TS1170 drives do not generate a TapeAlert when the environment goes out of bounds, just the LTO9/10 drives. We are gathering the drive environment values now (similar to how you do) so we can see what the drives think is going on. We are writing scripts to automatically stop/start the drives depending on the machine room environment.

Tim

5

Hello Vlado, Esther

Many thanks for your useful comments; we are exploring various options for humidity control in the machine room.

We have now in place a cron script that brings CTA drives down when they exceed either 25C temperature and/or 70% relative humidity for TS1160 and LTO9 drives (50% in the case of TS1170 drives). Given the data loss we suffered due to high humidity we would not want to change the humidity max values. But what do you think about the max temperature of 25C? Should be treated as an absolute value or allow some leeway to it? I mean, is it ok for any drive to operate in temperatures above 25C (but presumably < 30C)?

Best,

George