Hello,
What is the choice for the software RAID of a typical EOSCTA disk server containing 16 SSDs: LVM or ZFS? What are you using at CERN and how much does this choice affect I/O performance?
Thanks,
George
Hello @george_patargias,
I can only speak for my development environment at AARNet, which is aimed at a relatively small 8 drive LTO-7 setup.
My testing and experience shows that the IBM LTO-7 drives are only capable of about 270 MByte/s reliably, given the data going in is already compressed and encrypted before it is put on the CTA spool. I rounded it up to 300 MByte/s, meaning 2400 MByte/s, and ensured the layout of the SAS connections in my server can deliver this.
A development system I’m using has 12 SSDs, and 48 hard drives, each benchmarked at about 550 MByte/s and 250 MByte/s respectively on sequential reads with a 64k block size.
I realised pretty quickly that the limiting factor is actually my fibre channel path to the tape drives, and not the disk, so I’ve had to deal with multipathing for performance.
That said, I am running Ceph on the SSDs and EOS on the raw spindles. We use Kubernetes to run our software, so each EOS spindle is a FST with a single FS, and of course each SSD is an individual OSD.
Within a few percent and so long as I’m not using tiny files on EOS, cta-admin tells me the system is writing tape media at what I expect out of the system.
In other words, I’ve not found a reason to worry too much about RAID, given that both Ceph and EOS take care of availability due to replication for me.
I too am interested in the hardware specification CERN are using.
Hello @davidjericho,
Many thanks for these usefull comments.