CTA / mhVTL test bed - howto

Hello All,

I am the developer of the mhVTL and would like to setup a test environment (on a Mac laptop) to test CTA / mhVTL interoperability and look at ways to improve mhVTL to better emulate real hardware. I would love to see the functionality improved such that all ‘custom’ handling for mhVTL was removed.

My current base line is Veritas NetBackup, and as I no longer work for Veritas - I was looking for other platforms. Besides, commercial backup vendors have all gone chasing disk with very little effort in tape.

I am hoping I can get away with a ‘minimal’ install of CTA (if such a thing exists) with the ‘tape’ & ‘library’ pieces. An ‘all-in-one’ VM would be perfect for my needs.

I have had a brief review of the install instructions and sort of lost my way down the rabbit hole.

Thanks in advance for any tips / directions through the install instructions.

Regards
Mark

Hi Mark

What problems have you found? I would recommend to install the containerised installation of CTA.

Are you using a VM with Centos7?

Regards,
Jorge

Thanks, let me retry using a known platform an a single set of instructions

The system bootstrap finished without error

Tuning the system for EOS
System bootstrap finished!
[markh@mrhcta vmBootstrap]$ su - cta

however (using above ‘containerised installation of CTA’) instructions are failing at this point.

Creating source rpm
+ rm -rf /home/cta/CTA-build-srpm
+ mkdir -p /home/cta/CTA-build-srpm
+ cd /home/cta/CTA-build-srpm
+ cmake3 -DPackageOnly:Bool=true ../CTA
./bootstrapCTA.sh: line 45: cmake3: command not found

Off to look for cmake3 :slight_smile:

sudo yum install -y cmake3

That was easy :slight_smile:

Next hurdle is

2023-02-07 12:12:54 (171 MB/s) - ‘/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle’ saved [1011/1011]

+ sudo wget https://git.ceph.com/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph
--2023-02-07 12:12:54--  https://git.ceph.com/release.asc
Resolving git.ceph.com (git.ceph.com)... 8.43.84.129
Connecting to git.ceph.com (git.ceph.com)|8.43.84.129|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-07 12:12:57 ERROR 403: Forbidden.

Place https://git.ceph.com/release.asc into a web browser and I get the same ‘403: Forbidden’ error

Hi Mark, thanks for notifying us about this.
This not long ago, but it looks like something changed on the Ceph side.

The instructions at Install Ceph Storage Cluster — Ceph Documentation now say to use https://download.ceph.com/keys/release.asc instead of the key above. I did a quick test by updating the key location in the bootstrapCTA.sh script and it seems to work.
We’ll update the script in our repo, but in the mean time you could do the same replacement

Should I keep on the same thread - or create a new one per build issue ?

The issue now seems to be the SPEC (cta.spec.in) requirement for ‘gtest-devel-1.12’
Can’t seem to find one suitable for CentOS7 (rpmfind bringing up Fedora - but this won’t install on CentOS)
FWIW: CentOS7 is installing gtest-1.6.0-2

Thanks in advance.

Ceph issue also during the ‘prepareImage.sh’

[cta@mrhcta ~]$ cd CTA/continuousintegration/buildtree_runner/
[cta@mrhcta buildtree_runner (297-ci-archive-retrieve-tests-with-gfal2)]$ ./prepareImage.sh
<snip>

Removing intermediate container a059f3714ab7
Step 8/13 : RUN wget https://git.ceph.com/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph
 ---> Running in 8d123d27a19f

--2023-02-08 07:35:54--  https://git.ceph.com/release.asc
Resolving git.ceph.com (git.ceph.com)... 8.43.84.129
Connecting to git.ceph.com (git.ceph.com)|8.43.84.129|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-08 07:35:56 ERROR 403: Forbidden.

The command '/bin/sh -c wget https://git.ceph.com/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph' returned a non-zero code: 8

This thread is fine!
Sorry about this one - you’ve found a gap in our k8s setup.
We redistribute gtest-devel-1.12 through our public repository at Index of /cta-4/el-7/cta-dependencies/x86_64 . The repo is added by the rpm for our public binary release, but it looks like we’ve forgotten to add it for the public edition of the dev environment bootstrap process. We’ll adjust the scripts, and in the mean time you can work around this issue by:

  1. Install the CTA public repositories:
    • yum-config-manager --add-repo CTA/cta-release/cta-public.repo
  2. Edit /etc/yum.repos.d/cta-public.repo and set the enabled=1 for cta-public-4 and cta-public-4-depend.
  3. Might not be needed depending on your system/previous installs, but I ran into a conflict with gmock-devel-1.6.0 on my dev machine. If you run into the same:
    • yum erase gmock-devel
  4. Then re-run bootstrapCTA.sh

Thanks for the info

I’ll most likely rebuild the CentOS7 VM as I’ve butchered the current one trying to build gtest from git source. Certainly not a reproducible env any more…

Minor typo in install instructions.

CTA build tree¶

The user should then login as the and then run the next step - bootstrapCTA.sh:

I think it’s missing “cta user”
e.g.
then login as the cta user and then run the next step

Not sure if this is harmless - but reporting anyway

ifconfig: command not found - during bootstrapKubernetes.sh

Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
Setting up kubernetes DNS...
./bootstrapKubernetes.sh: line 50: ifconfig: command not found
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

So far - so good, when attempting to ‘prepare CTA instance’ - missing jq package.

Creating pods in instance
pod "init" created
Waiting for init../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
../create_instance.sh: line 303: jq: command not found
^C
[cta@markhcta orchestration (main)]$ sudo yum install -y jq

Thanks for these reports Mark, I’ll create a merge request with the corrections.

This one should have been installed by bootstrapSystem.sh <user>, as part of

echo Installing minimal tools and tape tools
sudo yum install -y centos-release-scl-rh
sudo yum install -y git cmake cmake3 rpm-build devtoolset-8 vim gdb cgdb strace ltrace screen tig lsscsi mt-st mtx sg3_utils jq psmisc yum-plugin-versionlock krb5-workstation wget yum-utils epel-release

Did this script fail as well?

If the script failed, I didn’t notice it… the scripts all ran to completion. I don’t suppose the install output is captured (tee) into a log anywhere

I’ve been busy with work last couple of days hence haven’t made much more progress

Re-running from scratch - not getting cmake3 or jq

[markh@mrh-cta vmBootstrap]$ ./bootstrapSystem.sh 
Initializing krb5.conf...
‘krb5.conf’ -> ‘/etc/krb5.conf’
Adding user 'cta'
Changing password for user cta.
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
Installing minimal tools and tape tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.aarnet.edu.au
 * extras: mirror.aarnet.edu.au
 * updates: mirror.aarnet.edu.au
Resolving Dependencies
--> Running transaction check
---> Package centos-release-scl-rh.noarch 0:2-3.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

<snip>

Installed size: 20 k
Downloading packages:
centos-release-scl-rh-2-3.el7.centos.noarch.rpm                                                                                                                                                                 |  12 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : centos-release-scl-rh-2-3.el7.centos.noarch                                                                                                                                                                         1/1 
  Verifying  : centos-release-scl-rh-2-3.el7.centos.noarch                                                                                                                                                                         1/1 

Installed:
  centos-release-scl-rh.noarch 0:2-3.el7.centos                                                                                                                                                                                        

Complete!
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.aarnet.edu.au
 * centos-sclo-rh: mirror.aktkn.sg
 * extras: mirror.aarnet.edu.au
 * updates: mirror.aarnet.edu.au
centos-sclo-rh                                                                                                                                                                                                  | 3.0 kB  00:00:00     
centos-sclo-rh/x86_64/primary_db                                                                                                                                                                                | 3.4 MB  00:00:01     
Package git-1.8.3.1-23.el7_8.x86_64 already installed and latest version
No package cmake3 available.
Package 2:vim-enhanced-7.4.629-8.el7_9.x86_64 already installed and latest version
No package cgdb available.
No package tig available.
Package lsscsi-0.27-6.el7.x86_64 already installed and latest version
No package jq available.
Resolving Dependencies
--> Running transaction check
---> Package cmake.x86_64 0:2.8.12.2-2.el7 will be installed
--> Processing Dependency: libarchive.so.13()(64bit) for package: cmake-2.8.12.2-2.el7.x86_64
---> Package devtoolset-8.x86_64 0:8.1-1.el7 will be installed

Looks like I miss out on cmake3, cgdb, tig & jq
I missed this bit first time as the install script does exit successfully.

Complete!
Getting CTA sources for cta...
Cloning into 'CTA'...
remote: Enumerating objects: 253857, done.
remote: Counting objects: 100% (133/133), done.
remote: Compressing objects: 100% (123/123), done.
remote: Total 253857 (delta 75), reused 19 (delta 5), pack-reused 253724
Receiving objects: 100% (253857/253857), 113.11 MiB | 2.02 MiB/s, done.
Resolving deltas: 100% (204677/204677), done.
Tuning the system for EOS
System bootstrap finished!

Go figure - manually installing the above missing packages works…

vmBootstrap]$ sudo yum install -y cmake3 jq cgdb tig
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
 * base: mirror.aarnet.edu.au
 * centos-sclo-rh: download.nus.edu.sg
 * epel: ftp.riken.jp
 * extras: mirror.aarnet.edu.au
 * updates: mirror.aarnet.edu.au
epel/x86_64/primary_db                                                                                                                                                                                          | 7.0 MB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package cgdb.x86_64 0:0.6.8-1.el7 will be installed
---> Package cmake3.x86_64 0:3.17.5-1.el7 will be installed
--> Processing Dependency: cmake3-data = 3.17.5-1.el7 for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: librhash.so.0()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: libuv.so.1()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
--> Processing Dependency: libzstd.so.1()(64bit) for package: cmake3-3.17.5-1.el7.x86_64
---> Package jq.x86_64 0:1.6-2.el7 will be installed
--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64
---> Package tig.x86_64 0:2.4.0-1.el7 will be installed