Hello,
Can you please point us to the exact procedure that needs to be followed
upgrade the (Oracle) CTA DB schema. We are aware of the following documentation https://eoscta.docs.cern.ch/catalogue/upgrade/
It looks like one needs to create the “liquidbase upgrade file” which is the actual upgrade sql script and then run this script against the CTA DB schema. Since you have alreade dy done this for quite a few upgrades and put the upgrade scripts here
https://gitlab.cern.ch/cta/CTA/-/tree/master/catalogue/migrations/liquibase/oracle
can you please confirm if we can use one of these for the CTA DB schema upgrade we aim? For example, we are now running 4.0.5 schema version and we want to upgrade to 4.1.
Thanks,
George
Hello George,
The procedure to update the CTA DB Schema is explained in this documentation
Yes, to update from 4.0 to 4.1 you should follow the backward compatible upgrade documentation
Cheers,
Jorge
Hi Jorge,
Thanks for this. So, to get from the 4.0 to the 4.6 version we need to perform the upgrade of the CTA DB schema in steps: 1) 4.0 to 4.1
2) 4.1 to 4.2 3)… right?
George
Hi George,
We have not yet deployed the 4.6 schema ourselves. We plan to deploy it next week.
4.6 is an intermediate schema, it is a step on the way to some backwards-incompatible schema changes—there will be more upgrades shortly afterwards. We have made a lot of schema changes to move tape drive statuses out of the objectstore into the DB, and to support new features like disk reservations. We hope this will settle down soon.
Upgrading through several schema versions will be complicated. At certain points we have added new columns with NOT NULL constraints. We do this by adding the column without the constraint in one schema version, then deploying the software, populating the column, then upgrading to a new schema version which adds the constraint. We are still refining our procedures for this.
I recommend that you take a snapshot/clone of your current DB and try to upgrade it on a test system and see how it goes.
Regards,
Michael
In the Release Notes, you can see which CTA releases require a DB upgrade. I recommend that you follow the same steps that we did:
For each schema version :
- Upgrade the schema
- Upgrade the CTA software to the corresponding version listed in the Release Notes
- Populate any new columns in the DB if necessary
Upgrading the software is necessary to get the new cta-admin commands to populate the DB. Upgrading to the next schema version without first populating new columns will break when the NOT NULL constraint is applied.
Hi Michael,
Thanks for clarifying in such a detail!
In the Release notes, I noticed that certain CTA version have a feature that reads like "Upgraded EOS to 4.8.67 " (in the case of version 4.4.0-1).
Is this the version of the EOS client rpms that a particular CTA version requires? Is this also the EOS version that needs to be installed on the
EOS MGM and FST nodes?
Can we upgrade from EOS 4.8.45, that we are running now, to EOS 4.8.78 while keeping CTA to the 4.0.5?
George
Also, if you could confirm once again the the EOS and CTA versions you are running in production that would be great. We intend to bring our versions in line with yours.
Hi George,
usually we upgrade CTA before EOS: you can upgrade CTA and leave your EOS version to 4.8.45.
The reason for this is that we have plenty of eoscta EOS instances and 1 CTA tape backend.
So basically upgrade CTA to the targetted version (we are running cta version 4.6.0) and then look in the CTA CI version lock file which EOS version is coming along with this CTA version.
For version 4.6.0-1 the eos version is 4.8.75 (but you should go straight to 4.8.78).
There is another thread dealing with finding the recommended software matrix: EOSCTA service versions
Julien
Hi Julien,
Many thanks again for this. We upgraded he CTA DB and then the Frontend and the tape server without any issue. We are struggling a bit with EOS though: upgrading to either 4.8.75 or 4.8.78 resulted in the MGM entering a failed state (segmentation fault) whenever an eos client command was issued.
Have you come across this issue and is there a EOS upgade procedure that has been formalised? The one we found here EOS Version update - EOS-Ops documentation is not very informative. I think we may have missed a step…
Best,
George
Hi George,
I am sorry to hear that.
Could you check that the rpms you installed are at the same version as the ones in the corresponding CI ctaeos pod?
Be extra vigilent for eos-xrootd, eos-XXX, quarkXXX and xrootd-XXX packages.
EOS consistent rpm list should be something like:
eos-debuginfo-4.8.10-1.el7.cern.x86_64
eos-folly-deps-2019.11.11.00-1.el7.cern.x86_64
eos-ns-inspect-4.8.78-1.el7.cern.x86_64
eos-server-4.8.78-1.el7.cern.x86_64
eos-folly-2019.11.11.00-1.el7.cern.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64
eos-xrootd-debuginfo-4.12.8-1.el7.cern.x86_64
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64
eos-client-4.8.78-1.el7.cern.x86_64
quarkdb-0.4.2-1.el7.cern.x86_64
quarkdb-debuginfo-0.4.2-1.el7.cern.x86_64
Then xrootd@quarkdb service runs OK with the following xrootd rpms:
xrootd-server-4.12.6-1.el7.x86_64
xrootd-client-libs-4.12.6-1.el7.x86_64
xrootd-selinux-4.12.6-1.el7.noarch
xrootd-client-4.12.6-1.el7.x86_64
xrootd-libs-4.12.6-1.el7.x86_64
xrootd-4.12.6-1.el7.x86_64
xrootd-debuginfo-4.12.6-1.el7.x86_64
xrootd-server-libs-4.12.6-1.el7.x86_64
This is maybe a bit of an overkill list, but check if quarkdb is OK first and then look for eos- rpm consistency.
Cheers,
Julien
Hi Julien,
Thanks for this. quarkDB is running fine with a quorum, a leader and all nodes have a node-health green. Our rpm list is consitent with yours (see below) with the only difference being that we have installed XRootD 4.12.8 (the latest XRootD 4 version from the EOS citrine deps repo) which is what we have in production and appears to be working fine.
Best,
George
eos-fusex-core-4.8.78-1.el7.cern.x86_64
eos-xrootd-debuginfo-4.12.8-1.el7.cern.x86_64
eos-scitokens-1.2.0-1.el7.cern.x86_64
eos-test-4.8.78-1.el7.cern.x86_64
eos-protobuf3-3.5.1-5.el7.cern.eos.x86_64
eos-fuse-sysv-4.8.78-1.el7.cern.x86_64
eos-fusex-4.8.78-1.el7.cern.x86_64
eos-testkeytab-4.8.78-1.el7.cern.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64
eos-folly-2019.11.11.00-1.el7.cern.x86_64
libmicrohttpd-0.9.38-eos.yves.el7.cern.x86_64
eos-fuse-core-4.8.78-1.el7.cern.x86_64
eos-fusex-selinux-4.8.78-1.el7.cern.x86_64
eos-ns-inspect-4.8.78-1.el7.cern.x86_64
eos-folly-deps-2019.11.11.00-1.el7.cern.x86_64
eos-client-4.8.78-1.el7.cern.x86_64
eos-fuse-4.8.78-1.el7.cern.x86_64
eos-server-4.8.78-1.el7.cern.x86_64
eos-archive-4.8.78-1.el7.cern.x86_64
quarkdb-0.4.2-1.el7.cern.x86_64
python2-xrootd-4.12.8-1.el7.x86_64
eos-xrootd-debuginfo-4.12.8-1.el7.cern.x86_64
xrootd-server-4.12.8-1.el7.x86_64
xrootd-4.12.8-1.el7.x86_64
xrootd-voms-4.12.8-1.el7.x86_64
xrootd-libs-4.12.8-1.el7.x86_64
eos-xrootd-4.12.8-1.el7.cern.x86_64
xrootd-server-libs-4.12.8-1.el7.x86_64
xrootd-selinux-4.12.8-1.el7.noarch
xrootd-alicetokenacc-1.3.1-1.x86_64
xrootd-client-4.12.8-1.el7.x86_64
xrootd-client-libs-4.12.8-1.el7.x86_64
Hi George,
Yesterday we upgraded CTA to v4.6.1-1 and we recommend that you follow this upgrade. This upgrades the DB schema version to v4.6.
Please note that this upgrade is not completely transparent: cta-taped v4.6.0-1 is not fully compatible with schema v4.6. This means that you should put the drives down, then stop cta-taped before you carry out the upgrade. Then upgrade the schema and the software and all should be good.
There will be a further DB schema upgrade for the upcoming CTA release v4.7. We hope that this will be the last schema upgrade for a while. At least, I am trying to get all anticipated schema changes into this release.
Best regards,
Michael
I have reorganised and revised the documentation on how to do DB schema upgrades, see Upgrading the schema.
Hi Michael,
Many thanks for this. Is your EOS version still 4.8.78 even after upgrading to CTA 4.6.1-1?
George
EOS and CTA can be upgraded independently. Upgrading CTA by itself will not change the EOS version. We are still running 4.8.78 in production but in CI we are running 4.8.79.
Hi Michael,
Are you still on 4.6.1-1 version? Can you please let me know when are you planning to upgrade to 4.7?
Thanks,
George
Last week we did the upgrade in production to 4.7.0-1 with schema version 10.0. You should be able to follow.
Hi Michael,
Do you foresee any issues with the migration of our CASTOR Facilities instance directly to the CTA DB schema v10.0? Or we need to perform the migration to 4.6 schema and then upgrade to v10.0?
George
You should upgrade to 4.6 and then to 10.0.
The upgrade scripts are only provided for upgrading between specific versions. There is no script to upgrade 4.5to10.0.sql for example.