Upgrade

Before the upgrade

  • Before upgrading Horizon, you should always take a full database backup. Remind that upgrading is a one-way operation, and that you’ll need to restore to an older database state if you wish to rollback the upgrade.

  • Check the release notes for the version you’re upgrading to for known defects that might impact you. Also, carefully read the Specific upgrade instructions for any particular instructions for the version you’re upgrading to.

  • Upgrades should follow the below order to ensure no errors when using High Availability

This new migration workflow is only available when upgrading from a 2.7.x instance. Please refer to 2.7.x documentation to upgrade to this version, and then use this page.

Installing the migration tool

The migration tool should be updated to the latest available version before each migration.

  • RPM

  • Debian

  • Kubernetes

  • Docker

In order to install Horizon Migration Tool, the server must have access to a repository (mirror, ISO file, …​) of the linux distribution you are using in order to be able to install the dependencies of the software. Horizon Migration Tool package has the following dependencies:

  • java-17-openjdk-headless

Please note that these packages may have their own dependencies.

Installation from the EverTrust repository

Create a /etc/yum.repos.d/horizon-migration.repo file containing the EverTrust repository info:

[horizon-migration]
enabled=1
name=Horizon Migration Tool Repository
baseurl=https://repo.evertrust.io/repository/horizon-migration-rpm/
gpgcheck=0
username=<username>
password=<password>

Replace <username> and <password> with the credentials you were provided.

You can then run the following to install the latest Horizon Migration Tool version:

# yum install horizon-migration

To prevent unattended upgrades when running yum update, you should pin the Horizon Migration Tool version by adding

exclude=horizon-migration

at the end of the /etc/yum.repos.d/horizon-migration.repo file after installing Horizon Migration Tool.

Installing from RPM

Download the latest RPM for Horizon Migration Tool on the Official EVERTRUST repository.

Upload the file 'horizon-migration-<latest>.noarch.rpm' to the server;

Access the server with an account with administrative privileges;

Install the Horizon Migration Tool package with the following command:

# yum localinstall /root/horizon-migration-<latest>.noarch.rpm

In order to install Horizon Migration Tool, the server must have access to a repository (mirror, ISO file, …​) of the linux distribution you are using in order to be able to install the dependencies of the software. Horizon Migration Tool package has the following dependencies:

  • openjdk-17-jre-headless

Please note that these packages may have their own dependencies.

Installation from the EverTrust repository

If you haven’t already, to add the EVERTRUST repository to your APT repositories, run the following commands:

  1. Install the required tools (gpg)

    # sudo apt install gnupg

  2. Download and install the EVERTRUST GPG key

    # curl https://evertrust.io/.well-known/apt/gpg.pub | sudo gpg -o /usr/share/keyrings/evertrust.gpg --dearmor

  3. Add the repository

    # echo "deb [ arch=all signed-by=/usr/share/keyrings/evertrust.gpg ] https://repo.evertrust.io/repository/apt all main" | sudo tee /etc/apt/sources.list.d/evertrust.list

Once the repository has been added, authentication to it must be provided. To do so, edit the /etc/apt/auth.conf file and add the following lines:

machine repo.evertrust.io
login <your EVERTRUST login>
password <your EVERTRUST password>

Once the repository has been added, run the following command to update the APT repository list.

# sudo apt update

You can then run the following command to install the latest Horizon Migration Tool version:

# sudo apt install horizon-migration

Installing from DEB

Download the latest DEB for Horizon Migration Tool on the Official EVERTRUST repository.

Upload the file 'horizon-migration-<latest>_all.deb' to the server;

Access the server with an account with administrative privileges;

Install the Horizon Migration Tool package with the following command:

# apt install /root/horizon-migration-<latest>_all.deb

No installation is required as the helm chart will pull the appropriate image.

The docker image is available on the registry:

$ docker pull registry.evertrust.io/horizon-migration:latest

Pre-migration checks

Before upgrading, some checks should be run to ensure compatibility with the newer version:

  • RPM

  • Debian

  • Kubernetes

  • Docker

Before planning an upgrade, the migration tool should be upgraded to the latest version. It is recommended to install it on one of the Horizon machine for a seamless experience.

Once upgraded, run the following command to check the database compatibility:

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version>

In most cases, horizon-migration can detect the version you’re upgrading from by checking the database. However, the source version can be specified using --from flag:

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version> \
  --from <source version>

If you wish to run the migration from another machine that the one Horizon is installed on, or if you wish to migrate another database, the --mongo-uri flag should be added to access to MongoDB.

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version> \
  --mongo-uri <mongo uri>

Before planning an upgrade, the migration tool should be upgraded to the latest version. It is recommended to install it on one of the Horizon machine for a seamless experience.

Once upgraded, run the following command to check the database compatibility:

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version>

In most cases, horizon-migration can detect the version you’re upgrading from by checking the database. However, the source version can be specified using --from flag:

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version> \
  --from <source version>

If you wish to run the migration from another machine that the one Horizon is installed on, or if you wish to migrate another database, the --mongo-uri flag should be added to access to MongoDB.

$ /opt/evertrust/horizon-migration/bin/horizon-migration check \
  --to <target version> \
  --mongo-uri <mongo uri>

By default, the chart will automatically create a Job that checks database compatibility before an upgrade. This job will be assigned the helm.sh/hook: pre-upgrade annotation, which means they’ll run before the upgrade is actually performed. If the job fails, the upgrade will also fail before proceeding with altering the database.

Should you wish to disable the automatic upgrade mechanism, just set the upgradeCompatibilityCheck.enabled key to false. If you still wish to manually run the pre-migration script, you can use this Kubernetes manifest as a starting point:

apiVersion: batch/v1
kind: Job
metadata:
  name: horizon-migration
spec:
  template:
    spec:
      containers:
        - name: horizon-migration
          image: registry.evertrust.io/horizon-migration:latest
          imagePullPolicy: IfNotPresent
          args: [
              "check",
              "-y",
              "--mongo-uri", "$(MONGODB_URI)",
              "--ignore-empty-from",
              "--to", "<target-version>" (1)
          ]
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  name: horizon (2)
                  key: mongoUri
      restartPolicy: Never
  backoffLimit: 0
1 The target version should be updated to match the desired upgrade path. The source version will be inferred from database.
2 The secret name and key should match where you store the Horizon MongoDB URI, so it will be injected as an environment variable to the Pod.

You can run the migration tool on any computer that has access to your MongoDB database, provided that it has Docker installed:

$ docker run -it --rm registry.evertrust.io/horizon-migration:latest check \
  --mongo-uri <mongo uri> \
  --to <target version> \
  --from <source version> # This is required when upgrading from an older version of Horizon

Upgrade Horizon

  • RPM

  • Debian

  • Kubernetes

  • Docker

Follow the installation procedure again to retrieve the new version.

For High Availability instances, starting from 2.8, nodes can be updated one by one. Upgraded nodes will be idle and serve a maintenance page until the database migration (next step) has ended.

Follow the installation procedure again to retrieve the new version.

For High Availability instances, starting from 2.8, nodes can be updated one by one. Upgraded nodes will be idle and serve a maintenance page until the database migration (next step) has ended.

When upgrading Horizon, you’ll need to pull the latest version of the chart:

$ helm repo update evertrust

Verify that you now have the latest version of Horizon (through the App version column):

$ helm search repo evertrust/horizon
NAME                       	CHART VERSION	APP VERSION	DESCRIPTION
evertrust/horizon          	0.9.3        	2.8.0      	EverTrust Horizon Helm chart

Launch an upgrade by specifying the new version of the chart through the --version flag in your command:

$ helm upgrade <horizon> evertrust/horizon \
 --values override-values.yaml \
 --version 0.9.3
We recommended that you only change values you need to customize in your values.yml file to ensure smooth upgrading.

Pull the new image from the repository.

Migrate the database

  • RPM

  • Debian

  • Kubernetes

  • Docker

Using the horizon-migration tool used at the first step, the migration can now be run using the following command:

$ /opt/evertrust/horizon-migration/bin/horizon-migration migrate --to <target version>

Just like the check command, additional flags should be added when running on another machine.

Using the horizon-migration tool used at the first step, the migration can now be run using the following command:

$ /opt/evertrust/horizon-migration/bin/horizon-migration migrate --to <target version>

Just like the check command, additional flags should be added when running on another machine.

By default, a chart upgrade will dispatch a migration job annotated with helm.sh/hook: post-upgrade, meaning it will run once all nodes have been upgraded to the target version. While the rollout and the migration script are running, nodes running a newer version will be put in maintenance mode. Maintenance mode will automatically exit once the upgrade script finishes running.

Should you wish to disable the automatic upgrade mechanism, just set the upgrade.enabled key to false. If you still wish to manually run the migration script, you can use this Kubernetes manifest as a starting point:

apiVersion: batch/v1
kind: Job
metadata:
  name: horizon-migration
spec:
  template:
    spec:
      containers:
        - name: horizon-migration
          image: registry.evertrust.io/horizon-migration:latest
          imagePullPolicy: IfNotPresent
          args: [
              "migrate",
              "-y",
              "--mongo-uri", "$(MONGODB_URI)",
              "--ignore-empty-from",
              "--to", "<target-version>" (1)
          ]
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  name: horizon (2)
                  key: mongoUri
      restartPolicy: Never
  backoffLimit: 0
1 The target version should be updated to match the desired upgrade path. The source version will be inferred from database.
2 The secret name and key should match where you store the Horizon MongoDB URI, so it will be injected as an environment variable to the Pod.

You can run the script on any computer that has access to your MongoDB database, provided that it has Docker installed:

$ docker run -it --rm registry.evertrust.io/horizon-migration:latest migrate \
   --mongo-uri <mongo uri> \
   --to <target version> \
   --from <source version>

Specific upgrade instructions

These steps should be followed in addition to the common upgrade procedure found in the above standard upgrade instructions.

Upgrading from a version prior to 2.8.0

When upgrading from a version prior to 2.8.0, you must manually create a new keyset using Tinkey.

The migration tool will fail if the keyset is not generated before running the upgrade.

Different keyset templates are available depending on your security requirements.To maintain the same encryption level as your current installation, use the following mapping:

If you are unsure about which mode to use, contact the EVERTRUST support team.

Follow the steps in the Tinkey page to setup Tinkey, create your Keyset and make it available for Horizon.

Once your keyset has been created, migration can occur as specified by the steps above. However for the 2.8 migration, specific information must be provided to the migration tool to handle the encryption of secrets. This depends on your current encryption mechanism.

  • SSV

  • Others

If you are running the migration tool on the same server as the Horizon installation, and using the same user, the tool will pick up the current configuration automatically, and requires no additional configuration.

If you are using the Helm Chart, upgrading it should also handle the migration automatically.

If the migration tool does not run in the same context, the required configuration can be provided using environment variables:

Variable Description

HORIZON_MIG_280_SSV_PASSWORD

The SSV password

HORIZON_MIG_280_KEYSET

The raw Tink keyset to use. To define if HORIZON_MIG_280_KEYSET_PATH is not defined

HORIZON_MIG_280_KEYSET_PATH

Path to the Tink keyset file. To define if HORIZON_MIG_280_KEYSET is not defined

HORIZON_MIG_280_MASTER_KEY_URI

Tink Master key URI if the keyset is wrapped

HORIZON_MIG_280_CREDENTIALS_PATH

Path to the credentials file to combine with the Master key if using a KMS

This will migrate all vaults to the tink keyset provided.

If you wish to use other vaults configuration, the path to a full configuration file for the vault migration can be provided.

The migration file must be provided in the HORIZON_MIG_280_VAULT_CONF_PATH environment variable. This will override all other behaviors.

This file defines all the source and destination vaults, and how to modify them. Here is an example file that describes migrating to an SHV vault to a keyset (that can contain any key, for example an equivalent PKCS#11 CBC Derived key).

vault {
  (1)
  source {
    configuration = "source_default"
    escrow = "source_default"
    transient = "source_default"
  }
  (2)
  destination {
    configuration = "dest_default"
    escrow = "dest_default"
  }
}

(3)
vaults {
  source_default {
    module_path = "/usr/lib64/pkcs11/p11.so"
    slot_id = "12345678"
    pin = "1234"
    label = "horizon_masterkey"
    allow_master_key_gen = false
    vault_type = "shv"
  }
  dest_default {
    vault_type = "tink"
    path = "/opt/horizon/etc/horizon.keyset"
  }
}
1 The previous vault types and the associated vault to use.
2 The target vault types. The transient vault is now considered part of the configuration vault.
3 The definition of the vaults (how to encrypt the data.)

If your current vault configuration does not only use SSV, the migration tool must be configured using a configuration file.

The migration file must be provided in the HORIZON_MIG_280_VAULT_CONF_PATH environment variable. This will override all other behaviors.

This file defines all the source and destination vaults, and how to modify them. Here is an example file that describes migrating to an SHV vault to a keyset (that can contain any key, for example an equivalent PKCS#11 CBC Derived key).

vault {
  (1)
  source {
    configuration = "source_default"
    escrow = "source_default"
    transient = "source_default"
  }
  (2)
  destination {
    configuration = "dest_default"
    escrow = "dest_default"
  }
}

(3)
vaults {
  source_default {
    module_path = "/usr/lib64/pkcs11/p11.so"
    slot_id = "12345678"
    pin = "1234"
    label = "horizon_masterkey"
    allow_master_key_gen = false
    vault_type = "shv"
  }
  dest_default {
    vault_type = "tink"
    path = "/opt/horizon/etc/horizon.keyset"
  }
}
1 The previous vault types and the associated vault to use.
2 The target vault types. The transient vault is now considered part of the configuration vault.
3 The definition of the vaults (how to encrypt the data.)

Architecture-specific instructions

  • RPM

  • Debian

  • Kubernetes

  • Docker

No specific instructions.

No specific instructions.

Upgrading with a StatefulSet

When using a StatefulSet in Kubernetes (for instance when Analytics are configured with persistence), you’ll need to take extra steps before launching the upgrade to ensure that pods get shutdown while the migration script is running:

  • the StatefulSet strategy should be set to RollingUpdate;

  • the StatefulSet resource should be scaled down to 1 replica:

    $ kubectl scale --replicas=1 rs/{release-name}
If not scaled down to one replica, multiple versions of Horizon could write to the database and corrupt data.

Upgrading to 0.3.0

  • Loggers are now configured with an array instead of a dictionary. Check the values.yaml format and update your override values.yaml accordingly.

  • The init database parameters (initDatabase, initUsername and initPassword) have been renamed and moved to mongodb.horizon.

Upgrading to 0.5.0 - The ingress definition has changed. The rules and tls keys have been removed in favor of a more user-friendly hostname that will autoconfigure the ingress rules, and a boolean tls key that will enable TLS on that ingress. Check the Ingress section.

Upgrading to 0.9.0 - clientCertificateDefaultParsingType has been removed and is no longer supported by Horizon. Explicitly set the clientCertificateHeader or use ingress autoconfiguration to continue using client certificate authentication. - ingress.type will now be strictly validated. It may fail if you use an unsupported value. - mailer.port, mailer.tls and mailer.ssl are no longer set by default. You must now explicitly set if you want to use them.

Upgrading to 0.11.0 - New Lease CRD is added. - akka.conf has been replaced with pekko.conf. It may fail if you use custom configuration otherwise it will be handled by the helm chart.

Upgrade to 0.16.0 - Switching to native Kubernetes leases implementation. CRDs leases aren’t used anymore.

Upgrade to 1.0.0

  • This version drops support for the Bitnami MongoDB subchart. Instead, a new temporaryDatabase key controls whether a temporary MongoDB instance should be created for the duration of the upgrade. To migrate from the Bitnami MongoDB subchart to a temporary instance or an external MongoDB database, you can use the mongodump and mongorestore utilities.

Upgrade to 2.0.0

  • vaults section is no longer supported.

  • legacySsvPassword must be set to allow migration of existing secrets.

  • defaultVault must be configured with the correct values.

  • defaultVault.keyset must be generated, stored as a secret, and properly referenced before performing the upgrade.

No specific instructions.