Upgrade

Before the upgrade

  • Before upgrading Horizon, you should always take a full database backup. Remind that upgrading is a one-way operation, and that you’ll need to restore to an older database state if you wish to rollback the upgrade.

  • Upgrades should run when Horizon is stopped. Currently, Horizon doesn’t support rolling (HA) migrations.

  • Check the release notes for the version you’re upgrading to for known defects that might impact you. Also, carefully read the Specific upgrade instructions for any particular instructions for the version you’re upgrading to.

Upgrade Horizon

  • RPM

  • Kubernetes

You must retrieve the latest Horizon RPM from the EVERTRUST repository manually using the user credentials you were provided.

Access the server through SSH with an account with administrative privileges.

Install the Horizon package with the following command:

$ yum install horizon-2.7.X-1.noarch.rpm

After installing, services must be reloaded to take the change into account

$ systemctl daemon-reload

When upgrading Horizon, you’ll need to pull the latest version of the chart:

$ helm repo update evertrust

Verify that you now have the latest version of Horizon (through the App version column):

$ helm search repo evertrust/horizon
NAME                       	CHART VERSION	APP VERSION	DESCRIPTION
evertrust/horizon          	0.9.3        	2.7.0      	EverTrust Horizon Helm chart

Launch an upgrade by specifying the new version of the chart through the --version flag in your command:

$ helm upgrade <horizon> evertrust/horizon \
 --values override-values.yaml \
 --version 0.9.3
We recommended that you only change values you need to customize in your values.yml file to ensure smooth upgrading.

Migrate the database

Horizon requires that you run a script called horizon-upgrade before installing a newer version of Horizon. This script will migrate the database schemas for compatibility with the new version.

  • RPM

  • Kubernetes

  • Docker

Since version 2.1.0, horizon-upgrade comes bundled with Horizon. Therefore, after each upgrade, you should run horizon-upgrade to check whether new migrations should be run.

Access the server through SSH with an account with administrative privileges.

Run the following command:

$ /opt/horizon/sbin/horizon-upgrade -t <target version>

In most cases, horizon-upgrade can detect the version you’re upgrading from by checking the database. However, when upgrading from version prior to 2.1.0, you will encounter the following error:

*** Unable to infer the source version from your database. Specify it explicitly with the -s flag. ***

You’ll have to explicitly tell horizon-upgrade which version you are upgrading from. To do that, simply set the source version explicitly with the -s flag:

$ /opt/horizon/sbin/horizon-upgrade -t <target version> -s <source version>

Similarly, horizon-upgrade will try to use the MongoDB URI that was configured by the Horizon configuration utility. If it fails to auto-detect your database URI or you wish to migrate another database, specify the URI explicitly using the -m flag:

$ /opt/horizon/sbin/horizon-upgrade -t <target version> -m "<mongo uri>"

The upgrade script requires a MongoDB client to connect to your database (either mongo or mongosh). If no client is installed on the host where Horizon is running, consider installing the standalone mongosh client or running the upgrade script from another host that has access to the database.

By default, the chart will automatically create a Job that runs an upgrade script when it detects that the Horizon version has changed between two releases.

Should you wish to disable the automatic upgrade mechanism, just set the upgrade.enabled key to false. You’ll need to manually run the upgrade script, you can use the dockerized version of the script provided that your host device has access to the Horizon’s MongoDB database:

apiVersion: batch/v1
kind: Job
metadata:
  name: horizon-upgrade
spec:
  template:
    spec:
      containers:
        - name: horizon-upgrade
          image: registry.evertrust.io/horizon-upgrade:2.7.x
          imagePullPolicy: IfNotPresent
          args: [
              "-y",
              "-m", "$(MONGODB_URI)",
              "-s", "<source version>", (1)
              "-t", "<target version>"
          ]
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  name: horizon (2)
                  key: mongoUri
      restartPolicy: Never
  backoffLimit: 0
1 The source and target values should be updated to match the desired upgrade path
2 The secret name and key should match where you store the Horizon MongoDB URI, so it will be injected as an environment variable to the Pod.
When upgrading from an old version of Horizon, you may need to explicitly specify the version you’re upgrading from using the upgrade.from key. The created pod will pull an image named horizon-upgrade:2.7.x, so make sure this image will be available to the cluster when upgrading.

You can run the script on any computer that has access to your MongoDB database, provided that it has Docker installed:

$ docker run -it --rm registry.evertrust.io/horizon-upgrade:2.7.x \
   -m <mongo uri> \
   -t <target version> \
   -s <source version> # This is required when upgrading from an older version of Horizon

Specific upgrade instructions

These steps should be followed in addition to the common upgrade procedure found in the above standard upgrade instructions.

Kubernetes

Upgrading with a StatefulSet

When using a StatefulSet in Kubernetes (for instance when Analytics are configured with persistence), you’ll need to take extra steps before launching the upgrade to ensure that pods get shutdown while the migration script is running:

  • the StatefulSet strategy should be set to RollingUpdate;

  • the StatefulSet resource should be scaled down to 1 replica:

    $ kubectl scale --replicas=1 rs/{release-name}
If not scaled down to one replica, multiple versions of Horizon could write to the database and corrupt data.

Upgrading to 0.3.0

  • Loggers are now configured with an array instead of a dictionary. Check the values.yaml format and update your override values.yaml accordingly.

  • The init dabatabase parameters (initDatabase, initUsername and initPassword) have been renamed and moved to mongodb.horizon.

Upgrading to 0.5.0

  • The ingress definition has changed. The rules and tls keys have been removed in favor of a more user-friendly hostname that will autoconfigure the ingress rules, and a boolean tls key that will enable TLS on that ingress. Check the Ingress section.

Upgrading to 0.9.0

  • clientCertificateDefaultParsingType has been removed and is no longer supported by Horizon. Explicitly set the clientCertificateHeader or use ingress autoconfiguration to continue using client certificate authentication.

  • ingress.type will now be strictly validated. It may fail if you use an unsupported value.

  • mailer.port, mailer.tls and mailer.ssl are no longer set by default. You must now explicitly set if you want to use them.

Upgrading to 0.11.0

  • New Lease CRD is added.

  • akka.conf has been replaced with pekko.conf. It may fail if you use custom configuration otherwise it will be handled by the helm chart.

Upgrade to 0.16.0

  • Switching to native kubernetes leases implementation. CRDs leases aren’t used anymore.

RPM

Upgrading from a version prior to 2.1.0

You must manually create an entry to pass an event seal secret to Horizon in the /etc/default/horizon file. horizon-config won’t do that automatically.

To do so, open the /etc/default/horizon file with a text editor:

$ vi /etc/default/horizon

And add a new line under the Horizon variables section:

# Horizon variables
HORIZON_NOTIFICATION_SMTP_HOST=127.0.0.1
HORIZON_HOSTNAME=
HORIZON_DEFAULT_SSV_KEY=
HORIZON_EVENT_SEAL_SECRET=changeme # <- this one

Then, near the end of the file, after the # Setting Horizon Mongo DB uri section, create a new section for the event seal secret:

# Setting the Horizon event seal secret
JAVA_OPTS="$JAVA_OPTS -Dhorizon.event.seal.secret=${HORIZON_EVENT_SEAL_SECRET}"

Horizon won’t boot if the HORIZON_EVENT_SEAL_SECRET is set to changeme. Therefore, you should set your secret to something hard to guess. Refer to the Initial Configuration guide to learn how to generate a seal secret with horizon-config.