Advanced usage

Some edge use-cases might not have been included in the previous installation documentation, for clarity purposes. You may find some of them below.

Running behind a Docker registry proxy

If your installation environment requires you to whitelist images that can be pulled by the Kubernetes cluster, you must whitelist the registry.evertrust.io/horizon and registry.evertrust.io/horizon-upgrade images.

Leases

To ensure clustering issues get resolved as fast as possible, Horizon can use a CRD (Custom Resource Definition) named Lease (akka.io/v1/leases). We strongly recommend that you use this mechanism, however it implies that you have the necessary permissions to install CRDs onto your server. In case you don’t, the feature can be disabled by passing the --skip-crds flag to the Helm command when installing the chart, and setting the leases.enabled key to false. If you want to manually install the CRD, you can check the crds/leases.yml file.

Injecting extra configuration

Extra Horizon configuration can be injected to the bundled application.conf file to modify low-level behavior of Horizon. This should be used carefully as it may cause things to break. To do so, use the extraConfig value in your values.yaml file:

This can be done with the following edits to your values.yaml file:

extraConfig: |
    play.server.http.port = 9999
    horizon {
      notification.mail.attachment.extension.der = "der"
    }

Extra configurations are included at the end of the config file, overriding any previously set config value.

An exhaustive list of configuration options can be found on the Overridable configuration parameters page.

Custom startup scripts

Sometimes, you’ll want to run scripts each time the container starts up in order to configure files in the container or set environment variables. To do so, you’ll need to mount shell scripts into the /docker-entrypoint.d/ directory in the container. Using the Helm chart, this can be achieved easily using the following values.yaml overrides:

extraVolumes:
  - name: horizon-entrypoint-scripts
    configMap:
      name: horizon-entrypoint-scripts

extraVolumeMounts:
  - name: horizon-entrypoint-scripts
    mountPath: /docker-entrypoint.d/

Given you’ve previously create a ConfigMap called horizon-entrypoint-scripts:

apiVersion: v1
kind: ConfigMap
metadata:
  name: horizon-entrypoint-scripts
data:
  run-on-startup.sh: |
    echo "Hello World !"

By design, Horizon is configured to run as an unprivileged user inside the container to follow industry best practices. This means that your scripts won’t be able to perform privileged operations on the container, such as trusting custom CAs. If you do want to overcome this problem, you can run the container as root, even though it is generally discouraged.

Networking overview

When installed in HA, Horizon sends messages to other running instances in its cluster. To form the cluster and set up networking between nodes, Horizon is relying on Pekko, a framework for building clusterized applications. Understanding how clustering works is important when building deployments with highly specific needs or when preparing a disaster recovery plan.

When deployed on multiple nodes inside a Kubernetes cluster, the following steps are followed:

Discovery: the discovery process locates all nodes that will be used to form a cluster. It relies on a third-party to give that information, such as a DNS record or the Kubernetes API (which is the default when deploying using the Helm Chart). For documentation, see Pekko Discovery.
Bootstrap: once each node in the cluster has the address of every other node, nodes start to contact each other. This is done though Pekko Management, a tool for helping nodes coordinate. For documentation, see Pekko Management.
Remoting: the cluster is now formed, nodes can communicate with each other. This uses Pekko Remoting, a higher level protocol for serializing data over multiple transports. Typically, TCP is used. For documentation, see Pekko Remoting.

This clustering process can be summarized by the below diagram:

Sequence diagram of the cluster management of Horizon

sequenceDiagram
    autonumber
    rect rgb(191, 223, 255)
    Pod1 ->> Kubernetes API: Discovery request
    destroy Kubernetes API
    Kubernetes API ->> Pod1: Returns other pods addresses
    end
    Note right of Pod2: 1-2: Discovery process


    rect rgb(156, 250, 152)
    Pod1 ->> Pod2: Contact Pekko Management
    Pod2 ->> Pod1: Returns already contacted nodes

    break when an existing cluster is found
    Pod1 ->> Pod2: Joins the existing cluster
    end

    break when no existing cluster is found
    Pod1 ->> Pod1: Self-joins and create cluster
    Pod2 ->> Pod1: Joins the created cluster
    end
    end

    Note over Pod1,Pod2: Leader election is performed at this point

    Note right of Pod2: 3-7: Bootstrap process

    rect rgb(250, 148, 142)
    Pod1 ->> Pod2: Exchanges actor messages
    Pod2 ->> Pod1: Exchanges actor messages
    end

    Note right of Pod2: 8-9: Remoting

Traffic between different nodes is described in the below table:

Table 1. Traffic detail for Horizon clustering
Traffic type	Diagram color	Protocol	Port
Kubernetes API	Blue	HTTP	443
Pekko Management	Green	HTTP	7626 (by default)
Pekko Remote	Red	TCP (by default)	17355