Advanced usage

Some edge use-cases might not have been included in the previous installation documentation, for clarity purposes. You may find some of them below.

Running behind a container registry proxy

If your installation environment requires you to whitelist images that can be pulled by the Kubernetes cluster, you must whitelist the registry.evertrust.io/horizon and registry.evertrust.io/horizon-upgrade images. It is then possible to override the images being pulled by setting the global.imageRegistry key in your values.yaml file to point to your private registry:

global:
  imageRegistry: <YOUR-PRIVATE-REGISTRY>

Leases

To ensure clustering issues get resolved as fast as possible, Horizon can use Kubernetes leases. We strongly recommend that you use this safety mechanism. However, the feature can be disabled by setting the leases.enabled key to false.

Trusting custom CAs

Available from image 2.7.5-1

When your application needs to establish secure connections with services that use certificates signed by custom Certificate Authorities, you need to import these CA certificates into your system’s trust store. This documentation shows how to accomplish this using Kubernetes ConfigMaps or Secrets.

1. Import Your Custom CA Certificate

Choose one of the following methods to store your CA certificate:

A. Using a ConfigMap

extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: custom-ca-certificates
    data:
      company-internal-ca.pem: |
        -----BEGIN CERTIFICATE-----
        MIIGfjCCBGagAwIBAg...
        -----END CERTIFICATE-----

B. Using a Secret

extraObjects:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: custom-ca-certificates
    data:
      company-internal-ca.pem: <YOUR_CA_CERTIFICATE_BASE64_ENCODED>

To encode your certificate in base64, use: cat your-ca.pem | base64 -w 0

2. Configure Your Application to Trust the CA Certificate

After creating the ConfigMap or Secret with your CA certificate, configure your application to use it by setting the appropriate environment variable:

A. Loading from ConfigMap

environment:
  - name: SYSTEM_CA_TRUST
    valueFrom:
      configMapKeyRef:
        name: custom-ca-certificates
        key: company-internal-ca.pem

B. Loading from Secret

environment:
  - name: SYSTEM_CA_TRUST
    valueFrom:
      secretKeyRef:
        name: custom-ca-certificates
        key: company-internal-ca.pem

Notes

The SYSTEM_CA_TRUST environment variable is used by the application to add the provided certificate to the system’s trusted certificate store.
You can provide multiple CA certificates by concatenating multiple certificates under a single key (ensure each certificate begins with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----)

Example Complete Deployment using Horizon Helm Chart

environment:
  - name: SYSTEM_CA_TRUST
    valueFrom:
      configMapKeyRef:
        name: custom-ca-certificates
        key: company-internal-ca.pem

extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: custom-ca-certificates
    data:
      company-internal-ca.pem: |
        -----BEGIN CERTIFICATE-----
        MIIGfjCCBGagAwIBAg...
        -----END CERTIFICATE-----

Custom startup scripts

Sometimes, you’ll want to run scripts each time the container starts up in order to configure files in the container or set environment variables. To do so, you’ll need to mount shell scripts into the /docker-entrypoint.d/ directory in the container.

Networking overview

When installed in HA, Horizon sends messages to other running instances in its cluster. To form the cluster and set up networking between nodes, Horizon is relying on Pekko, a framework for building clusterized applications. Understanding how clustering works is important when building deployments with highly specific needs or when preparing a disaster recovery plan.

When deployed on multiple nodes inside a Kubernetes cluster, the following steps are followed:

Discovery: the discovery process locates all nodes that will be used to form a cluster. It relies on a third-party to give that information, such as a DNS record or the Kubernetes API (which is the default when deploying using the Helm Chart). For documentation, see Pekko Discovery.
Bootstrap: once each node in the cluster has the address of every other node, nodes start to contact each other. This is done though Pekko Management, a tool for helping nodes coordinate. For documentation, see Pekko Management.
Remoting: the cluster is now formed, nodes can communicate with each other. This uses Pekko Remoting, a higher level protocol for serializing data over multiple transports. Typically, TCP is used. For documentation, see Pekko Remoting.

This clustering process can be summarized by the below diagram:

Figure 1. Sequence diagram of the cluster management of Horizon

Traffic between different nodes is described in the below table:

Table 1. Traffic detail for Horizon clustering
Traffic type	Diagram color	Protocol	Port
Kubernetes API	Blue	HTTP	443
Pekko Management	Green	HTTP	7626 (by default)
Pekko Remote	Red	TCP (by default)	17355