Production checklist

Even though the Helm Chart makes installing Horizon a breeze, you’ll still have to set up a few things to make Horizon resilient enough to operate in a production environment.

Operating the database

All persistant data used by Horizon is stored in the underlying MongoDB database. Therefore, the database should be operated securely and backed up regularly.

When installing the chart, you face multiple options regarding your database:

By default, a local MongoDB standalone instance will be spawned in your cluster, using the bitnami/mongodb chart. No additional configuration is required but it is not production ready out of the box. You can configure the chart as you would normally below the mongodb key:
```
mongodb:
  architecture: replicaset
  # Any other YAML value from the chart docs
```
If you want to use an existing MongoDB instance, provide the externalDatabase.uri value. The URI should be treated as a secret as it must include credentials:
```
externalDatabase:
  uri:
    valueFrom:
      secretKeyRef:
        name: <secret name>
        key: <secret key>
```

The chart doesn’t manage the database. You are still in charge of making sure that the database is correctly backed up. You could either back up manually using mongodump or use a managed service such as MongoDB Atlas, which will take care of the backups for you.

Managing secrets

Storing secrets is a crucial part of your Horizon installation. On cloud-native installations like on Kubernetes, we recommend using SSV (Secure Software Vault) to encrypt sensitive data: a master passphrase will be used to encrypt and decrypt data before they enter the database. Alongside with other application secrets like your MongoDB URI (containing your credentials or certificate). We recommend that you create Kubernetes secrets beforehand or inject them directly into the pod.

Values that should be treated as secrets in this chart are:

Name Description Impact on loss

Name	Description	Impact on loss
`vaults.*.master_password`	SSV password used to encrypt sensitive data in database.	Highest impact: database would be unusable
`events.secret`	Secret used to sign and chain events.	Moderate impact: events integrity would be unverifiable
`externalDatabase.uri`	External database URI, containing a username and password.	Low impact: reset the MongoDB password
`appSecret`	Application secret use to encrypt session data.	Low impact: sessions would be reset
`mailer.password`	SMTP server password	Low impact: reset the SMTP password

vaults.*.master_password

SSV password used to encrypt sensitive data in database.

Highest impact: database would be unusable

events.secret

Secret used to sign and chain events.

Moderate impact: events integrity would be unverifiable

externalDatabase.uri

External database URI, containing a username and password.

Low impact: reset the MongoDB password

appSecret

Application secret use to encrypt session data.

Low impact: sessions would be reset

mailer.password

SMTP server password

Low impact: reset the SMTP password

For each of these values, either:

leave the field empty, so that a secret will be automatically generated.

derive the secret value from an existing Kubernetes secret:

appSecret:
  valueFrom:
    secretKeyRef:
      name: <secret name>
      key: <secret key>

Always store auto-generated secrets in a safe place after they’re generated. If you ever uninstall your Helm chart, the deletion of the SSV secret will lead to the impossibility of recovering most of your data.

High availability

By default, the chart will configure a single-pod deployment. This deployment method is fine for testing but not ready for production as a single failure could take down the entire application. Instead, we recommend that you set up a Horizon cluster using at least 3 pods.

In order to do that, configure an horizontalAutoscaler in your override-values.yaml file:

horizontalAutoscaler:
  enabled: true
  minReplicas: 3
  maxReplicas: 3

Use nodeAffinity to spread your Horizon cluster Pods among multiple nodes in different availability zones to reduce the risk of Single Point of Failure.

If your cluster setup requires specific configurations (that could be due to network or configuration constraints), we encourage you to check out the Networking overview section of the documentation.

Configuring ingresses

The recommended way to access Horizon is behind a reverse proxy, known in the Kubernetes world as "ingress controllers". However, Horizon requires that the reverse proxy in front of it (that also terminates the TLS connection) requests certificate client authentication (also known as mTLS).

To create an ingress upon installation, simply set the following keys in your override-values.yaml file:

ingress:
  enabled: true
  hostname: horizon.lab
  tls: true

Identify CAs which will require certificate authentication

You’ll need to gather a list of CAs that will emit certificates which will be able to authenticate to Horizon. To identify them, ask yourself whether the certificates signed by these CAs will:

renew using the EST protocol (used by the Horizon Client)
be used to authenticate users to Horizon (either through API or via the UI)
be used to authenticate the WinHorizon component (in an Active Directory environment)

Other use-cases might also require you to authenticate with a client certificate.

Configure your ingress to require a client certificate

Configuration for mTLS depends on the ingress controller that you use. The following ingress controllers are officially supported by EVERTRUST, and we strongly advise to use one of them with Horizon. However, almost any ingress controller can be configured to correctly request client certificates manually.

ingress-nginx

The Horizon Helm Chart supports autoconfiguring ingress-nginx. To enable client certificate authentication, simply set the following values in the values-override.yaml file:

ingress:
  enabled: true
  type: nginx
  clientCertificateAuth: true
  hostname: horizon.lab
  tls: true

Skip to the Ensure certificate authentication is effective section to test your configuration.

ingress-nginx doesn’t require a list of CAs trusted for client authentication, so any certificate may be submitted by a connecting client. If you wish to specify a list of CAs, disable autoconfiguration and manually configure your ingress using annotations following the ingress-nginx documentation.

Traefik

The Horizon Helm Chart supports autoconfiguring Traefik. To enable client certificate authentication, simply set the following values in the values-override.yaml file:

ingress:
  enabled: true
  type: traefik
  clientCertificateAuth: true
  hostname: horizon.lab
  tls: true

Skip to the Ensure certificate authentication is effective section to test your configuration.

Traefik doesn’t require a list of CAs trusted for client authentication, so any certificate may be submitted by a connecting client. If you wish to specify a list of CAs, disable autoconfiguration and manually configure your ingress using annotations following the Traefik documentation.

Other ingress controllers

If you do not wish or cannot use autoconfiguration, you should ensure your ingress controller is correctly configured to enable all Horizon features.

When requiring client certificates for authentication, the web server should not perform checks to validate that the certificate is signed by a trusted CA. Instead, the certificate should be sent to Horizon through a request header, base64-encoded. The header name used can be controlled using the clientCertificateHeader.
Some endpoints should not be server over HTTPS, in particular those used for SCEP enrollment. You may want to create an HTTP-only ingress for serving paths prefixed by /scep and /certsrv, and prevent those from redirecting to HTTPS.

The cert-auth-proxy component, maintained by EverTrust, can be used to add client certificate authentication to any ingress controller which supports passthrough TLS.

Ensure certificate authentication is effective

To ensure that Horizon can properly decode certificates being sent by clients, get a certificate from a CA configured for client authentication in a cert.pem file and its associated key in a key.pem file.

Then, run the following curl command :

$ curl -k --cert cert.pem --key key.pem https://<Horizon URL>/api/v1/security/principals/self

If Horizon returns an error, or states that the principal is not authenticated (through a 204 HTTP code), then certificate authentication is incorrectly configured.

Instead, information about the certificate should be returned in the principal key :

{
  "identity": {
    "identifier": "CN=User, O=EVERTRUST, C=FR", (1)
    "name": "User",
    "identityProviderType": "X509", (2)
    "identityProviderName": "EVERTRUST CA"
  },
  "permissions": [],
  "roles": null,
  "teams": null,
  "preferences": null,
  "customDashboards": null
}

1	The DN of the certificate is used as the principal identifier.
2	The identity provider is of type X509.