Advanced usage

Some edge use-cases might not have been included in the previous installation documentation, for clarity purposes. You may find some of them below.

Running behind a container registry proxy

If your installation environment requires you to whitelist images that can be pulled by the Kubernetes cluster, you must whitelist the registry.evertrust.io/stream and registry.evertrust.io/stream-upgrade images. It is then possible to override the images being pulled by setting the global.imageRegistry key in your values.yaml file to point to your private registry:

global:
  imageRegistry: <YOUR-PRIVATE-REGISTRY>

Leases

To ensure clustering issues get resolved as fast as possible, Stream can use Kubernetes leases. We strongly recommend that you use this safety mechanism. However, the feature can be disabled by setting the leases.enabled key to false.

Injecting extra configuration

Extra Stream configuration can be injected to the bundled application.conf file to modify low-level behavior of Stream. This should be used carefully as it may cause things to break. To do so, use the extraConfig value in your values.yaml file:

This can be done with the following edits to your values.yaml file:

extraConfig: |
    play.server.http.port = 9999
    stream {
      ...
    }

Extra configurations are included at the end of the config file, overriding any previously set config value.

Custom startup scripts

Sometimes, you’ll want to run scripts each time the container starts up in order to configure files in the container or set environment variables. To do so, you’ll need to mount shell scripts into the /docker-entrypoint.d/ directory in the container. Using the Helm chart, this can be achieved easily using the following values.yaml overrides:

extraVolumes:
  - name: stream-entrypoint-scripts
    configMap:
      name: stream-entrypoint-scripts

extraVolumeMounts:
  - name: stream-entrypoint-scripts
    mountPath: /docker-entrypoint.d/

Given you’ve previously create a ConfigMap called stream-entrypoint-scripts:

apiVersion: v1
kind: ConfigMap
metadata:
  name: stream-entrypoint-scripts
data:
  run-on-startup.sh: |
    echo "Hello World !"

By design, Stream is configured to run as an unprivileged user inside the container to follow industry best practices. This means that your scripts won’t be able to perform privileged operations on the container, such as trusting custom CAs. If you do want to overcome this problem, you can run the container as root, even though it is generally discouraged.

Networking overview

When installed in HA, Stream sends messages to other running instances in its cluster. To form the cluster and set up networking between nodes, Stream is relying on Pekko, a framework for building clusterized applications. Understanding how clustering works is important when building deployments with highly specific needs or when preparing a disaster recovery plan.

When deployed on multiple nodes inside a Kubernetes cluster, the following steps are followed:

Discovery: the discovery process locates all nodes that will be used to form a cluster. It relies on a third-party to give that information, such as a DNS record or the Kubernetes API (which is the default when deploying using the Helm Chart). For documentation, see Pekko Discovery.
Bootstrap: once each node in the cluster has the address of every other node, nodes start to contact each other. This is done though Pekko Management, a tool for helping nodes coordinate. For documentation, see Pekko Management.
Remoting: the cluster is now formed, nodes can communicate with each other. This uses Pekko Remoting, a higher level protocol for serializing data over multiple transports. Typically, TCP is used. For documentation, see Pekko Remoting.

This clustering process can be summarized by the below diagram:

Figure 1. Sequence diagram of the cluster management of Stream

Traffic between different nodes is described in the below table:

Table 1. Traffic detail for Stream clustering
Traffic type	Diagram color	Protocol	Port
Kubernetes API	Blue	HTTP	443
Pekko Management	Green	HTTP	7626 (by default)
Pekko Remote	Red	TCP (by default)	17355