Troubleshooting

Horizon Doctor

Horizon Doctor is currently only available for deployments on CentOS/RHEL. To troubleshoot deployments on Kubernetes, use built-in tools like events and logs.

Horizon doctor is a tool that performs checks on your Horizon installation as well as its required dependencies to ensure that everything is configured properly. The tool is targeted towards troubleshooting during installation or update procedures. Note that the tool requires root permissions to run.

Performed checks

At the moment, Horizon Doctor checks for:

OS checks

  • Checks for installed Horizon version, MongoDB version, Java version, Nginx version, OS Version.

    • If the OS is a RedHat distribution, checks if the RedHat subscription is active

    • If Mongo is not installed locally, it notices it as an information log

  • Checks for SELinux's configuration: throws a warning if it is enabled, says ok if it is on permissive or disabled

  • Checks for the status of the necessary services: postfix, mongod, nginx and horizon.

    • If the postfix service is running, tries to connect via a TCP SYN on the port 25 of the relayhost specified in the /etc/postfix/main.cf file and throws an error if it can’t.

  • Checks how long the Horizon service has been running for.

  • Checks if there is an NTP service active on the machine and checks if the system clock is synchronized with the NTP service.

Config checks

  • Checks for existence and permissions of the configuration file: the permissions are expected to be at least 640 and the file is supposed to belong to horizon:horizon

  • Checks for existence and permissions of the licence file: the permissions are expected to be at least 640 and the file is supposed to belong to horizon:horizon.

  • Checks for existence and permissions of the vault file: the permissions are expected to be at least 640 and the file is supposed to belong to horizon:horizon.

  • Checks for the permission of the Horizon directory (default: /opt/horizon): the permission is expected to be at least 755.

  • Checks for the existence of the symbolic link for nginx configuration and runs an nginx -t test.

  • Retrieves the Java heap size parameters that were set for Horizon and throws a warning if the default ones are used (min = 2048 and max = 3072).

  • Retrieves the Horizon DNS hostname and stores it for a later test (throws an error if it has not been set).

  • Checks for the Horizon Play Secret and Horizon Event Seal Secret: these are the Horizon application secrets and should be different from default value thus Horizon Doctor throws an error if either of them is equal to the default value (changeme).

  • Retrieves the MongoDB URI (throws a warning if MongoDB is running on localhost; throws an error if MongoDB is running on an external instance but the authSource=admin parameter is missing from the URI).

  • Parses the Horizon license file to retrieve its expiration date as well as the license details (number of holders per category).

Network checks

  • Runs a MongoDB ping on the URI, then checks for the database used in the URI (throws a warning if the database used is not called horizon; throws an error if no database is specified in the URI).

  • Checks for PEKKO High Availability settings: if no node hostname is set up, skips the remaining HA checks. If 2 nodes are set up, retrieves which node is running the doctor and checks for the other node. If 3 nodes are set up, retrieves which node is running the doctor and checks for the other 2 nodes. The check runs as:

    • if curl is installed, runs a curl request on the Node hostname at alive on the management port (default is 7626), and if alive runs another curl request on the Node hostname at /ready on the management port. Both requests should return HTTP/200 if ok, 000 otherwise.

    • if curl is not installed, uses the built-in Linux TCP socket to run TCP SYN checks on both the HA communication port (default is 17355) and the management port (default is 7626) on the Node hostname.

  • Checks for firewall configuration. Currently only supports firewalld (RHEL) and a netstat test.

    • The netstat part will run a netstat command to check if the JVM listening socket is active (listening on port 9000). If netstat is not installed, it will skip this test.

    • The firewalld part will check if the HTTP and HTTPS services are opened in the firewall and if it detected a HA configuration, it will check if the HA ports (both of them) are allowed through the firewalld. If firewalld is not installed or not active, it will skip this test.

  • Checks if IPv6 is active in every network interface and throws a warning if it is the case (specifying the interface with IPv6 turned on).

TLS checks

  • Checks for existence and permissions of the Horizon server certificate file: the permissions are expected to be at least 640 and the file is supposed to belong to the nginx group.

  • Parses the Horizon server certificate file: it should be constituted of the actual TLS server certificate first, then of every certificate of the trust chain (order being leaf to root). It throws a warning if the certificate is self-signed or raises an error if the trust chain has not been imported. It otherwise tries to reconstitute the certificate trust chain via the openssl verify command, and throws an error if it cannot.

  • Parses the Horizon server certificate file and checks if the Horizon hostname is present in the SAN DNS names of the certificate, throws an error if it is not there.

Log packing option

If the Horizon doctor is launched with the -l option, it will pack the logs of the last 7 days (in /opt/horizon/var/log) as well as the startup logs (the /var/log/horizon/horizon.log file) and create a tar archive.

The -l option accepts an optional parameter that should be an integer (1-99) and will pack the logs of the last n days instead, as well as the startup logs.

Note that the Horizon doctor will still perform all of its check; the log packing is done at the very end of the program.

Example of call to pack the logs of the last 7 days:

$  horizon-doctor -l

Example of call to pack the logs of the last 30 days:

$ horizon-doctor -l 30

Saving the doctor’s output

If the Horizon doctor is launched with the -o option, it will perform all of its checks and save the output in the specified file instead of displaying it into the stdout (default is the command line interface).

If you use the option, you must provide a filepath in a writable directory.

Example of call to save the output in a file named horizon-doctor.out instead of the stdout:

$ horizon-doctor -o horizon-doctor.out

Help menu

To display Horizon doctor’s help menu, use the -h option.

Additional checks

  • Ensure that you are using an up-to-date web browser when trying to access the Horizon web interface.

  • Ensure that Javascript in turned on in your web browser.

  • Ensure that your user machine can access the server where Horizon was installed.

  • If several hostnames have been set up for the Horizon interface, ensure that every single one of them is present in the TLS certificate SAN DNS names.