Logging

Horizon is able to format output log thanks to logback.

Horizon defines a default rolling file appender named RUN.

The default configuration keeps the technical logs for 30 days into files with the following naming convention:

horizon.log-<yyyy-MM-dd>.log

Those files are available under the /opt/horizon/var/log directory

Directly sending logs to your syslog server

1. Access the EverTrust Horizon server through SSH with an account with administrative privileges;

2. Using an editor like vi, open the horizon-logback.xml file located at /opt/horizon/etc/horizon-logback.xml ;

3. Edit the appender named "SYSLOG" to change the IP address for the syslogHost to redirect to your own syslog server. As an example, if your syslog server is on 192.168.1.2 and the Horizon logs must be processed by the LOCAL6 facility, the syslog appender should look like this:

<appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
    <syslogHost>192.168.1.2</syslogHost>
    <facility>LOCAL6</facility>
    <suffixPattern>%msg%n</suffixPattern>
</appender>

4. Still in the horizon-logback.xml file, update any logger with the SYSLOG appender ref and ensure that the log level is set to "INFO":

<logger name="event" level="INFO">
    <appender-ref ref="SYSLOG"/>
</logger>

5. Save your modifications and restart the Horizon service:

$ systemctl restart horizon

The functional logs from Horizon should now be received by your remote syslog server:

horizon {"code": "SERVICE-STOP","details":[{"key":"horizonVersion","value":"2.5.0"},{"key":"message","value":"Service successfully stopped"}],"module":"service","node":"horizon","timestamp":1674054152149,"status":"success"}
horizon {"code": "SERVICE-START","details":[{"key":"horizonVersion","value":"2.5.0"},{"key":"message","value":"Service successfully started"}],"module":"service","node":"horizon","timestamp":1674054170567,"status":"success"}

Using the local syslog server for filtering and forwarding

Alternatively, you might want to use a local syslog instance to add grok filtering to your logs before forwarding them to your own syslog server. To do so, ensure that you have a syslog instance running (like rsyslog), then:

1. Access the EverTrust Horizon server through SSH with an account with administrative privileges;

2. With an editor like vi, edit the /etc/rsyslog.d/horizon.conf (or create it if it does not exist yet) to add this line:

local6.*                                                @REMOTE_SYSLOG_HOSTNAME

Don’t forget to replace the REMOTE_SYSLOG_HOSTNAME to the IP or DNS name of your remote syslog server. As an example, if your syslog server is on 192.168.1.2, the line should look like this:

local6.*                                                @192.168.1.2

Note that you must set up your syslog host to accept UDP traffic on a specific port (here, we are going to use the default port which is 514) and catch the local6 facility logs, however the configuration of your own syslog host is out of the scope of this document.

3. Edit the /etc/rsyslog.conf file to uncomment the module and input lines of the UDP section:

#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

They should look like this after uncommenting:

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

4. Restart your syslog service:

$ systemctl restart rsyslog

JSON Logging

1. Access the EverTrust Horizon server through SSH with an account with administrative privileges;

2. Using an editor like vi, open the horizon-logback.xml file located at /opt/horizon/etc/horizon-logback.xml ;

3. To send the JSON logs to a syslog server: Edit or create the appender named "JSON_SYSLOG" to change the IP address for the syslogHost to redirect to your own syslog server. As an example, if your syslog server is on 192.168.1.2 and the Horizon logs must be processed by the LOCAL6 facility, the syslog appender should look like this:

<conversionRule conversionWord="syslogStart" converterClass="ch.qos.logback.classic.pattern.SyslogStartConverter"/>
<appender name="JSON_SYSLOG" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
    <host>192.168.1.2</host>
    <port>514</port>
    <layout class="net.logstash.logback.layout.LogstashLayout">
      <prefix class="ch.qos.logback.classic.PatternLayout">
        <pattern>%syslogStart{LOCAL6}</pattern>
      </prefix>
      <fieldNames>
        <timestamp>time</timestamp>
        <logger>logger</logger>
        <thread>thread</thread>
        <level>severity</level>
        <stackTrace>exception</stackTrace>
      </fieldNames>
      <customFields>{"app":"horizon", "hostname":"${HOSTNAME}"}</customFields>
    </layout>
</appender>

To send the JSON logs to the local console: Edit or create the appender named "STDOUT" to change the encoder to a JSON one. The final configuration should look like this.

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <fieldNames>
          <timestamp>time</timestamp>
          <logger>logger</logger>
          <thread>thread</thread>
          <level>severity</level>
          <stackTrace>exception</stackTrace>
        </fieldNames>
        <customFields>{"app":"horizon", "hostname":"${HOSTNAME}"}</customFields>
    </encoder>
  </appender>

4. Still in the horizon-logback.xml file, update any logger with the appender ref and ensure that the log level is not OFF:

<logger name="event" level="INFO">
    <appender-ref ref="JSON_SYSLOG"/>
</logger>

or

<logger name="controllers" level="INFO">
    <appender-ref ref="STDOUT"/>
</logger>

5. Save your modifications and restart the Horizon service:

$ systemctl restart horizon

The functional logs from Horizon should now be in JSON Format:

Console : {"time": "2023-08-16T16:12:54.481+02:00","@version":"1","message":"[Actor pkimanager] - Registering PKI Queue 'slowed-queue' (cluster wide: 'false')","logger":"actors.pki.PKIManagerActor","thread":"application-blocking-io-dispatcher-43","severity":"INFO","level_value":20000,"HOSTNAME":"horizon.evertrust","application.home":"/opt/horizon","kamonSpanId":"c5a74b959971c7ee","kamonTraceId":"b1ccb54c9eb7e493","kamonSpanName":"/ui","app":"horizon","hostname":"horizon.evertrust"}
Syslog : 2023-08-16T16:12:54+02:00 horizon.evertrust {"time": "2023-08-16T16:12:54.482+02:00","@version":"1","message":"[Actor pkimanager] - Registering PKI Queue 'test' (cluster wide: 'false')","logger":"actors.pki.PKIManagerActor","thread":"application-blocking-io-dispatcher-43","severity":"INFO","level_value":20000,"HOSTNAME":"horizon.evertrust","application.home":"/opt/horizon","kamonSpanId":"c5a74b959971c7ee","kamonTraceId":"b1ccb54c9eb7e493","kamonSpanName":"/ui","app":"horizon","hostname":"horizon.evertrust"}