In the Linux operating system, many applications send logs to syslog, the primary logging mechanism. Best security practices often dictate that it is ideal to send your logs to a separate system, like Loggly, a SaaS solution for log data management.
There are many reasons to do this, but the primary ones are:
Backup of critical log data Ability to verify log integrity and detect data changes Consolidation of logs across systems
Loggly has many features, as shown below, but one feature, in particular, is Loggly’s agent-free deployment ability.
Monitoring multiple cloud systems, internal logs, and application logs Integration with third-party systems, such as Slack, GitHub, or PagerDuty Analysis and visualization of data using graphs and KPIs Agent-free deployment
In this article, we are going to explain how to integrate Loggly with rsyslog, a common and powerful replacement for syslogd. This means that there is no agent needed on a Linux system, and rsyslog can be configured to directly send the logs to Loggly.
Installing rsyslog
Many systems come with rsyslog preinstalled, but if it is not, a few steps are needed for installation. Even though rsyslog is commonly included in the repositories of many distributions, it may not be the latest version. In this article, Ubuntu is the distribution of choice, notably that of 18.04. Instructions for other distributions are located here.
Adding the rsyslog Repository
In the case of Ubuntu, we need to add the Adiscon PPA repository.
Installing rsyslog
There are two main packages we need to install for rsyslog. Beyond the base package itself, the rsyslog-gnutls package allows us to use an encrypted connection to the Loggly service.
Adding the Loggly TLS Certificates
Before we configure rsyslog with TLS, we first need to download and make accessible the Loggly certificates.
Verifying the Work Directory Permissions
There are some cases where the /var/spool/rsyslog permissions are incorrect, and the following code will verify that the permissions are set correctly (in the context of Ubuntu 18.04).
Configuring Loggly
Over time, the configuration directives and syntax have changed for rsyslog. Often, there is a mix of the old and new directives available. Below is a default configuration for rsyslog using the new syntax only. The primary reason to modify this configuration is to point rsyslog to the certificate that we just downloaded.
After signing in to your Loggly account, you will need to create a customer token. This is located under Source Setup > Customer Tokens. As there is a new user interface coming, included below are screenshots for both the old and new interfaces.
Once you have navigated to Customer Tokens, click on the “Add New” button to generate a new token. It is best to give this token a description.
Finally, you will have a token that you can use. Copy this for use later in our configuration files.
Configuring rsyslog for Loggly
The final step for configuring rsyslog for Loggly is to define our configuration file and restart rsyslog. Below is a default configuration file located in /etc/rsyslog.d/22-remote.conf that will tell rsyslog to send syslog events to Loggly.
Within this configuration, we define the format for sending the logs over, which helps Loggly to properly categorize the logs. The action configuration is for sending all Syslog messages to Loggly by default. Finally for this configuration to take effect, we need to restart rsyslog.
rsyslog Troubleshooting
If you are encountering issues with the configuration, which tend to be either permissions or a mis-configuration, then you can view the current log here: /var/log/rsyslog.log. Additionally, to turn on a higher level of logging, add the following lines to your rsyslog.conf file and restart rsyslog.
Conclusion
The combination of rsyslog and Loggly is a powerful one. Allowing you to consolidate, validate, and analyze your logs is important to maintaining a proper security posture. You will quickly find that the advanced features of Loggly are well worth the setup when used in conjunction with your advanced logging configuration for rsyslog.
