Syslog Tutorial: Features, Code Examples, Tutorials & More (2024)

Syslog is a standard for sending and receiving notification messages–in a particular format–from various network devices. The messages include time stamps, event messages, severity, host IP addresses, diagnostics and more. In terms of its built-in severity level, it can communicate a range between level 0, an Emergency, level 5, a Warning, System Unstable, critical and level 6 and 7 which are Informational and Debugging.

Moreover, Syslog is open-ended. Syslog was designed to monitor network devices and systems to send out notification messages if there are any issues their functionality–it also sends out alerts for pre-defined events and monitors suspicious activity via the change log/event log of participating network devices.

The Syslog protocol was initially written by Eric Allman and is defined in RFC 3164. The messages are sent across IP networks to the event message collectors or syslog servers. Syslog uses the User Datagram Protocol (UDP), port 514, to communicate. Although, syslog servers do not send back an acknowledgment of receipt of the messages. Since 2009, syslog has been standardized by the IETF in RFC 5424.

Screenshot via Flylib

Today, it has gained wide support on many operating systems including almost all versions of Linux, Unix, and MacOS. For Microsoft Windows, Syslog is supported through open sources and commercial third-party libraries.

Benefits of Logging

In the simplest definition, logging is the act of keeping a log. Sysadmins have engaged in an ongoing debate over what level of detail to log their system data. There is the tradeoff between using up disk space too quickly and not having enough data in your logs.

Yet, the benefits of logging are still wide-ranging—especially when troubleshooting code. It is necessary to have a standardized and centralized system to generate, record and log messages. Furthermore, it helps to improve your ability to control and use logging data. Here are a few other benefits:

• Reduce the number of trouble tickets
• Reduce downtime
• Decrease the volume of business interruptions
• Promote preventative troubleshooting

Without logging, it can become a nightmare to search for a single transaction that may have been processed on any of your servers. With centralized logging, you get a correlated view of all of the log data. In contrast, reviewing each log file separately can become quite time-consuming. This is why using Syslog to forward local log messages to a remote log analytics server has become the standard for logging solutions.

Components of Syslog Servers

You now understand how Syslog offers a central repository for logs from multiple sources. To achieve this objective, Syslog servers have several components including:

• A Syslog listener–The listener gathers and processes syslog data sent over UDP port 514. Although, there is no acknowledgment receipt and messages aren’t guaranteed to arrive.
• A database–Syslog servers need databases to store the massive amounts of data for quick access.
• Management and filtering software–Since there can be enormous amounts of data, it can take excessive amounts of time to find specific log entries. The syslog server needs help to automate the work, as well as to filter to view specified log messages.

Filtering Specifics

To illustrate, it can extract messages based on specific parameters such as a critical event or device name. You can also use the filter to avoid seeing certain types of entries through the Negative Filter rule. If you wanted, you could show all of the critical log messages from a firewall.

How It Works

There are three different layers within the Syslog standard, which are:

• Syslog content (information contained in an event message)
• Syslog application (generates, interprets, routes and stores messages)
• Syslog transport (transmits the messages)

In addition, applications can be configured to send messages to several destinations. There are also alarms that give instant notifications for events such as:

• Hardware errors
• Application failures
• Lost contact
• Mis-configuration

Furthermore, alarms can be set up to send notifications via SMS, pop-up messages, email, HTTP and more. Since the process is automated, the IT team will get immediate notifications of there is a sudden failure of any of the devices.(Image via Wikimedia Commons)

Syslog Servers

Syslog servers are used to send diagnostic and monitoring data. The data can then be analyzed for system monitoring, network maintenance and more. Since the Syslog protocol is supported by a wide swath of devices, they can conveniently log information into the Syslog server.

SNMP datacan be used to assess any failure points quickly. Syslog servers can also have automatedevents to trigger alerts that help to prevent downtime or outages.

List of a Few Windows-Based Syslog Servers

1. Kiwi Syslog Server. This server is simple to install and generates reports in plain text or HTML. The software handles Syslog and SNMP, even from Linux and UNIX hosts. It is compatible with Win XP 32/64, Win 2003 32/64, Windows Vista 32/64, Win7 32/64, Windows 2008 R2 32/64, Windows 8, Windows Server 2012 & 2012 R2.

2. PRTG. This adds a sensor to the PRTG monitoring to enable Syslog ability. It focuses on SNMP and Syslog protocol data. It is compatible with any Windows 64-bit environment with Windows Server 2012 R2.

3. SNMPSoft Sys-log Watcher. This is a dedicated syslog server for a wide variety of devices. It can also parse and manage non-standard Syslog. It is compatible with Windows XP through Windows 10.

4. The Dude. This system is used for general network management with a built-in syslog server. Furthermore, it comes with functionality for remote logging through the RouterOS. It is compatible with Windows 2000 or newer. Yet, it also runs on Linux or MacOS using Wine/Darwine.

5. Visual Syslog Server. This is a lighter syslog option that looks at alerts in real time. Thresholds can be configured to trigger both scripts and programs. It is compatible with Windows XP, Vista, 7, 8, 8.1, as well as Windows Server 2003, 2008, 2012.

Mac-OS Syslog Servers

For the Mac OS X, you can use Splunk–which enables system monitoring and syslog events. In fact, Splunk is known as the tool for operational intelligence. Also, you can configure Splunk as a forwarder to your central monitoring server. To configure the Mac OS X Syslogd you would:

1. Open a terminal window

2. Make a backup copy of syslogd.conf into the /tmp folder by typing

 $ cp /etc/syslog.conf /tmp/syslog.conf.bkp

3. Open the configuration file in the editor of your choice

$ sudo vi /etc/syslog.conf

Password: The ‘sudo’ command is used to execute vi with “root” privileges.

4. Replace the IP address 192.168.1.12 with the IP address if your Splunk server’s network interface

5. Type ‘i’ in vi to enter the insert mode (text entry). Make sure to use tabs and not spaces between the selector and action fields.

6. Save and Exit. Save the file by typing

’:wq <enter>’

7 Restart the ‘syslogd’ service, and test it by typing

$ ps -e | grep syslogd5070 ?? 2:33.75 /usr/sbin/syslogd

These are the commands that restart the service:

$ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist$ sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

To log an event, open a new Terminal window and type:

$ logger -s -p user.info Testing splunk syslog forwarding

The Syslog Format

Syslog has a standard definition and format of the log message defined by RFC 5424. As a result, it is composed of aheader, structured-data (SD) and a message.Within the header, you will see a description of the type such as:

• Priority
• Version
• Timestamp
• Hostname
• Application
• Process id
• Message id

Then, you will see structured-data which have data blocks in the “key=value” format within square brackets. After the SD, you will see the detailed log message, which is encoded in UTF-8.

For example, the following message:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

Corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

Syslog Messages

Syslog messages are used to report levels of Emergency and Warnings with regards to software or hardware issues. To illustrate, a system restart will be sent through the Notice level. A system reload will be sent through the Informational level. If debug commands are outputted, it is conveyed through the Debug level.

Screenshot via Cisco

Here are the Syslog Message Levels:

• Emergency Messages–System is unavailable and unusable (Could be a “panic” condition due to a natural disaster)
• Alert Messages–Action needs to be taken immediately (an example is loss of backup ISP connection)
• Critical Messages–Critical conditions (this could be a loss of primary ISP connection)
• Error Messages–Error conditions (must be resolved within a specified time frame)
• Warning Messages–Warning conditions (indicates an error may occur if action is not taken)
• Notification Messages–Things are normal, but this is still a significant condition (immediate action is usually not required)
• Informational Messages–Informational messages (for reporting and measuring)
• Debugging Messages–Debug-level messages (Offers information around debugging apps)

Here are some additional examples from TechZone. This system error message:

May 11 10:40:48 scrooge disk-health-nurse[26783]: [ID 702911 user.error] m:SY-mon-full-500 c:H : partition health measures for /var did not suffice - still using 96% of partition spaceWe can split into the following columns:

Column 1 = "May 11 10:40:48" > TimestampColumn 2 = "scrooge" > LoghostColumn 3 = "disk-health-nurse[26783]:" > Application/ProcessColumn 4 = "[ID 702911 user.error]" > Syslog facility.levelColumn 5 = "m:SY-mon-full-500" > Message IDColumn 6 = "c:H : partition health..." > Message [possibly including rid, sid, ip]

And, here’s an example of a summary message:

May 11 10:00:39 scrooge SG_child[808]: [ID 748625 user.info] m:WR-SG-SUMMARY c:X vhost:iscrooge61.seclutions.com:80 (http) GET / => http://bali/ , status:200 , redirection URL:<n/a> , referer:<n/a> , mapping:bali , request size: 421 , backend response size: 12960 , audit token:- , time statistics (microseconds): [request total 16617 , allow/deny filters 1290 , backend responsiveness 11845 , response processing 1643 , ICAP reqmod <n/a> , ICAP respmod <n/a> ] timestamp: [2012-05-11 10:00:39] [ rid:T6zHJ38AAAEAAAo2BCwAAAMk sid:910e5dd02df49434d0db9b445ebba975 ip:172.18.61.2 ]

Which contains the following columns:

Column 1 = "May 11 10:00:39" > TimestampColumn 2 = "scrooge" > LoghostColumn 3 = "SG_child[808]:" > Application/ProcessColumn 4 = "[ID 748625 user.info]" > Syslog facility.levelColumn 5 = "m:WR-SG-SUMMARY" > Message IDColumn 6 = "c:X vhost:..." > Message [including time statistics and rid, sid, ip]

The Most Important Log Files to Track and Monitor

Monitoring log files is critical because it helps you manage any errors in the functioning of your OS. Some the types of relevant information you will receive includes:

• User issues
• Security breaches
• Hard disk crashes or power outages

Of course, there are log files of high priority that you should always keep track of. The log files include:

• /var/log/messages–Contains most system messages
• /var/log/secure–Authentication messages
• /var/log/cron–Logs Cron job activities
• /var/log/maillog–Mail transactions

If you were to look into /var/log/messages, you would find:

• The timestamp
• The hostname of the executing program
• Name of the utility that prompted the message
• The action that took place

Pros and Cons of Syslog

One problematic scenario is when your /var/log/messages file fills up due to logging misconfiguration. Plus, there will be times when your system’s logging will cause unforeseen issues. This is why it is imperative to understand how to control your logging and where your logs are saved. Plus, there may be some packet loss if there is a large burst of network traffic.

Also, the fact that Syslog is based on UDP means there can be issues with reliability. On the other hand, as systems become more complex, it becomes increasingly important to collect and monitor all relevant data produced by applications.

We analyze this data to determine how systems are behaving. Plus, logs are considered a reliable data source for understanding current system statistics and making trend predictions. Not to mention, logs are used for activities such as troubleshooting such or rolling back the system after a failure accident.

Best Practices

In terms of securing your log files, you will have many devices to generate that data. Still, it is a best practice to funnel all logs to a dedicated host. We should harden and secure this host.

Moreover, you only want to open up the syslog port in all firewalls between you and the UDP/514. If you have a geographic network, then you should have a local loghost—at each location– that sends data to the central loghost.

You can also rotate a log file once it reaches a particular size. Nonetheless, the UNIX logrotate utility will continue to write the log information to a new file after rotating the old file. Here are the keys to use:

• /usr/sbin/logrotate–The logrotate command
• /etc/cron.daily/logrotate–The shell script that executes the logrotate command on a daily basis.
• /etc/logrotate.conf– global configuration for log rotation
• /etc/logrotate.d–For individual packages

To rotate a log file for every 1KB, use the logrotate.conf below

$ cat logrotate.conf/tmp/output.log {size 1kcreate 700 bala balarotate 4}

It gives you three options:

• size 1k–logrotate only runs if the file size is equal to or greater than this size
• create–rotate the original file and create a new file with configured users, groups and permissions
• rotate–this keeps only the most recent four rotated log files

Since an increasing number of organizations are moving to the cloud, the need for log management tools and services has never been greater. It is good to have centralized logs, but you also need the right tools to analyze them effectively. Going through files individually will drive you nuts.

Several Free and Paid Tools For Centralized Logging

1. Retrace–one of Stackify’s developer tools, and the only developer tool that combines APM, errors, metrics, and monitoring with logging to provide a fully-integrated, multi-environment tool that gives you app performance superpowers.

2. Loggly–this is a cloud-based management and analytics service provider that has a free and paid plan starting at $49 per month. With their dynamic field explorer, you get a bird’s eye view of your logs. It also comes with power full-text searches.

3. logz.io–this tool has free and paid plans starting at $89 per month. It has an interface powered by Kibana that lets you search through millions of records easily. You can also filter your results with custom parameters.

4. Splunk–this is a popular tool that has been around since 2003. It also comes in free and paid plans. The paid plan pricing depends on the amount of volume your process. In addition, it comes with powerful drill-downs that let you go back in time with ad-hoc queries.

5. Logstash–this is a free open source tool for managing and collecting your events and logs. Plus, you can use it along with Kibana.

6. AWStats–this free analysis tool has a community of thousands of users. This is because it lets you generate streamlined reports, as well as exporting the log data in varying formats for analysis offline. Moreover, you can run it on almost every popular platform.

Conclusion

For a list of 51 useful log management tools, check out our listhere.Syslog Better Logging Tutorial

• Understand logging in Linux
• Understanding Syslog: Servers, Messages & Security
• Understanding syslog conf – targeting specific types / tags of logs
• Setting up a centralized Linux log server

Syslog is a widely used tool that makes management of complex networks much more straightforward. Plus, with the right tools, you don’t have to worry about the volume of data since you can filter easily and conveniently.