Learn about Logging and Log Rotation on Linux

Updated on : Nov 9, 2018 Category : Linux/ Unix

Ever thought how the logs on Linux Machine or Linux Web Server are rotated? Wondering what are those tarballs in your /var/log directory.

Today we will share some information about Logging and their rotation on Linux.

Logging has become important in today’s date. As Logging helps a developer to debug the issue and errors. A Linux system has many subsystems and applications running. We use system logging to gather data about our running system from the moment it boots. And Sometimes, these logs are checked just to make sure if everything is running fine without any problems. Linux logs provide a timeline of events for the Linux operating system, applications, and system, and are a valuable troubleshooting tool when you encounter issues. Essentially, analyzing log files is the first thing an administrator needs to do when an issue is discovered. Files are stored in plain-text and can be found in the /var/log directory and subdirectory. There are Linux logs for everything: system, kernel, package managers, boot processes, Apache, MySQL, etc.

There are a few most important directory sections of the logs. Most directories can be grouped into one of four categories:

  • Application Logs: file of events that are logged by a software application. It contains errors, informational events, and warnings.
  • Event Logs: which records a variety of events, including system error messages, system startups, and system shutdowns
  • Service Logs: They contain messages about the server, including the services and applications running on it.
  • System Logs: They contain the Linux operating system, applications, and syslogs, and are a valuable troubleshooting tool when you encounter issues.

A few other directories and their uses include:

  • /var/log/daemon.log: tracks services running in the background that perform important tasks.
  • /var/log/btmp: recordings of failed login attempts.
  • /var/log/utmp: current login state, by user.
  • /var/log/wtmp: login/logout history.
  • /var/log/lastlog: information about the last logins for all users. This binary file can be read by command lastlog.
  • /var/log/pureftp.log: runs the pureftp process that listens for FTP connections. All connections, FTP logins, and authentication failures get logged here.
  • /var/log/spooler: rarely used and often empty. When used, it contains messages from USENET.
  • /var/log/xferlog: contains all FTP file transfer sessions, including information about the file name and user initiating FTP transfers.

As we know Linux Operating System is very stable and does not require any reboots or encounter the crashes frequently. Linux Logging has a special mechanism of log rotation in their system. This means that logging will be done for all the activities on the machine when it is running.  This will create a huge pile of logs in the /var/log directory which will occupy tremendous Disk Space on your machine. So as to prevent such problems, In Linux log rotation is typically performed using the logrotate command. The command can be used to email logs to a systems administrator after log rotation. Dated logs may also be compressed.  The command is often run as a cron job, which has the effect of fully automatic log rotation. Typically, a new logfile is created periodically, and the old logfile is renamed by appending a “1” to the name. Each time a new log file is started, and all rotated log files are compressed by gzip into the .gz. Old logfiles whose number exceeds a threshold can then be deleted or archived off-line to save space.

The configuration file for logrotate is located at /etc/logrotate.conf where you can find configs like below image.

logrotate config

The following is a list of some of the directives you can specify in your logrotate configuration file:

  • weekly — Specifies the rotation of log files to be done weekly. Similar directives include:

    • daily
    • monthly

    • yearly

       

  • compress — Enables compression of rotated log files. Similar directives include:

    • nocompress
    • compresscmd — Specifies the command to be used for compressing.
    • uncompresscmd
    • compressext — Specifies what extension is to be used for compressing.
    • compressoptions — Specifies any options to be passed to the compression program used.

    • delaycompress — Postpones the compression of log files to the next rotation of log files.

  • rotate INTEGER — Specifies the number of rotations a log file undergoes before it is removed or mailed to a specific address. If the value 0 is specified, old log files are removed instead of rotated.

  • mail ADDRESS — This option enables mailing of log files that have been rotated as many times as is defined by the rotate directive to the specified address. Similar directives include:
    • nomail
    • mailfirst — Specifies that the just-rotated log files are to be mailed, instead of the about-to-expire log files.
    • maillast — Specifies that the about-to-expire log files are to be mailed, instead of the just-rotated log files. This is the default option when mail is enabled.

    Comparison of Appending vs. Rotation of Log File

    Even if the application writes a new log file after each restart, how is it saving in the current log? Is it appending to one single which turns into a massive file? Linux systems are not known for frequent reboots or crashes: applications can run for very long periods without even blinking and this is a benefit, but this could be a problem because that can also make the log file very large.  If we are trying to analyze the root cause of a connection failure that happened last week, we could easily be searching through tens of thousands of lines.

    It is recommended to configure the application to rotate its log file once every day. Because it becomes manageable for a starter. It’s much easier to find a file name with a specific date time pattern than to search through one file for that date’s entries. Files are also much smaller: you don’t think the editor has frozen when you open a log file. Secondly, if you are sending the log file over the network to a different location or logs backup server– perhaps a nightly backup job copying to a centralized log server – it doesn’t use up your network’s bandwidth. Third and final, it helps with your log retention. If you want to check old log entries, it’s easier to delete files older than a particular date than to have an application parsing one single large file.

    For how much time should be a Retention of Log File

    That definitely comes down to business requirement. We could be asked to keep one month’s worth of logging information, or it may be a regulatory requirement to keep ten years’ worth of data. Whatever it is, logs need to go from the server at one time or other.

    In our opinion, unless otherwise required, keep at least a month’s worth of log files online, plus copy them to a secondary location like a logging server. Anything older than that can be offloaded to a separate media or server.