Log Files and System Monitoring

In this lesson, we'll explore how Linux systems keep track of what's happening through log files and how to monitor system performance. Understanding logs and monitoring is crucial for troubleshooting issues, maintaining system health, and ensuring your Ubuntu server runs smoothly.

Learning Goals:

• Understand the Linux logging system and common log file locations
• Use essential tools to read and analyze log files
• Monitor system resources (CPU, memory, disk, network)
• Set up basic log rotation and management
• Use monitoring tools to identify performance bottlenecks

Understanding Linux Logging System

Linux uses a centralized logging system that collects messages from the kernel, system services, and applications. Most modern Ubuntu systems use systemd-journald for system logs and rsyslog for persistent logging.

Common Log File Locations

Most system logs are stored in /var/log/ directory. Here are the key log files:

Exploring /var/log directory

ls -la /var/log/

Key log files include:

• /var/log/syslog - General system activity logs
• /var/log/auth.log - Authentication and authorization logs
• /var/log/kern.log - Kernel messages and warnings
• /var/log/dpkg.log - Package management activities
• /var/log/apache2/ - Web server logs (if installed)
• /var/log/mysql/ - Database logs (if installed)

Reading and Analyzing Log Files

Using tail, head, and grep

The most common tools for log analysis are tail, head, and grep:

Monitoring logs in real-time

# Watch system logs in real-time
tail -f /var/log/syslog

# Show last 50 lines of auth log
tail -n 50 /var/log/auth.log

# Search for specific terms in logs
grep "error" /var/log/syslog
grep -i "failed" /var/log/auth.log

Using journalctl (systemd systems)

For systems using systemd, journalctl provides powerful log querying capabilities:

Using journalctl for log analysis

# View all logs from current boot
journalctl -b

# Follow new log entries in real-time
journalctl -f

# Show logs for a specific service
journalctl -u ssh

# Filter logs by time range
journalctl --since "2024-01-15 09:00:00" --until "2024-01-15 17:00:00"

# Show error priority messages only
journalctl -p err

tip

Combine journalctl with grep for more specific searches: journalctl -u nginx | grep "error"

System Monitoring Tools

Monitoring System Resources

Using top and htop

Monitoring processes with top and htop

# Basic process monitoring
top

# Install and use htop (more user-friendly)
sudo apt update && sudo apt install htop
htop

Using vmstat and iostat

System performance monitoring

# Virtual memory statistics
vmstat 1 5  # Update every 1 second, 5 times

# I/O statistics (install sysstat first)
sudo apt install sysstat
iostat -x 1 3  # Extended stats, every 1 second, 3 reports

Disk Space Monitoring

Checking disk usage

# Check disk space
df -h

# Check inode usage
df -i

# Find large files and directories
du -sh /var/log/*
du -h --max-depth=1 /home | sort -hr

Network Monitoring

Network monitoring tools

# Real-time network traffic
sudo apt install nethogs
sudo nethogs

# Network statistics
netstat -tulpn
ss -tulpn

# Monitor network connections in real-time
sudo apt install iftop
sudo iftop

Log Rotation and Management

Linux systems automatically rotate logs to prevent them from consuming all disk space. The logrotate service handles this:

Understanding logrotate configuration

# View logrotate configuration
cat /etc/logrotate.conf

# Check service-specific logrotate configurations
ls /etc/logrotate.d/

# Manually run logrotate to test configuration
sudo logrotate -d /etc/logrotate.conf  # Dry run
sudo logrotate -f /etc/logrotate.conf  # Force rotation

Example of a custom logrotate configuration:

/etc/logrotate.d/myapp

/var/log/myapp/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 644 myuser mygroup
    postrotate
        /usr/bin/systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

Setting Up Basic Monitoring

Creating a Simple Monitoring Script

Let's create a basic shell script to monitor system health:

monitor-system.sh

#!/bin/bash

# System monitoring script
echo "=== System Health Check ==="
echo "Timestamp: $(date)"
echo

# CPU Load
echo "CPU Load:"
uptime
echo

# Memory Usage
echo "Memory Usage:"
free -h
echo

# Disk Space
echo "Disk Space:"
df -h / /home /var
echo

# Top 5 processes by CPU
echo "Top 5 CPU Processes:"
ps aux --sort=-%cpu | head -6
echo

# Check critical services
echo "Service Status:"
systemctl is-active ssh > /dev/null && echo "✓ SSH: Active" || echo "✗ SSH: Inactive"
systemctl is-active cron > /dev/null && echo "✓ Cron: Active" || echo "✗ Cron: Inactive"

Make it executable and run it:

chmod +x monitor-system.sh
./monitor-system.sh

warning

Be careful when running monitoring scripts in production. Continuous monitoring can impact system performance if not properly configured.

Common Pitfalls

• Ignoring log rotation: Logs can fill up your disk space if rotation isn't properly configured
• Not monitoring disk inodes: Even with free space, a system can fail if inodes are exhausted
• Overlooking authentication logs: Failed login attempts can indicate security breaches
• Missing service-specific logs: Applications often have their own log directories outside /var/log
• Not setting up log retention policies: Keeping logs forever wastes space, deleting them too soon loses troubleshooting data

Summary

In this lesson, you've learned how to work with Linux log files and monitor system performance. You can now:

• Navigate and analyze system logs using tail, grep, and journalctl
• Monitor system resources with tools like top, htop, and vmstat
• Manage log rotation to prevent disk space issues
• Create basic monitoring scripts to track system health
• Identify and troubleshoot common system issues through log analysis

Effective log management and system monitoring are essential skills for maintaining healthy Linux systems and quickly resolving issues when they occur.

Quiz

Show quiz

1. Which command would you use to monitor new entries in the system log in real-time? a) head -f /var/log/syslog b) tail -f /var/log/syslog c) watch /var/log/syslog d) live /var/log/syslog

2. What is the primary purpose of the logrotate service? a) To encrypt log files for security b) To compress and archive old logs to prevent disk space exhaustion c) To send logs to a remote server d) To analyze log patterns for anomalies

3. Which directory typically contains most system log files in Ubuntu? a) /etc/log/ b) /usr/log/ c) /var/log/ d) /tmp/log/

4. What does the command journalctl -u ssh display? a) All system logs except SSH-related messages b) Only error messages from the SSH service c) Log entries specifically for the SSH service d) A summary of SSH connection statistics

5. Which tool provides a more user-friendly alternative to the top command? a) vmstat b) iostat c) htop d) netstat

---

Answers:

1. b) tail -f /var/log/syslog
2. b) To compress and archive old logs to prevent disk space exhaustion
3. c) /var/log/
4. c) Log entries specifically for the SSH service
5. c) htop