Learn › Linux for Production Support › Arjun automates the boring work

Linux for Production Support Ch 25 / 32 Intermediate

⏰

Arjun automates the boring work

crontab, systemd timers, flock — scheduling jobs that actually run reliably

⏱ 12 min 6 commands 5 takeaways

⏰

In this chapter

Arjun

Ops engineer automating 14 manual tasks

The story

Arjun had a list of 14 manual tasks he did every day and every week. Check disk space. Archive old logs. Run a database backup. Send a summary report. Restart the cache service every Sunday morning before traffic picked up.

His senior engineer said: if you do it more than once on a schedule, automate it. Arjun spent one afternoon building a cron system. He never did those 14 tasks manually again.

CRON — THE LINUX TASK SCHEDULER

cron runs commands at scheduled times. The crontab file stores the schedule.

crontab -e          # open your personal cron schedule for editing
crontab -l          # list your current cron jobs
crontab -r          # remove all your cron jobs (careful!)
sudo crontab -l     # see root's cron jobs
sudo crontab -u vijay -l    # see another user's cron jobs

CRON SYNTAX — 5 FIELDS

minute  hour  day-of-month  month  day-of-week  command
  *       *        *          *         *         /path/to/command

# Field ranges:
# minute:       0-59
# hour:         0-23
# day-of-month: 1-31
# month:        1-12
# day-of-week:  0-7 (0 and 7 are both Sunday)

# Special values:
# *   = every (every minute, every hour, etc.)
# */5 = every 5 (every 5 minutes, every 5 hours, etc.)
# 1-5 = range (Monday through Friday)
# 1,3,5 = list (Monday, Wednesday, Friday)

REAL EXAMPLES

# Every day at 2am:
0 2 * * * /usr/local/bin/backup.sh

# Every 15 minutes:
*/15 * * * * /usr/local/bin/health-check.sh

# Every Sunday at 3am:
0 3 * * 0 /usr/local/bin/weekly-restart.sh

# First day of every month at midnight:
0 0 1 * * /usr/local/bin/monthly-report.sh

# Every weekday (Monday-Friday) at 8am:
0 8 * * 1-5 /usr/local/bin/morning-report.sh

# Every hour between 9am and 6pm on weekdays:
0 9-18 * * 1-5 /usr/local/bin/hourly-check.sh

# Use https://crontab.guru to visually build cron expressions

PRODUCTION CRON BEST PRACTICES

Always use absolute paths in cron — cron has a minimal environment:

# Wrong: assumes PATH is set correctly:
*/5 * * * * df -h | mail -s "Disk" admin@company.com

# Right: full paths everywhere:
*/5 * * * * /bin/df -h | /usr/bin/mail -s "Disk" admin@company.com
# Or set PATH at top of crontab:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Always log cron output:

# Redirect output to log file:
0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
# 2>&1 captures both stdout and stderr

# Discard output (if you don't want emails):
*/5 * * * * /usr/local/bin/health-check.sh > /dev/null 2>&1

Prevent overlapping runs for long tasks:

*/5 * * * * flock -n /tmp/mylock.lock /usr/local/bin/slow-task.sh
# flock -n skips if already running — no overlapping processes

SYSTEMD TIMERS — THE MODERN ALTERNATIVE TO CRON

systemd timers are more powerful: they log to journalctl, handle missed runs, and have better dependency management.

# Create a service for the job:
sudo nano /etc/systemd/system/backup.service
[Unit]
Description=Daily database backup

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup.sh
StandardOutput=journal
StandardError=journal

# Create the timer:
sudo nano /etc/systemd/system/backup.timer
[Unit]
Description=Run backup daily at 2am

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true     # run immediately if missed (e.g. server was off at 2am)

[Install]
WantedBy=timers.target

# Enable and start:
sudo systemctl daemon-reload
sudo systemctl enable backup.timer
sudo systemctl start backup.timer

# Check status:
systemctl list-timers                   # all timers and when they last/next ran
journalctl -u backup.service            # logs from the backup job

ARJUN'S 14 AUTOMATED TASKS

# /etc/cron.d/ops-tasks (system-wide cron, runs as specified user)

# Disk check every 15 minutes:
*/15 * * * * root /usr/local/bin/disk-check.sh >> /var/log/ops/disk.log 2>&1

# Log archive every night at 1am:
0 1 * * * root find /opt/app/logs -name "*.log" -mtime +7 | xargs gzip -9 >> /var/log/ops/archive.log 2>&1

# Database backup every day at 2am:
0 2 * * * postgres pg_dump appdb | gzip > /backups/appdb_$(date +\%Y\%m\%d).sql.gz

# Delete backups older than 30 days every Sunday at 4am:
0 4 * * 0 root find /backups -name "*.gz" -mtime +30 -delete

# Weekly cache restart Sunday 3am:
0 3 * * 0 root systemctl restart redis && echo "$(date): Redis restarted" >> /var/log/ops/restart.log

# Monthly disk report first day of month:
0 8 1 * * root df -h | mail -s "Monthly disk report $(date +\%B)" admin@company.com

All 14 tasks automated. Arjun uses that time to build better systems instead.

Key takeaways

Cron syntax is: minute hour day month weekday command — use crontab.guru to build expressions visually

Always use absolute paths in cron — cron runs with a minimal PATH that may not include your usual commands

Always redirect output: >> /var/log/myjob.log 2>&1 — cron runs silently and errors disappear otherwise

flock -n /tmp/lock.lock command prevents overlapping runs — use for slow jobs that run frequently

systemd timers are better for important jobs — they log to journalctl and handle missed runs automatically

Commands from this chapter

$ crontab -e

Open cron schedule editor — add or modify scheduled jobs

$ crontab -l

List all current cron jobs for your user

$ 0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Run backup every day at 2am with logging

$ flock -n /tmp/job.lock /usr/local/bin/slow-job.sh

Run job only if not already running

$ systemctl list-timers

List all systemd timers and when they last and next ran

$ journalctl -u myservice.service --since today

View logs from a systemd timer job