Essential Linux Administration & Bash Automation

This covers the defensive bash habits, set -euo pipefail, log rotation, health checks, and error trapping, that keep an unattended script or scheduled job from failing silently.

set -euo pipefail and why it matters

Every bash script I write starts with this line, right after the shebang:

#!/usr/bin/env bash
set -euo pipefail

Each flag closes a real hole:

• -e exits the script the instant a command fails. Without it, a failed cd /app/releases/current still lets the next line run in the wrong directory.
• -u treats an unset variable as an error. This catches the classic typo ($RELEASE_DIR misspelled as $RELEASE_DR) before it silently expands to an empty string and turns rm -rf "$RELEASE_DR/old" into rm -rf /old.
• pipefail makes a pipeline fail if any command in it fails, rather than only checking the last one's exit code. cat missing_file.txt | grep foo will otherwise report success because grep exited fine.

A log-rotation script that doesn't take down the disk with it

The other recurring incident: an app (or a script) logs to a flat file forever, nobody notices, and one day /var/log, or worse, /, hits 100% and takes the whole host down with it. logrotate handles this for most system services, but plenty of app logs, cron output, and custom job logs never get wired into it. This is the script I drop in as a stopgap:

#!/usr/bin/env bash
set -euo pipefail
 
LOG_DIR="/var/log/myapp"
RETENTION_DAYS=14
 
if [[ ! -d "$LOG_DIR" ]]; then
  echo "ERROR: log directory $LOG_DIR does not exist" >&2
  exit 1
fi
 
# Compress logs older than 1 day that aren't already compressed
find "$LOG_DIR" -maxdepth 1 -type f -name "*.log" -mtime +1 -exec gzip {} \;
 
# Delete compressed logs past retention
find "$LOG_DIR" -maxdepth 1 -type f -name "*.log.gz" -mtime "+${RETENTION_DAYS}" -print -delete
 
echo "Log cleanup complete: $(date -Iseconds)"

A health-check script you can actually schedule

For anything client-facing, I want something running on a schedule that hits the real endpoint, rather than only checking systemctl status:

#!/usr/bin/env bash
set -euo pipefail
 
URL="https://api.example.com/healthz"
TIMEOUT=5
EXPECTED_CODE=200
 
status_code=$(curl --silent --output /dev/null --write-out "%{http_code}" \
  --max-time "$TIMEOUT" "$URL" || echo "000")
 
if [[ "$status_code" -ne "$EXPECTED_CODE" ]]; then
  echo "HEALTH CHECK FAILED: $URL returned $status_code" >&2
  exit 1
fi
 
echo "OK: $URL returned $status_code"
exit 0

What matters here is the exit code: it's what lets cron, systemd, or a monitoring agent (Nagios, Prometheus's blackbox exporter) decide whether to page someone.

Cron vs systemd timers

For simple, single-host schedules, cron is still fine:

*/5 * * * * /usr/local/bin/healthcheck.sh >> /var/log/healthcheck.log 2>&1

I still see this line shipped without the >> ... 2>&1 half, which means any output, including the error message you need, vanishes.

But once a job needs dependency ordering, retries, or resource limits, I reach for a systemd timer instead. Same if you just want its logs in journalctl rather than a flat file you have to rotate yourself:

# /etc/systemd/system/healthcheck.service
[Unit]
Description=API health check
 
[Service]
Type=oneshot
ExecStart=/usr/local/bin/healthcheck.sh

# /etc/systemd/system/healthcheck.timer
[Unit]
Description=Run healthcheck.service every 5 minutes
 
[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
Persistent=true
 
[Install]
WantedBy=timers.target

Enable it with systemctl enable --now healthcheck.timer. From then on, journalctl -u healthcheck.service gives you every run, its output, and its exit code, and it survives a reboot without a stray @reboot cron entry.

Logging and trapping errors so failures don't stay silent

The last habit is what keeps a failure from staying silent for weeks: make failure loud on purpose.

#!/usr/bin/env bash
set -euo pipefail
 
LOG_FILE="/var/log/myapp/etl.log"
 
log() {
  echo "$(date -Iseconds) $*" | tee -a "$LOG_FILE"
}
 
on_error() {
  local exit_code=$?
  local line_no=$1
  log "ERROR: script failed at line $line_no with exit code $exit_code"
  # send a Slack/PagerDuty alert here instead of hoping someone reads the log
  exit "$exit_code"
}
 
trap 'on_error $LINENO' ERR
 
log "Starting ETL run"
# ... actual work happens here ...
log "ETL run completed successfully"

trap ... ERR runs whenever a command fails (with -e active), and $LINENO tells you exactly where. That one addition turns "the job stopped somewhere three weeks ago" into "the job failed at line 42 at 03:14 this morning, and here's the alert that fired the moment it happened."

They're six habits, applied every time. The difference is whether you hear about a failure from a monitoring alert within minutes, or from someone else weeks later.