Learn › Linux for Production Support › Deepak runs things in parallel

Linux for Production Support Ch 17 / 32 Expert

⚡

Deepak runs things in parallel

xargs, parallel execution, advanced scripting — automate work across many servers at once

⏱ 13 min 5 commands 5 takeaways

⚡

In this chapter

Deepak

Platform engineer, growing SaaS

The story

Deepak inherited a 200-server infrastructure. Every month, the team manually updated SSL certificates on all 200 servers. It took two engineers an entire day, involved SSH-ing into each server one by one, and had broken production twice due to human error.

He automated it in one weekend.

Sequential approach - too slow:

for server in server-{001..200}; do
    ssh $server "certbot renew --quiet"
done
# 200 servers x 30 seconds each = 100 minutes

Parallel with xargs - 20 servers at once:

printf '%s

' server-{001..200} | xargs -P 20 -I {} ssh {} "certbot renew --quiet"

# -P 20 means 20 parallel jobs
# -I {} is the placeholder for each server name
# Same 200 servers now completes in under 10 minutes

GNU parallel gives more control:

parallel -j 20 --tag --line-buffer         'ssh {} "certbot renew --quiet"'         ::: server-{001..200}
# --tag prefixes each output line with the server name
# --line-buffer prevents output from different servers mixing

Error handling in parallel scripts:

#!/bin/bash
set -euo pipefail

LOG_FILE="/var/log/cert_renewal_$(date +%Y%m%d).log"
FAILED=()

renew_cert() {
    local server=$1
    if ssh -o ConnectTimeout=10 "$server" "certbot renew --quiet" 2>&1; then
        echo "$(date +%H:%M:%S) OK: $server" | tee -a "$LOG_FILE"
        return 0
    else
        echo "$(date +%H:%M:%S) FAIL: $server" | tee -a "$LOG_FILE"
        return 1
    fi
}
export -f renew_cert
export LOG_FILE

while IFS= read -r server; do
    renew_cert "$server" || FAILED+=("$server")
done < <(cat servers.txt)

if [ ${#FAILED[@]} -gt 0 ]; then
    printf '%s

' "${FAILED[@]}" | mail -s "Cert renewal failures" ops@company.com

    exit 1
fi

The export -f function_name line is critical. It exports the bash function to subprocesses so that xargs and parallel can call it.

Process substitution avoids temporary files:

# Diff the output of two commands without creating temp files
diff <(ls /opt/app/v1/) <(ls /opt/app/v2/)

# Safe loop over command output
while IFS= read -r line; do
    process "$line"
done < <(command_that_produces_output)

Heredoc embeds multi-line content in scripts:

cat > /etc/nginx/sites-available/myapp << 'HEREDOC'
server {
    listen 80;
    server_name api.company.com;
    location / {
        proxy_pass http://localhost:8080;
    }
}
HEREDOC

nginx -t && systemctl reload nginx

Complete cert renewal script with AWS integration:

#!/bin/bash
set -euo pipefail

SERVERS=$(aws ec2 describe-instances         --filters "Name=tag:Environment,Values=production"         --query 'Reservations[*].Instances[*].PrivateIpAddress'         --output text)

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a /var/log/cert_renewal.log; }

renew_server() {
    local ip=$1
    if timeout 60 ssh -o StrictHostKeyChecking=no "ec2-user@$ip"             "sudo certbot renew --quiet" 2>&1; then
        echo "OK:$ip"
    else
        echo "FAIL:$ip"
    fi
}
export -f renew_server

RESULTS=$(echo "$SERVERS" | xargs -P 20 -I {} bash -c 'renew_server "$@"' _ {})
FAILED=$(echo "$RESULTS" | grep "^FAIL:" | cut -d: -f2)

if [ -n "$FAILED" ]; then
    log "FAILED: $FAILED"
    exit 1
fi
log "All certs renewed successfully"

Deepak runs this from cron monthly. Two engineers get back an entire workday. Zero human errors.

Key takeaways

xargs -P 20 runs 20 parallel jobs — turning a 100-minute sequential job into a 10-minute parallel one

export -f function_name makes a bash function available to subprocesses — required for xargs and parallel

Process substitution with <() avoids creating temporary files when comparing command outputs

Heredoc (cat > file << HEREDOC) embeds multi-line config files directly in scripts

Always collect failures in parallel scripts and report them — partial failure is worse than total failure

Commands from this chapter

$ cat servers.txt | xargs -P 20 -I {} ssh {} 'certbot renew --quiet'

Renew SSL certs on 20 servers simultaneously

$ parallel -j 20 --tag 'ssh {} uptime' ::: server-{001..100}

Run a command on 100 servers in parallel with labeled output

$ diff <(ls dir1/) <(ls dir2/)

Diff output of two commands without temp files

$ while IFS= read -r line; do echo $line; done < <(command)

Safe loop over command output

$ export -f my_function

Export bash function for use in subprocesses and xargs