Learn › Linux for Production Support › Nandini extends the disk at 3am

Linux for Production Support Ch 26 / 32 Advanced

💿

Nandini extends the disk at 3am

LVM, partitions, swap, filesystem resize — adding space without downtime

⏱ 13 min 6 commands 5 takeaways

💿

In this chapter

Nandini

SRE, 3am disk emergency

The story

Nandini got a 3am alert: root partition at 97%. The app was still running but only because it was writing logs to a separate mount. She had 20 minutes before the root partition hit 100% and everything stopped.

She needed to add disk space without rebooting, without losing data, and without the app going down.

This is the moment you need to understand LVM.

DISK PARTITIONS — THE BASICS

Linux organises storage as:

Physical disks (sda, sdb, nvme0n1) → Partitions (sda1, sda2) → Filesystems (ext4, xfs)

lsblk                           # tree view of all disks and partitions
fdisk -l                        # detailed partition info (needs sudo)
df -h                           # filesystem usage (mounted partitions)
du -sh /*                       # disk usage by directory

Understanding lsblk output:

NAME        SIZE  TYPE  MOUNTPOINT
sda          50G  disk
+-sda1        1G  part  /boot
+-sda2       49G  part
  +-vg0-root 30G  lvm   /
  +-vg0-var  19G  lvm   /var
sdb         200G  disk
+-sdb1      200G  part

LVM — LOGICAL VOLUME MANAGER

LVM adds a layer of abstraction between physical disks and filesystems. This means you can:

- Resize partitions without unmounting them

- Add a new disk and extend an existing partition into it

- Take snapshots for backups

The three layers:

1. Physical Volume (PV) — a disk or partition given to LVM

2. Volume Group (VG) — a pool of storage from one or more PVs

3. Logical Volume (LV) — a virtual partition carved from the VG

pvs                             # list physical volumes
vgs                             # list volume groups
lvs                             # list logical volumes
lsblk                           # see the full picture

NANDINI'S 3AM FIX: ADD A DISK AND EXTEND ROOT

Step 1: The cloud provider added a new 100GB disk. It appeared as /dev/sdb.

lsblk                           # confirm sdb is there
fdisk -l /dev/sdb               # see the new disk

Step 2: Create a physical volume on the new disk.

sudo pvcreate /dev/sdb

Step 3: Extend the existing volume group to include the new disk.

sudo vgextend vg0 /dev/sdb
vgs                             # confirm VG now shows more free space

Step 4: Extend the logical volume for root.

sudo lvextend -L +90G /dev/vg0/root   # add 90GB
# or use all free space:
sudo lvextend -l +100%FREE /dev/vg0/root

Step 5: Resize the filesystem (online — no unmount needed).

sudo resize2fs /dev/vg0/root    # for ext4
sudo xfs_growfs /               # for xfs

df -h /                         # confirm new size

Total time: 4 minutes. No reboot. No downtime. Root partition went from 97% to 22%.

CREATING A NEW PARTITION FROM SCRATCH

Adding a new data disk (not LVM — simpler for separate data partitions):

# New disk appeared as /dev/sdb:
sudo fdisk /dev/sdb
# Inside fdisk: n (new), p (primary), 1 (partition 1), Enter, Enter, w (write)

# Format it:
sudo mkfs.ext4 /dev/sdb1

# Create mount point:
sudo mkdir /data

# Mount it:
sudo mount /dev/sdb1 /data

# Make it permanent (auto-mount on boot):
# Get the UUID:
sudo blkid /dev/sdb1

# Add to /etc/fstab:
sudo nano /etc/fstab
# Add line: UUID=your-uuid-here /data ext4 defaults 0 2

# Test fstab entry without rebooting:
sudo mount -a
df -h /data             # confirm it mounted

SWAP SPACE — LINUX VIRTUAL MEMORY

Swap is Linux's page file — disk space used as overflow memory when RAM is full.

swapon --show           # is swap enabled? how much?
free -h                 # shows swap usage
cat /proc/sys/vm/swappiness    # how aggressively Linux uses swap (10 is good for servers)

# Add a swap file (without a dedicated partition):
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Make it permanent in /etc/fstab:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

CHECKING DISK HEALTH

sudo smartctl -a /dev/sda           # SMART health data (install: sudo apt install smartmontools)
sudo smartctl -t short /dev/sda     # run a quick self-test
sudo badblocks -v /dev/sdb          # scan for bad sectors (use on new disks before formatting)

# Filesystem check (must be unmounted or on boot):
sudo fsck /dev/sdb1

Nandini added a checklist to her runbook: before any server gets 85% full, add a disk and extend. Never let it hit 97% again. LVM makes it a 4-minute job.

Key takeaways

LVM lets you extend a partition while it is running — no unmounting, no downtime, takes 4 minutes

lsblk shows the full disk tree: physical disks, partitions, LVM volumes, and mount points

Adding disk to LVM: pvcreate disk, vgextend group disk, lvextend volume, resize2fs — four commands

UUID in /etc/fstab instead of device name — UUIDs do not change when disks are reordered on reboot

sudo mount -a tests your fstab changes without rebooting — always test before restarting

Commands from this chapter

$ lsblk

See all disks, partitions, and mount points as a tree

$ sudo pvcreate /dev/sdb

Initialise a new disk as an LVM physical volume

$ sudo vgextend vg0 /dev/sdb

Add a new disk to an existing volume group

$ sudo lvextend -l +100%FREE /dev/vg0/root && sudo resize2fs /dev/vg0/root

Extend root volume using all free space and resize filesystem

$ sudo blkid /dev/sdb1

Get UUID for /etc/fstab entry

$ sudo mount -a

Test /etc/fstab changes without rebooting