Unit 3 Bonus - Storage

NOTE: This is an optional bonus section. You do not need to read it, but if you're interested in digging deeper, this is for you.

When storage issues arise, troubleshooting step by step ensures a quick resolution. This guide flows logically, covering the most common issues you might face, from slow performance to filesystem corruption.

🔍 Step 1: Is Storage Performance Slow?

If everything feels sluggish, your disk might be the bottleneck.

Check:

# Monitor disk I/O, latency, and throughput
iostat -xz 1

# Identify processes consuming high I/O
pidstat -d 1

# Real-time disk activity monitoring
iostat -dx 1

• If I/O wait is high, it means the CPU is waiting on slow disk operations.
• If certain processes are consuming all disk bandwidth, they might be the cause.

Fix:

1. Identify and stop unnecessary high I/O processes:

   # Forcefully terminate a process (use with caution)
   kill -9 <PID>
   

2. Optimize filesystem writes (for ext4):

   # Enable writeback mode for better performance
   tune2fs -o journal_data_writeback /dev/sdX
   

3. Reduce excessive metadata writes:

   # Disable access time updates and set commit interval
   mount -o noatime,commit=60 /mnt/data
   

4. If using LVM, extend the volume to reduce fragmentation:

   # Add 5GB to volume
   lvextend -L +5G /dev/examplegroup/lv_data
   

🔍 Step 2: Is the Filesystem Full? ("No Space Left on Device")

👉 Disk space exhaustion is one of the most common causes of storage failures.

Check:

# Show disk usage per filesystem
df -hT

# Find the biggest files
du -ahx / | sort -rh | head -20

• If a filesystem is 100% full, it prevents writes and can cause application crashes.
• If there's space but files still won't write, check Step 4 (Corrupted Filesystem).

Fix:

1. Find and remove large unnecessary files:

   # Remove specific log file
   rm -f /var/log/large_old_log.log
   

2. Truncate logs safely without deleting them:

   # Clear log contents while preserving file
   truncate -s 0 /var/log/syslog
   
   # Limit journal size
   journalctl --vacuum-size=100M
   

3. Expand disk space if using LVM:

   # Extend logical volume
   lvextend -L +10G /dev/examplegroup/lv_data
   
   # Resize filesystem
   resize2fs /dev/examplegroup/lv_data  # for ext4
   xfs_growfs /mnt/data                 # for XFS
   

🔍 Step 3: Are Mounts Failing? (LVM, fstab, NFS, SMB)

If files suddenly disappear or applications complain about missing storage, a mount issue may be the cause.

Check:

# View current mounts
mount | grep /mnt/data

# Check block devices
lsblk

# Verify permanent mount configuration
cat /etc/fstab

Fix:

1. Manually remount the filesystem (if missing):

   # Remount all fstab entries
   mount -a
   

2. Ensure correct fstab entry for persistence:

   # Add to /etc/fstab (replace UUID with actual value)
   UUID=xxx-yyy-zzz /mnt/data ext4 defaults 0 2
   

3. If an LVM mount is missing after reboot, reactivate it:

   # Activate volume groups
   vgchange -ay
   
   # Mount the logical volume
   mount /dev/examplegroup/lv_data /mnt/data
   

4. For NFS issues, check connectivity and restart services:

   # Check NFS exports
   showmount -e <NFS_SERVER_IP>
   
   # Restart NFS service
   systemctl restart nfs-server
   

🔍 Step 4: Is the Filesystem Corrupted?

👉 Power losses, unexpected shutdowns, and failing drives can cause corruption.

Check:

# Check kernel error messages
dmesg | grep -i "error"

# Check filesystem integrity (non-destructive)
fsck.ext4 -n /dev/sdX  # for ext4
xfs_repair -n /dev/sdX  # for XFS

Fix:

1. Repair the filesystem (if unmounted):

   # Unmount first
   umount /dev/sdX
   
   # Run filesystem repair
   fsck -y /dev/sdX    # for ext4
   xfs_repair /dev/sdX  # for XFS
   

2. If corruption is severe, restore from backup:

   # Restore using rsync
   rsync -av /backup/mnt_data /mnt/data/
   

🔍 Step 5: Are You Out of Inodes?

You might have disk space but still can't create files? Check your inodes!

Check:

# Check inode usage
df -i

# Count files in current directory
find . -type f | wc -l

• If inode usage shows 100%, you can't create new files even with free space.
• This happens when you have too many small files.

Fix:

1. Clean up temporary files:

   # Remove old files in /tmp
   rm -rf /tmp/*
   
   # Clean package cache (Debian/Ubuntu)
   apt-get clean
   

2. Find and remove unnecessary files:

   # List directories with most files
   du -a | sort -n -r | head -n 10
   

Downloads