File-system issues can cause a Linux system, such as Red Hat Enterprise Linux, to fail during boot, leading to system downtime and operational disruption. This guide provides step-by-step instructions on detecting, troubleshooting, and repairing common file-system problems encountered during boot.

Detecting File-system Issues

During the boot process, if the system detects file-system corruption or inconsistencies, it typically initiates an automatic file-system check (fsck). If the file-system check fails or encounters errors that it cannot fix automatically, the boot process may halt, requiring manual intervention.

Signs of file-system issues include:

• Errors displayed during boot such as “unexpected inconsistency; run fsck manually.”
• System enters emergency mode or maintenance mode.

Accessing Emergency or Rescue Mode

When the system halts due to file-system issues, it may enter emergency mode automatically. If it does not:

1. Restart your system.
2. Interrupt the boot loader (GRUB) by pressing e at the GRUB menu.
3. Append the keyword emergency or rescue at the end of the kernel command line, then boot by pressing Ctrl + X.

Manually Running File-system Checks

After booting into emergency mode:

1. Identify the problematic file system:

   blkid
   lsblk

2. Run a file-system check manually:

   fsck -y /dev/<partition-name>

• -y automatically answers “yes” to fix issues detected.

1. Wait for the process to complete. If issues persist, you might need to run fsck multiple times.

Mounting the File System

If the file-system check resolves all issues:

1. Mount the file system manually to verify it can be mounted:

   mount /dev/<partition-name> /mnt

2. Check that the file system is accessible and files are intact:

   ls /mnt

Updating the Boot Configuration

If the file system is fixed and mounted successfully:

• Ensure the /etc/fstab file correctly references file-system devices:

  vi /mnt/etc/fstab

• Ensure no incorrect or outdated UUIDs or device names are listed.

Rebooting the System

After successful repair and verification:

umount /mnt
reboot

Preventive Maintenance

• Regularly monitor logs (/var/log/messages) and journal (journalctl -xb) for early indications of file-system problems.
• Schedule periodic file-system checks and backups.
  
  

Repairing from a GRUB > Prompt

If your system drops you into a grub> or grub rescue> prompt, it typically means GRUB can’t find its configuration or boot files. This guide walks you through identifying your Linux installation and booting manually.

Find your Hard drive
ls
Output Example:
(hd0) (hd0,msdos2) (hd0,msdos1)

List Root partition on each drive til you find your Install
ls (hd0,msdos2)/
Check for the directory listing, if nothing, move to next

Now that we have found the proper drive lets boot to it

grub> set root=(hd0,msdos2)
grub> linux /boot/vmlinu (tab complete) root=/dev/sda1 (or the root linux partition)
grub> initrd /boot/initrd (tab complete)
grub> boot

Repairing from a GRUB Rescue > Prompt

Find your Hard drive

ls
Output Example:
(hd0) (hd0,msdos2) (hd0,msdos1)

List Root partition on each drive til you find your Install
ls (hd0,msdos2)/
Check for the directory listing, if nothing, move to next

Now that we have found the proper drive lets boot to it

grub rescue> set prefix=(hd0,msdos2)/boot/grub
grub rescue> set root=(hd0,msdos2)
grub rescue> insmod normal
grub rescue> normal
grub rescue> insmod linux
grub rescue> linux /boot/vmlinu (tab complete) root=/dev/sda1 (or the root linux partition)
grub rescue> initrd /boot/initrd (tab complete)
grub rescue> boot

Fix GRUB Permanently

Now that we are booted in from GRUB Rescue, we can begin work with repairing our grub permanently. First we rebuild the /boot/grub/grub.cfg file:

Debian-based Distributions use update-grub
Other Distributions use grub-mkconfig -o /boot/grub/grub.cfg

With the Configuration rebuilt, we now simply need to reinstall grub

grub-install /dev/sda

Note: /dev/sda is the DEVICE…NOT THE PARTITION.

Thoughts

Proactively monitoring and repairing file-system issues promptly ensures system reliability and minimizes downtime. Effective troubleshooting and consistent preventive maintenance practices are essential for maintaining a healthy Linux environment.