Prompt.cz

A Practical Guide to Linux

User Tools

Site Tools


troubleshooting

Troubleshooting


1. Fixing boot failure (RHEL/CentOS 7, 8)

1.1. Damaged master boot record (MBR) or missing files in the /boot/efi partition

1. Boot from an installation source (live CD, installation DVD or network).

2. When the boot loader menu appears, select the "rescue" entry using the up/down arrow keys.

3. Select option 1 (Continue) and press "Enter" to mount the system under "/mnt/sysimage".

4. Press "Enter" to open a shell.

5. Change the root directory to "/mnt/sysimage":

sh-4.4# chroot /mnt/sysimage


6. Reinstall the GRUB 2 boot loader to the MBR of the boot device:

sh-4.4# grub2-install /dev/sda

or reinstall the "grub2-efi" and "shim" packages:

sh-4.4# yum reinstall grub2-efi shim


7. Exit from both the chroot and rescue shell:

sh-4.4# exit
sh-4.4# exit

(The system reboots, performs a full SELinux relabel, and then reboots again.)

1.2. Wrong entries in the GRUB 2 configuration file

A. Using rescue mode

1. Reboot the system.

2. When the boot loader menu appears, select the "rescue" entry using the up/down arrow keys.

3. Log in to the rescue mode.

4. Create a new GRUB 2 configuration file:

# grub2-mkconfig -o /boot/grub2/grub.cfg

(on a system with the BIOS firmware)
or

# grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg

(on a system with the UEFI firmware)

5. Reboot the system:

# reboot


B. Using GRUB 2 shell

1. Reboot the system.

2. When the boot loader menu appears, press any key to interrupt the boot process, except "Enter".

3. Select the kernel using the up/down arrow keys.

4. Press the "c" key to enter the GRUB 2 shell.

5. Load the XFS and LVM modules:

grub> insmod xfs
grub> insmod lvm


6. List available devices:

grub> ls
(lvm/vg00-home_lv) (lvm/vg00-tmp_lv) (lvm/vg00-var_lv) (lvm/vg00-usr_lv) (lvm/v
g00-root_lv) (lvm/vg00-swap_lv) (hd0) (hd0,msdos2) (hd0,msdos1)


7. Display information about the first partition on the first disk:

grub> ls (hd0,msdos1)
        Partition hd0,msdos1: Filesystem type xfs, UUID
a5eeb5e9-9cd4-437c-b8bb-3d928a0daf5e - Partition start at 1024KiB - Total size
524288KiB


8. List the contents of the potential boot partition:

grub> ls (hd0,msdos1)/
efi/ grub2/ grub/ System.map-4.18.0-305.7.1.el8_4.x86_64 config-4.18.0-305.7.1.
el8_4.x86_64 vmlinuz-4.18.0-305.7.1.el8_4.x86_64 initramfs-0-rescue-73a3b006041
74b4f9843524821768b43.img vmlinuz-0-rescue-73a3b00604174b4f9843524821768b43 ini
tramfs-4.18.0-305.7.1.el8_4.x86_64.img initramfs-4.18.0-305.7.1.el8_4.x86_64kdu
mp.img


9. Set the boot partition:

grub> set root=(hd0,msdos1)


10. Load the kernel and specify the root file system:

grub> linux (hd0,msdos1)/vmlinuz-4.18.0-305.7.1.el8_4.x86_64 root=/dev/mapper/v

g00-root_lv

(Use the "Tab" key for the path completion to the kernel.)

11. Load the initial RAM disk:

grub> initrd (hd0,msdos1)/initramfs-4.18.0-305.7.1.el8_4.x86_64.img

(Use the "Tab" key for the path completion to the initramfs image.)

12. Boot the system:

grub> boot


13. Create a new GRUB 2 configuration file:

# grub2-mkconfig -o /boot/grub2/grub.cfg

(on a system with the BIOS firmware)
or

# grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg

(on a system with the UEFI firmware)

14. Reboot the system:

# reboot


1.3. Wrong entries in /etc/fstab

1. Reboot the system.

2. When the boot loader menu appears, press any key to interrupt the boot process, except "Enter".

3. Select the kernel using the up/down arrow keys.

4. Press the "e" key to edit the entry.

5. Use the cursor keys to navigate to the line that starts with "linux", "linux16" or "linuxefi".

6. Move the cursor to the end of the line by pressing the "End" key.

7. Append "systemd.unit=emergency.target" to boot into the emergency mode.

8. Press Ctrl+x to boot the system using the modified configuration.

9. Log in to the emergency mode.

10. Check which file systems are currently mounted:

# mount


11. Remount the root file system in read-write mode:

# mount -o remount,rw /


12. Mount all the file systems listed in /etc/fstab:

# mount -a


13. Edit /etc/fstab in case of errors (non-existent UUID / mount point or incorrect mount options):

# vi /etc/fstab


14. Mount again all the file systems to verify that /etc/fstab is correct:

# mount -a


15. Reboot the system:

# reboot

2. Fixing root login failure

2.1. RHEL/CentOS 7, 8

1. Reboot the system.

2. When the boot loader menu appears, press any key to interrupt the boot process, except "Enter".

3. Select the kernel using the up/down arrow keys.

4. Press the "e" key to edit the entry.

5. Use the cursor keys to navigate to the line that starts with "linux", "linux16" or "linuxefi".

6. Move the cursor to the end of the line by pressing the "End" key.

7. Append "rd.break" to break the boot process and get a shell prompt.

8. Press Ctrl+x to boot the system using the modified configuration.

9. After a shell from initramfs starts, remount the "/sysroot" file system in read-write mode:

switch_root:/# mount -o remount,rw /sysroot


10. Change the root directory to "/sysroot" and run a new shell:

switch_root:/# chroot /sysroot


11. Reset the root password:

sh-4.4# passwd


12. Enable the SELinux relabeling process on the next system boot (if SELinux is used):

sh-4.4# touch /.autorelabel


13. Exit from the shell:

sh-4.4# exit
switch_root:/# exit

(The system continues booting, performs a full SELinux relabel, and then reboots again.)

2.2. RHEL/CentOS 4, 5, 6

1. Reboot the system.

2. When the boot loader menu appears, press any key to interrupt the boot process, except "Enter".

3. Select the kernel using the up/down arrow keys.

4. Press the "e" key to edit the entry.

5. Use the cursor keys to navigate to the line that starts with "kernel".

6. Press again the "e" key to edit the entry.

7. Append "s" or "single" at the end of the line.

8. Press "Enter" to accept the changes and then press "b" to boot the system using the modified configuration.

9. Reset the root password.

10. Exit from the shell.


3. Fixing software installation failure


"Kernel panic"

=>

# yum reinstall kernel


"Yum Error: database disk image is malformed"

=>

# yum clean dbcache

or

# yum clean all


"Depsolving loop limit reached" / "<package_name-versionX> is a duplicate with <package_name-versionY>"

(duplicate packages from a failed yum transaction)
=>

# yum-complete-transaction

or

# package-cleanup --cleandupes
# yum update

or

# tar -cvzf /tmp/rpm_db.tar.gz /var/lib/{rpm,yum}
# yum check duplicates 2> /dev/null | awk '/duplicate/ {print $NF}' | xargs rpm -e --justdb --nodeps
# yum update


"error: rpmdbNextIterator: skipping h#..."

(corrupted RPM database)
=>

# lsof | grep /var/lib/rpm
# rm -f /var/lib/rpm/__db*
# tar -cvzf rpmdb.tar.gz /var/lib/rpm
# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
# mv /var/lib/rpm/Packages{,.broken}
# /usr/lib/rpm/rpmdb_dump /var/lib/rpm/Packages.broken | /usr/lib/rpm/rpmdb_load /var/lib/rpm/Packages
# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
# rm -f /var/lib/rpm/Packages.broken
# rpm --rebuilddb
# rpm -qa > /dev/null

4. Fixing file execution failure


"No execute permissions for /bin/chmod"

=>

# setfacl -m u::rwx,g::rx,o::rx /bin/chmod

or

# cp /bin/chmod /bin/chmod.broken
# install -m 0755 /bin/chmod.broken /bin/chmod
# rm -f /bin/chmod.broken

or

# perl -e 'chmod 0755, "/bin/chmod"'

or

# /lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod

or

# rpm -qf /bin/chmod
coreutils-8.30-8.el8.x86_64
# yum reinstall coreutils

or

# cp /bin/ls /bin/ls.old && cat /bin/chmod > /bin/ls.old && /bin/ls.old 755 /bin/chmod && rm -f /bin/ls.old

or copy the file from another server:

# scp <user>@<IP>:/bin/chmod /tmp
# mv /tmp/chmod /bin
# chown root:root /bin/chmod