How to fix file system errors

How to fix file system errors

File system errors can occur when a unit is not rebooted or shutdown properly.  For example when there is a power loss while the system is actively writing to files.  Nodegrid OS v5.0.9 and newer run 'fsck -p' on every reboot and can fix minor errors without manual intervention.  Nodegrid OS v6.0.26 and newer run 'fsck -fy' on every reboot and prior to applying an upgrade.  It is strongly recommended to upgrade to the latest Nodegrid OS version available to prevent potential file system errors and have the automated repair ability.  There are two situations where a manual fsck is required:
  1. If the unit with file system errors is on an older version
  2. If the errors are not cleared after a proper reboot.

Check the logs for file system errors:

  1. Log into the admin CLI
  2. Access the root shell using: shell sudo su -
  3. Search dmesg for disk related lines using: dmesg | grep -i sda
  4. Note: Not all lines reflect errors.  If there is doubt, please send this output to support@zpesystems.com for review.

Manual file system check:

  1. Reboot the system
  2. Press Esc after the PXE boot and before the normal GRUB menu appears 
    1. The time window for this menu is very short.  Press ESC repeatedly once PXE boot is seen to increase the success rate.
  3. You should see a GRUB menu with 3 options: Chain boot, Rescue Mode, & Secure Erase 
    1. If 4 options show, it is the regular GRUB menu instead of the one required; see Notes below.
  4. Choose Rescue Mode
  5. Run these commands:
    1. Be sure to replace # with the number found in the dmesg errors.
    2. If there are any errors, send the output to support@zpesystems.com for review.

umount /dev/sda#

fsck.ext4 -fy /dev/sda#

  1. After the scan completes you can verify the repair was successful by using:
fsck.ext4 -fn /dev/sda#
  1. To reboot the system and restore normal operation, use:
reboot -f
or
exit

Notes: If you're getting the "target is busy" or "/dev/sda# is mounted" error, you may have missed steps 2 & 3. Refer to the attached screencast, which includes example of running manual steps on /dev/sda8.
    1. If you are having difficulty accessing the recommended GRUB menu and repairing sda8, you can use this command to unmount sda8: umount_sda8.sh

For BTRFS filesystem:

  1. Log into the admin CLI
  2. Access the root shell using: shell sudo su -
  3. Below are the commands to list and check the filesystem errors
    1. Scan for btrfs filesystem
      btrfs device scan
    2. List btrfs filesystems
      btrfs filesystem show
    3. Check for errors. Replace <device> with the devid found from previous command
      btrfs check --repair --force <device>
    4. Show device IO errors statistics. Replace # with the number having btrfs filesystem
      btrfs device stats /dev/sda#

Reformatting the device
Another option to fix the file system errors is to reformat the device. Note that the factory reset process won't solve partition issues, the process only returns the configuration files to the factory state.

Before starting the process, backup your unit.
GUI: Navigate System :: Toolkit :: Apply Settings - Save the file to your Local Computer.
CLI:  Run save_settings
set destination=local_system
set filename=Mybackup
commit
finish
The file will be available under /var/local/file_manager/admin_group/config_backup
Then you would have the transfer the file from your Nodegrid to your system.




    • Related Articles

    • How to save diagnostic data for support

      The Diagnostic Data tool will allow you to quickly and easily gather data critical to facilitate case resolution when working with the ZPE Support team. It builds a collection file which contains logs, key configurations, and command outputs helpful ...

    • How to configure TFTP so that you can PUT a file

      INTERNAL USE ONLY WARNING: Following the below could compromise your NG system Edit the TFTP configuration file: /etc/xinetd.d/tftpd so that the content is as follow: service tftp { disable = no socket_type = dgram protocol = udp flags = IPv6 wait = ...

    • Improved NodeGrid WebUI: Apply Settings using Local System option

      To Save the current configuration of your NodeGrid, follow the steps below (5.x.x and 6.x.x NodeGrid Versions) WebUI 1. Go to System, Toolkit, and click on Save Settings icon. 2. Select the option where you want to save the configuration: a. Local ...

    • System Configuration Checksum

      Nodegrid system configuration checksum is a feature which creats a checksum for all the configuration files inside Nodegrid. This provides administrators a quick tool to verify periodically if the configuration has changed, and can be used for ...

    • How to review and adjust System Profile

      The system profile determines the initial functionality available; full details are available here: System Profile To check the current System profile from GUI from Security :: Services :: General Services: To check the current System profile from ...