Operating System

Harvester runs on an OpenSUSE-based OS. The OS is an artifact produced by the elemental-toolkit. The following sections contain information and tips to help users troubleshoot OS-related issues.

How to log in to a Harvester node

Users can log in to a Harvester node with the username rancher and the password or SSH keypair provided during installation. The user rancher can execute privileged commands without entering a password:

  1. # Run a privileged command
  2. rancher@node1:~> sudo blkid
  4. # Or become root
  5. rancher@node1:~> sudo -i
  6. node1:~ # blkid

How can I install packages? Why are some paths read-only?

The OS file system, like a container image, is image-based and immutable except in some directories.

We recommend using a toolbox container to run programs not packaged in the Harvester OS for debugging purposes. Please see this article to learn how to build and run a toolbox container.

The Harvester OS also provides a way to enable the read-write mode temporarily. Please follow the following steps:

Operating System - 图1caution

Enabling read-write mode might break your system if files are modified. Please use it at your own risk.

  • For version v0.3.0, we need to apply a workaround first to make some directories non-overlaid after enabling read-write mode. On a running Harvester node, run the following command as root:

    1. cat > /oem/91_hack.yaml <<'EOF'
    2. name: "Rootfs Layout Settings for debugrw"
    3. stages:
    4.   rootfs:
    5.     - if: 'grep -q root=LABEL=COS_STATE /proc/cmdline && grep -q rd.cos.debugrw /proc/cmdline'
    6.       name: "Layout configuration for debugrw"
    7.       environment_file: /run/cos/cos-layout.env
    8.       environment:
    9.         RW_PATHS: " "
    10. EOF

  • Reboot the system to GRUB menu. Press ESC to stay on the menu.

    Operating System - 图2

  • Press e on first menuentry. Append rd.cos.debugrw to the linux (loop0)$kernel $kernelcmd line. Press Ctrl + x to boot the system.

    Operating System - 图3

How to permanently edit kernel parameters

Operating System - 图4note

The following steps are a workaround. Harvester will inform the community once a permanent resolution is in place.

  • Re-mount state directory in rw mode:

    1. # blkid -L COS_STATE
    2. /dev/vda2
    3. # mount -o remount,rw /dev/vda2 /run/initramfs/cos-state

  • Edit the grub config file and append parameters to the linux (loop0)$kernel $kernelcmd line. The following example adds a nomodeset parameter:

    1. # vim /run/initramfs/cos-state/grub2/grub.cfg
    2. menuentry "${display_name}" --id cos {
    3.   # label is kept around for backward compatibility
    4.   set label=${active_label}
    5.   set img=/cOS/active.img
    6.   loopback $loopdev /$img
    7.   source ($loopdev)/etc/cos/bootargs.cfg
    8.   linux ($loopdev)$kernel $kernelcmd ${extra_cmdline} ${extra_active_cmdline} nomodeset
    9.   initrd ($loopdev)$initramfs
    10. }

  • Reboot for changes to take effect.

How to change the default GRUB boot menu entry

To change the default entry, first check the --id attribute of a menu entry. Grub menu entries are located in the following files:

  • /run/initramfs/cos-state/grub2/grub.cfg: Contains the default, fallback, and recovery entries
  • /run/initramfs/cos-state/grubcustom: Contains the debug entry

In the following example, the id of the entry is debug.

  1. # cat \
  2.     /run/initramfs/cos-state/grub2/grub.cfg \
  3.     /run/initramfs/cos-state/grubcustom
  5. <...>
  6. menuentry "${display_name} (debug)" --id debug {
  7.   search --no-floppy --set=root --label COS_STATE
  8.   set img=/cOS/active.img
  9.   set label=COS_ACTIVE
  10.   loopback loop0 /$img
  11.   set root=($root)
  12.   source (loop0)/etc/cos/bootargs.cfg
  13.   linux (loop0)$kernel $kernelcmd ${extra_cmdline} ${extra_passive_cmdline} ${crash_kernel_params}
  14.   initrd (loop0)$initramfs
  15. }

You can configure the default entry by running the following commands:

  1. # mount -o remount,rw /run/initramfs/cos-state
  2. # grub2-editenv /run/initramfs/cos-state/grub_oem_env set saved_entry=debug

If necessary, you can undo the change by running the command grub2-editenv /run/initramfs/cos-state/grub_oem_env unset saved_entry.

How to debug a system crash or hang

Collect crash log

If kernel panic traces are not recorded in the system log when a system crashes, one reliable way to locate the crash log is to use a serial console.

To enable outputting of kernel messages to a serial console, please use the following steps:

  • Boot the system to GRUB menu. Press ESC to stay on the menu. Operating System - 图5

  • Press e on first menuentry. Append console=ttyS0,115200n8 to the linux (loop0)$kernel $kernelcmd line. Press Ctrl + x to boot the system.

    Operating System - 图6

Operating System - 图7note

Adjust the console options according to your environment. Make sure to append the console= string at the end of the line.

  • Connect to the serial port to capture logs.

Collect crash dumps

For kernel panic crashes, you can use kdump to collect crash dumps.

By default, the OS is booted without the kdump feature enabled. Users can enable the feature by selecting the debug menuentry when booting, as in the following example:

Operating System - 图8

When a system crashes, a crash dump will be stored in the /var/crash/<time> directory. Providing the crash dump to developers helps them to troubleshoot and resolve issues.