Operating System

Harvester runs on an OpenSUSE-based OS. The OS is an artifact produced by the cOS toolkit. The following sections contain information and tips to help users troubleshoot OS-related issues.

How to log into a Harvester node

Users can log into a Harvester node with the username rancher and the password or SSH keypair provided during installation. The user rancher can execute privileged commands without entering a password:

  1. # Run a privileged command
  2. [email protected]:~> sudo blkid
  4. # Or become root
  5. [email protected]:~> sudo -i
  6. node1:~ # blkid

How can I install packages? Why are some paths read-only?

The OS file system, like a container image, is image-based and immutable except in some directories. To temporarily enable the read-write mode, please use the following steps:

Warning

Enabling read-write mode might break your system if files are modified. Please use it at your own risk.

  • For version v0.3.0, we need to apply a workaround first to make some directories non-overlaid after enabling read-write mode. On a running Harvester node, run the following command as root:

    1. cat > /oem/91_hack.yaml <<'EOF'
    2. name: "Rootfs Layout Settings for debugrw"
    3. stages:
    4.   rootfs:
    5.     - if: 'grep -q root=LABEL=COS_ACTIVE /proc/cmdline && grep -q rd.cos.debugrw /proc/cmdline'
    6.       name: "Layout configuration for debugrw"
    7.       environment_file: /run/cos/cos-layout.env
    8.       environment:
    9.         RW_PATHS: " "
    10. EOF

  • Reboot the system to GRUB menu. Press ESC to stay on the menu. Operating System - 图1

  • Press e on first menuentry. Append rd.cos.debugrw to the linux (loop0)$kernel $kernelcmd line. Press Ctrl + x to boot the system. Operating System - 图2

How to permanently edit kernel parameters

Note

The following steps are a workaround. Harvester will inform the community once a permanent resolution is in place.

  • Re-mount state directory in rw mode:

    1. # blkid -L COS_STATE
    2. /dev/vda2
    3. # mount -o remount,rw /dev/vda2 /run/initramfs/cos-state

  • Edit the grub config file and append parameters to the linux (loop0)$kernel $kernelcmd line. The following example adds a nomodeset parameter:

    1. # vim /run/initramfs/cos-state/grub2/grub.cfg
    2. menuentry "Harvester ea6e7f5-dirty" --id cos {
    3.   search.fs_label COS_STATE root
    4.   set img=/cOS/active.img
    5.   set label=COS_ACTIVE
    6.   loopback loop0 /$img
    7.   set root=($root)
    8.   source (loop0)/etc/cos/bootargs.cfg
    9.   linux (loop0)$kernel $kernelcmd nomodeset
    10.   initrd (loop0)$initramfs
    11. }

  • Reboot for changes to take effect.

How to change the default GRUB boot menu entry

To change the default entry, first check the --id attribute of a menu entry, as in the following example:

  1. # cat /run/initramfs/cos-state/grub2/grub.cfg
  3. <...>
  4. menuentry "Harvester ea6e7f5-dirty (debug)" --id cos-debug {
  5.   search.fs_label COS_STATE root
  6.   set img=/cOS/active.img
  7.   set label=COS_ACTIVE
  8.   loopback loop0 /$img

The id of the above entry is cos-debug. We can then set the default entry by:

  1. # grub2-editenv /oem/grubenv set saved_entry=cos-debug

How to debug a system crash or hang

Collect crash log

If kernel panic traces are not recorded in the system log when a system crashes, one reliable way to locate the crash log is to use a serial console.

To enable outputting of kernel messages to a serial console, please use the following steps:

  • Boot the system to GRUB menu. Press ESC to stay on the menu. Operating System - 图3

  • Press e on first menuentry. Append console=ttyS0,115200n8 to the linux (loop0)$kernel $kernelcmd line. Press Ctrl + x to boot the system.

    Operating System - 图4

    Note

    Adjust the console options according to your environment. Make sure to append the console= string at the end of the line.

    • Connect to the serial port to capture logs.

    Collect crash dumps

    For kernel panic crashes, you can use kdump to collect crash dumps.

By default, the OS is booted without the kdump feature enabled. Users can enable the feature by selecting the debug menuentry when booting, as in the following example:

Operating System - 图5

When a system crashes, a crash dump will be stored in the /var/crash/<time> directory. Providing the crash dump to developers helps them to troubleshoot and resolve issues.