Troubleshoot image rendering

In this section, you’ll learn how to enable logging for the image renderer and you’ll find the most common issues.

Enable debug logging

To troubleshoot the image renderer, different kind of logs are available.

You can enable debug log messages for rendering in the Grafana configuration file and inspect the Grafana server logs.

  1. [log]
  2. filters = rendering:debug

You can also enable more logs in image renderer service itself by:

Missing libraries

The plugin and rendering service uses Chromium browser which depends on certain libraries. If you don’t have all of those libraries installed in your system you may encounter errors when trying to render an image, e.g.

  1. Rendering failed: Error: Failed to launch chrome!/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome:
  2. error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

In general you can use the ldd utility to figure out what shared libraries are not installed in your system:

  1. cd <grafana-image-render plugin directory>
  2. ldd chrome-linux/chrome
  3.       linux-vdso.so.1 (0x00007fff1bf65000)
  4.       libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2047945000)
  5.       libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2047924000)
  6.       librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f204791a000)
  7.       libX11.so.6 => not found
  8.       libX11-xcb.so.1 => not found
  9.       libxcb.so.1 => not found
  10.       libXcomposite.so.1 => not found
  11.         ...

Ubuntu:

On Ubuntu 18.10 the following dependencies have been confirmed as needed for the image rendering to function.

  1. libx11-6 libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrender1 libxtst6 libglib2.0-0 libnss3 libcups2  libdbus-1-3 libxss1 libxrandr2 libgtk-3-0 libgtk-3-0 libasound2 libxcb-dri3-0 libgbm1 libxshmfence1

Debian:

On Debian 9 (Stretch) the following dependencies have been confirmed as needed for the image rendering to function.

  1. libx11 libcairo bcairo2 libcairo2 libxtst6 libxcomposite1 libx11-xcb1 libxcursor1 libxdamage1 libnss3 libcups libcups2 libXss libXss1 libxss1 libxrandr2 libasound2 libatk1.0-0 libatk-bridge2.0-0 libpangocairo-1.0-0 libgtk-3-0 libgbm1 libxshmfence1

On Debian 10 (Buster) the following dependencies have been confirmed as needed for the image rendering to function.

  1. libxdamage1 libxext6 libxi6 libxtst6 libnss3 libnss3 libcups2 libxss1 libxrandr2 libasound2 libatk1.0-0 libatk-bridge2.0-0 libpangocairo-1.0-0 libpango-1.0-0 libcairo2 libatspi2.0-0 libgtk3.0-cil libgdk3.0-cil libx11-xcb-dev libgbm1 libxshmfence1

Centos:

On a minimal Centos installation, the following dependencies have been confirmed as needed for the image rendering to function:

  1. libXcomposite libXdamage libXtst cups libXScrnSaver pango atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy liberation-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time trousers xdg-utils xkeyboard-config alsa-lib

Certificate signed by internal certificate authorities

In many cases, Grafana runs on internal servers and uses certificates that have not been signed by a CA (Certificate Authority) known to Chrome, and therefore cannot be validated. Chrome internally uses NSS (Network Security Services) for cryptographic operations such as the validation of certificates.

If you are using the Grafana Image Renderer with a Grafana server that uses a certificate signed by such a custom CA (for example a company-internal CA), rendering images will fail and you will see messages like this in the Grafana log:

  1. t=2019-12-04T12:39:22+0000 lvl=error msg="Render request failed" logger=rendering error=map[] url="https://192.168.106.101:3443/d-solo/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m&from=1575438321300&to=1575459921300&var-Host=master1&panelId=4&width=1000&height=500&tz=Europe%2FBerlin&render=1" timestamp=0001-01-01T00:00:00.000Z
  2. t=2019-12-04T12:39:22+0000 lvl=error msg="Rendering failed." logger=context userId=1 orgId=1 uname=admin error="Rendering failed: Error: net::ERR_CERT_AUTHORITY_INVALID at https://192.168.106.101:3443/d-solo/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m&from=1575438321300&to=1575459921300&var-Host=master1&panelId=4&width=1000&height=500&tz=Europe%2FBerlin&render=1"
  3. t=2019-12-04T12:39:22+0000 lvl=error msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/render/d-solo/zxDJxNaZk/graphite-metrics status=500 remote_addr=192.168.106.101 time_ms=310 size=1722 referer="https://grafana.xxx-xxx/d/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m"

If this happens, then you have to add the certificate to the trust store. If you have the certificate file for the internal root CA in the file internal-root-ca.crt.pem, then use these commands to create a user specific NSS trust store for the Grafana user (grafana for the purpose of this example) and execute the following steps:

Linux:

  1. [root@server ~]# [ -d /usr/share/grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb
  2. [root@merver ~]# certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n internal-root-ca -t C -i /etc/pki/tls/certs/internal-root-ca.crt.pem
  3. [root@server ~]# chown -R grafana: /usr/share/grafana/.pki/nssdb

Windows:

  1. certutil –addstore "Root" <path>/internal-root-ca.crt.pem

Custom Chrome/Chromium

As a last resort, if you already have Chrome or Chromium installed on your system, then you can configure the Grafana Image renderer plugin to use this instead of the pre-packaged version of Chromium.

Note: Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not compatible with the Grafana Image renderer plugin.

To override the path to the Chrome/Chromium executable in plugin mode, set an environment variable and make sure that it’s available for the Grafana process. For example:

  1. export GF_PLUGIN_RENDERING_CHROME_BIN="/usr/bin/chromium-browser"

In remote rendering mode, you need to set the environment variable or update the configuration file and make sure that it’s available for the image rendering service process:

  1. CHROME_BIN="/usr/bin/chromium-browser"
  1. {
  2.   "rendering": {
  3.     "chromeBin": "/usr/bin/chromium-browser"
  4.   }
  5. }