Reporting and analyzing crashes (segfaults))

So you managed to break Julia. Congratulations! Collected here are some general procedures you can undergo for common symptoms encountered when something goes awry. Including the information from these debugging steps can greatly help the maintainers when tracking down a segfault or trying to figure out why your script is running slower than expected.

If you’ve been directed to this page, find the symptom that best matches what you’re experiencing and follow the instructions to generate the debugging information requested. Table of symptoms:

Version/Environment info

No matter the error, we will always need to know what version of Julia you are running. When Julia first starts up, a header is printed out with a version number and date. Please also include the output of versioninfo() (exported from the InteractiveUtils standard library) in any report you create:

  1. julia> using InteractiveUtils
  3. julia> versioninfo()
  4. Julia Version 1.7.0
  5. Commit 3bf9d17731 (2021-11-30 12:12 UTC)
  6. Platform Info:
  7.   OS: Linux (x86_64-pc-linux-gnu)
  8.   CPU: Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz
  9.   WORD_SIZE: 64
  10.   LIBM: libopenlibm
  11.   LLVM: libLLVM-12.0.1 (ORCJIT, haswell)
  12. Environment:
  13.   JULIA_CPU_TARGET = generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)
  14.   JULIA_PRECOMPILE = 0

Segfaults during bootstrap (sysimg.jl))

Segfaults toward the end of the make process of building Julia are a common symptom of something going wrong while Julia is preparsing the corpus of code in the base/ folder. Many factors can contribute toward this process dying unexpectedly, however it is as often as not due to an error in the C-code portion of Julia, and as such must typically be debugged with a debug build inside of gdb. Explicitly:

Create a debug build of Julia:

  1. $ cd <julia_root>
  2. $ make debug

Note that this process will likely fail with the same error as a normal make incantation, however this will create a debug executable that will offer gdb the debugging symbols needed to get accurate backtraces. Next, manually run the bootstrap process inside of gdb:

  1. $ cd base/
  2. $ gdb -x ../contrib/debug_bootstrap.gdb

This will start gdb, attempt to run the bootstrap process using the debug build of Julia, and print out a backtrace if (when) it segfaults. You may need to hit <enter> a few times to get the full backtrace. Create a gist with the backtrace, the version info, and any other pertinent information you can think of and open a new issue on Github with a link to the gist.

Segfaults when running a script

The procedure is very similar to Segfaults during bootstrap (sysimg.jl)). Create a debug build of Julia, and run your script inside of a debugged Julia process:

  1. $ cd <julia_root>
  2. $ make debug
  3. $ gdb --args usr/bin/julia-debug <path_to_your_script>

Note that gdb will sit there, waiting for instructions. Type r to run the process, and bt to generate a backtrace once it segfaults:

  1. (gdb) r
  2. Starting program: /home/sabae/src/julia/usr/bin/julia-debug ./test.jl
  3. ...
  4. (gdb) bt

Create a gist with the backtrace, the version info, and any other pertinent information you can think of and open a new issue on Github with a link to the gist.

Errors during Julia startup

Occasionally errors occur during Julia’s startup process (especially when using binary distributions, as opposed to compiling from source) such as the following:

  1. $ julia
  2. exec: error -5

These errors typically indicate something is not getting loaded properly very early on in the bootup phase, and our best bet in determining what’s going wrong is to use external tools to audit the disk activity of the julia process:

  • On Linux, use strace:

    1. $ strace julia

  • On OSX, use dtruss:

    1. $ dtruss -f julia

Create a gist with the strace/ dtruss output, the version info, and any other pertinent information and open a new issue on Github with a link to the gist.

Other generic segfaults or unreachables reached

As mentioned elsewhere, julia has good integration with rr for generating traces; this includes, on Linux, the ability to automatically run julia under rr and share the trace after a crash. This can be immensely helpful when debugging such crashes and is strongly encouraged when reporting crash issues to the JuliaLang/julia repo. To run julia under rr automatically, do:

  1. julia --bug-report=rr

To generate the rr trace locally, but not share, you can do:

  1. julia --bug-report=rr-local

Note that this is only works on Linux. The blog post on Time Travelling Bug Reporting has many more details.

Glossary

A few terms have been used as shorthand in this guide:

  • <julia_root> refers to the root directory of the Julia source tree; e.g. it should contain folders such as base, deps, src, test, etc…..