Development Guidelines

Dask is a community maintained project. We welcome contributions in the formof bug reports, documentation, code, design proposals, and more.This page provides resources on how best to contribute.

Where to ask for help

Dask conversation happens in the following places:

  • Stack Overflow #dask tag: for usage questions
  • GitHub Issue Tracker: for discussions around new features or established bugs
  • Gitter chat: for real-time discussionFor usage questions and bug reports we strongly prefer the use of Stack Overflowand GitHub issues over gitter chat. GitHub and Stack Overflow are more easilysearchable by future users and so is more efficient for everyone’s time.Gitter chat is generally reserved for community discussion.

Separate Code Repositories

Dask maintains code and documentation in a few git repositories hosted on theGitHub dask organization, https://github.com/dask. This includes the primaryrepository and several other repositories for different components. Anon-exhaustive list follows:

Git and GitHub can be challenging at first. Fortunately good materials existon the internet. Rather than repeat these materials here, we refer you toPandas’ documentation and links on this subject athttps://pandas.pydata.org/pandas-docs/stable/contributing.html

Issues

The community discusses and tracks known bugs and potential features in theGitHub Issue Tracker. If you have a new idea or have identified a bug, thenyou should raise it there to start public discussion.

If you are looking for an introductory issue to get started with development,then check out the “good first issue” label, which contains issues that are goodfor starting developers. Generally, familiarity with Python, NumPy, Pandas, andsome parallel computing are assumed.

Development Environment

Download code

Make a fork of the main Dask repository andclone the fork:

  1. git clone https://github.com/<your-github-username>/dask

Contributions to Dask can then be made by submitting pull requests on GitHub.

Install

You may want to install larger dependencies like NumPy and Pandas using abinary package manager like conda. You can skip this step if you alreadyhave these libraries, don’t care to use them, or have sufficient buildenvironment on your computer to compile them when installing with pip:

  1. conda install -y numpy pandas scipy bokeh psutil

Install Dask and dependencies:

  1. cd dask
  2. pip install -e ".[complete]"

For development, Dask uses the following additional dependencies:

  1. pip install pytest moto

Run Tests

Dask uses py.test for testing. You can run tests from the main dask directoryas follows:

  1. py.test dask --verbose --doctest-modules

Contributing to Code

Dask maintains development standards that are similar to most PyData projects. These standards includelanguage support, testing, documentation, and style.

Python Versions

Dask supports Python versions 3.5, 3.6, and 3.7.Name changes are handled by the dask/compatibility.py file.

Test

Dask employs extensive unit tests to ensure correctness of code both for todayand for the future. Test coverage is expected for all code contributions.

Tests are written in a py.test style with bare functions:

  1. def test_fibonacci():
  2.     assert fib(0) == 0
  3.     assert fib(1) == 0
  4.     assert fib(10) == 55
  5.     assert fib(8) == fib(7) + fib(6)
  6.  
  7.     for x in [-3, 'cat', 1.5]:
  8.         with pytest.raises(ValueError):
  9.             fib(x)

These tests should compromise well between covering all branches and fail casesand running quickly (slow test suites get run less often).

You can run tests locally by running py.test in the local dask directory:

  1. py.test dask --verbose

You can also test certain modules or individual tests for faster response:

  1. py.test dask/dataframe --verbose
  2.  
  3. py.test dask/dataframe/tests/test_dataframe_core.py::test_set_index

Tests run automatically on the Travis.ci and Appveyor continuous testingframeworks on every push to every pull request on GitHub.

Tests are organized within the various modules’ subdirectories:

  1. dask/array/tests/test_*.py
  2. dask/bag/tests/test_*.py
  3. dask/dataframe/tests/test_*.py
  4. dask/diagnostics/tests/test_*.py

For the Dask collections like Dask Array and Dask DataFrame, behavior istypically tested directly against the NumPy or Pandas libraries using theassert_eq functions:

  1. import numpy as np
  2. import dask.array as da
  3. from dask.array.utils import assert_eq
  4.  
  5. def test_aggregations():
  6.     nx = np.random.random(100)
  7.     dx = da.from_array(nx, chunks=(10,))
  8.  
  9.     assert_eq(nx.sum(), dx.sum())
  10.     assert_eq(nx.min(), dx.min())
  11.     assert_eq(nx.max(), dx.max())
  12.     ...

This technique helps to ensure compatibility with upstream libraries and tendsto be simpler than testing correctness directly. Additionally, by passing Daskcollections directly to the assert_eq function rather than call computemanually, the testing suite is able to run a number of checks on the lazycollections themselves.

Docstrings

User facing functions should roughly follow the numpydoc standard, includingsections for Parameters, Examples, and general explanatory prose.

By default, examples will be doc-tested. Reproducible examples in documentationis valuable both for testing and, more importantly, for communication of commonusage to the user. Documentation trumps testing in this case and clearexamples should take precedence over using the docstring as testing space.To skip a test in the examples add the comment # doctest: +SKIP directlyafter the line.

  1. def fib(i):
  2.     """ A single line with a brief explanation
  3.  
  4.     A more thorough description of the function, consisting of multiple
  5.     lines or paragraphs.
  6.  
  7.     Parameters
  8.     ----------
  9.     i: int
  10.          A short description of the argument if not immediately clear
  11.  
  12.     Examples
  13.     --------
  14.     >>> fib(4)
  15.     3
  16.     >>> fib(5)
  17.     5
  18.     >>> fib(6)
  19.     8
  20.     >>> fib(-1)  # Robust to bad inputs
  21.     ValueError(...)
  22.     """

Docstrings are currently tested under Python 3.6 on Travis.ci. You can testdocstrings with pytest as follows:

  1. py.test dask --doctest-modules

Docstring testing requires graphviz to be installed. This can be done via:

  1. conda install -y graphviz

Code Formatting

Dask uses Black andFlake8 to ensure a consistent codeformat throughout the project. black and flake8 can be installed withpip:

  1. pip install black flake8

and then run from the root of the Dask repository:

  1. black dask
  2. flake8 dask

to auto-format your code. Additionally, many editors have plugins that willapply black as you edit files.

Optionally, you may wish to setup pre-commit hooksto automatically run black and flake8 when you make a git commit. Thiscan be done by installing pre-commit:

  1. pip install pre-commit

and then running:

  1. pre-commit install

from the root of the Dask repository. Now black and flake8 will be runeach time you commit changes. You can skip these checks withgit commit —no-verify.

Contributing to Documentation

Dask uses Sphinx for documentation, hosted on https://readthedocs.org .Documentation is maintained in the RestructuredText markup language (.rstfiles) in dask/docs/source. The documentation consists both of proseand API documentation.

To build the documentation locally, first install the necessary requirements:

  1. cd docs/
  2. pip install -r requirements-docs.txt

Then build the documentation with make:

  1. make html

The resulting HTML files end up in the build/html directory.

You can now make edits to rst files and run make html again to updatethe affected pages.