Frequently Asked Questions

Why is pandas not enough?

pandas is a fantastic library for analysis of low-dimensional labelled data -if it can be sensibly described as “rows and columns”, pandas is probably theright choice. However, sometimes we want to use higher dimensional arrays(ndim > 2), or arrays for which the order of dimensions (e.g., columns vsrows) shouldn’t really matter. For example, the images of a movie can benatively represented as an array with four dimensions: time, row, column andcolor.

Pandas has historically supported N-dimensional panels, but deprecated them inversion 0.20 in favor of Xarray data structures. There are now built-in methodson both sides to convert between pandas and Xarray, allowing for more focusseddevelopment effort. Xarray objects have a much richer model of dimensionality -if you were using Panels:

• You need to create a new factory type for each dimensionality.

• You can’t do math between NDPanels with different dimensionality.

• Each dimension in a NDPanel has a name (e.g., ‘labels’, ‘items’,‘major_axis’, etc.) but the dimension names refer to order, not theirmeaning. You can’t specify an operation as to be applied along the “time”axis.

• You often have to manually convert collections of pandas arrays(Series, DataFrames, etc) to have the same number of dimensions.In contrast, this sort of data structure fits very naturally in anxarray Dataset.

You can read about switching from Panels to Xarray here.Pandas gets a lot of things right, but many science, engineering and complexanalytics use cases need fully multi-dimensional data structures.

How do xarray data structures differ from those found in pandas?

The main distinguishing feature of xarray’s DataArray over labeled arrays inpandas is that dimensions can have names (e.g., “time”, “latitude”,“longitude”). Names are much easier to keep track of than axis numbers, andxarray uses dimension names for indexing, aggregation and broadcasting. Not onlycan you write x.sel(time='2000-01-01') and x.mean(dim='time'), butoperations like x - x.mean(dim='time') always work, no matter the orderof the “time” dimension. You never need to reshape arrays (e.g., withnp.newaxis) to align them for arithmetic operations in xarray.

Should I use xarray instead of pandas?

It’s not an either/or choice! xarray provides robust support for convertingback and forth between the tabular data-structures of pandas and its ownmulti-dimensional data-structures.

That said, you should only bother with xarray if some aspect of data isfundamentally multi-dimensional. If your data is unstructured orone-dimensional, pandas is usually the right choice: it has better performancefor common operations such as groupby and you’ll find far more usageexamples online.

Why don’t aggregations return Python scalars?

xarray tries hard to be self-consistent: operations on a DataArray (resp.Dataset) return another DataArray (resp. Dataset) object. Inparticular, operations returning scalar values (e.g. indexing or aggregationslike mean or sum applied to all axes) will also return xarray objects.

Unfortunately, this means we sometimes have to explicitly cast our results fromxarray when using them in other libraries. As an illustration, the followingcode fragment

In [1]: arr = xr.DataArray([1, 2, 3])
 
In [2]: pd.Series({'x': arr[0], 'mean': arr.mean(), 'std': arr.std()})
Out[2]: 
x              <xarray.DataArray ()>\narray(1)
mean          <xarray.DataArray ()>\narray(2.)
std     <xarray.DataArray ()>\narray(0.816497)
dtype: object

does not yield the pandas DataFrame we expected. We need to specify the typeconversion ourselves:

In [3]: pd.Series({'x': arr[0], 'mean': arr.mean(), 'std': arr.std()}, dtype=float)
Out[3]: 
x       1.000000
mean    2.000000
std     0.816497
dtype: float64

Alternatively, we could use the item method or the float constructor toconvert values one at a time

In [4]: pd.Series({'x': arr[0].item(), 'mean': float(arr.mean())})
Out[4]: 
x       1.0
mean    2.0
dtype: float64

What is your approach to metadata?

We are firm believers in the power of labeled data! In addition to dimensionsand coordinates, xarray supports arbitrary metadata in the form of global(Dataset) and variable specific (DataArray) attributes (attrs).

Automatic interpretation of labels is powerful but also reduces flexibility.With xarray, we draw a firm line between labels that the library understands(dims and coords) and labels for users and user code (attrs). Forexample, we do not automatically interpret and enforce units or CFconventions. (An exception is serialization to and from netCDF files.)

An implication of this choice is that we do not propagate attrs throughmost operations unless explicitly flagged (some methods have a keep_attrsoption, and there is a global flag for setting this to be always True orFalse). Similarly, xarray does not check for conflicts between attrs whencombining arrays and datasets, unless explicitly requested with the optioncompat='identical'. The guiding principle is that metadata should not beallowed to get in the way.

What other netCDF related Python libraries should I know about?

netCDF4-python provides a lower level interface for working withnetCDF and OpenDAP datasets in Python. We use netCDF4-python internally inxarray, and have contributed a number of improvements and fixes upstream. xarraydoes not yet support all of netCDF4-python’s features, such as modifying fileson-disk.

Iris (supported by the UK Met office) provides similar tools for in-memory manipulation of labeled arrays, aimed specifically at weather andclimate data needs. Indeed, the Iris Cube was directinspiration for xarray’s DataArray. xarray and Iris take verydifferent approaches to handling metadata: Iris strictly interpretsCF conventions. Iris particularly shines at mapping, thanks to itsintegration with Cartopy.

UV-CDAT is another Python library that implements in-memory netCDF-likevariables and tools for working with climate data.

We think the design decisions we have made for xarray (namely, basing it onpandas) make it a faster and more flexible data analysis tool. That said, Irisand CDAT have some great domain specific functionality, and xarray includesmethods for converting back and forth between xarray and these libraries. Seeto_iris() and to_cdms2()for more details.

What other projects leverage xarray?

See section Xarray related projects.

How should I cite xarray?

If you are using xarray and would like to cite it in academic publication, wewould certainly appreciate it. We recommend two citations.

1. At a minimum, we recommend citing the xarray overview journal article,published in the Journal of Open Research Software.

   • Hoyer, S. & Hamman, J., (2017). xarray: N-D labeled Arrays andDatasets in Python. Journal of Open Research Software. 5(1), p.10.DOI: http://doi.org/10.5334/jors.148

     Here’s an example of a BibTeX entry:

     @article{hoyer2017xarray,  title     = {xarray: {N-D} labeled arrays and datasets in {Python}},  author    = {Hoyer, S. and J. Hamman},  journal   = {Journal of Open Research Software},  volume    = {5},  number    = {1},  year      = {2017},  publisher = {Ubiquity Press},  doi       = {10.5334/jors.148},  url       = {http://doi.org/10.5334/jors.148}}

2. You may also want to cite a specific version of the xarray package. Weprovide a Zenodo citation and DOIfor this purpose:

   

   An example BibTeX entry:

   @misc{xarray_v0_8_0,      author = {Stephan Hoyer and Clark Fitzgerald and Joe Hamman and others},      title  = {xarray: v0.8.0},      month  = aug,      year   = 2016,      doi    = {10.5281/zenodo.59499},      url    = {https://doi.org/10.5281/zenodo.59499}     }

What parts of xarray are considered public API?

As a rule, only functions/methods documented in our API Reference are consideredpart of xarray’s public API. Everything else (in particular, everything inxarray.core that is not also exposed in the top level xarray namespace)is considered a private implementation detail that may change at any time.

Objects that exist to facilitate xarray’s fluent interface on DataArray andDataset objects are a special case. For convenience, we document them inthe API docs, but only their methods and the DataArray/Datasetmethods/properties to construct them (e.g., .plot(), .groupby(),.str) are considered public API. Constructors and other details of theinternal classes used to implemented them (i.e.,xarray.plot.plotting._PlotMethods, xarray.core.groupby.DataArrayGroupBy,xarray.core.accessor_str.StringAccessor) are not.