Working with pandas

One of the most important features of xarray is the ability to convert to andfrom pandas objects to interact with the rest of the PyDataecosystem. For example, for plotting labeled data, we highly recommendusing the visualization built in to pandas itself or provided by the pandasaware libraries such as Seaborn.

Hierarchical and tidy data

Tabular data is easiest to work with when it meets the criteria fortidy data:

  • Each column holds a different variable.

  • Each rows holds a different observation.

In this “tidy data” format, we can represent any Dataset andDataArray in terms of pandas.DataFrame andpandas.Series, respectively (and vice-versa). The representationworks by flattening non-coordinates to 1D, and turning the tensor product ofcoordinate indexes into a pandas.MultiIndex.

Dataset and DataFrame

To convert any dataset to a DataFrame in tidy form, use theDataset.to_dataframe() method:

  1. In [1]: ds = xr.Dataset({'foo': (('x', 'y'), np.random.randn(2, 3))},
  2.    ...:                  coords={'x': [10, 20], 'y': ['a', 'b', 'c'],
  3.    ...:                          'along_x': ('x', np.random.randn(2)),
  4.    ...:                          'scalar': 123})
  5.    ...: 
  6.  
  7. In [2]: ds
  8. Out[2]: 
  9. <xarray.Dataset>
  10. Dimensions:  (x: 2, y: 3)
  11. Coordinates:
  12.   * x        (x) int64 10 20
  13.   * y        (y) <U1 'a' 'b' 'c'
  14.     along_x  (x) float64 0.1192 -1.044
  15.     scalar   int64 123
  16. Data variables:
  17.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
  18.  
  19. In [3]: df = ds.to_dataframe()
  20.  
  21. In [4]: df
  22. Out[4]: 
  23.            foo   along_x  scalar
  24. x  y                            
  25. 10 a  0.469112  0.119209     123
  26.    b -0.282863  0.119209     123
  27.    c -1.509059  0.119209     123
  28. 20 a -1.135632 -1.044236     123
  29.    b  1.212112 -1.044236     123
  30.    c -0.173215 -1.044236     123

We see that each variable and coordinate in the Dataset is now a column in theDataFrame, with the exception of indexes which are in the index.To convert the DataFrame to any other convenient representation,use DataFrame methods like reset_index(),stack() and unstack().

For datasets containing dask arrays where the data should be lazily loaded, see theDataset.to_dask_dataframe() method.

To create a Dataset from a DataFrame, use thefrom_dataframe() class method or the equivalentpandas.DataFrame.to_xarray method (pandasv0.18 or later):

  1. In [5]: xr.Dataset.from_dataframe(df)
  2. Out[5]: 
  3. <xarray.Dataset>
  4. Dimensions:  (x: 2, y: 3)
  5. Coordinates:
  6.   * x        (x) int64 10 20
  7.   * y        (y) object 'a' 'b' 'c'
  8. Data variables:
  9.     foo      (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
  10.     along_x  (x, y) float64 0.1192 0.1192 0.1192 -1.044 -1.044 -1.044
  11.     scalar   (x, y) int64 123 123 123 123 123 123

Notice that that dimensions of variables in the Dataset have nowexpanded after the round-trip conversion to a DataFrame. This is becauseevery object in a DataFrame must have the same indices, so we need tobroadcast the data of each array to the full size of the new MultiIndex.

Likewise, all the coordinates (other than indexes) ended up as variables,because pandas does not distinguish non-index coordinates.

DataArray and Series

DataArray objects have a complementary representation in terms of apandas.Series. Using a Series preserves the Dataset toDataArray relationship, because DataFrames are dict-like containersof Series. The methods are very similar to those for working withDataFrames:

  1. In [6]: s = ds['foo'].to_series()
  2.  
  3. In [7]: s
  4. Out[7]: 
  5. x   y
  6. 10  a    0.469112
  7.     b   -0.282863
  8.     c   -1.509059
  9. 20  a   -1.135632
  10.     b    1.212112
  11.     c   -0.173215
  12. Name: foo, dtype: float64
  13.  
  14. # or equivalently, with Series.to_xarray()
  15. In [8]: xr.DataArray.from_series(s)
  16. Out[8]: 
  17. <xarray.DataArray 'foo' (x: 2, y: 3)>
  18. array([[ 0.469112, -0.282863, -1.509059],
  19.        [-1.135632,  1.212112, -0.173215]])
  20. Coordinates:
  21.   * x        (x) int64 10 20
  22.   * y        (y) object 'a' 'b' 'c'

Both the from_series and from_dataframe methods use reindexing, so theywork even if not the hierarchical index is not a full tensor product:

  1. In [9]: s[::2]
  2. Out[9]: 
  3. x   y
  4. 10  a    0.469112
  5.     c   -1.509059
  6. 20  b    1.212112
  7. Name: foo, dtype: float64
  8.  
  9. In [10]: s[::2].to_xarray()
  10. Out[10]: 
  11. <xarray.DataArray 'foo' (x: 2, y: 3)>
  12. array([[ 0.469112,       nan, -1.509059],
  13.        [      nan,  1.212112,       nan]])
  14. Coordinates:
  15.   * x        (x) int64 10 20
  16.   * y        (y) object 'a' 'b' 'c'

Multi-dimensional data

Tidy data is great, but it sometimes you want to preserve dimensions instead ofautomatically stacking them into a MultiIndex.

DataArray.to_pandas() is a shortcut thatlets you convert a DataArray directly into a pandas object with the samedimensionality (i.e., a 1D array is converted to a Series,2D to DataFrame and 3D to Panel):

  1. In [11]: arr = xr.DataArray(np.random.randn(2, 3),
  2.    ....:                    coords=[('x', [10, 20]), ('y', ['a', 'b', 'c'])])
  3.    ....: 
  4.  
  5. In [12]: df = arr.to_pandas()
  6.  
  7. In [13]: df
  8. Out[13]: 
  9. y          a         b         c
  10. x                               
  11. 10 -0.861849 -2.104569 -0.494929
  12. 20  1.071804  0.721555 -0.706771

To perform the inverse operation of converting any pandas objects into a dataarray with the same shape, simply use the DataArrayconstructor:

  1. In [14]: xr.DataArray(df)
  2. Out[14]: 
  3. <xarray.DataArray (x: 2, y: 3)>
  4. array([[-0.861849, -2.104569, -0.494929],
  5.        [ 1.071804,  0.721555, -0.706771]])
  6. Coordinates:
  7.   * x        (x) int64 10 20
  8.   * y        (y) object 'a' 'b' 'c'

Both the DataArray and Dataset constructors directly convert pandasobjects into xarray objects with the same shape. This means that theypreserve all use of multi-indexes:

  1. In [15]: index = pd.MultiIndex.from_arrays([['a', 'a', 'b'], [0, 1, 2]],
  2.    ....:                                   names=['one', 'two'])
  3.    ....: 
  4.  
  5. In [16]: df = pd.DataFrame({'x': 1, 'y': 2}, index=index)
  6.  
  7. In [17]: ds = xr.Dataset(df)
  8.  
  9. In [18]: ds
  10. Out[18]: 
  11. <xarray.Dataset>
  12. Dimensions:  (dim_0: 3)
  13. Coordinates:
  14.   * dim_0    (dim_0) MultiIndex
  15.   - one      (dim_0) object 'a' 'a' 'b'
  16.   - two      (dim_0) int64 0 1 2
  17. Data variables:
  18.     x        (dim_0) int64 1 1 1
  19.     y        (dim_0) int64 2 2 2

However, you will need to set dimension names explicitly, either with thedims argument on in the DataArray constructor or by callingrename on the new object.

Transitioning from pandas.Panel to xarray

Panel, pandas’ data structure for 3D arrays, has alwaysbeen a second class data structure compared to the Series and DataFrame. Toallow pandas developers to focus more on its core functionality built aroundthe DataFrame, panads has deprecated Panel. It will be removed in pandas0.25.

xarray has most of Panel’s features, a more explicit API (particularly aroundindexing), and the ability to scale to >3 dimensions with the same interface.

As discussed elsewhere in the docs, there are two primary data structures inxarray: DataArray and Dataset. You can imagine a DataArray as an-dimensional pandas Series (i.e. a single typed array), and a Datasetas the DataFrame equivalent (i.e. a dict of aligned DataArray objects).

So you can represent a Panel, in two ways:

  • As a 3-dimensional DataArray,

  • Or as a Dataset containing a number of 2-dimensional DataArray objects.

Let’s take a look:

  1. In [19]: data = np.random.RandomState(0).rand(2, 3, 4)
  2.  
  3. In [20]: items = list('ab')
  4.  
  5. In [21]: major_axis = list('mno')
  6.  
  7. In [22]: minor_axis = pd.date_range(start='2000', periods=4, name='date')

With old versions of pandas (prior to 0.25), this could stored in a Panel:

  1. In [23]: pd.Panel(data, items, major_axis, minor_axis)
  2. Out[23]: 
  3. <class 'pandas.core.panel.Panel'>
  4. Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
  5. Items axis: a to b
  6. Major_axis axis: m to o
  7. Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-04 00:00:00

To put this data in a DataArray, write:

  1. In [24]: array = xr.DataArray(data, [items, major_axis, minor_axis])
  2.  
  3. In [25]: array
  4. Out[25]: 
  5. <xarray.DataArray (dim_0: 2, dim_1: 3, date: 4)>
  6. array([[[0.548814, 0.715189, 0.602763, 0.544883],
  7.         [0.423655, 0.645894, 0.437587, 0.891773],
  8.         [0.963663, 0.383442, 0.791725, 0.528895]],
  9.  
  10.        [[0.568045, 0.925597, 0.071036, 0.087129],
  11.         [0.020218, 0.83262 , 0.778157, 0.870012],
  12.         [0.978618, 0.799159, 0.461479, 0.780529]]])
  13. Coordinates:
  14.   * dim_0    (dim_0) <U1 'a' 'b'
  15.   * dim_1    (dim_1) <U1 'm' 'n' 'o'
  16.   * date     (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04

As you can see, there are three dimensions (each is also a coordinate). Two ofthe axes of were unnamed, so have been assigned dim_0 and dim_1respectively, while the third retains its name date.

You can also easily convert this data into Dataset:

  1. In [26]: array.to_dataset(dim='dim_0')
  2. Out[26]: 
  3. <xarray.Dataset>
  4. Dimensions:  (date: 4, dim_1: 3)
  5. Coordinates:
  6.   * dim_1    (dim_1) <U1 'm' 'n' 'o'
  7.   * date     (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
  8. Data variables:
  9.     a        (dim_1, date) float64 0.5488 0.7152 0.6028 ... 0.3834 0.7917 0.5289
  10.     b        (dim_1, date) float64 0.568 0.9256 0.07104 ... 0.7992 0.4615 0.7805

Here, there are two data variables, each representing a DataFrame on panel’sitems axis, and labelled as such. Each variable is a 2D array of therespective values along the items dimension.

While the xarray docs are relatively complete, a few items stand out for Panel users:

  • A DataArray’s data is stored as a numpy array, and so can only contain a singletype. As a result, a Panel that contains DataFrame objectswith multiple types will be converted to dtype=object. A Dataset ofmultiple DataArray objects each with its own dtype will allow originaltypes to be preserved.

  • Indexing is similar to pandas, but more explicit andleverages xarray’s naming of dimensions.

  • Because of those features, making much higher dimensional data is verypractical.

  • Variables in Dataset objects can use a subset of its dimensions. Forexample, you can have one dataset with Person x Score x Time, and another withPerson x Score.

  • You can use coordinates are used for both dimensions and for variables whichlabel the data variables, so you could have a coordinate Age, that labelledthe Person dimension of a Dataset of Person x Score x Time.

While xarray may take some getting used to, it’s worth it! If anything is unclear,please post an issue on GitHub orStackOverflow,and we’ll endeavor to respond to the specific case or improve the general docs.