Extending pandas

While pandas provides a rich set of methods, containers, and data types, yourneeds may not be fully satisfied. Pandas offers a few options for extendingpandas.

Registering custom accessors

Libraries can use the decoratorspandas.api.extensions.register_dataframe_accessor(),pandas.api.extensions.register_series_accessor(), andpandas.api.extensions.register_index_accessor(), to add additional“namespaces” to pandas objects. All of these follow a similar convention: youdecorate a class, providing the name of attribute to add. The class’sinit method gets the object being decorated. For example:

  1. @pd.api.extensions.registerdataframeaccessor("geo")class GeoAccessor:    def __init(self, pandas_obj):        self._validate(pandas_obj)        self._obj = pandas_obj
  2. @staticmethod
  3. def _validate(obj):
  4.     # verify there is a column latitude and a column longitude
  5.     if 'latitude' not in obj.columns or 'longitude' not in obj.columns:
  6.         raise AttributeError("Must have 'latitude' and 'longitude'.")
  8. @property
  9. def center(self):
  10.     # return the geographic center point of this DataFrame
  11.     lat = self._obj.latitude
  12.     lon = self._obj.longitude
  13.     return (float(lon.mean()), float(lat.mean()))
  15. def plot(self):
  16.     # plot this array's data on a map, e.g., using Cartopy
  17.     pass

Now users can access your methods using the geo namespace:

  1. >>> ds = pd.DataFrame({'longitude': np.linspace(0, 10),
  2. ...                    'latitude': np.linspace(0, 20)})
  3. >>> ds.geo.center
  4. (5.0, 10.0)
  5. >>> ds.geo.plot()
  6. # plots data on a map

This can be a convenient way to extend pandas objects without subclassing them.If you write a custom accessor, make a pull request adding it to ourPandas ecosystem page.

We highly recommend validating the data in your accessor’s init.In our GeoAccessor, we validate that the data contains the expected columns,raising an AttributeError when the validation fails.For a Series accessor, you should validate the dtype if the accessorapplies only to certain dtypes.

Extension types

New in version 0.23.0.

Warning

The pandas.api.extensions.ExtensionDtype and pandas.api.extensions.ExtensionArray APIs are new andexperimental. They may change between versions without warning.

Pandas defines an interface for implementing data types and arrays that _extend_NumPy’s type system. Pandas itself uses the extension system for some typesthat aren’t built into NumPy (categorical, period, interval, datetime withtimezone).

Libraries can define a custom array and data type. When pandas encounters theseobjects, they will be handled properly (i.e. not converted to an ndarray ofobjects). Many methods like pandas.isna() will dispatch to the extensiontype’s implementation.

If you’re building a library that implements the interface, please publicize iton Extension data types.

The interface consists of two classes.

ExtensionDtype

A pandas.api.extensions.ExtensionDtype is similar to a numpy.dtype object. It describes thedata type. Implementors are responsible for a few unique items like the name.

One particularly important item is the type property. This should be theclass that is the scalar type for your data. For example, if you were writing anextension array for IP Address data, this might be ipaddress.IPv4Address.

See the extension dtype source for interface definition.

New in version 0.24.0.

pandas.api.extension.ExtensionDtype can be registered to pandas to allow creation via a string dtype name.This allows one to instantiate Series and .astype() with a registered string name, forexample 'category' is a registered string accessor for the CategoricalDtype.

See the extension dtype dtypes for more on how to register dtypes.

ExtensionArray

This class provides all the array-like functionality. ExtensionArrays arelimited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via thedtype attribute.

Pandas makes no restrictions on how an extension array is created via itsnew or init, and puts no restrictions on how you store yourdata. We do require that your array be convertible to a NumPy array, even ifthis is relatively expensive (as it is for Categorical).

They may be backed by none, one, or many NumPy arrays. For example,pandas.Categorical is an extension array backed by two arrays,one for codes and one for categories. An array of IPv6 addresses maybe backed by a NumPy structured array with two fields, one for thelower 64 bits and one for the upper 64 bits. Or they may be backedby some other storage type, like Python lists.

See the extension array source for the interface definition. The docstringsand comments contain guidance for properly implementing the interface.

ExtensionArray Operator Support

New in version 0.24.0.

By default, there are no operators defined for the class ExtensionArray.There are two approaches for providing operator support for your ExtensionArray:

  • Define each of the operators on your ExtensionArray subclass.
  • Use an operator implementation from pandas that depends on operators that are already definedon the underlying elements (scalars) of the ExtensionArray.

Note

Regardless of the approach, you may want to set array_priorityif you want your implementation to be called when involved in binary operationswith NumPy arrays.

For the first approach, you define selected operators, e.g., add, le, etc. thatyou want your ExtensionArray subclass to support.

The second approach assumes that the underlying elements (i.e., scalar type) of the ExtensionArrayhave the individual operators already defined. In other words, if your ExtensionArraynamed MyExtensionArray is implemented so that each element is an instanceof the class MyExtensionElement, then if the operators are definedfor MyExtensionElement, the second approach will automaticallydefine the operators for MyExtensionArray.

A mixin class, ExtensionScalarOpsMixin supports this secondapproach. If developing an ExtensionArray subclass, for example MyExtensionArray,can simply include ExtensionScalarOpsMixin as a parent class of MyExtensionArray,and then call the methods _add_arithmetic_ops() and/or_add_comparison_ops() to hook the operators intoyour MyExtensionArray class, as follows:

  1. from pandas.api.extensions import ExtensionArray, ExtensionScalarOpsMixin
  2.  
  3. class MyExtensionArray(ExtensionArray, ExtensionScalarOpsMixin):
  4.     pass
  5.  
  6.  
  7. MyExtensionArray._add_arithmetic_ops()
  8. MyExtensionArray._add_comparison_ops()

Note

Since pandas automatically calls the underlying operator on eachelement one-by-one, this might not be as performant as implementing your ownversion of the associated operators directly on the ExtensionArray.

For arithmetic operations, this implementation will try to reconstruct a newExtensionArray with the result of the element-wise operation. Whetheror not that succeeds depends on whether the operation returns a resultthat’s valid for the ExtensionArray. If an ExtensionArray cannotbe reconstructed, an ndarray containing the scalars returned instead.

For ease of implementation and consistency with operations between pandasand NumPy ndarrays, we recommend not handling Series and Indexes in your binary ops.Instead, you should detect these cases and return NotImplemented.When pandas encounters an operation like op(Series, ExtensionArray), pandaswill

  • unbox the array from the Series (Series.array)
  • call result = op(values, ExtensionArray)
  • re-box the result in a Series

NumPy Universal Functions

Series implements array_ufunc. As part of the implementation,pandas unboxes the ExtensionArray from the Series, applies the ufunc,and re-boxes it if necessary.

If applicable, we highly recommend that you implement array_ufunc in yourextension array to avoid coercion to an ndarray. Seethe numpy documentationfor an example.

As part of your implementation, we require that you defer to pandas when a pandascontainer (Series, DataFrame, Index) is detected in inputs.If any of those is present, you should return NotImplemented. Pandas will take care ofunboxing the array from the container and re-calling the ufunc with the unwrapped input.

Testing extension arrays

We provide a test suite for ensuring that your extension arrays satisfy the expectedbehavior. To use the test suite, you must provide several pytest fixtures and inheritfrom the base test class. The required fixtures are found inhttps://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/conftest.py.

To use a test, subclass it:

  1. from pandas.tests.extension import base
  2.  
  3.  
  4. class TestConstructors(base.BaseConstructorsTests):
  5.     pass

See https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/base/init.pyfor a list of all the tests available.

Subclassing pandas data structures

Warning

There are some easier alternatives before considering subclassing pandas data structures.

This section describes how to subclass pandas data structures to meet more specific needs. There are two points that need attention:

  • Override constructor properties.
  • Define original properties

Note

You can find a nice example in geopandas project.

Override constructor properties

Each data structure has several constructor properties for returning a newdata structure as the result of an operation. By overriding these properties,you can retain subclasses through pandas data manipulations.

There are 3 constructor properties to be defined:

  • _constructor: Used when a manipulation result has the same dimensions as the original.
  • _constructor_sliced: Used when a manipulation result has one lower dimension(s) as the original, such as DataFrame single columns slicing.
  • _constructor_expanddim: Used when a manipulation result has one higher dimension as the original, such as Series.to_frame().

Following table shows how pandas data structures define constructor properties by default.

Property AttributesSeriesDataFrame
_constructorSeriesDataFrame
_constructor_slicedNotImplementedErrorSeries
_constructor_expanddimDataFrameNotImplementedError

Below example shows how to define SubclassedSeries and SubclassedDataFrame overriding constructor properties.

  1. class SubclassedSeries(pd.Series):
  2.  
  3.     @property
  4.     def _constructor(self):
  5.         return SubclassedSeries
  6.  
  7.     @property
  8.     def _constructor_expanddim(self):
  9.         return SubclassedDataFrame
  10.  
  11.  
  12. class SubclassedDataFrame(pd.DataFrame):
  13.  
  14.     @property
  15.     def _constructor(self):
  16.         return SubclassedDataFrame
  17.  
  18.     @property
  19.     def _constructor_sliced(self):
  20.         return SubclassedSeries
  1. >>> s = SubclassedSeries([1, 2, 3])
  2. >>> type(s)
  3. <class '__main__.SubclassedSeries'>
  4.  
  5. >>> to_framed = s.to_frame()
  6. >>> type(to_framed)
  7. <class '__main__.SubclassedDataFrame'>
  8.  
  9. >>> df = SubclassedDataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})
  10. >>> df
  11.    A  B  C
  12. 0  1  4  7
  13. 1  2  5  8
  14. 2  3  6  9
  15.  
  16. >>> type(df)
  17. <class '__main__.SubclassedDataFrame'>
  18.  
  19. >>> sliced1 = df[['A', 'B']]
  20. >>> sliced1
  21.    A  B
  22. 0  1  4
  23. 1  2  5
  24. 2  3  6
  25.  
  26. >>> type(sliced1)
  27. <class '__main__.SubclassedDataFrame'>
  28.  
  29. >>> sliced2 = df['A']
  30. >>> sliced2
  31. 0    1
  32. 1    2
  33. 2    3
  34. Name: A, dtype: int64
  35.  
  36. >>> type(sliced2)
  37. <class '__main__.SubclassedSeries'>

Define original properties

To let original data structures have additional properties, you should let pandas know what properties are added. pandas maps unknown properties to data names overriding getattribute. Defining original properties can be done in one of 2 ways:

  • Define _internal_names and _internal_names_set for temporary properties which WILL NOT be passed to manipulation results.
  • Define _metadata for normal properties which will be passed to manipulation results.Below is an example to define two original properties, “internal_cache” as a temporary property and “added_property” as a normal property
  1. class SubclassedDataFrame2(pd.DataFrame):
  2.  
  3.     # temporary properties
  4.     _internal_names = pd.DataFrame._internal_names + ['internal_cache']
  5.     _internal_names_set = set(_internal_names)
  6.  
  7.     # normal properties
  8.     _metadata = ['added_property']
  9.  
  10.     @property
  11.     def _constructor(self):
  12.         return SubclassedDataFrame2
  1. >>> df = SubclassedDataFrame2({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})
  2. >>> df
  3.    A  B  C
  4. 0  1  4  7
  5. 1  2  5  8
  6. 2  3  6  9
  7.  
  8. >>> df.internal_cache = 'cached'
  9. >>> df.added_property = 'property'
  10.  
  11. >>> df.internal_cache
  12. cached
  13. >>> df.added_property
  14. property
  15.  
  16. # properties defined in _internal_names is reset after manipulation
  17. >>> df[['A', 'B']].internal_cache
  18. AttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache'
  19.  
  20. # properties defined in _metadata are retained
  21. >>> df[['A', 'B']].added_property
  22. property

Plotting backends

Starting in 0.25 pandas can be extended with third-party plotting backends. Themain idea is letting users select a plotting backend different than the providedone based on Matplotlib. For example:

  1. >>> pd.set_option('plotting.backend', 'backend.module')
  2. >>> pd.Series([1, 2, 3]).plot()

This would be more or less equivalent to:

  1. >>> import backend.module
  2. >>> backend.module.plot(pd.Series([1, 2, 3]))

The backend module can then use other visualization tools (Bokeh, Altair,…)to generate the plots.

Libraries implementing the plotting backend should use entry pointsto make their backend discoverable to pandas. The key is "pandas_plotting_backends". For example, pandasregisters the default “matplotlib” backend as follows.

  1. # in setup.py
  2. setup(  # noqa: F821
  3.     ...,
  4.     entry_points={
  5.         "pandas_plotting_backends": [
  6.             "matplotlib = pandas:plotting._matplotlib",
  7.         ],
  8.     },
  9. )

More information on how to implement a third-party plotting backend can be found athttps://github.com/pandas-dev/pandas/blob/master/pandas/plotting/init.py#L1.