Intro to data structures

We’ll start with a quick, non-comprehensive overview of the fundamental datastructures in pandas to get you started. The fundamental behavior about datatypes, indexing, and axis labeling / alignment apply across all of theobjects. To get started, import NumPy and load pandas into your namespace:

  1. In [1]: import numpy as np
  2.  
  3. In [2]: import pandas as pd

Here is a basic tenet to keep in mind: data alignment is intrinsic. The linkbetween labels and data will not be broken unless done so explicitly by you.

We’ll give a brief intro to the data structures, then consider all of the broadcategories of functionality and methods in separate sections.

Series

Series is a one-dimensional labeled array capable of holding any datatype (integers, strings, floating point numbers, Python objects, etc.). The axislabels are collectively referred to as the index. The basic method to create a Series is to call:

  1. >>> s = pd.Series(data, index=index)

Here, data can be many different things:

  • a Python dict
  • an ndarray
  • a scalar value (like 5)

The passed index is a list of axis labels. Thus, this separates into a fewcases depending on what data is:

From ndarray

If data is an ndarray, index must be the same length as data. If noindex is passed, one will be created having values [0, …, len(data) - 1].

  1. In [3]: s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
  2.  
  3. In [4]: s
  4. Out[4]: 
  5. a    0.469112
  6. b   -0.282863
  7. c   -1.509059
  8. d   -1.135632
  9. e    1.212112
  10. dtype: float64
  11.  
  12. In [5]: s.index
  13. Out[5]: Index(['a', 'b', 'c', 'd', 'e'], dtype='object')
  14.  
  15. In [6]: pd.Series(np.random.randn(5))
  16. Out[6]: 
  17. 0   -0.173215
  18. 1    0.119209
  19. 2   -1.044236
  20. 3   -0.861849
  21. 4   -2.104569
  22. dtype: float64

Note

pandas supports non-unique index values. If an operationthat does not support duplicate index values is attempted, an exceptionwill be raised at that time. The reason for being lazy is nearly all performance-based(there are many instances in computations, like parts of GroupBy, where the indexis not used).

From dict

Series can be instantiated from dicts:

  1. In [7]: d = {'b': 1, 'a': 0, 'c': 2}
  2.  
  3. In [8]: pd.Series(d)
  4. Out[8]: 
  5. b    1
  6. a    0
  7. c    2
  8. dtype: int64

Note

When the data is a dict, and an index is not passed, the Series indexwill be ordered by the dict’s insertion order, if you’re using Pythonversion >= 3.6 and Pandas version >= 0.23.

If you’re using Python < 3.6 or Pandas < 0.23, and an index is not passed,the Series index will be the lexically ordered list of dict keys.

In the example above, if you were on a Python version lower than 3.6 or aPandas version lower than 0.23, the Series would be ordered by the lexicalorder of the dict keys (i.e. ['a', 'b', 'c'] rather than ['b', 'a', 'c']).

If an index is passed, the values in data corresponding to the labels in theindex will be pulled out.

  1. In [9]: d = {'a': 0., 'b': 1., 'c': 2.}
  2.  
  3. In [10]: pd.Series(d)
  4. Out[10]: 
  5. a    0.0
  6. b    1.0
  7. c    2.0
  8. dtype: float64
  9.  
  10. In [11]: pd.Series(d, index=['b', 'c', 'd', 'a'])
  11. Out[11]: 
  12. b    1.0
  13. c    2.0
  14. d    NaN
  15. a    0.0
  16. dtype: float64

Note

NaN (not a number) is the standard missing data marker used in pandas.

From scalar value

If data is a scalar value, an index must beprovided. The value will be repeated to match the length of index.

  1. In [12]: pd.Series(5., index=['a', 'b', 'c', 'd', 'e'])
  2. Out[12]: 
  3. a    5.0
  4. b    5.0
  5. c    5.0
  6. d    5.0
  7. e    5.0
  8. dtype: float64

Series is ndarray-like

Series acts very similarly to a ndarray, and is a valid argument to most NumPy functions.However, operations such as slicing will also slice the index.

  1. In [13]: s[0]
  2. Out[13]: 0.4691122999071863
  3.  
  4. In [14]: s[:3]
  5. Out[14]: 
  6. a    0.469112
  7. b   -0.282863
  8. c   -1.509059
  9. dtype: float64
  10.  
  11. In [15]: s[s > s.median()]
  12. Out[15]: 
  13. a    0.469112
  14. e    1.212112
  15. dtype: float64
  16.  
  17. In [16]: s[[4, 3, 1]]
  18. Out[16]: 
  19. e    1.212112
  20. d   -1.135632
  21. b   -0.282863
  22. dtype: float64
  23.  
  24. In [17]: np.exp(s)
  25. Out[17]: 
  26. a    1.598575
  27. b    0.753623
  28. c    0.221118
  29. d    0.321219
  30. e    3.360575
  31. dtype: float64

Note

We will address array-based indexing like s[[4, 3, 1]]in section.

Like a NumPy array, a pandas Series has a dtype.

  1. In [18]: s.dtype
  2. Out[18]: dtype('float64')

This is often a NumPy dtype. However, pandas and 3rd-party librariesextend NumPy’s type system in a few places, in which case the dtype wouldbe a ExtensionDtype. Some examples withinpandas are Categorical data and Nullable integer data type. See dtypesfor more.

If you need the actual array backing a Series, use Series.array.

  1. In [19]: s.array
  2. Out[19]: 
  3. <PandasArray>
  4. [ 0.4691122999071863, -0.2828633443286633, -1.5090585031735124,
  5.  -1.1356323710171934,  1.2121120250208506]
  6. Length: 5, dtype: float64

Accessing the array can be useful when you need to do some operation without theindex (to disable automatic alignment, for example).

Series.array will always be an ExtensionArray.Briefly, an ExtensionArray is a thin wrapper around one or more concrete arrays like anumpy.ndarray. Pandas knows how to take an ExtensionArray andstore it in a Series or a column of a DataFrame.See dtypes for more.

While Series is ndarray-like, if you need an actual ndarray, then useSeries.to_numpy().

  1. In [20]: s.to_numpy()
  2. Out[20]: array([ 0.4691, -0.2829, -1.5091, -1.1356,  1.2121])

Even if the Series is backed by a ExtensionArray,Series.to_numpy() will return a NumPy ndarray.

Series is dict-like

A Series is like a fixed-size dict in that you can get and set values by indexlabel:

  1. In [21]: s['a']
  2. Out[21]: 0.4691122999071863
  3.  
  4. In [22]: s['e'] = 12.
  5.  
  6. In [23]: s
  7. Out[23]: 
  8. a     0.469112
  9. b    -0.282863
  10. c    -1.509059
  11. d    -1.135632
  12. e    12.000000
  13. dtype: float64
  14.  
  15. In [24]: 'e' in s
  16. Out[24]: True
  17.  
  18. In [25]: 'f' in s
  19. Out[25]: False

If a label is not contained, an exception is raised:

  1. >>> s['f']
  2. KeyError: 'f'

Using the get method, a missing label will return None or specified default:

  1. In [26]: s.get('f')
  2.  
  3. In [27]: s.get('f', np.nan)
  4. Out[27]: nan

See also the section on attribute access.

Vectorized operations and label alignment with Series

When working with raw NumPy arrays, looping through value-by-value is usuallynot necessary. The same is true when working with Series in pandas.Series can also be passed into most NumPy methods expecting an ndarray.

  1. In [28]: s + s
  2. Out[28]: 
  3. a     0.938225
  4. b    -0.565727
  5. c    -3.018117
  6. d    -2.271265
  7. e    24.000000
  8. dtype: float64
  9.  
  10. In [29]: s * 2
  11. Out[29]: 
  12. a     0.938225
  13. b    -0.565727
  14. c    -3.018117
  15. d    -2.271265
  16. e    24.000000
  17. dtype: float64
  18.  
  19. In [30]: np.exp(s)
  20. Out[30]: 
  21. a         1.598575
  22. b         0.753623
  23. c         0.221118
  24. d         0.321219
  25. e    162754.791419
  26. dtype: float64

A key difference between Series and ndarray is that operations between Seriesautomatically align the data based on label. Thus, you can write computationswithout giving consideration to whether the Series involved have the samelabels.

  1. In [31]: s[1:] + s[:-1]
  2. Out[31]: 
  3. a         NaN
  4. b   -0.565727
  5. c   -3.018117
  6. d   -2.271265
  7. e         NaN
  8. dtype: float64

The result of an operation between unaligned Series will have the union ofthe indexes involved. If a label is not found in one Series or the other, theresult will be marked as missing NaN. Being able to write code without doingany explicit data alignment grants immense freedom and flexibility ininteractive data analysis and research. The integrated data alignment featuresof the pandas data structures set pandas apart from the majority of relatedtools for working with labeled data.

Note

In general, we chose to make the default result of operations betweendifferently indexed objects yield the union of the indexes in order toavoid loss of information. Having an index label, though the data ismissing, is typically important information as part of a computation. Youof course have the option of dropping labels with missing data via thedropna function.

Name attribute

Series can also have a name attribute:

  1. In [32]: s = pd.Series(np.random.randn(5), name='something')
  2.  
  3. In [33]: s
  4. Out[33]: 
  5. 0   -0.494929
  6. 1    1.071804
  7. 2    0.721555
  8. 3   -0.706771
  9. 4   -1.039575
  10. Name: something, dtype: float64
  11.  
  12. In [34]: s.name
  13. Out[34]: 'something'

The Series name will be assigned automatically in many cases, in particularwhen taking 1D slices of DataFrame as you will see below.

New in version 0.18.0.

You can rename a Series with the pandas.Series.rename() method.

  1. In [35]: s2 = s.rename("different")
  2.  
  3. In [36]: s2.name
  4. Out[36]: 'different'

Note that s and s2 refer to different objects.

DataFrame

DataFrame is a 2-dimensional labeled data structure with columns ofpotentially different types. You can think of it like a spreadsheet or SQLtable, or a dict of Series objects. It is generally the most commonly usedpandas object. Like Series, DataFrame accepts many different kinds of input:

  • Dict of 1D ndarrays, lists, dicts, or Series
  • 2-D numpy.ndarray
  • Structured or record ndarray
  • A Series
  • Another DataFrame

Along with the data, you can optionally pass index (row labels) andcolumns (column labels) arguments. If you pass an index and / or columns,you are guaranteeing the index and / or columns of the resultingDataFrame. Thus, a dict of Series plus a specific index will discard all datanot matching up to the passed index.

If axis labels are not passed, they will be constructed from the input databased on common sense rules.

Note

When the data is a dict, and columns is not specified, the DataFramecolumns will be ordered by the dict’s insertion order, if you are usingPython version >= 3.6 and Pandas >= 0.23.

If you are using Python < 3.6 or Pandas < 0.23, and columns is notspecified, the DataFrame columns will be the lexically ordered list of dictkeys.

From dict of Series or dicts

The resulting index will be the union of the indexes of the variousSeries. If there are any nested dicts, these will first be converted toSeries. If no columns are passed, the columns will be the ordered list of dictkeys.

  1. In [37]: d = {'one': pd.Series([1., 2., 3.], index=['a', 'b', 'c']),
  2.    ....:      'two': pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
  3.    ....: 
  4.  
  5. In [38]: df = pd.DataFrame(d)
  6.  
  7. In [39]: df
  8. Out[39]: 
  9.    one  two
  10. a  1.0  1.0
  11. b  2.0  2.0
  12. c  3.0  3.0
  13. d  NaN  4.0
  14.  
  15. In [40]: pd.DataFrame(d, index=['d', 'b', 'a'])
  16. Out[40]: 
  17.    one  two
  18. d  NaN  4.0
  19. b  2.0  2.0
  20. a  1.0  1.0
  21.  
  22. In [41]: pd.DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
  23. Out[41]: 
  24.    two three
  25. d  4.0   NaN
  26. b  2.0   NaN
  27. a  1.0   NaN

The row and column labels can be accessed respectively by accessing theindex and columns attributes:

Note

When a particular set of columns is passed along with a dict of data, thepassed columns override the keys in the dict.

  1. In [42]: df.index
  2. Out[42]: Index(['a', 'b', 'c', 'd'], dtype='object')
  3.  
  4. In [43]: df.columns
  5. Out[43]: Index(['one', 'two'], dtype='object')

From dict of ndarrays / lists

The ndarrays must all be the same length. If an index is passed, it mustclearly also be the same length as the arrays. If no index is passed, theresult will be range(n), where n is the array length.

  1. In [44]: d = {'one': [1., 2., 3., 4.],
  2.    ....:      'two': [4., 3., 2., 1.]}
  3.    ....: 
  4.  
  5. In [45]: pd.DataFrame(d)
  6. Out[45]: 
  7.    one  two
  8. 0  1.0  4.0
  9. 1  2.0  3.0
  10. 2  3.0  2.0
  11. 3  4.0  1.0
  12.  
  13. In [46]: pd.DataFrame(d, index=['a', 'b', 'c', 'd'])
  14. Out[46]: 
  15.    one  two
  16. a  1.0  4.0
  17. b  2.0  3.0
  18. c  3.0  2.0
  19. d  4.0  1.0

From structured or record array

This case is handled identically to a dict of arrays.

  1. In [47]: data = np.zeros((2, ), dtype=[('A', 'i4'), ('B', 'f4'), ('C', 'a10')])
  2.  
  3. In [48]: data[:] = [(1, 2., 'Hello'), (2, 3., "World")]
  4.  
  5. In [49]: pd.DataFrame(data)
  6. Out[49]: 
  7.    A    B         C
  8. 0  1  2.0  b'Hello'
  9. 1  2  3.0  b'World'
  10.  
  11. In [50]: pd.DataFrame(data, index=['first', 'second'])
  12. Out[50]: 
  13.         A    B         C
  14. first   1  2.0  b'Hello'
  15. second  2  3.0  b'World'
  16.  
  17. In [51]: pd.DataFrame(data, columns=['C', 'A', 'B'])
  18. Out[51]: 
  19.           C  A    B
  20. 0  b'Hello'  1  2.0
  21. 1  b'World'  2  3.0

Note

DataFrame is not intended to work exactly like a 2-dimensional NumPyndarray.

From a list of dicts

  1. In [52]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]
  2.  
  3. In [53]: pd.DataFrame(data2)
  4. Out[53]: 
  5.    a   b     c
  6. 0  1   2   NaN
  7. 1  5  10  20.0
  8.  
  9. In [54]: pd.DataFrame(data2, index=['first', 'second'])
  10. Out[54]: 
  11.         a   b     c
  12. first   1   2   NaN
  13. second  5  10  20.0
  14.  
  15. In [55]: pd.DataFrame(data2, columns=['a', 'b'])
  16. Out[55]: 
  17.    a   b
  18. 0  1   2
  19. 1  5  10

From a dict of tuples

You can automatically create a MultiIndexed frame by passing a tuplesdictionary.

  1. In [56]: pd.DataFrame({('a', 'b'): {('A', 'B'): 1, ('A', 'C'): 2},
  2.    ....:               ('a', 'a'): {('A', 'C'): 3, ('A', 'B'): 4},
  3.    ....:               ('a', 'c'): {('A', 'B'): 5, ('A', 'C'): 6},
  4.    ....:               ('b', 'a'): {('A', 'C'): 7, ('A', 'B'): 8},
  5.    ....:               ('b', 'b'): {('A', 'D'): 9, ('A', 'B'): 10}})
  6.    ....: 
  7. Out[56]: 
  8.        a              b      
  9.        b    a    c    a     b
  10. A B  1.0  4.0  5.0  8.0  10.0
  11.   C  2.0  3.0  6.0  7.0   NaN
  12.   D  NaN  NaN  NaN  NaN   9.0

From a Series

The result will be a DataFrame with the same index as the input Series, andwith one column whose name is the original name of the Series (only if no othercolumn name provided).

Missing data

Much more will be said on this topic in the Missing datasection. To construct a DataFrame with missing data, we use np.nan torepresent missing values. Alternatively, you may pass a numpy.MaskedArrayas the data argument to the DataFrame constructor, and its masked entries willbe considered missing.

Alternate constructors

DataFrame.from_dict

DataFrame.from_dict takes a dict of dicts or a dict of array-like sequencesand returns a DataFrame. It operates like the DataFrame constructor exceptfor the orient parameter which is 'columns' by default, but which can beset to 'index' in order to use the dict keys as row labels.

  1. In [57]: pd.DataFrame.from_dict(dict([('A', [1, 2, 3]), ('B', [4, 5, 6])]))
  2. Out[57]: 
  3.    A  B
  4. 0  1  4
  5. 1  2  5
  6. 2  3  6

If you pass orient='index', the keys will be the row labels. In thiscase, you can also pass the desired column names:

  1. In [58]: pd.DataFrame.from_dict(dict([('A', [1, 2, 3]), ('B', [4, 5, 6])]),
  2.    ....:                        orient='index', columns=['one', 'two', 'three'])
  3.    ....: 
  4. Out[58]: 
  5.    one  two  three
  6. A    1    2      3
  7. B    4    5      6

DataFrame.from_records

DataFrame.from_records takes a list of tuples or an ndarray with structureddtype. It works analogously to the normal DataFrame constructor, except thatthe resulting DataFrame index may be a specific field of the structureddtype. For example:

  1. In [59]: data
  2. Out[59]: 
  3. array([(1, 2., b'Hello'), (2, 3., b'World')],
  4.       dtype=[('A', '<i4'), ('B', '<f4'), ('C', 'S10')])
  5.  
  6. In [60]: pd.DataFrame.from_records(data, index='C')
  7. Out[60]: 
  8.           A    B
  9. C               
  10. b'Hello'  1  2.0
  11. b'World'  2  3.0

Column selection, addition, deletion

You can treat a DataFrame semantically like a dict of like-indexed Seriesobjects. Getting, setting, and deleting columns works with the same syntax asthe analogous dict operations:

  1. In [61]: df['one']
  2. Out[61]: 
  3. a    1.0
  4. b    2.0
  5. c    3.0
  6. d    NaN
  7. Name: one, dtype: float64
  8.  
  9. In [62]: df['three'] = df['one'] * df['two']
  10.  
  11. In [63]: df['flag'] = df['one'] > 2
  12.  
  13. In [64]: df
  14. Out[64]: 
  15.    one  two  three   flag
  16. a  1.0  1.0    1.0  False
  17. b  2.0  2.0    4.0  False
  18. c  3.0  3.0    9.0   True
  19. d  NaN  4.0    NaN  False

Columns can be deleted or popped like with a dict:

  1. In [65]: del df['two']
  2.  
  3. In [66]: three = df.pop('three')
  4.  
  5. In [67]: df
  6. Out[67]: 
  7.    one   flag
  8. a  1.0  False
  9. b  2.0  False
  10. c  3.0   True
  11. d  NaN  False

When inserting a scalar value, it will naturally be propagated to fill thecolumn:

  1. In [68]: df['foo'] = 'bar'
  2.  
  3. In [69]: df
  4. Out[69]: 
  5.    one   flag  foo
  6. a  1.0  False  bar
  7. b  2.0  False  bar
  8. c  3.0   True  bar
  9. d  NaN  False  bar

When inserting a Series that does not have the same index as the DataFrame, itwill be conformed to the DataFrame’s index:

  1. In [70]: df['one_trunc'] = df['one'][:2]
  2.  
  3. In [71]: df
  4. Out[71]: 
  5.    one   flag  foo  one_trunc
  6. a  1.0  False  bar        1.0
  7. b  2.0  False  bar        2.0
  8. c  3.0   True  bar        NaN
  9. d  NaN  False  bar        NaN

You can insert raw ndarrays but their length must match the length of theDataFrame’s index.

By default, columns get inserted at the end. The insert function isavailable to insert at a particular location in the columns:

  1. In [72]: df.insert(1, 'bar', df['one'])
  2.  
  3. In [73]: df
  4. Out[73]: 
  5.    one  bar   flag  foo  one_trunc
  6. a  1.0  1.0  False  bar        1.0
  7. b  2.0  2.0  False  bar        2.0
  8. c  3.0  3.0   True  bar        NaN
  9. d  NaN  NaN  False  bar        NaN

Assigning new columns in method chains

Inspired by dplyr’smutate verb, DataFrame has an assign()method that allows you to easily create new columns that are potentiallyderived from existing columns.

  1. In [74]: iris = pd.read_csv('data/iris.data')
  2.  
  3. In [75]: iris.head()
  4. Out[75]: 
  5.    SepalLength  SepalWidth  PetalLength  PetalWidth         Name
  6. 0          5.1         3.5          1.4         0.2  Iris-setosa
  7. 1          4.9         3.0          1.4         0.2  Iris-setosa
  8. 2          4.7         3.2          1.3         0.2  Iris-setosa
  9. 3          4.6         3.1          1.5         0.2  Iris-setosa
  10. 4          5.0         3.6          1.4         0.2  Iris-setosa
  11.  
  12. In [76]: (iris.assign(sepal_ratio=iris['SepalWidth'] / iris['SepalLength'])
  13.    ....:      .head())
  14.    ....: 
  15. Out[76]: 
  16.    SepalLength  SepalWidth  PetalLength  PetalWidth         Name  sepal_ratio
  17. 0          5.1         3.5          1.4         0.2  Iris-setosa     0.686275
  18. 1          4.9         3.0          1.4         0.2  Iris-setosa     0.612245
  19. 2          4.7         3.2          1.3         0.2  Iris-setosa     0.680851
  20. 3          4.6         3.1          1.5         0.2  Iris-setosa     0.673913
  21. 4          5.0         3.6          1.4         0.2  Iris-setosa     0.720000

In the example above, we inserted a precomputed value. We can also pass ina function of one argument to be evaluated on the DataFrame being assigned to.

  1. In [77]: iris.assign(sepal_ratio=lambda x: (x['SepalWidth'] / x['SepalLength'])).head()
  2. Out[77]: 
  3.    SepalLength  SepalWidth  PetalLength  PetalWidth         Name  sepal_ratio
  4. 0          5.1         3.5          1.4         0.2  Iris-setosa     0.686275
  5. 1          4.9         3.0          1.4         0.2  Iris-setosa     0.612245
  6. 2          4.7         3.2          1.3         0.2  Iris-setosa     0.680851
  7. 3          4.6         3.1          1.5         0.2  Iris-setosa     0.673913
  8. 4          5.0         3.6          1.4         0.2  Iris-setosa     0.720000

assign always returns a copy of the data, leaving the originalDataFrame untouched.

Passing a callable, as opposed to an actual value to be inserted, isuseful when you don’t have a reference to the DataFrame at hand. This iscommon when using assign in a chain of operations. For example,we can limit the DataFrame to just those observations with a Sepal Lengthgreater than 5, calculate the ratio, and plot:

  1. In [78]: (iris.query('SepalLength > 5')
  2.    ....:      .assign(SepalRatio=lambda x: x.SepalWidth / x.SepalLength,
  3.    ....:              PetalRatio=lambda x: x.PetalWidth / x.PetalLength)
  4.    ....:      .plot(kind='scatter', x='SepalRatio', y='PetalRatio'))
  5.    ....: 
  6. Out[78]: <matplotlib.axes._subplots.AxesSubplot at 0x7f453ca05f90>

../_images/basics_assign.pngSince a function is passed in, the function is computed on the DataFramebeing assigned to. Importantly, this is the DataFrame that’s been filteredto those rows with sepal length greater than 5. The filtering happens first,and then the ratio calculations. This is an example where we didn’thave a reference to the filtered DataFrame available.

The function signature for assign is simply **kwargs. The keysare the column names for the new fields, and the values are either a valueto be inserted (for example, a Series or NumPy array), or a functionof one argument to be called on the DataFrame. A copy of the originalDataFrame is returned, with the new values inserted.

Changed in version 0.23.0.

Starting with Python 3.6 the order of kwargs is preserved. This allowsfor dependent assignment, where an expression later in kwargs can referto a column created earlier in the same assign().

  1. In [79]: dfa = pd.DataFrame({"A": [1, 2, 3],
  2.    ....:                     "B": [4, 5, 6]})
  3.    ....: 
  4.  
  5. In [80]: dfa.assign(C=lambda x: x['A'] + x['B'],
  6.    ....:            D=lambda x: x['A'] + x['C'])
  7.    ....: 
  8. Out[80]: 
  9.    A  B  C   D
  10. 0  1  4  5   6
  11. 1  2  5  7   9
  12. 2  3  6  9  12

In the second expression, x['C'] will refer to the newly created column,that’s equal to dfa['A'] + dfa['B'].

To write code compatible with all versions of Python, split the assignment in two.

  1. In [81]: dependent = pd.DataFrame({"A": [1, 1, 1]})
  2.  
  3. In [82]: (dependent.assign(A=lambda x: x['A'] + 1)
  4.    ....:           .assign(B=lambda x: x['A'] + 2))
  5.    ....: 
  6. Out[82]: 
  7.    A  B
  8. 0  2  4
  9. 1  2  4
  10. 2  2  4

Warning

Dependent assignment may subtly change the behavior of your code betweenPython 3.6 and older versions of Python.

If you wish to write code that supports versions of python before and after 3.6,you’ll need to take care when passing assign expressions that

  • Update an existing column
  • Refer to the newly updated column in the same assign

For example, we’ll update column “A” and then refer to it when creating “B”.

  1. >>> dependent = pd.DataFrame({"A": [1, 1, 1]})
  2. >>> dependent.assign(A=lambda x: x["A"] + 1, B=lambda x: x["A"] + 2)

For Python 3.5 and earlier the expression creating B refers to the“old” value of A, [1, 1, 1]. The output is then

  1.    A  B
  2. 0  2  3
  3. 1  2  3
  4. 2  2  3

For Python 3.6 and later, the expression creating A refers to the“new” value of A, [2, 2, 2], which results in

  1.    A  B
  2. 0  2  4
  3. 1  2  4
  4. 2  2  4

Indexing / selection

The basics of indexing are as follows:

OperationSyntaxResult
Select columndf[col]Series
Select row by labeldf.loc[label]Series
Select row by integer locationdf.iloc[loc]Series
Slice rowsdf[5:10]DataFrame
Select rows by boolean vectordf[bool_vec]DataFrame

Row selection, for example, returns a Series whose index is the columns of theDataFrame:

  1. In [83]: df.loc['b']
  2. Out[83]: 
  3. one              2
  4. bar              2
  5. flag         False
  6. foo            bar
  7. one_trunc        2
  8. Name: b, dtype: object
  9.  
  10. In [84]: df.iloc[2]
  11. Out[84]: 
  12. one             3
  13. bar             3
  14. flag         True
  15. foo           bar
  16. one_trunc     NaN
  17. Name: c, dtype: object

For a more exhaustive treatment of sophisticated label-based indexing andslicing, see the section on indexing. We will address thefundamentals of reindexing / conforming to new sets of labels in thesection on reindexing.

Data alignment and arithmetic

Data alignment between DataFrame objects automatically align on both thecolumns and the index (row labels). Again, the resulting object will have theunion of the column and row labels.

  1. In [85]: df = pd.DataFrame(np.random.randn(10, 4), columns=['A', 'B', 'C', 'D'])
  2.  
  3. In [86]: df2 = pd.DataFrame(np.random.randn(7, 3), columns=['A', 'B', 'C'])
  4.  
  5. In [87]: df + df2
  6. Out[87]: 
  7.           A         B         C   D
  8. 0  0.045691 -0.014138  1.380871 NaN
  9. 1 -0.955398 -1.501007  0.037181 NaN
  10. 2 -0.662690  1.534833 -0.859691 NaN
  11. 3 -2.452949  1.237274 -0.133712 NaN
  12. 4  1.414490  1.951676 -2.320422 NaN
  13. 5 -0.494922 -1.649727 -1.084601 NaN
  14. 6 -1.047551 -0.748572 -0.805479 NaN
  15. 7       NaN       NaN       NaN NaN
  16. 8       NaN       NaN       NaN NaN
  17. 9       NaN       NaN       NaN NaN

When doing an operation between DataFrame and Series, the default behavior isto align the Series index on the DataFrame columns, thus broadcastingrow-wise. For example:

  1. In [88]: df - df.iloc[0]
  2. Out[88]: 
  3.           A         B         C         D
  4. 0  0.000000  0.000000  0.000000  0.000000
  5. 1 -1.359261 -0.248717 -0.453372 -1.754659
  6. 2  0.253128  0.829678  0.010026 -1.991234
  7. 3 -1.311128  0.054325 -1.724913 -1.620544
  8. 4  0.573025  1.500742 -0.676070  1.367331
  9. 5 -1.741248  0.781993 -1.241620 -2.053136
  10. 6 -1.240774 -0.869551 -0.153282  0.000430
  11. 7 -0.743894  0.411013 -0.929563 -0.282386
  12. 8 -1.194921  1.320690  0.238224 -1.482644
  13. 9  2.293786  1.856228  0.773289 -1.446531

In the special case of working with time series data, if the DataFrame indexcontains dates, the broadcasting will be column-wise:

  1. In [89]: index = pd.date_range('1/1/2000', periods=8)
  2.  
  3. In [90]: df = pd.DataFrame(np.random.randn(8, 3), index=index, columns=list('ABC'))
  4.  
  5. In [91]: df
  6. Out[91]: 
  7.                    A         B         C
  8. 2000-01-01 -1.226825  0.769804 -1.281247
  9. 2000-01-02 -0.727707 -0.121306 -0.097883
  10. 2000-01-03  0.695775  0.341734  0.959726
  11. 2000-01-04 -1.110336 -0.619976  0.149748
  12. 2000-01-05 -0.732339  0.687738  0.176444
  13. 2000-01-06  0.403310 -0.154951  0.301624
  14. 2000-01-07 -2.179861 -1.369849 -0.954208
  15. 2000-01-08  1.462696 -1.743161 -0.826591
  16.  
  17. In [92]: type(df['A'])
  18. Out[92]: pandas.core.series.Series
  19.  
  20. In [93]: df - df['A']
  21. Out[93]: 
  22.             2000-01-01 00:00:00  2000-01-02 00:00:00  2000-01-03 00:00:00  2000-01-04 00:00:00  2000-01-05 00:00:00  2000-01-06 00:00:00  2000-01-07 00:00:00  2000-01-08 00:00:00   A   B   C
  23. 2000-01-01                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  24. 2000-01-02                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  25. 2000-01-03                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  26. 2000-01-04                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  27. 2000-01-05                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  28. 2000-01-06                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  29. 2000-01-07                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN
  30. 2000-01-08                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN NaN NaN NaN

Warning

  1. df - df['A']

is now deprecated and will be removed in a future release. The preferred wayto replicate this behavior is

  1. df.sub(df['A'], axis=0)

For explicit control over the matching and broadcasting behavior, see thesection on flexible binary operations.

Operations with scalars are just as you would expect:

  1. In [94]: df * 5 + 2
  2. Out[94]: 
  3.                    A         B         C
  4. 2000-01-01 -4.134126  5.849018 -4.406237
  5. 2000-01-02 -1.638535  1.393469  1.510587
  6. 2000-01-03  5.478873  3.708672  6.798628
  7. 2000-01-04 -3.551681 -1.099880  2.748742
  8. 2000-01-05 -1.661697  5.438692  2.882222
  9. 2000-01-06  4.016548  1.225246  3.508122
  10. 2000-01-07 -8.899303 -4.849247 -2.771039
  11. 2000-01-08  9.313480 -6.715805 -2.132955
  12.  
  13. In [95]: 1 / df
  14. Out[95]: 
  15.                    A         B          C
  16. 2000-01-01 -0.815112  1.299033  -0.780489
  17. 2000-01-02 -1.374179 -8.243600 -10.216313
  18. 2000-01-03  1.437247  2.926250   1.041965
  19. 2000-01-04 -0.900628 -1.612966   6.677871
  20. 2000-01-05 -1.365487  1.454041   5.667510
  21. 2000-01-06  2.479485 -6.453662   3.315381
  22. 2000-01-07 -0.458745 -0.730007  -1.047990
  23. 2000-01-08  0.683669 -0.573671  -1.209788
  24.  
  25. In [96]: df ** 4
  26. Out[96]: 
  27.                     A         B         C
  28. 2000-01-01   2.265327  0.351172  2.694833
  29. 2000-01-02   0.280431  0.000217  0.000092
  30. 2000-01-03   0.234355  0.013638  0.848376
  31. 2000-01-04   1.519910  0.147740  0.000503
  32. 2000-01-05   0.287640  0.223714  0.000969
  33. 2000-01-06   0.026458  0.000576  0.008277
  34. 2000-01-07  22.579530  3.521204  0.829033
  35. 2000-01-08   4.577374  9.233151  0.466834

Boolean operators work as well:

  1. In [97]: df1 = pd.DataFrame({'a': [1, 0, 1], 'b': [0, 1, 1]}, dtype=bool)
  2.  
  3. In [98]: df2 = pd.DataFrame({'a': [0, 1, 1], 'b': [1, 1, 0]}, dtype=bool)
  4.  
  5. In [99]: df1 & df2
  6. Out[99]: 
  7.        a      b
  8. 0  False  False
  9. 1  False   True
  10. 2   True  False
  11.  
  12. In [100]: df1 | df2
  13. Out[100]: 
  14.       a     b
  15. 0  True  True
  16. 1  True  True
  17. 2  True  True
  18.  
  19. In [101]: df1 ^ df2
  20. Out[101]: 
  21.        a      b
  22. 0   True   True
  23. 1   True  False
  24. 2  False   True
  25.  
  26. In [102]: -df1
  27. Out[102]: 
  28.        a      b
  29. 0  False   True
  30. 1   True  False
  31. 2  False  False

Transposing

To transpose, access the T attribute (also the transpose function),similar to an ndarray:

  1. # only show the first 5 rows
  2. In [103]: df[:5].T
  3. Out[103]: 
  4.    2000-01-01  2000-01-02  2000-01-03  2000-01-04  2000-01-05
  5. A   -1.226825   -0.727707    0.695775   -1.110336   -0.732339
  6. B    0.769804   -0.121306    0.341734   -0.619976    0.687738
  7. C   -1.281247   -0.097883    0.959726    0.149748    0.176444

DataFrame interoperability with NumPy functions

Elementwise NumPy ufuncs (log, exp, sqrt, …) and various other NumPy functionscan be used with no issues on Series and DataFrame, assuming the data withinare numeric:

  1. In [104]: np.exp(df)
  2. Out[104]: 
  3.                    A         B         C
  4. 2000-01-01  0.293222  2.159342  0.277691
  5. 2000-01-02  0.483015  0.885763  0.906755
  6. 2000-01-03  2.005262  1.407386  2.610980
  7. 2000-01-04  0.329448  0.537957  1.161542
  8. 2000-01-05  0.480783  1.989212  1.192968
  9. 2000-01-06  1.496770  0.856457  1.352053
  10. 2000-01-07  0.113057  0.254145  0.385117
  11. 2000-01-08  4.317584  0.174966  0.437538
  12.  
  13. In [105]: np.asarray(df)
  14. Out[105]: 
  15. array([[-1.2268,  0.7698, -1.2812],
  16.        [-0.7277, -0.1213, -0.0979],
  17.        [ 0.6958,  0.3417,  0.9597],
  18.        [-1.1103, -0.62  ,  0.1497],
  19.        [-0.7323,  0.6877,  0.1764],
  20.        [ 0.4033, -0.155 ,  0.3016],
  21.        [-2.1799, -1.3698, -0.9542],
  22.        [ 1.4627, -1.7432, -0.8266]])

DataFrame is not intended to be a drop-in replacement for ndarray as itsindexing semantics and data model are quite different in places from an n-dimensionalarray.

Series implements array_ufunc, which allows it to work with NumPy’suniversal functions.

The ufunc is applied to the underlying array in a Series.

  1. In [106]: ser = pd.Series([1, 2, 3, 4])
  2.  
  3. In [107]: np.exp(ser)
  4. Out[107]: 
  5. 0     2.718282
  6. 1     7.389056
  7. 2    20.085537
  8. 3    54.598150
  9. dtype: float64

Changed in version 0.25.0: When multiple Series are passed to a ufunc, they are aligned beforeperforming the operation.

Like other parts of the library, pandas will automatically align labeled inputsas part of a ufunc with multiple inputs. For example, using numpy.remainder()on two Series with differently ordered labels will align before the operation.

  1. In [108]: ser1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])
  2.  
  3. In [109]: ser2 = pd.Series([1, 3, 5], index=['b', 'a', 'c'])
  4.  
  5. In [110]: ser1
  6. Out[110]: 
  7. a    1
  8. b    2
  9. c    3
  10. dtype: int64
  11.  
  12. In [111]: ser2
  13. Out[111]: 
  14. b    1
  15. a    3
  16. c    5
  17. dtype: int64
  18.  
  19. In [112]: np.remainder(ser1, ser2)
  20. Out[112]: 
  21. a    1
  22. b    0
  23. c    3
  24. dtype: int64

As usual, the union of the two indices is taken, and non-overlapping values are filledwith missing values.

  1. In [113]: ser3 = pd.Series([2, 4, 6], index=['b', 'c', 'd'])
  2.  
  3. In [114]: ser3
  4. Out[114]: 
  5. b    2
  6. c    4
  7. d    6
  8. dtype: int64
  9.  
  10. In [115]: np.remainder(ser1, ser3)
  11. Out[115]: 
  12. a    NaN
  13. b    0.0
  14. c    3.0
  15. d    NaN
  16. dtype: float64

When a binary ufunc is applied to a Series and Index, the Seriesimplementation takes precedence and a Series is returned.

  1. In [116]: ser = pd.Series([1, 2, 3])
  2.  
  3. In [117]: idx = pd.Index([4, 5, 6])
  4.  
  5. In [118]: np.maximum(ser, idx)
  6. Out[118]: 
  7. 0    4
  8. 1    5
  9. 2    6
  10. dtype: int64

NumPy ufuncs are safe to apply to Series backed by non-ndarray arrays,for example SparseArray (see Sparse calculation). If possible,the ufunc is applied without converting the underlying data to an ndarray.

Console display

Very large DataFrames will be truncated to display them in the console.You can also get a summary using info().(Here I am reading a CSV version of the baseball dataset from the plyrR package):

  1. In [119]: baseball = pd.read_csv('data/baseball.csv')
  2.  
  3. In [120]: print(baseball)
  4.        id     player  year  stint team  lg   g   ab   r    h  X2b  X3b  hr   rbi   sb   cs  bb    so  ibb  hbp   sh   sf  gidp
  5. 0   88641  womacto01  2006      2  CHN  NL  19   50   6   14    1    0   1   2.0  1.0  1.0   4   4.0  0.0  0.0  3.0  0.0   0.0
  6. 1   88643  schilcu01  2006      1  BOS  AL  31    2   0    1    0    0   0   0.0  0.0  0.0   0   1.0  0.0  0.0  0.0  0.0   0.0
  7. ..    ...        ...   ...    ...  ...  ..  ..  ...  ..  ...  ...  ...  ..   ...  ...  ...  ..   ...  ...  ...  ...  ...   ...
  8. 98  89533   aloumo01  2007      1  NYN  NL  87  328  51  112   19    1  13  49.0  3.0  0.0  27  30.0  5.0  2.0  0.0  3.0  13.0
  9. 99  89534  alomasa02  2007      1  NYN  NL   8   22   1    3    1    0   0   0.0  0.0  0.0   0   3.0  0.0  0.0  0.0  0.0   0.0
  10.  
  11. [100 rows x 23 columns]
  12.  
  13. In [121]: baseball.info()
  14. <class 'pandas.core.frame.DataFrame'>
  15. RangeIndex: 100 entries, 0 to 99
  16. Data columns (total 23 columns):
  17. id        100 non-null int64
  18. player    100 non-null object
  19. year      100 non-null int64
  20. stint     100 non-null int64
  21. team      100 non-null object
  22. lg        100 non-null object
  23. g         100 non-null int64
  24. ab        100 non-null int64
  25. r         100 non-null int64
  26. h         100 non-null int64
  27. X2b       100 non-null int64
  28. X3b       100 non-null int64
  29. hr        100 non-null int64
  30. rbi       100 non-null float64
  31. sb        100 non-null float64
  32. cs        100 non-null float64
  33. bb        100 non-null int64
  34. so        100 non-null float64
  35. ibb       100 non-null float64
  36. hbp       100 non-null float64
  37. sh        100 non-null float64
  38. sf        100 non-null float64
  39. gidp      100 non-null float64
  40. dtypes: float64(9), int64(11), object(3)
  41. memory usage: 18.1+ KB

However, using to_string will return a string representation of theDataFrame in tabular form, though it won’t always fit the console width:

  1. In [122]: print(baseball.iloc[-20:, :12].to_string())
  2.        id     player  year  stint team  lg    g   ab   r    h  X2b  X3b
  3. 80  89474  finlest01  2007      1  COL  NL   43   94   9   17    3    0
  4. 81  89480  embreal01  2007      1  OAK  AL    4    0   0    0    0    0
  5. 82  89481  edmonji01  2007      1  SLN  NL  117  365  39   92   15    2
  6. 83  89482  easleda01  2007      1  NYN  NL   76  193  24   54    6    0
  7. 84  89489  delgaca01  2007      1  NYN  NL  139  538  71  139   30    0
  8. 85  89493  cormirh01  2007      1  CIN  NL    6    0   0    0    0    0
  9. 86  89494  coninje01  2007      2  NYN  NL   21   41   2    8    2    0
  10. 87  89495  coninje01  2007      1  CIN  NL   80  215  23   57   11    1
  11. 88  89497  clemero02  2007      1  NYA  AL    2    2   0    1    0    0
  12. 89  89498  claytro01  2007      2  BOS  AL    8    6   1    0    0    0
  13. 90  89499  claytro01  2007      1  TOR  AL   69  189  23   48   14    0
  14. 91  89501  cirilje01  2007      2  ARI  NL   28   40   6    8    4    0
  15. 92  89502  cirilje01  2007      1  MIN  AL   50  153  18   40    9    2
  16. 93  89521  bondsba01  2007      1  SFN  NL  126  340  75   94   14    0
  17. 94  89523  biggicr01  2007      1  HOU  NL  141  517  68  130   31    3
  18. 95  89525  benitar01  2007      2  FLO  NL   34    0   0    0    0    0
  19. 96  89526  benitar01  2007      1  SFN  NL   19    0   0    0    0    0
  20. 97  89530  ausmubr01  2007      1  HOU  NL  117  349  38   82   16    3
  21. 98  89533   aloumo01  2007      1  NYN  NL   87  328  51  112   19    1
  22. 99  89534  alomasa02  2007      1  NYN  NL    8   22   1    3    1    0

Wide DataFrames will be printed across multiple rows bydefault:

  1. In [123]: pd.DataFrame(np.random.randn(3, 12))
  2. Out[123]: 
  3.           0         1         2         3         4         5         6         7         8         9        10        11
  4. 0 -0.345352  1.314232  0.690579  0.995761  2.396780  0.014871  3.357427 -0.317441 -1.236269  0.896171 -0.487602 -0.082240
  5. 1 -2.182937  0.380396  0.084844  0.432390  1.519970 -0.493662  0.600178  0.274230  0.132885 -0.023688  2.410179  1.450520
  6. 2  0.206053 -0.251905 -2.213588  1.063327  1.266143  0.299368 -0.863838  0.408204 -1.048089 -0.025747 -0.988387  0.094055

You can change how much to print on a single row by setting the display.widthoption:

  1. In [124]: pd.set_option('display.width', 40)  # default is 80
  2.  
  3. In [125]: pd.DataFrame(np.random.randn(3, 12))
  4. Out[125]: 
  5.           0         1         2         3         4         5         6         7         8         9        10        11
  6. 0  1.262731  1.289997  0.082423 -0.055758  0.536580 -0.489682  0.369374 -0.034571 -2.484478 -0.281461  0.030711  0.109121
  7. 1  1.126203 -0.977349  1.474071 -0.064034 -1.282782  0.781836 -1.071357  0.441153  2.353925  0.583787  0.221471 -0.744471
  8. 2  0.758527  1.729689 -0.964980 -0.845696 -1.340896  1.846883 -1.328865  1.682706 -1.717693  0.888782  0.228440  0.901805

You can adjust the max width of the individual columns by setting display.max_colwidth

  1. In [126]: datafile = {'filename': ['filename_01', 'filename_02'],
  2.    .....:             'path': ["media/user_name/storage/folder_01/filename_01",
  3.    .....:                      "media/user_name/storage/folder_02/filename_02"]}
  4.    .....: 
  5.  
  6. In [127]: pd.set_option('display.max_colwidth', 30)
  7.  
  8. In [128]: pd.DataFrame(datafile)
  9. Out[128]: 
  10.       filename                           path
  11. 0  filename_01  media/user_name/storage/fo...
  12. 1  filename_02  media/user_name/storage/fo...
  13.  
  14. In [129]: pd.set_option('display.max_colwidth', 100)
  15.  
  16. In [130]: pd.DataFrame(datafile)
  17. Out[130]: 
  18.       filename                                           path
  19. 0  filename_01  media/user_name/storage/folder_01/filename_01
  20. 1  filename_02  media/user_name/storage/folder_02/filename_02

You can also disable this feature via the expand_frame_repr option.This will print the table in one block.

DataFrame column attribute access and IPython completion

If a DataFrame column label is a valid Python variable name, the column can beaccessed like an attribute:

  1. In [131]: df = pd.DataFrame({'foo1': np.random.randn(5),
  2.    .....:                    'foo2': np.random.randn(5)})
  3.    .....: 
  4.  
  5. In [132]: df
  6. Out[132]: 
  7.        foo1      foo2
  8. 0  1.171216 -0.858447
  9. 1  0.520260  0.306996
  10. 2 -1.197071 -0.028665
  11. 3 -1.066969  0.384316
  12. 4 -0.303421  1.574159
  13.  
  14. In [133]: df.foo1
  15. Out[133]: 
  16. 0    1.171216
  17. 1    0.520260
  18. 2   -1.197071
  19. 3   -1.066969
  20. 4   -0.303421
  21. Name: foo1, dtype: float64

The columns are also connected to the IPythoncompletion mechanism so they can be tab-completed:

  1. In [5]: df.fo<TAB>  # noqa: E225, E999
  2. df.foo1  df.foo2