数据类型

The main types stored in pandas objects are float, int, bool, datetime64[ns] and datetime64[ns, tz], timedelta[ns], category and object. In addition these dtypes have item sizes, e.g. int64 and int32. See Series with TZ for more detail on datetime64[ns, tz] dtypes.

A convenient dtypes attribute for DataFrame returns a Series with the data type of each column.

  1. In [344]: dft = pd.DataFrame(dict(A = np.random.rand(3),
  2.    .....:                         B = 1,
  3.    .....:                         C = 'foo',
  4.    .....:                         D = pd.Timestamp('20010102'),
  5.    .....:                         E = pd.Series([1.0]*3).astype('float32'),
  6.    .....:                                     F = False,
  7.    .....:                                     G = pd.Series([1]*3,dtype='int8')))
  8.    .....: 
  10. In [345]: dft
  11. Out[345]: 
  12.           A  B    C          D    E      F  G
  13. 0  0.809585  1  foo 2001-01-02  1.0  False  1
  14. 1  0.128238  1  foo 2001-01-02  1.0  False  1
  15. 2  0.775752  1  foo 2001-01-02  1.0  False  1
  17. In [346]: dft.dtypes
  18. Out[346]: 
  19. A           float64
  20. B             int64
  21. C            object
  22. D    datetime64[ns]
  23. E           float32
  24. F              bool
  25. G              int8
  26. dtype: object

On a Series object, use the dtype attribute.

  1. In [347]: dft['A'].dtype
  2. Out[347]: dtype('float64')

If a pandas object contains data with multiple dtypes in a single column, the dtype of the column will be chosen to accommodate all of the data types (object is the most general).

  1. # these ints are coerced to floats
  2. In [348]: pd.Series([1, 2, 3, 4, 5, 6.])
  3. Out[348]: 
  4. 0    1.0
  5. 1    2.0
  6. 2    3.0
  7. 3    4.0
  8. 4    5.0
  9. 5    6.0
  10. dtype: float64
  12. # string data forces an ``object`` dtype
  13. In [349]: pd.Series([1, 2, 3, 6., 'foo'])
  14. Out[349]: 
  15. 0      1
  16. 1      2
  17. 2      3
  18. 3      6
  19. 4    foo
  20. dtype: object

The number of columns of each type in a DataFrame can be found by calling get_dtype_counts().

  1. In [350]: dft.get_dtype_counts()
  2. Out[350]: 
  3. float64           1
  4. float32           1
  5. int64             1
  6. int8              1
  7. datetime64[ns]    1
  8. bool              1
  9. object            1
  10. dtype: int64

Numeric dtypes will propagate and can coexist in DataFrames. If a dtype is passed (either directly via the dtype keyword, a passed ndarray, or a passed Series, then it will be preserved in DataFrame operations. Furthermore, different numeric dtypes will NOT be combined. The following example will give you a taste.

  1. In [351]: df1 = pd.DataFrame(np.random.randn(8, 1), columns=['A'], dtype='float32')
  3. In [352]: df1
  4. Out[352]: 
  5.           A
  6. 0  0.890400
  7. 1  0.283331
  8. 2 -0.303613
  9. 3 -1.192210
  10. 4  0.065420
  11. 5  0.455918
  12. 6  2.008328
  13. 7  0.188942
  15. In [353]: df1.dtypes
  16. Out[353]: 
  17. A    float32
  18. dtype: object
  20. In [354]: df2 = pd.DataFrame(dict( A = pd.Series(np.random.randn(8), dtype='float16'),
  21.    .....:                         B = pd.Series(np.random.randn(8)),
  22.    .....:                         C = pd.Series(np.array(np.random.randn(8), dtype='uint8')) ))
  23.    .....: 
  25. In [355]: df2
  26. Out[355]: 
  27.           A         B    C
  28. 0 -0.454346  0.200071  255
  29. 1 -0.916504 -0.557756  255
  30. 2  0.640625 -0.141988    0
  31. 3  2.675781 -0.174060    0
  32. 4 -0.007866  0.258626    0
  33. 5 -0.204224  0.941688    0
  34. 6 -0.100098 -1.849045    0
  35. 7 -0.402100 -0.949458    0
  37. In [356]: df2.dtypes
  38. Out[356]: 
  39. A    float16
  40. B    float64
  41. C      uint8
  42. dtype: object

defaults

By default integer types are int64 and float types are float64, regardless of platform (32-bit or 64-bit). The following will all result in int64 dtypes.

  1. In [357]: pd.DataFrame([1, 2], columns=['a']).dtypes
  2. Out[357]: 
  3. a    int64
  4. dtype: object
  6. In [358]: pd.DataFrame({'a': [1, 2]}).dtypes
  7. Out[358]: 
  8. a    int64
  9. dtype: object
  11. In [359]: pd.DataFrame({'a': 1 }, index=list(range(2))).dtypes
  12. Out[359]: 
  13. a    int64
  14. dtype: object

Note that Numpy will choose platform-dependent types when creating arrays. The following WILL result in int32 on 32-bit platform.

  1. In [360]: frame = pd.DataFrame(np.array([1, 2]))

upcasting

Types can potentially be upcasted when combined with other types, meaning they are promoted from the current type (e.g. int to float).

  1. In [361]: df3 = df1.reindex_like(df2).fillna(value=0.0) + df2
  3. In [362]: df3
  4. Out[362]: 
  5.           A         B      C
  6. 0  0.436054  0.200071  255.0
  7. 1 -0.633173 -0.557756  255.0
  8. 2  0.337012 -0.141988    0.0
  9. 3  1.483571 -0.174060    0.0
  10. 4  0.057555  0.258626    0.0
  11. 5  0.251695  0.941688    0.0
  12. 6  1.908231 -1.849045    0.0
  13. 7 -0.213158 -0.949458    0.0
  15. In [363]: df3.dtypes
  16. Out[363]: 
  17. A    float32
  18. B    float64
  19. C    float64
  20. dtype: object

The values attribute on a DataFrame return the lower-common-denominator of the dtypes, meaning the dtype that can accommodate ALL of the types in the resulting homogeneous dtyped NumPy array. This can force some upcasting.

  1. In [364]: df3.values.dtype
  2. Out[364]: dtype('float64')

astype

You can use the astype() method to explicitly convert dtypes from one to another. These will by default return a copy, even if the dtype was unchanged (pass copy=False to change this behavior). In addition, they will raise an exception if the astype operation is invalid.

Upcasting is always according to the numpy rules. If two different dtypes are involved in an operation, then the more general one will be used as the result of the operation.

  1. In [365]: df3
  2. Out[365]: 
  3.           A         B      C
  4. 0  0.436054  0.200071  255.0
  5. 1 -0.633173 -0.557756  255.0
  6. 2  0.337012 -0.141988    0.0
  7. 3  1.483571 -0.174060    0.0
  8. 4  0.057555  0.258626    0.0
  9. 5  0.251695  0.941688    0.0
  10. 6  1.908231 -1.849045    0.0
  11. 7 -0.213158 -0.949458    0.0
  13. In [366]: df3.dtypes
  14. Out[366]: 
  15. A    float32
  16. B    float64
  17. C    float64
  18. dtype: object
  20. # conversion of dtypes
  21. In [367]: df3.astype('float32').dtypes
  22. Out[367]: 
  23. A    float32
  24. B    float32
  25. C    float32
  26. dtype: object

Convert a subset of columns to a specified type using astype().

  1. In [368]: dft = pd.DataFrame({'a': [1,2,3], 'b': [4,5,6], 'c': [7, 8, 9]})
  3. In [369]: dft[['a','b']] = dft[['a','b']].astype(np.uint8)
  5. In [370]: dft
  6. Out[370]: 
  7.    a  b  c
  8. 0  1  4  7
  9. 1  2  5  8
  10. 2  3  6  9
  12. In [371]: dft.dtypes
  13. Out[371]: 
  14. a    uint8
  15. b    uint8
  16. c    int64
  17. dtype: object

New in version 0.19.0.

Convert certain columns to a specific dtype by passing a dict to astype().

  1. In [372]: dft1 = pd.DataFrame({'a': [1,0,1], 'b': [4,5,6], 'c': [7, 8, 9]})
  3. In [373]: dft1 = dft1.astype({'a': np.bool, 'c': np.float64})
  5. In [374]: dft1
  6. Out[374]: 
  7.        a  b    c
  8. 0   True  4  7.0
  9. 1  False  5  8.0
  10. 2   True  6  9.0
  12. In [375]: dft1.dtypes
  13. Out[375]: 
  14. a       bool
  15. b      int64
  16. c    float64
  17. dtype: object

Note: When trying to convert a subset of columns to a specified type using astype() and loc(), upcasting occurs.

loc() tries to fit in what we are assigning to the current dtypes, while [] will overwrite them taking the dtype from the right hand side. Therefore the following piece of code produces the unintended result.

  1. In [376]: dft = pd.DataFrame({'a': [1,2,3], 'b': [4,5,6], 'c': [7, 8, 9]})
  3. In [377]: dft.loc[:, ['a', 'b']].astype(np.uint8).dtypes
  4. Out[377]: 
  5. a    uint8
  6. b    uint8
  7. dtype: object
  9. In [378]: dft.loc[:, ['a', 'b']] = dft.loc[:, ['a', 'b']].astype(np.uint8)
  11. In [379]: dft.dtypes
  12. Out[379]: 
  13. a    int64
  14. b    int64
  15. c    int64
  16. dtype: object

object conversion

pandas offers various functions to try to force conversion of types from the object dtype to other types. In cases where the data is already of the correct type, but stored in an object array, the DataFrame.infer_objects() and Series.infer_objects() methods can be used to soft convert to the correct type.

  1. In [380]: import datetime
  3. In [381]: df = pd.DataFrame([[1, 2],
  4.    .....:                    ['a', 'b'],
  5.    .....:                    [datetime.datetime(2016, 3, 2), datetime.datetime(2016, 3, 2)]])
  6.    .....: 
  8. In [382]: df = df.T
  10. In [383]: df
  11. Out[383]: 
  12.    0  1                    2
  13. 0  1  a  2016-03-02 00:00:00
  14. 1  2  b  2016-03-02 00:00:00
  16. In [384]: df.dtypes
  17. Out[384]: 
  18. 0    object
  19. 1    object
  20. 2    object
  21. dtype: object

Because the data was transposed the original inference stored all columns as object, which infer_objects will correct.

  1. In [385]: df.infer_objects().dtypes
  2. Out[385]: 
  3. 0             int64
  4. 1            object
  5. 2    datetime64[ns]
  6. dtype: object

The following functions are available for one dimensional object arrays or scalars to perform hard conversion of objects to a specified type:

  • to_numeric() (conversion to numeric dtypes)

    1. In [386]: m = ['1.1', 2, 3]
    3. In [387]: pd.to_numeric(m)
    4. Out[387]: array([ 1.1,  2. ,  3. ])

  • to_datetime() (conversion to datetime objects)

    1. In [388]: import datetime
    3. In [389]: m = ['2016-07-09', datetime.datetime(2016, 3, 2)]
    5. In [390]: pd.to_datetime(m)
    6. Out[390]: DatetimeIndex(['2016-07-09', '2016-03-02'], dtype='datetime64[ns]', freq=None)

  • to_timedelta() (conversion to timedelta objects)

    1. In [391]: m = ['5us', pd.Timedelta('1day')]
    3. In [392]: pd.to_timedelta(m)
    4. Out[392]: TimedeltaIndex(['0 days 00:00:00.000005', '1 days 00:00:00'], dtype='timedelta64[ns]', freq=None)

To force a conversion, we can pass in an errors argument, which specifies how pandas should deal with elements that cannot be converted to desired dtype or object. By default, errors='raise', meaning that any errors encountered will be raised during the conversion process. However, if errors='coerce', these errors will be ignored and pandas will convert problematic elements to pd.NaT (for datetime and timedelta) or np.nan (for numeric). This might be useful if you are reading in data which is mostly of the desired dtype (e.g. numeric, datetime), but occasionally has non-conforming elements intermixed that you want to represent as missing:

  1. In [393]: import datetime
  3. In [394]: m = ['apple', datetime.datetime(2016, 3, 2)]
  5. In [395]: pd.to_datetime(m, errors='coerce')
  6. Out[395]: DatetimeIndex(['NaT', '2016-03-02'], dtype='datetime64[ns]', freq=None)
  8. In [396]: m = ['apple', 2, 3]
  10. In [397]: pd.to_numeric(m, errors='coerce')
  11. Out[397]: array([ nan,   2.,   3.])
  13. In [398]: m = ['apple', pd.Timedelta('1day')]
  15. In [399]: pd.to_timedelta(m, errors='coerce')
  16. Out[399]: TimedeltaIndex([NaT, '1 days'], dtype='timedelta64[ns]', freq=None)

The errors parameter has a third option of errors='ignore', which will simply return the passed in data if it encounters any errors with the conversion to a desired data type:

  1. In [400]: import datetime
  3. In [401]: m = ['apple', datetime.datetime(2016, 3, 2)]
  5. In [402]: pd.to_datetime(m, errors='ignore')
  6. Out[402]: array(['apple', datetime.datetime(2016, 3, 2, 0, 0)], dtype=object)
  8. In [403]: m = ['apple', 2, 3]
  10. In [404]: pd.to_numeric(m, errors='ignore')
  11. Out[404]: array(['apple', 2, 3], dtype=object)
  13. In [405]: m = ['apple', pd.Timedelta('1day')]
  15. In [406]: pd.to_timedelta(m, errors='ignore')
  16. Out[406]: array(['apple', Timedelta('1 days 00:00:00')], dtype=object)

In addition to object conversion, to_numeric() provides another argument downcast, which gives the option of downcasting the newly (or already) numeric data to a smaller dtype, which can conserve memory:

  1. In [407]: m = ['1', 2, 3]
  3. In [408]: pd.to_numeric(m, downcast='integer')   # smallest signed int dtype
  4. Out[408]: array([1, 2, 3], dtype=int8)
  6. In [409]: pd.to_numeric(m, downcast='signed')    # same as 'integer'
  7. Out[409]: array([1, 2, 3], dtype=int8)
  9. In [410]: pd.to_numeric(m, downcast='unsigned')  # smallest unsigned int dtype
  10. Out[410]: array([1, 2, 3], dtype=uint8)
  12. In [411]: pd.to_numeric(m, downcast='float')     # smallest float dtype
  13. Out[411]: array([ 1.,  2.,  3.], dtype=float32)

As these methods apply only to one-dimensional arrays, lists or scalars; they cannot be used directly on multi-dimensional objects such as DataFrames. However, with apply(), we can “apply” the function over each column efficiently:

  1. In [412]: import datetime
  3. In [413]: df = pd.DataFrame([['2016-07-09', datetime.datetime(2016, 3, 2)]] * 2, dtype='O')
  5. In [414]: df
  6. Out[414]: 
  7.             0                    1
  8. 0  2016-07-09  2016-03-02 00:00:00
  9. 1  2016-07-09  2016-03-02 00:00:00
  11. In [415]: df.apply(pd.to_datetime)
  12. Out[415]: 
  13.            0          1
  14. 0 2016-07-09 2016-03-02
  15. 1 2016-07-09 2016-03-02
  17. In [416]: df = pd.DataFrame([['1.1', 2, 3]] * 2, dtype='O')
  19. In [417]: df
  20. Out[417]: 
  21.      0  1  2
  22. 0  1.1  2  3
  23. 1  1.1  2  3
  25. In [418]: df.apply(pd.to_numeric)
  26. Out[418]: 
  27.      0  1  2
  28. 0  1.1  2  3
  29. 1  1.1  2  3
  31. In [419]: df = pd.DataFrame([['5us', pd.Timedelta('1day')]] * 2, dtype='O')
  33. In [420]: df
  34. Out[420]: 
  35.      0                1
  36. 0  5us  1 days 00:00:00
  37. 1  5us  1 days 00:00:00
  39. In [421]: df.apply(pd.to_timedelta)
  40. Out[421]: 
  41.                 0      1
  42. 0 00:00:00.000005 1 days
  43. 1 00:00:00.000005 1 days

gotchas

Performing selection operations on integer type data can easily upcast the data to floating. The dtype of the input data will be preserved in cases where nans are not introduced. See also Support for integer NA.

  1. In [422]: dfi = df3.astype('int32')
  3. In [423]: dfi['E'] = 1
  5. In [424]: dfi
  6. Out[424]: 
  7.    A  B    C  E
  8. 0  0  0  255  1
  9. 1  0  0  255  1
  10. 2  0  0    0  1
  11. 3  1  0    0  1
  12. 4  0  0    0  1
  13. 5  0  0    0  1
  14. 6  1 -1    0  1
  15. 7  0  0    0  1
  17. In [425]: dfi.dtypes
  18. Out[425]: 
  19. A    int32
  20. B    int32
  21. C    int32
  22. E    int64
  23. dtype: object
  25. In [426]: casted = dfi[dfi>0]
  27. In [427]: casted
  28. Out[427]: 
  29.      A   B      C  E
  30. 0  NaN NaN  255.0  1
  31. 1  NaN NaN  255.0  1
  32. 2  NaN NaN    NaN  1
  33. 3  1.0 NaN    NaN  1
  34. 4  NaN NaN    NaN  1
  35. 5  NaN NaN    NaN  1
  36. 6  1.0 NaN    NaN  1
  37. 7  NaN NaN    NaN  1
  39. In [428]: casted.dtypes
  40. Out[428]: 
  41. A    float64
  42. B    float64
  43. C    float64
  44. E      int64
  45. dtype: object

While float dtypes are unchanged.

  1. In [429]: dfa = df3.copy()
  3. In [430]: dfa['A'] = dfa['A'].astype('float32')
  5. In [431]: dfa.dtypes
  6. Out[431]: 
  7. A    float32
  8. B    float64
  9. C    float64
  10. dtype: object
  12. In [432]: casted = dfa[df2>0]
  14. In [433]: casted
  15. Out[433]: 
  16.           A         B      C
  17. 0       NaN  0.200071  255.0
  18. 1       NaN       NaN  255.0
  19. 2  0.337012       NaN    NaN
  20. 3  1.483571       NaN    NaN
  21. 4       NaN  0.258626    NaN
  22. 5       NaN  0.941688    NaN
  23. 6       NaN       NaN    NaN
  24. 7       NaN       NaN    NaN
  26. In [434]: casted.dtypes
  27. Out[434]: 
  28. A    float32
  29. B    float64
  30. C    float64
  31. dtype: object