缺失数据相关基础知识

何时/为何数据丢失？

Some might quibble over our usage of missing. By “missing” we simply mean NA (“not available”) or “not present for whatever reason”. Many data sets simply arrive with missing data, either because it exists and was not collected or it never existed. For example, in a collection of financial time series, some of the time series might start on different dates. Thus, values prior to the start date would generally be marked as missing.

In pandas, one of the most common ways that missing data is introduced into a data set is by reindexing. For example:

  1. In [1]: df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f', 'h'],
  2.    ...:                   columns=['one', 'two', 'three'])
  3.    ...: 
  5. In [2]: df['four'] = 'bar'
  7. In [3]: df['five'] = df['one'] > 0
  9. In [4]: df
  10. Out[4]: 
  11.         one       two     three four   five
  12. a -0.166778  0.501113 -0.355322  bar  False
  13. c -0.337890  0.580967  0.983801  bar  False
  14. e  0.057802  0.761948 -0.712964  bar   True
  15. f -0.443160 -0.974602  1.047704  bar  False
  16. h -0.717852 -1.053898 -0.019369  bar  False
  18. In [5]: df2 = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
  20. In [6]: df2
  21. Out[6]: 
  22.         one       two     three four   five
  23. a -0.166778  0.501113 -0.355322  bar  False
  24. b       NaN       NaN       NaN  NaN    NaN
  25. c -0.337890  0.580967  0.983801  bar  False
  26. d       NaN       NaN       NaN  NaN    NaN
  27. e  0.057802  0.761948 -0.712964  bar   True
  28. f -0.443160 -0.974602  1.047704  bar  False
  29. g       NaN       NaN       NaN  NaN    NaN
  30. h -0.717852 -1.053898 -0.019369  bar  False

Values considered “missing”

As data comes in many shapes and forms, pandas aims to be flexible with regard to handling missing data. While NaN is the default missing value marker for reasons of computational speed and convenience, we need to be able to easily detect this value with data of different types: floating point, integer, boolean, and general object. In many cases, however, the Python None will arise and we wish to also consider that “missing” or “not available” or “NA”.

Note: If you want to consider inf and -inf to be “NA” in computations, you can set pandas.options.mode.use_inf_as_na = True.

To make detecting missing values easier (and across different array dtypes), pandas provides the isna() and notna() functions, which are also methods on Series and DataFrame objects:

  1. In [7]: df2['one']
  2. Out[7]: 
  3. a   -0.166778
  4. b         NaN
  5. c   -0.337890
  6. d         NaN
  7. e    0.057802
  8. f   -0.443160
  9. g         NaN
  10. h   -0.717852
  11. Name: one, dtype: float64
  13. In [8]: pd.isna(df2['one'])
  14. Out[8]: 
  15. a    False
  16. b     True
  17. c    False
  18. d     True
  19. e    False
  20. f    False
  21. g     True
  22. h    False
  23. Name: one, dtype: bool
  25. In [9]: df2['four'].notna()
  26. Out[9]: 
  27. a     True
  28. b    False
  29. c     True
  30. d    False
  31. e     True
  32. f     True
  33. g    False
  34. h     True
  35. Name: four, dtype: bool
  37. In [10]: df2.isna()
  38. Out[10]: 
  39.      one    two  three   four   five
  40. a  False  False  False  False  False
  41. b   True   True   True   True   True
  42. c  False  False  False  False  False
  43. d   True   True   True   True   True
  44. e  False  False  False  False  False
  45. f  False  False  False  False  False
  46. g   True   True   True   True   True
  47. h  False  False  False  False  False

警告

One has to be mindful that in Python (and NumPy), the nan's don’t compare equal, but None’s do. Note that pandas/NumPy uses the fact that np.nan != np.nan , and treats None like np.nan.

  2. In [11]: None == None
  3. Out[11]: True

So as compared to above, a scalar equality comparison versus a None/np.nan doesn’t provide useful information.

  2. In [13]: df2['one'] == np.nan
  3. Out[13]: 
  4. a    False
  5. b    False
  6. c    False
  7. d    False
  8. e    False
  9. f    False
  10. g    False
  11. h    False
  12. Name: one, dtype: bool