重复数据

If you want to identify and remove duplicate rows in a DataFrame, there are two methods that will help: duplicated and drop_duplicates. Each takes as an argument the columns to use to identify duplicated rows.

  • duplicated returns a boolean vector whose length is the number of rows, and which indicates whether a row is duplicated.
  • drop_duplicates removes duplicate rows.

By default, the first observed row of a duplicate set is considered unique, but each method has a keep parameter to specify targets to be kept.

  • keep='first' (default): mark / drop duplicates except for the first occurrence.
  • keep='last': mark / drop duplicates except for the last occurrence.
  • keep=False: mark / drop all duplicates.
  1. In [268]: df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'two', 'two', 'three', 'four'],
  2.    .....:                     'b': ['x', 'y', 'x', 'y', 'x', 'x', 'x'],
  3.    .....:                     'c': np.random.randn(7)})
  4.    .....: 
  6. In [269]: df2
  7. Out[269]: 
  8.        a  b         c
  9. 0    one  x -1.067137
  10. 1    one  y  0.309500
  11. 2    two  x -0.211056
  12. 3    two  y -1.842023
  13. 4    two  x -0.390820
  14. 5  three  x -1.964475
  15. 6   four  x  1.298329
  17. In [270]: df2.duplicated('a')
  18. Out[270]: 
  19. 0    False
  20. 1     True
  21. 2    False
  22. 3     True
  23. 4     True
  24. 5    False
  25. 6    False
  26. dtype: bool
  28. In [271]: df2.duplicated('a', keep='last')
  29. Out[271]: 
  30. 0     True
  31. 1    False
  32. 2     True
  33. 3     True
  34. 4    False
  35. 5    False
  36. 6    False
  37. dtype: bool
  39. In [272]: df2.duplicated('a', keep=False)
  40. Out[272]: 
  41. 0     True
  42. 1     True
  43. 2     True
  44. 3     True
  45. 4     True
  46. 5    False
  47. 6    False
  48. dtype: bool
  50. In [273]: df2.drop_duplicates('a')
  51. Out[273]: 
  52.        a  b         c
  53. 0    one  x -1.067137
  54. 2    two  x -0.211056
  55. 5  three  x -1.964475
  56. 6   four  x  1.298329
  58. In [274]: df2.drop_duplicates('a', keep='last')
  59. Out[274]: 
  60.        a  b         c
  61. 1    one  y  0.309500
  62. 4    two  x -0.390820
  63. 5  three  x -1.964475
  64. 6   four  x  1.298329
  66. In [275]: df2.drop_duplicates('a', keep=False)
  67. Out[275]: 
  68.        a  b         c
  69. 5  three  x -1.964475
  70. 6   four  x  1.298329

Also, you can pass a list of columns to identify duplications.

  1. In [276]: df2.duplicated(['a', 'b'])
  2. Out[276]: 
  3. 0    False
  4. 1    False
  5. 2    False
  6. 3    False
  7. 4     True
  8. 5    False
  9. 6    False
  10. dtype: bool
  12. In [277]: df2.drop_duplicates(['a', 'b'])
  13. Out[277]: 
  14.        a  b         c
  15. 0    one  x -1.067137
  16. 1    one  y  0.309500
  17. 2    two  x -0.211056
  18. 3    two  y -1.842023
  19. 5  three  x -1.964475
  20. 6   four  x  1.298329

To drop duplicates by index value, use Index.duplicated then perform slicing. The same set of options are available for the keep parameter.

  1. In [278]: df3 = pd.DataFrame({'a': np.arange(6),
  2.    .....:                     'b': np.random.randn(6)},
  3.    .....:                    index=['a', 'a', 'b', 'c', 'b', 'a'])
  4.    .....: 
  6. In [279]: df3
  7. Out[279]: 
  8.    a         b
  9. a  0  1.440455
  10. a  1  2.456086
  11. b  2  1.038402
  12. c  3 -0.894409
  13. b  4  0.683536
  14. a  5  3.082764
  16. In [280]: df3.index.duplicated()
  17. Out[280]: array([False,  True, False, False,  True,  True], dtype=bool)
  19. In [281]: df3[~df3.index.duplicated()]
  20. Out[281]: 
  21.    a         b
  22. a  0  1.440455
  23. b  2  1.038402
  24. c  3 -0.894409
  26. In [282]: df3[~df3.index.duplicated(keep='last')]
  27. Out[282]: 
  28.    a         b
  29. c  3 -0.894409
  30. b  4  0.683536
  31. a  5  3.082764
  33. In [283]: df3[~df3.index.duplicated(keep=False)]
  34. Out[283]: 
  35.    a         b
  36. c  3 -0.894409