tracemalloc —- 跟踪内存分配

3.4 新版功能.

源代码：Lib/tracemalloc.py

The tracemalloc module is a debug tool to trace memory blocks allocated byPython. It provides the following information:

  • Traceback where an object was allocated

  • Statistics on allocated memory blocks per filename and per line number:total size, number and average size of allocated memory blocks

  • Compute the differences between two snapshots to detect memory leaks

To trace most memory blocks allocated by Python, the module should be startedas early as possible by setting the PYTHONTRACEMALLOC environmentvariable to 1, or by using -X tracemalloc command lineoption. The tracemalloc.start() function can be called at runtime tostart tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most recentframe (1 frame). To store 25 frames at startup: set thePYTHONTRACEMALLOC environment variable to 25, or use the-X tracemalloc=25 command line option.

例子

显示前10项

显示内存分配最多的10个文件：

  1. import tracemalloc
  2.  
  3. tracemalloc.start()
  4.  
  5. # ... run your application ...
  6.  
  7. snapshot = tracemalloc.take_snapshot()
  8. top_stats = snapshot.statistics('lineno')
  9.  
  10. print("[ Top 10 ]")
  11. for stat in top_stats[:10]:
  12.     print(stat)

Python测试套件的输出示例：

  1. [ Top 10 ]
  2. <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
  3. <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
  4. /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
  5. /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
  6. /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
  7. /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
  8. <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
  9. <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
  10. <string>:5: size=49.7 KiB, count=148, average=344 B
  11. /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB

We can see that Python loaded 4855 KiB data (bytecode and constants) frommodules and that the collections module allocated 244 KiB to buildnamedtuple types.

See Snapshot.statistics() for more options.

计算差异

获取两个快照并显示差异：

  1. import tracemalloc
  2. tracemalloc.start()
  3. # ... start your application ...
  4.  
  5. snapshot1 = tracemalloc.take_snapshot()
  6. # ... call the function leaking memory ...
  7. snapshot2 = tracemalloc.take_snapshot()
  8.  
  9. top_stats = snapshot2.compare_to(snapshot1, 'lineno')
  10.  
  11. print("[ Top 10 differences ]")
  12. for stat in top_stats[:10]:
  13.     print(stat)

Example of output before/after running some tests of the Python test suite:

  1. [ Top 10 differences ]
  2. <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
  3. /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
  4. /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
  5. <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
  6. /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
  7. /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
  8. /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
  9. /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
  10. /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
  11. /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B

We can see that Python has loaded 8173 KiB of module data (bytecode andconstants), and that this is 4428 KiB more than had been loaded before thetests, when the previous snapshot was taken. Similarly, the linecachemodule has cached 940 KiB of Python source code to format tracebacks, allof it since the previous snapshot.

If the system has little free memory, snapshots can be written on disk usingthe Snapshot.dump() method to analyze the snapshot offline. Then use theSnapshot.load() method reload the snapshot.

Get the traceback of a memory block

Code to display the traceback of the biggest memory block:

  1. import tracemalloc
  2.  
  3. # Store 25 frames
  4. tracemalloc.start(25)
  5.  
  6. # ... run your application ...
  7.  
  8. snapshot = tracemalloc.take_snapshot()
  9. top_stats = snapshot.statistics('traceback')
  10.  
  11. # pick the biggest memory block
  12. stat = top_stats[0]
  13. print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
  14. for line in stat.traceback.format():
  15.     print(line)

Example of output of the Python test suite (traceback limited to 25 frames):

  1. 903 memory blocks: 870.1 KiB
  2.   File "<frozen importlib._bootstrap>", line 716
  3.   File "<frozen importlib._bootstrap>", line 1036
  4.   File "<frozen importlib._bootstrap>", line 934
  5.   File "<frozen importlib._bootstrap>", line 1068
  6.   File "<frozen importlib._bootstrap>", line 619
  7.   File "<frozen importlib._bootstrap>", line 1581
  8.   File "<frozen importlib._bootstrap>", line 1614
  9.   File "/usr/lib/python3.4/doctest.py", line 101
  10.     import pdb
  11.   File "<frozen importlib._bootstrap>", line 284
  12.   File "<frozen importlib._bootstrap>", line 938
  13.   File "<frozen importlib._bootstrap>", line 1068
  14.   File "<frozen importlib._bootstrap>", line 619
  15.   File "<frozen importlib._bootstrap>", line 1581
  16.   File "<frozen importlib._bootstrap>", line 1614
  17.   File "/usr/lib/python3.4/test/support/__init__.py", line 1728
  18.     import doctest
  19.   File "/usr/lib/python3.4/test/test_pickletools.py", line 21
  20.     support.run_doctest(pickletools)
  21.   File "/usr/lib/python3.4/test/regrtest.py", line 1276
  22.     test_runner()
  23.   File "/usr/lib/python3.4/test/regrtest.py", line 976
  24.     display_failure=not verbose)
  25.   File "/usr/lib/python3.4/test/regrtest.py", line 761
  26.     match_tests=ns.match_tests)
  27.   File "/usr/lib/python3.4/test/regrtest.py", line 1563
  28.     main()
  29.   File "/usr/lib/python3.4/test/__main__.py", line 3
  30.     regrtest.main_in_temp_cwd()
  31.   File "/usr/lib/python3.4/runpy.py", line 73
  32.     exec(code, run_globals)
  33.   File "/usr/lib/python3.4/runpy.py", line 160
  34.     "__main__", fname, loader, pkg_name)

We can see that the most memory was allocated in the importlib module toload data (bytecode and constants) from modules: 870.1 KiB. The traceback iswhere the importlib loaded data most recently: on the import pdbline of the doctest module. The traceback may change if a new module isloaded.

Pretty top

Code to display the 10 lines allocating the most memory with a pretty output,ignoring <frozen importlib._bootstrap> and <unknown> files:

  1. import linecache
  2. import os
  3. import tracemalloc
  4.  
  5. def display_top(snapshot, key_type='lineno', limit=10):
  6.     snapshot = snapshot.filter_traces((
  7.         tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
  8.         tracemalloc.Filter(False, "<unknown>"),
  9.     ))
  10.     top_stats = snapshot.statistics(key_type)
  11.  
  12.     print("Top %s lines" % limit)
  13.     for index, stat in enumerate(top_stats[:limit], 1):
  14.         frame = stat.traceback[0]
  15.         # replace "/path/to/module/file.py" with "module/file.py"
  16.         filename = os.sep.join(frame.filename.split(os.sep)[-2:])
  17.         print("#%s: %s:%s: %.1f KiB"
  18.               % (index, filename, frame.lineno, stat.size / 1024))
  19.         line = linecache.getline(frame.filename, frame.lineno).strip()
  20.         if line:
  21.             print('    %s' % line)
  22.  
  23.     other = top_stats[limit:]
  24.     if other:
  25.         size = sum(stat.size for stat in other)
  26.         print("%s other: %.1f KiB" % (len(other), size / 1024))
  27.     total = sum(stat.size for stat in top_stats)
  28.     print("Total allocated size: %.1f KiB" % (total / 1024))
  29.  
  30. tracemalloc.start()
  31.  
  32. # ... run your application ...
  33.  
  34. snapshot = tracemalloc.take_snapshot()
  35. display_top(snapshot)

Python测试套件的输出示例：

  1. Top 10 lines
  2. #1: Lib/base64.py:414: 419.8 KiB
  3.     _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars]
  4. #2: Lib/base64.py:306: 419.8 KiB
  5.     _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars]
  6. #3: collections/__init__.py:368: 293.6 KiB
  7.     exec(class_definition, namespace)
  8. #4: Lib/abc.py:133: 115.2 KiB
  9.     cls = super().__new__(mcls, name, bases, namespace)
  10. #5: unittest/case.py:574: 103.1 KiB
  11.     testMethod()
  12. #6: Lib/linecache.py:127: 95.4 KiB
  13.     lines = fp.readlines()
  14. #7: urllib/parse.py:476: 71.8 KiB
  15.     for a in _hexdig for b in _hexdig}
  16. #8: <string>:5: 62.0 KiB
  17. #9: Lib/_weakrefset.py:37: 60.0 KiB
  18.     self.data = set()
  19. #10: Lib/base64.py:142: 59.8 KiB
  20.     _b32tab2 = [a + b for a in _b32tab for b in _b32tab]
  21. 6220 other: 3602.8 KiB
  22. Total allocated size: 5303.1 KiB

See Snapshot.statistics() for more options.

API

函数

  • tracemalloc.clear_traces()
  • Clear traces of memory blocks allocated by Python.

See also stop().

  • tracemalloc.getobject_traceback(_obj)
  • Get the traceback where the Python object obj was allocated.Return a Traceback instance, or None if the tracemallocmodule is not tracing memory allocations or did not trace the allocation ofthe object.

See also gc.get_referrers() and sys.getsizeof() functions.

  • tracemalloc.get_traceback_limit()
  • Get the maximum number of frames stored in the traceback of a trace.

The tracemalloc module must be tracing memory allocations toget the limit, otherwise an exception is raised.

The limit is set by the start() function.

  • tracemalloc.get_traced_memory()
  • Get the current size and peak size of memory blocks traced by thetracemalloc module as a tuple: (current: int, peak: int).
  • tracemalloc.get_tracemalloc_memory()
  • Get the memory usage in bytes of the tracemalloc module used to storetraces of memory blocks.Return an int.
  • tracemalloc.is_tracing()
  • True if the tracemalloc module is tracing Python memoryallocations, False otherwise.

See also start() and stop() functions.

  • tracemalloc.start(nframe: int=1)
  • Start tracing Python memory allocations: install hooks on Python memoryallocators. Collected tracebacks of traces will be limited to nframe_frames. By default, a trace of a memory block only stores the most recentframe: the limit is 1. _nframe must be greater or equal to 1.

Storing more than 1 frame is only useful to compute statistics groupedby 'traceback' or to compute cumulative statistics: see theSnapshot.compare_to() and Snapshot.statistics() methods.

Storing more frames increases the memory and CPU overhead of thetracemalloc module. Use the get_tracemalloc_memory() functionto measure how much memory is used by the tracemalloc module.

The PYTHONTRACEMALLOC environment variable(PYTHONTRACEMALLOC=NFRAME) and the -X tracemalloc=NFRAMEcommand line option can be used to start tracing at startup.

See also stop(), is_tracing() and get_traceback_limit()functions.

  • tracemalloc.stop()
  • Stop tracing Python memory allocations: uninstall hooks on Python memoryallocators. Also clears all previously collected traces of memory blocksallocated by Python.

Call take_snapshot() function to take a snapshot of traces beforeclearing them.

See also start(), is_tracing() and clear_traces()functions.

  • tracemalloc.take_snapshot()
  • Take a snapshot of traces of memory blocks allocated by Python. Return a newSnapshot instance.

The snapshot does not include memory blocks allocated before thetracemalloc module started to trace memory allocations.

Tracebacks of traces are limited to get_traceback_limit() frames. Usethe nframe parameter of the start() function to store more frames.

The tracemalloc module must be tracing memory allocations to take asnapshot, see the start() function.

See also the get_object_traceback() function.

域过滤器

  • class tracemalloc.DomainFilter(inclusive: bool, domain: int)
  • Filter traces of memory blocks by their address space (domain).

3.6 新版功能.

  • inclusive
  • If inclusive is True (include), match memory blocks allocatedin the address space domain.

If inclusive is False (exclude), match memory blocks not allocatedin the address space domain.

  • domain
  • Address space of a memory block (int). Read-only property.

过滤器

  • class tracemalloc.Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None)
  • 对内存块的跟踪进行筛选。

See the fnmatch.fnmatch() function for the syntax offilename_pattern. The '.pyc' file extension isreplaced with '.py'.

示例：

  • Filter(True, subprocess.file) only includes traces of thesubprocess module

  • Filter(False, tracemalloc.file) excludes traces of thetracemalloc module

  • Filter(False, "<unknown>") excludes empty tracebacks

在 3.5 版更改: The '.pyo' file extension is no longer replaced with '.py'.

在 3.6 版更改: Added the domain attribute.

  • domain
  • Address space of a memory block (int or None).

tracemalloc uses the domain 0 to trace memory allocations made byPython. C extensions can use other domains to trace other resources.

  • inclusive
  • If inclusive is True (include), only match memory blocks allocatedin a file with a name matching filename_pattern at line numberlineno.

If inclusive is False (exclude), ignore memory blocks allocated ina file with a name matching filename_pattern at line numberlineno.

  • lineno

  • Line number (int) of the filter. If lineno is None, the filtermatches any line number.

  • filename_pattern

  • Filename pattern of the filter (str). Read-only property.

  • all_frames

  • If all_frames is True, all frames of the traceback are checked. Ifall_frames is False, only the most recent frame is checked.

This attribute has no effect if the traceback limit is 1. See theget_traceback_limit() function and Snapshot.traceback_limitattribute.

Frame

  • class tracemalloc.Frame
  • Frame of a traceback.

The Traceback class is a sequence of Frame instances.

  • filename

  • 文件名(str).

  • lineno

  • 行号 (int).

快照

  • class tracemalloc.Snapshot
  • Snapshot of traces of memory blocks allocated by Python.

The take_snapshot() function creates a snapshot instance.

  • compareto(_old_snapshot: Snapshot, key_type: str, cumulative: bool=False)
  • Compute the differences with an old snapshot. Get statistics as a sortedlist of StatisticDiff instances grouped by key_type.

See the Snapshot.statistics() method for key_type and _cumulative_parameters.

The result is sorted from the biggest to the smallest by: absolute valueof StatisticDiff.size_diff, StatisticDiff.size, absolutevalue of StatisticDiff.count_diff, Statistic.count andthen by StatisticDiff.traceback.

  • dump(filename)
  • 将快照写入文件

使用 load() 重载快照。

  • filtertraces(_filters)
  • Create a new Snapshot instance with a filtered tracessequence, filters is a list of DomainFilter andFilter instances. If filters is an empty list, return a newSnapshot instance with a copy of the traces.

All inclusive filters are applied at once, a trace is ignored if noinclusive filters match it. A trace is ignored if at least one exclusivefilter matches it.

在 3.6 版更改: DomainFilter instances are now also accepted in filters.

  • classmethod load(filename)
  • 从文件载入快照。

另见 dump().

  • statistics(key_type: str, cumulative: bool=False)
  • 获取 Statistic 信息列表，按 key_type 分组排序：

key_type

描述

'filename'

filename

'lineno'

文件名和行号

'traceback'

回溯

If cumulative is True, cumulate size and count of memory blocks ofall frames of the traceback of a trace, not only the most recent frame.The cumulative mode can only be used with key_type equals to'filename' and 'lineno'.

The result is sorted from the biggest to the smallest by:Statistic.size, Statistic.count and then byStatistic.traceback.

  • traceback_limit

  • Maximum number of frames stored in the traceback of traces:result of the get_traceback_limit() when the snapshot was taken.

  • traces

  • Traces of all memory blocks allocated by Python: sequence ofTrace instances.

The sequence has an undefined order. Use the Snapshot.statistics()method to get a sorted list of statistics.

统计

  • class tracemalloc.Statistic
  • 统计内存分配

Snapshot.statistics() 返回 Statistic 实例的列表。.

参见 StatisticDiff 类。

  • count

  • 内存块数（整形）。

  • size

  • Total size of memory blocks in bytes (int).

  • traceback

  • Traceback where the memory block was allocated, Tracebackinstance.

StatisticDiff

  • class tracemalloc.StatisticDiff
  • Statistic difference on memory allocations between an old and a newSnapshot instance.

Snapshot.compare_to() returns a list of StatisticDiffinstances. See also the Statistic class.

  • count

  • Number of memory blocks in the new snapshot (int): 0 ifthe memory blocks have been released in the new snapshot.

  • count_diff

  • Difference of number of memory blocks between the old and the newsnapshots (int): 0 if the memory blocks have been allocated inthe new snapshot.

  • size

  • Total size of memory blocks in bytes in the new snapshot (int):0 if the memory blocks have been released in the new snapshot.

  • size_diff

  • Difference of total size of memory blocks in bytes between the old andthe new snapshots (int): 0 if the memory blocks have beenallocated in the new snapshot.

  • traceback

  • Traceback where the memory blocks were allocated, Tracebackinstance.

跟踪

  • class tracemalloc.Trace
  • Trace of a memory block.

The Snapshot.traces attribute is a sequence of Traceinstances.

在 3.6 版更改: Added the domain attribute.

  • domain
  • Address space of a memory block (int). Read-only property.

tracemalloc uses the domain 0 to trace memory allocations made byPython. C extensions can use other domains to trace other resources.

  • size

  • Size of the memory block in bytes (int).

  • traceback

  • Traceback where the memory block was allocated, Tracebackinstance.

回溯

  • class tracemalloc.Traceback
  • Sequence of Frame instances sorted from the oldest frame to themost recent frame.

A traceback contains at least 1 frame. If the tracemalloc modulefailed to get a frame, the filename "<unknown>" at line number 0 isused.

When a snapshot is taken, tracebacks of traces are limited toget_traceback_limit() frames. See the take_snapshot() function.

The Trace.traceback attribute is an instance of Tracebackinstance.

在 3.7 版更改: Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.

  • format(limit=None, most_recent_first=False)
  • Format the traceback as a list of lines with newlines. Use thelinecache module to retrieve lines from the source code.If limit is set, format the limit most recent frames if limit_is positive. Otherwise, format the abs(limit) oldest frames.If _most_recent_first is True, the order of the formatted framesis reversed, returning the most recent frame first instead of last.

Similar to the traceback.format_tb() function, except thatformat() does not include newlines.

示例:

  1. print("Traceback (most recent call first):")
  2. for line in traceback:
  3.     print(line)

输出:

  1. Traceback (most recent call first):
  2.   File "test.py", line 9
  3.     obj = Object()
  4.   File "test.py", line 12
  5.     tb = tracemalloc.get_object_traceback(f())