multiprocessing.shared_memory —- 可从进程直接访问的共享内存

源代码:Lib/multiprocessing/shared_memory.py

3.8 新版功能.

该模块提供了一个 SharedMemory 类，用于分配和管理多核或对称多处理器（SMP）机器上进程间的共享内存。为了协助管理不同进程间的共享内存生命周期，multiprocessing.managers 模块也提供了一个 BaseManager 的子类： SharedMemoryManager。

In this module, shared memory refers to "System V style" shared memory blocks(though is not necessarily implemented explicitly as such) and does not referto "distributed shared memory". This style of shared memory permits distinctprocesses to potentially read and write to a common (or shared) region ofvolatile memory. Processes are conventionally limited to only have access totheir own process memory space but shared memory permits the sharingof data between processes, avoiding the need to instead send messages betweenprocesses containing that data. Sharing data directly via memory can providesignificant performance benefits compared to sharing data via disk or socketor other communications requiring the serialization/deserialization andcopying of data.

  • class multiprocessing.sharedmemory.SharedMemory(_name=None, create=False, size=0)
  • Creates a new shared memory block or attaches to an existing sharedmemory block. Each shared memory block is assigned a unique name.In this way, one process can create a shared memory block with aparticular name and a different process can attach to that same sharedmemory block using that same name.

As a resource for sharing data across processes, shared memory blocksmay outlive the original process that created them. When one processno longer needs access to a shared memory block that might still beneeded by other processes, the close() method should be called.When a shared memory block is no longer needed by any process, theunlink() method should be called to ensure proper cleanup.

name is the unique name for the requested shared memory, specified asa string. When creating a new shared memory block, if None (thedefault) is supplied for the name, a novel name will be generated.

create controls whether a new shared memory block is created (True)or an existing shared memory block is attached (False).

size specifies the requested number of bytes when creating a new sharedmemory block. Because some platforms choose to allocate chunks of memorybased upon that platform's memory page size, the exact size of the sharedmemory block may be larger or equal to the size requested. When attachingto an existing shared memory block, the size parameter is ignored.

  • close()

  • Closes access to the shared memory from this instance. In order toensure proper cleanup of resources, all instances should callclose() once the instance is no longer needed. Note that callingclose() does not cause the shared memory block itself to bedestroyed.

  • unlink()

  • Requests that the underlying shared memory block be destroyed. Inorder to ensure proper cleanup of resources, unlink() should becalled once (and only once) across all processes which have needfor the shared memory block. After requesting its destruction, ashared memory block may or may not be immediately destroyed andthis behavior may differ across platforms. Attempts to access datainside the shared memory block after unlink() has been called mayresult in memory access errors. Note: the last process relinquishingits hold on a shared memory block may call unlink() andclose() in either order.

  • buf

  • A memoryview of contents of the shared memory block.

  • name

  • Read-only access to the unique name of the shared memory block.

  • size

  • Read-only access to size in bytes of the shared memory block.

The following example demonstrates low-level use of SharedMemoryinstances:

  1. >>> from multiprocessing import shared_memory
  2. >>> shm_a = shared_memory.SharedMemory(create=True, size=10)
  3. >>> type(shm_a.buf)
  4. <class 'memoryview'>
  5. >>> buffer = shm_a.buf
  6. >>> len(buffer)
  7. 10
  8. >>> buffer[:4] = bytearray([22, 33, 44, 55])  # Modify multiple at once
  9. >>> buffer[4] = 100                           # Modify single byte at a time
  10. >>> # Attach to an existing shared memory block
  11. >>> shm_b = shared_memory.SharedMemory(shm_a.name)
  12. >>> import array
  13. >>> array.array('b', shm_b.buf[:5])  # Copy the data into a new array.array
  14. array('b', [22, 33, 44, 55, 100])
  15. >>> shm_b.buf[:5] = b'howdy'  # Modify via shm_b using bytes
  16. >>> bytes(shm_a.buf[:5])      # Access via shm_a
  17. b'howdy'
  18. >>> shm_b.close()   # Close each SharedMemory instance
  19. >>> shm_a.close()
  20. >>> shm_a.unlink()  # Call unlink only once to release the shared memory

The following example demonstrates a practical use of the SharedMemoryclass with NumPy arrays, accessing thesame numpy.ndarray from two distinct Python shells:

  1. >>> # In the first Python interactive shell
  2. >>> import numpy as np
  3. >>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
  4. >>> from multiprocessing import shared_memory
  5. >>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
  6. >>> # Now create a NumPy array backed by shared memory
  7. >>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
  8. >>> b[:] = a[:]  # Copy the original data into shared memory
  9. >>> b
  10. array([1, 1, 2, 3, 5, 8])
  11. >>> type(b)
  12. <class 'numpy.ndarray'>
  13. >>> type(a)
  14. <class 'numpy.ndarray'>
  15. >>> shm.name  # We did not specify a name so one was chosen for us
  16. 'psm_21467_46075'
  17.  
  18. >>> # In either the same shell or a new Python shell on the same machine
  19. >>> import numpy as np
  20. >>> from multiprocessing import shared_memory
  21. >>> # Attach to the existing shared memory block
  22. >>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
  23. >>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
  24. >>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
  25. >>> c
  26. array([1, 1, 2, 3, 5, 8])
  27. >>> c[-1] = 888
  28. >>> c
  29. array([  1,   1,   2,   3,   5, 888])
  30.  
  31. >>> # Back in the first Python interactive shell, b reflects this change
  32. >>> b
  33. array([  1,   1,   2,   3,   5, 888])
  34.  
  35. >>> # Clean up from within the second Python shell
  36. >>> del c  # Unnecessary; merely emphasizing the array is no longer used
  37. >>> existing_shm.close()
  38.  
  39. >>> # Clean up from within the first Python shell
  40. >>> del b  # Unnecessary; merely emphasizing the array is no longer used
  41. >>> shm.close()
  42. >>> shm.unlink()  # Free and release the shared memory block at the very end
  • class multiprocessing.managers.SharedMemoryManager([address[, authkey]])
  • A subclass of BaseManager which can beused for the management of shared memory blocks across processes.

A call to start() on aSharedMemoryManager instance causes a new process to be started.This new process's sole purpose is to manage the life cycleof all shared memory blocks created through it. To trigger the releaseof all shared memory blocks managed by that process, callshutdown() on the instance.This triggers a SharedMemory.unlink() call on all of theSharedMemory objects managed by that process and thenstops the process itself. By creating SharedMemory instancesthrough a SharedMemoryManager, we avoid the need to manually trackand trigger the freeing of shared memory resources.

This class provides methods for creating and returning SharedMemoryinstances and for creating a list-like object (ShareableList)backed by shared memory.

Refer to multiprocessing.managers.BaseManager for a descriptionof the inherited address and authkey optional input arguments and howthey may be used to connect to an existing SharedMemoryManager servicefrom other processes.

  • SharedMemory(size)

  • Create and return a new SharedMemory object with thespecified size in bytes.

  • ShareableList(sequence)

  • Create and return a new ShareableList object, initializedby the values from the input sequence.

The following example demonstrates the basic mechanisms of aSharedMemoryManager:

  1. >>> from multiprocessing.managers import SharedMemoryManager
  2. >>> smm = SharedMemoryManager()
  3. >>> smm.start()  # Start the process that manages the shared memory blocks
  4. >>> sl = smm.ShareableList(range(4))
  5. >>> sl
  6. ShareableList([0, 1, 2, 3], name='psm_6572_7512')
  7. >>> raw_shm = smm.SharedMemory(size=128)
  8. >>> another_sl = smm.ShareableList('alpha')
  9. >>> another_sl
  10. ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221')
  11. >>> smm.shutdown()  # Calls unlink() on sl, raw_shm, and another_sl

The following example depicts a potentially more convenient pattern for usingSharedMemoryManager objects via the with statement toensure that all shared memory blocks are released after they are no longerneeded:

  1. >>> with SharedMemoryManager() as smm:
  2. ...     sl = smm.ShareableList(range(2000))
  3. ...     # Divide the work among two processes, storing partial results in sl
  4. ...     p1 = Process(target=do_work, args=(sl, 0, 1000))
  5. ...     p2 = Process(target=do_work, args=(sl, 1000, 2000))
  6. ...     p1.start()
  7. ...     p2.start()  # A multiprocessing.Pool might be more efficient
  8. ...     p1.join()
  9. ...     p2.join()   # Wait for all work to complete in both processes
  10. ...     total_result = sum(sl)  # Consolidate the partial results now in sl

When using a SharedMemoryManager in a with statement, theshared memory blocks created using that manager are all released when thewith statement's code block finishes execution.

  • class multiprocessing.sharedmemory.ShareableList(_sequence=None, *, name=None)
  • Provides a mutable list-like object where all values stored within arestored in a shared memory block. This constrains storable values toonly the int, float, bool, str (less than 10M bytes each),bytes (less than 10M bytes each), and None built-in data types.It also notably differs from the built-in list type in that theselists can not change their overall length (i.e. no append, insert, etc.)and do not support the dynamic creation of new ShareableListinstances via slicing.

sequence is used in populating a new ShareableList full of values.Set to None to instead attach to an already existingShareableList by its unique shared memory name.

name is the unique name for the requested shared memory, as describedin the definition for SharedMemory. When attaching to anexisting ShareableList, specify its shared memory block's uniquename while leaving sequence set to None.

  • count(value)

  • Returns the number of occurrences of value.

  • index(value)

  • Returns first index position of value. Raises ValueError ifvalue is not present.

  • format

  • Read-only attribute containing the struct packing format used byall currently stored values.

  • shm

  • The SharedMemory instance where the values are stored.

The following example demonstrates basic use of a ShareableListinstance:

  1. >>> from multiprocessing import shared_memory
  2. >>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42])
  3. >>> [ type(entry) for entry in a ]
  4. [<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]
  5. >>> a[2]
  6. -273.154
  7. >>> a[2] = -78.5
  8. >>> a[2]
  9. -78.5
  10. >>> a[2] = 'dry ice'  # Changing data types is supported as well
  11. >>> a[2]
  12. 'dry ice'
  13. >>> a[2] = 'larger than previously allocated storage space'
  14. Traceback (most recent call last):
  15.   ...
  16. ValueError: exceeds available storage for existing str
  17. >>> a[2]
  18. 'dry ice'
  19. >>> len(a)
  20. 7
  21. >>> a.index(42)
  22. 6
  23. >>> a.count(b'howdy')
  24. 0
  25. >>> a.count(b'HoWdY')
  26. 1
  27. >>> a.shm.close()
  28. >>> a.shm.unlink()
  29. >>> del a  # Use of a ShareableList after call to unlink() is unsupported

The following example depicts how one, two, or many processes may access thesame ShareableList by supplying the name of the shared memory blockbehind it:

  1. >>> b = shared_memory.ShareableList(range(5))         # In a first process
  2. >>> c = shared_memory.ShareableList(name=b.shm.name)  # In a second process
  3. >>> c
  4. ShareableList([0, 1, 2, 3, 4], name='...')
  5. >>> c[-1] = -999
  6. >>> b[-1]
  7. -999
  8. >>> b.shm.close()
  9. >>> c.shm.close()
  10. >>> c.shm.unlink()