Speed

https://d33wubrfki0l68.cloudfront.net/f33b6d6a18c20fd1a83677c8ea09be0a08e43231/e3c2f/_images/33175625804_e225b90f3e_k_d.jpg
CPython, the most commonly used implementation of Python, is slow for CPU boundtasks. PyPy is fast.

Using a slightly modified version of David Beazley’s CPU bound test code(added loop for multiple tests), you can see the difference between CPythonand PyPy’s processing.

  1. # PyPy
  2. $ ./pypy -V
  3. Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:10)
  4. [PyPy 1.7.0 with GCC 4.4.3]
  5. $ ./pypy measure2.py
  6. 0.0683999061584
  7. 0.0483210086823
  8. 0.0388588905334
  9. 0.0440690517426
  10. 0.0695300102234
  1. # CPython
  2. $ ./python -V
  3. Python 2.7.1
  4. $ ./python measure2.py
  5. 1.06774401665
  6. 1.45412397385
  7. 1.51485204697
  8. 1.54693889618
  9. 1.60109114647

Context

The GIL

The GIL (Global Interpreter Lock) is how Python allows multiple threads tooperate at the same time. Python’s memory management isn’t entirely thread-safe,so the GIL is required to prevent multiple threads from running the samePython code at once.

David Beazley has a great guide on how the GIL operates. He also covers thenew GIL in Python 3.2. His results show that maximizing performance in aPython application requires a strong understanding of the GIL, how it affectsyour specific application, how many cores you have, and where your applicationbottlenecks are.

C Extensions

The GIL

Special care must be taken when writing C extensions to make sure youregister your threads with the interpreter.

C Extensions

Cython

Cython implements a superset of the Python languagewith which you are able to write C and C++ modules for Python. Cython alsoallows you to call functions from compiled C libraries. Using Cython allowsyou to take advantage of Python’s strong typing of variables and operations.

Here’s an example of strong typing with Cython:

  1. def primes(int kmax):
  2. """Calculation of prime numbers with additional
  3. Cython keywords"""
  4.  
  5.     cdef int n, k, i
  6.     cdef int p[1000]
  7.     result = []
  8.     if kmax > 1000:
  9.         kmax = 1000
  10.     k = 0
  11.     n = 2
  12.     while k < kmax:
  13.         i = 0
  14.         while i < k and n % p[i] != 0:
  15.             i = i + 1
  16.         if i == k:
  17.             p[k] = n
  18.             k = k + 1
  19.             result.append(n)
  20.         n = n + 1
  21.     return result

This implementation of an algorithm to find prime numbers has some additionalkeywords compared to the next one, which is implemented in pure Python:

  1. def primes(kmax):
  2. """Calculation of prime numbers in standard Python syntax"""
  3.  
  4.     p = range(1000)
  5.     result = []
  6.     if kmax > 1000:
  7.         kmax = 1000
  8.     k = 0
  9.     n = 2
  10.     while k < kmax:
  11.         i = 0
  12.         while i < k and n % p[i] != 0:
  13.             i = i + 1
  14.         if i == k:
  15.             p[k] = n
  16.             k = k + 1
  17.             result.append(n)
  18.         n = n + 1
  19.     return result

Notice that in the Cython version you declare integers and integer arraysto be compiled into C types while also creating a Python list:

  1. def primes(int kmax):
  2.     """Calculation of prime numbers with additional
  3.     Cython keywords"""
  4.  
  5.     cdef int n, k, i
  6.     cdef int p[1000]
  7.     result = []
  1. def primes(kmax):
  2.     """Calculation of prime numbers in standard Python syntax"""
  3.  
  4.     p = range(1000)
  5.     result = []

What is the difference? In the upper Cython version you can see thedeclaration of the variable types and the integer array in a similar way asin standard C. For example cdef int n,k,i in line 3. This additional typedeclaration (i.e. integer) allows the Cython compiler to generate moreefficient C code from the second version. While standard Python code is savedin .py files, Cython code is saved in .pyx files.

What’s the difference in speed? Let’s try it!

  1. import time
  2. #activate pyx compiler
  3. import pyximport
  4. pyximport.install()
  5. #primes implemented with Cython
  6. import primesCy
  7. #primes implemented with Python
  8. import primes
  9.  
  10. print "Cython:"
  11. t1= time.time()
  12. print primesCy.primes(500)
  13. t2= time.time()
  14. print "Cython time: %s" %(t2-t1)
  15. print ""
  16. print "Python"
  17. t1= time.time()
  18. print primes.primes(500)
  19. t2= time.time()
  20. print "Python time: %s" %(t2-t1)

These lines both need a remark:

  1. import pyximport
  2. pyximport.install()

The pyximport module allows you to import .pyx files (e.g.,primesCy.pyx) with the Cython-compiled version of the primes_function. The _pyximport.install() command allows the Python interpreter tostart the Cython compiler directly to generate C code, which is automaticallycompiled to a .so C library. Cython is then able to import thislibrary for you in your Python code, easily and efficiently. With thetime.time() function you are able to compare the time between these 2different calls to find 500 prime numbers. On a standard notebook (dual coreAMD E-450 1.6 GHz), the measured values are:

  1. Cython time: 0.0054 seconds
  2.  
  3. Python time: 0.0566 seconds

And here is the output of an embedded ARM beaglebone machine:

  1. Cython time: 0.0196 seconds
  2.  
  3. Python time: 0.3302 seconds

Pyrex

Shedskin?

Concurrency

Concurrent.futures

The concurrent.futures module is a module in the standard library thatprovides a “high-level interface for asynchronously executing callables”. Itabstracts away a lot of the more complicated details about using multiplethreads or processes for concurrency, and allows the user to focus onaccomplishing the task at hand.

The concurrent.futures module exposes two main classes, theThreadPoolExecutor and the ProcessPoolExecutor. The ThreadPoolExecutorwill create a pool of worker threads that a user can submit jobs to. These jobswill then be executed in another thread when the next worker thread becomesavailable.

The ProcessPoolExecutor works in the same way, except instead of using multiplethreads for its workers, it will use multiple processes. This makes it possibleto side-step the GIL; however, because of the way things are passed to workerprocesses, only picklable objects can be executed and returned.

Because of the way the GIL works, a good rule of thumb is to use aThreadPoolExecutor when the task being executed involves a lot of blocking(i.e. making requests over the network) and to use a ProcessPoolExecutorexecutor when the task is computationally expensive.

There are two main ways of executing things in parallel using the twoExecutors. One way is with the map(func, iterables) method. This worksalmost exactly like the builtin map() function, except it will executeeverything in parallel.

  1. from concurrent.futures import ThreadPoolExecutor
  2. import requests
  3.  
  4. def get_webpage(url):
  5.     page = requests.get(url)
  6.     return page
  7.  
  8. pool = ThreadPoolExecutor(max_workers=5)
  9.  
  10. my_urls = ['http://google.com/']*10  # Create a list of urls
  11.  
  12. for page in pool.map(get_webpage, my_urls):
  13.     # Do something with the result
  14.     print(page.text)

For even more control, the submit(func, args, *kwargs) method will schedulea callable to be executed ( as func(args, *kwargs)) and returns a Futureobject that represents the execution of the callable.

The Future object provides various methods that can be used to check on theprogress of the scheduled callable. These include:

cancel()
Attempt to cancel the call.
cancelled()
Return True if the call was successfully cancelled.
running()
Return True if the call is currently being executed and cannot becancelled.
done()
Return True if the call was successfully cancelled or finished running.
result()
Return the value returned by the call. Note that this call will block untilthe scheduled callable returns by default.
exception()
Return the exception raised by the call. If no exception was raised thenthis returns None. Note that this will block just like result().
adddone_callback(fn)
Attach a callback function that will be executed (as _fn(future)) when thescheduled callable returns.
  1. from concurrent.futures import ProcessPoolExecutor, as_completed
  2.  
  3. def is_prime(n):
  4.     if n % 2 == 0:
  5.         return n, False
  6.  
  7.     sqrt_n = int(n**0.5)
  8.     for i in range(3, sqrt_n + 1, 2):
  9.         if n % i == 0:
  10.             return n, False
  11.     return n, True
  12.  
  13. PRIMES = [
  14.     112272535095293,
  15.     112582705942171,
  16.     112272535095293,
  17.     115280095190773,
  18.     115797848077099,
  19.     1099726899285419]
  20.  
  21. futures = []
  22. with ProcessPoolExecutor(max_workers=4) as pool:
  23.     # Schedule the ProcessPoolExecutor to check if a number is prime
  24.     # and add the returned Future to our list of futures
  25.     for p in PRIMES:
  26.         fut = pool.submit(is_prime, p)
  27.         futures.append(fut)
  28.  
  29. # As the jobs are completed, print out the results
  30. for number, result in as_completed(futures):
  31.     if result:
  32.         print("{} is prime".format(number))
  33.     else:
  34.         print("{} is not prime".format(number))

The concurrent.futures module contains two helper functions for working withFutures. The as_completed(futures) function returns an iterator over the listof futures, yielding the futures as they complete.

The wait(futures) function will simply block until all futures in the list offutures provided have completed.

For more information, on using the concurrent.futures module, consult theofficial documentation.

threading

The standard library comes with a threading module that allows a user towork with multiple threads manually.

Running a function in another thread is as simple as passing a callable andits arguments to Thread’s constructor and then calling start():

  1. from threading import Thread
  2. import requests
  3.  
  4. def get_webpage(url):
  5.     page = requests.get(url)
  6.     return page
  7.  
  8. some_thread = Thread(get_webpage, 'http://google.com/')
  9. some_thread.start()

To wait until the thread has terminated, call join():

  1. some_thread.join()

After calling join(), it is always a good idea to check whether the thread isstill alive (because the join call timed out):

  1. if some_thread.is_alive():
  2.     print("join() must have timed out.")
  3. else:
  4.     print("Our thread has terminated.")

Because multiple threads have access to the same section of memory, sometimesthere might be situations where two or more threads are trying to write to thesame resource at the same time or where the output is dependent on the sequenceor timing of certain events. This is called a data race or race condition.When this happens, the output will be garbled or you may encounter problemswhich are difficult to debug. A good example is this Stack Overflow post_.

The way this can be avoided is by using a Lock that each thread needs toacquire before writing to a shared resource. Locks can be acquired and releasedthrough either the contextmanager protocol (with statement), or by usingacquire() and release() directly. Here is a (rather contrived) example:

  1. from threading import Lock, Thread
  2.  
  3. file_lock = Lock()
  4.  
  5. def log(msg):
  6.     with file_lock:
  7.         open('website_changes.log', 'w') as f:
  8.             f.write(changes)
  9.  
  10. def monitor_website(some_website):
  11.     """
  12.     Monitor a website and then if there are any changes,
  13.     log them to disk.
  14.     """
  15.     while True:
  16.         changes = check_for_changes(some_website)
  17.         if changes:
  18.             log(changes)
  19.  
  20. websites = ['http://google.com/', ... ]
  21. for website in websites:
  22.     t = Thread(monitor_website, website)
  23.     t.start()

Here, we have a bunch of threads checking for changes on a list of sites andwhenever there are any changes, they attempt to write those changes to a fileby calling log(changes). When log() is called, it will wait to acquirethe lock with with file_lock:. This ensures that at any one time, only onethread is writing to the file.

Spawning Processes

Multiprocessing

原文: https://docs.python-guide.org/scenarios/speed/