Using an HTTP / HTTPS proxy

Note

  1. Support for HTTP proxies is available in Libcloud v0.16.0 and higher.
  2. Support for HTTPS proxies is available in Libcloud v2.5.1-dev and higher.
  3. In versions prior to v2.5.1-dev, driver.connection.set_http_proxy() method is broken and you need to use driver.connection.connection.set_http_proxy() instead.

Libcloud supports using an HTTP / HTTPS proxy for outgoing HTTP and HTTPS requests.

Proxy support has been tested with the following Python versions:

  • Python 2.7 / PyPy
  • Python 3.4
  • Python 3.6
  • Python 3.7

You can specify which HTTP(s) proxy to use using one of the approaches described below:

  • By setting http_proxy / https_proxy environment variable (this setting is system / process wide)
  • By passing http_proxy argument to the libcloud.common.base.LibcloudConnection class constructor (this setting is local to the connection instance)
  • By calling libcloud.common.base.LibcloudConnection.set_http_proxy() method aka driver.connection.connection.set_http_proxy (this setting is local to the connection instance)

Known limitations

  • Only HTTP basic access authentication proxy authorization method is supported
  • If you are using HTTPS proxy you need to configure Libcloud to use CA cert bundle path which is used by the proxy server. See an example below on how to do that.

Examples

This section includes some code examples which show how to use an HTTP(s) proxy with Libcloud.

1. Using http_proxy / htps_proxy environment variable

By setting http_proxy / https_proxy environment variable you can specify which proxy to use for all of the outgoing requests for a duration / life-time of the process or a script.

Without authentication (http proxy):

  1. http_proxy=http://<proxy hostname>:<proxy port> python my_script.py

Without authentication (https proxy):

  1. http_proxy=https://<proxy hostname>:<proxy port> python my_script.py
  2. # or
  3. https_proxy=https://<proxy hostname>:<proxy port> python my_script.py

With basic auth authentication (http proxy):

  1. http_proxy=http://<username>:<password>@<proxy hostname>:<proxy port> python my_script.py

2. Passing proxy_url argument to the connection class constructor

. note:

  1. Some drivers don't correctly pass ``proxy_url`` argument to the connection
  2. class and don't support ``proxy_url`` constructor argument.
  4. If you pass this argument to the driver constructor, but it doesn't appear
  5. to be working, it's likely the driver doesn't support this method.
  7. In such scenarios, you are advised to use some other method of setting a
  8. proxy (e.g. by setting an environment variable or by using
  9. :meth:`libcloud.common.base.LibcloudConnection.set_http_proxy` method).

By passing proxy_url argument to the libcloud.common.base.Connection class constructor, you can specify which proxy to use for a particular connection.

  1. from libcloud.compute.types import Provider
  2. from libcloud.compute.providers import get_driver
  4. HTTP_PROXY_URL_NO_AUTH_1 = 'http://<proxy hostname 1>:<proxy port 2>'
  5. HTTPS_PROXY_URL_NO_AUTH_1 = 'https://<proxy hostname 1>:<proxy port 2>'
  7. cls = get_driver(Provider.RACKSPACE)
  9. # 1. Use http proxy
  10. driver = cls('username', 'api key', region='ord',
  11.              proxy_url=HTTP_PROXY_URL_NO_AUTH_1)
  13. # 2. Use https proxy
  14. driver = cls('username', 'api key', region='ord',
  15.              proxy_url=HTTPS_PROXY_URL_NO_AUTH_1)

3. Calling set_http_proxy method

Calling set_http_proxy method allows you to specify which proxy to use for all the outgoing requests which follow set_http_proxy method call.

This method also allows you to use a different proxy for each request as shown in the example below.

  1. from pprint import pprint
  3. from libcloud.compute.types import Provider
  4. from libcloud.compute.providers import get_driver
  6. HTTP_PROXY_URL_NO_AUTH_1 = 'http://<proxy hostname 1>:<proxy port 2>'
  7. HTTP_PROXY_URL_NO_AUTH_2 = 'http://<proxy hostname 1>:<proxy port 2>'
  8. HTTP_PROXY_URL_BASIC_AUTH = 'http://<user>:<pass>@<proxy hostname>:<port>'
  10. cls = get_driver(Provider.RACKSPACE)
  11. driver = cls('username', 'api key', region='ord')
  13. # Use proxy 1 for this request
  14. driver.connection.connection.set_http_proxy(proxy_url=HTTP_PROXY_URL_NO_AUTH_1)
  15. pprint(driver.list_nodes())
  17. # Use proxy 2 for this request
  18. driver.connection.connection.set_http_proxy(proxy_url=HTTP_PROXY_URL_NO_AUTH_2)
  19. pprint(driver.list_nodes())

4. Using an HTTPS proxy

This example shows how to use an HTTPS proxy.

  1. import os.path
  2. from pprint import pprint
  4. from libcloud.compute.types import Provider
  5. from libcloud.compute.providers import get_driver
  7. import libcloud.security
  9. HTTPS_PROXY_URL_NO_AUTH = 'https://<proxy hostname 1>:<proxy port 2>'
  11. # 1. Use a custom CA bundle which is used by proxy server
  12. # This example uses CA cert bundle used by mitmproxy proxy server
  13. libcloud.security.CA_CERTS_PATH = os.path.expanduser(
  14.     '~/.mitmproxy/mitmproxy-ca-cert.pem')
  16. # User an https proxy for subsequent requests
  17. cls = get_driver(Provider.RACKSPACE)
  18. driver = cls('username', 'api key')
  20. driver.connection.connection.set_http_proxy(proxy_url=HTTPS_PROXY_URL_NO_AUTH)
  21. pprint(driver.list_nodes())

To use an HTTPS proxy, you also need to configure Libcloud to use CA cert bundle which is used by the HTTPS proxy server, to verify outgoing https request. If you don’t do that, you will see errors similar to the one below:

  1. SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Keep in mind that you will also receive a similar error message if you try to use HTTP proxy for HTTPS requests.