One of the big requests whilst we were replacing httplib with the requests package in 2.0 was why didn’t we use a HTTP library that supports asynchronous API calls.

The intention for 2.0 and replacing the HTTP backend classes was to improve the usability of the project, by making SSL certificates easier to manage, improving the maintainability of our source code by using an active 3rd party package and also improving performance and stability.

Apache Libcloud already has documentation on threaded libraries like gevent and callback-based libraries like Twisted, see using libcloud in multithreaded environments for examples.

PEP 492, implemented in Python 3.5 provides a new coroutine protocol using methods, __await__ for classes, a coroutine method wrapper, or a method that returns a coroutine object. Also async iterators and context managers have been introduced.

We would like to take advantage of the new language features by offering APIs in Apache Libcloud without breaking backward compatibility and compatibility for users of <Python 3.5.

Use cases for this would be:

• Being able to fetch Node or StorageObjects from multiple geographies or drivers simultaneously.
• Being able to quickly upload or download storage objects by parallelizing operations on the StorageDriver.
• Being able to call a long-running API method (e.g. generate report), whilst running other code.

Design 1 - async context managers PR 1016

This design would allow drivers to operate in 2 modes, the first is for synchronous method calls, they return list or object data as per usual. The second mode, API methods like NodeDriver.list_nodes would return a coroutine object and could be awaited or gathered using an event loop.

import asyncio

from integration.driver.test import TestNodeDriver
from libcloud.async_util import AsyncSession

driver = TestNodeDriver('apache', 'libcloud')

async def run():
    # regular API call
    nodes = driver.list_nodes()

    async with AsyncSession(driver) as async_instance:
        nodes = await async_instance.list_nodes()

    assert len(nodes) == 2

loop = asyncio.get_event_loop()
loop.run_until_complete(run())
loop.close()

Design 2 - Additional methods in each driver for coroutines PR 1027

This is the second design concept for async support in Libcloud.

The concept here is to have Asynchronous Mixins, LibcloudConnection uses requests and LibcloudAsyncConnection uses aiohttp for async transport see

The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the API for the drivers to consume see

The drivers then use this mixin for their custom connection classes, e.g.

class GoogleStorageConnection(ConnectionUserAndKey, AsyncConnection):
    ...

They then inherit from libcloud.storage.base.StorageAsyncDriver, which uses a new set of base methods, e.g. iterate_containers_async and can be implemented like this:

        async def iterate_containers_async(self):
            response = await self.connection.request_async('/')
            if response.status == httplib.OK:
                containers = self._to_containers(obj=response.object,
                                                 xpath='Buckets/Bucket')
                return containers
    
            raise LibcloudError('Unexpected status code: %s' % (response.status),
                                driver=self)

Now the consumer can more or less do this:

from libcloud.storage.providers import get_driver
from libcloud.storage.types import Provider

import asyncio

GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE)
driver = GoogleStorageDriver(key=KEY, secret=SECRET)

def do_stuff_with_object(obj):
    print(obj)

async def run():
    tasks = []
    async for container in driver.iterate_containers_async():
        async for obj in driver.iterate_container_objects_async(container):
            tasks.append(asyncio.ensure_future(do_stuff_with_object(obj)))
    await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())
loop.close()

Design 3 - Initializer with “async” mode

This option is similar to 2, except that if a driver is instantiated with “async=True”, then all driver class methods would return coroutine objects. Internally, it would patch the Connection class with the AsyncConnection class.

The downside of this is that all method calls to a driver would need to be awaited or used by an event loop.

from libcloud.storage.providers import get_driver
from libcloud.storage.types import Provider

import asyncio

GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE)
driver = GoogleStorageDriver(key=KEY, secret=SECRET, async=True)

def do_stuff_with_object(obj):
    print(obj)

async def run():
    tasks = []
    async for container in driver.iterate_containers():
        async for obj in driver.iterate_container_objects(container):
            tasks.append(asyncio.ensure_future(do_stuff_with_object(obj)))
    await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())
loop.close()

Give us feedback

Got a better idea? Have an API or design, the question we’re asking is “if you wanted to use Libcloud for an async application, what would the code look like?” This helps us design the API and the implementation details can follow.

Feel free to comment on the mailing list or on the pull requests, or raise your own pull-request with an API design.

We are pleased to announce the release of Libcloud 2.0.0rc2.

This release brings many new features, improvements, bug-fixes, and drivers.

Release highlights

• Apache Libcloud 2.0 series replaces the use of Python httplib with a hard dependency on the requests package. Users’ no longer have to specific Certificate Authority bundles when using Apache Libcloud
• 10% performance improvement through the use of HTTP sessions
• Support for buffered IO streams for storage drivers
• Support for Python 3.6, deprecation of Python 3.2

A detailed description of the 2.0 changes is documented here

Note that 2.0.0rc1 was not released to PyPi as 4 breaking issues were discovered by users. The changelog for both 2.0.0rc2 and rc1 is below.

Release highlights for 2.0.0rc2

Compute

• Fix a bug in profitbricks driver where listing snapshots would request a malformed URL
• Fix LIBCLOUD-806 bug where vsphere driver cannot be instantiated
• [google compute] Improve performance of list nodes by caching volume information.

Common

• Fix LIBCLOUD_DEBUG trying to decompress already decompressed responses
• Added an integration test API and a test suite for validating functionality without mocking any libcloud subsystems
• Fix for Linode classes since 2.0x
• Fix CertificateConnection not correctly signing requests in 2.0rc1, impacted Azure classic driver, OpenStack and Docker driver
• Change Cloudscale to cloudscale.ch.
• Explicitly check if response is None in RawResponse class

Compute

• Outscale SAS doc improvements and logo update
• [GCE] Allow preemptible instances to be created
• Add support for forcing detachment of EBS volumes to EC2 driver
• Fix Public IP not assigned when creating NIC on Azure ARM
• [ONAPP] Add list images support for OnApp driver
• [EC2] Add r4 instance types for AWS
• [EC2] support for AWS eu-west-2 and ca-central-1 regions
• [EC2] Add P2 GPU instance types
• [EC2] Add method to modify snapshot attribute for EC2
• [Linode] Add start, stop instance methods and fix incorrect state TERMINATED to STOPPED
• [EC2] Add ENA support for EC2 compute images
• [Azure ARM] fix typeerror on ex_list_nics
• [GCE] allow delete instances from managed group

Storage

• Reintroduce S3 multipart upload support with signature v4

Changes Apache Libcloud 2.0.0rc1

Common

• Fix DEBUG mode, also add support for using io.StringIO as the file handle when calling libcloud.enable_debug
• Introduction of the requests package as the mechanism for making HTTP requests for all drivers
• Fix bug where custom port and secure flag would not get propagated to connection class
• Fix bug where custom port would not get propagated to connection
• Fix bug where instantiating a connection from URL and then requesting an action with a leading / would lead to a malformed URL

Compute

• Fix a bug in profitbricks driver where listing snapshots would request a malformed URL
• Fix LIBCLOUD-806 bug where vsphere driver cannot be instantiated
• [google compute] Improve performance of list nodes by caching volume information.

Full change log can be found at here.

Special thank you

I would like to wish a special thank you to all of our community contributors for their ongoing support to the project.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==2.0.0rc2

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==2.0.0rc2

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We are pleased to announce the release of Libcloud 1.5.0.

This release brings many new features, improvements, bug-fixes, and drivers.

Release highlights

• [azure] New method for accessing rate cards.
• [openstack] Add new Connection class to support VOMS proxys to keystone servers.
• [ec2] Added m4 instances to us-gov and brazil, added m4.16xlarge to all.
• Add new CloudScale.ch driver
• [dimensiondata] Added support for 2.4 API, added support for image import, cloning. Add feature for changing NIC VLANs, add feature for changing NIC order for a server.

Full change log can be found at here.

Special thank you

I would like to wish a special thank you to all of our community contributors for their ongoing support to the project.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.5.0

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.5.0

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We are pleased to announce the release of Libcloud 1.4.0.

This release brings many new features, improvements, bug-fixes, and drivers.

Release highlights

The release includes a new Azure ARM driver and an Amazon Application Load Balancer (ALB) driver.

Full change log can be found at here.

Special thank you

I would like to wish a special thank you to all of our community contributors for their ongoing support to the project.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.4.0

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.4.0

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We are pleased to announce the release of Libcloud 1.3.0.

This release brings many new features, improvements, bug-fixes, and drivers.

Important changes

• RunAbove driver is now the OVH cloud driver because of changes in the platform. Users will get a deprecated error message and pointed to the website for more information.
• Fixed support for SLES/OpenSUSE 12, now checks the default certificate path (does not impact users using certifi).
• DigitalOcean v1 API has been deprecated in favour of the new 2.0 API.

Release highlights

Rancher Driver

Mario Loria contributed a full Rancher driver for our container abstraction interface. Documentation is available with examples of usage, you can use the driver to deploy containers, services, stacks or operate and maintain existing deployments! Thanks Mario.

from libcloud.container.types import Provider
from libcloud.container.providers import get_driver
from libcloud.container.base import ContainerImage

driver = get_driver(Provider.RANCHER)

connection = driver("MYRANCHERACCESSKEY", "MYRANCHERSECRETKEY",
                    host="17.23.66.4", port=443)

image = ContainerImage("hastebin", "hastebin", "rlister/hastebin", "latest",
                       driver=None)

new_service = connection.ex_deploy_service(name="excitingservice", image=image,
                                           environmentid="1e2",
                                           environment={
                                               "STORAGE_TYPE": "file"
                                           })

New API

As well as the direct get_driver API, there is now a short-hand API for users to choose.

import libcloud

cls = libcloud.get_driver(libcloud.DriverType.COMPUTE, libcloud.DriverType.COMPUTE.RACKSPACE)

Full change log can be found at here.

Special thank you

I would like to wish a special thank you to all of our community contributors for their ongoing support to the project.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.3.0

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.3.0

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.