Blog

Libcloud 1.1.0 released

We are pleased to announce the release of Libcloud 1.1.0.

This is a first release after v1.0.0. It includes various improvements and bug-fixes.

Release highlights

  • Support for automatic SNI (SSL extension) using the hostname supplied to connect to.
  • Various improvements in the libvirt driver
  • Various improvements in the DimensionData driver
  • Various improvements in the Aliyun driver

Full change log can be found at here.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.1.0

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.1.0

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/v1.1.0/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don't forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

Libcloud 1.0.0 released

We are pleased to announce the release of Libcloud 1.0.0.

This first release in the 1.0 series which means it brings many new features, improvements, bug-fixes, and drivers. The 1.0 series includes 2 new driver types, container-as-a-service and backup-as-a-service.

Release highlights

This includes:

  • New backup-as-a-service and container-as-a-service driver types
  • Deprecated drivers that were no longer available such as Ninefold, IBM SCE more details
  • Add support for authenticating against Keystone and OpenStack based clouds using OpenID Connect tokens.
  • GCE nodes can be launched in a subnetwork and support for subnetworks
  • Add Image Family support for GCE
  • Add new Perth, Australia and Manila, Philippines region to the CloudSigma v2 driver.
  • Add new S3 RGW storage driver.
  • The Amazon EC2 driver has been changed to use region codes instead of separate drivers for each region.
  • Introduce new list_regions class method on the base driver class
  • Support for Dimension Data backup
  • Added NSOne, LuaDNS, NearlyFreeSpeech.NET, BuddyNS, DNSPod DNS drivers
  • Added Aliyun compute, load balancer and storage drivers
  • Added Outscale storage driver

Full change log can be found at here.

Special thank you

I would like to wish a special thank you to all of our community contributors for their ongoing support to the project.

Tomaz Muraus, Paul Querna, Jerry Chen, Jeff Dunham, Eric Johnson, Jed Smith, Rick Wright, Roman Bogorodskiy, Mathew Baldwin, David Crossland, Alex Gaynor, Allard Hoeve, Sebastien Goasguen, Franck Cuny, Ken Dreyer, Javier Castillo II, Wido den Hollander, Michael Bennett, Anthony Monthe, Lior Goikhburg, Geunwoo Shin, Juan Font, "schaubl", Misha Brukman, Oltjano Terpollari, Peter Schmidt, Ming Sheng, Gertjan Oude Lohuis, Markos Gogoulos, Bernard Paques, Evgeny Egorochkin, Oleg, Atsushi Sasaki, Miguel Caballer, Kumar, Gavin McDonald, Roeland Kuipers, Brian Curtin, Ivan Kusalic, Vanč Levstik, None, Anže Pečar, "MrBasset", None, Javier M. Mellid, Greg Hill, None, Stefan Friesel, Mark Maglana, Alejandro Gabriel Pereira, Kyle Long, Teemu Vesala, Filipe Silva, Jay, Bernard Kerckenaere, John Obelenus, Jon Chen, Larry Lui,

Important breaking change- Amazon EC2 driver changes

The Amazon EC2 API was updated to consolidate the regional-based drivers into a single driver with a region argument in the constructor.

Amazon Instances should now be instantiated using the following syntax:

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

cls = get_driver(Provider.EC2)
driver = cls('access key', 'secret key', region='us-east-1')

This brings the Amazon API inline with the other drivers, makes it easier to maintain and switch between regions.

Bug fixes

General

  • Fix a bug with consuming stdout and stderr in the paramiko SSH client which would manifest itself under very rare condition when a consumed chunk only contained a single byte or part of a multi byte UTF-8 character. [Lakshmi Kannan, Tomaz Muraus]

  • Increase default chunk size from 1024 to 4096 bytes in the paramiko SSH client. This results in smaller number of receive calls on the average. [Tomaz Muraus]

  • Fix to Dimension Data API address for Middle-East and Africa (GITHUB-700) [Anthony Shaw]

  • Throw a more user-friendly exception on "No address associated with hostname". (GITHUB-711, GITHUB-714, LIBCLOUD-803) [Tomaz Muraus, Scott Crunkleton]

  • Remove deprecated provider constants with the region in the name and related driver classes (e.g. EC2_US_EAST, etc.).

    Those drivers have moved to single provider constant + region constructor argument model. [Tomaz Muraus]

New or deprecated drivers

Compute

  • Deprecated IBM SCE, HP Helion, OpSource, Ninefold and CloudFrames drivers, removed driver code and tests. (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • Introduced error messages (libcloud.compute.deprecated) for deprecated drivers (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • New Compute drivers- BSNL, Indosat, Med-1, NTT-America, Internet Solutions (GITHUB-700) [Anthony Shaw]

  • New driver for Aliyun Elastic Compute Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Storage

  • Added Outscale storage driver (GITHUB-730) [Javier M. Mellid]

  • New driver for Aliyun OSS Storage Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Loadbalancer

  • New driver for Aliyun SLB Loadbalancer Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

DNS

  • Added NearlyFreeSpeech.net (NSFN) driver [Ken Drayer]

  • Added Lua DNS driver [Oltjano Terpollari]

  • Added NSOne driver [Oltjano Terpollari]

Bug fixes

  • Fix a bug in the GoDaddy driver - make sure host attribute on the connection class is correctly set to the hostname. [Tomaz Muraus]

  • Fix handling of MX records in the Gandi driver. (GITHUB-718) [Ryan Lee]

Improvements

Compute

  • Introduce new list_regions class method on the base driver class. This method is to be used with provider drivers which support multiple regions and region constructor argument. It allows users to enumerate available / supported regions. [Tomaz Muraus]

  • [dimension data] added support for VMWare tools VM information inside list_nodes responses (GITHUB-734) [Jeff Dunham]

  • [ec2] added exencrypted and exkmskeyid optional parameters to the create volume method (GITHUB-729) [Viktor Ognev]

  • [dimension data] added support for managing host anti-affinity rules, added paging support to all supported calls and added support for requesting priority ordering when creating ACL rules (GITHUB-726) [Jeff Dunham]

  • Addition of Dimension Data Australia federal government region to dimension data drivers. (GITHUB-700) [Anthony Shaw]

  • [openstack] when creating floating IPs, added pool_id as an optional argument (GITHUB-725) [marko-p]

  • [google compute] Added setMachineType method to allow for changing sizes of instances (GITHUB-721) [Eric Johnson]

  • [google compute] allow bypassing image search in standard project list (GITHUB-713) [Max Illfelder]

  • Add support for requesting a MKS token for accessing the remote console in VMware vCloud driver (GITHUB-706) [Juan Font Alonso]

  • Add support in VMware vCloud driver for v5.5 API, with snapshot support (GITHUB-658) [Juan Font Alonso]

  • Added support for adding a family to an image on Google Compute Driver (GITHUB-704) [Max Illfelder]

  • Fix to set default signature version for AWS Seoul region to v4, removed non-supported size (hs1.xlarge) (GITHUB-684) [Geunwoo Shin]

  • Support filtering by location in list_nodes for dimension data compute driver fix lack of paging support (GITHUB-691) [Jeff Dunham]

  • Support for filtering by IPv4, IPv6, network, network domain, VLAN in Dimension data driver. (GITHUB-694) [Jeff Dunham]

  • Added Node.created_at which, on supported drivers, contains the datetime the node was first started. (GITHUB-698) [Allard Hoeve] [Rick van de Loo]

Storage

  • Improvements to Google Auth for Storage and Compute and MIME bug fix (LIBCLOUD-800, GITHUB-689) [Scott Crunkleton]

  • Implement get_container, get_object and upload_object_via_stream methods in the Backblaze B2 storage driver.

    Note: Backblaze API doesn't upload streaming uploads so when using upload_object_via_stream whole file is read and buffered in memory. (GITHUB-696) [Jay jshridha]

Backup

  • Dimension Data - added additional testing, fixed bug on client response naming, added support for adding backup clients to a backup enabled node. (GITHUB-692, GITHUB-693, GITHUB-695) [Jeff Dunham]

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.0.0

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.0.0

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don't forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

Libcloud 1.0.0-rc2 released

We are pleased to announce the release of Libcloud 1.0.0-rc2.

This the second pre-release in the 1.0.0 series which means it brings many new features, improvements, bug-fixes, and DNS drivers.

Release highlights

This includes:

  • Deprecated drivers that were no longer available such as Ninefold, IBM SCE more details
  • The Amazon EC2 driver has been changed to use region codes instead of separate drivers for each region.
  • Introduce new list_regions class method on the base driver class
  • Support for Dimension Data backup
  • Added NSOne, LuaDNS, NearlyFreeSpeech.NET DNS drivers
  • Added Aliyun compute, load balancer and storage drivers
  • Added Outscale storage driver

Full change log can be found at here.

Important breaking change- Amazon EC2 driver changes

The Amazon EC2 API was updated to consolidate the regional-based drivers into a single driver with a region argument in the constructor.

Amazon Instances should now be instantiated using the following syntax:

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

cls = get_driver(Provider.EC2, region='us-east-i1')
driver = cls('access key', 'secret key')

This brings the Amazon API inline with the other drivers, makes it easier to maintain and switch between regions.

Bug fixes

General

  • Fix a bug with consuming stdout and stderr in the paramiko SSH client which would manifest itself under very rare condition when a consumed chunk only contained a single byte or part of a multi byte UTF-8 character. [Lakshmi Kannan, Tomaz Muraus]

  • Increase default chunk size from 1024 to 4096 bytes in the paramiko SSH client. This results in smaller number of receive calls on the average. [Tomaz Muraus]

  • Fix to Dimension Data API address for Middle-East and Africa (GITHUB-700) [Anthony Shaw]

  • Throw a more user-friendly exception on "No address associated with hostname". (GITHUB-711, GITHUB-714, LIBCLOUD-803) [Tomaz Muraus, Scott Crunkleton]

  • Remove deprecated provider constants with the region in the name and related driver classes (e.g. EC2_US_EAST, etc.).

    Those drivers have moved to single provider constant + region constructor argument model. [Tomaz Muraus]

New or deprecated drivers

Compute

  • Deprecated IBM SCE, HP Helion, OpSource, Ninefold and CloudFrames drivers, removed driver code and tests. (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • Introduced error messages (libcloud.compute.deprecated) for deprecated drivers (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • New Compute drivers- BSNL, Indosat, Med-1, NTT-America, Internet Solutions (GITHUB-700) [Anthony Shaw]

  • New driver for Aliyun Elastic Compute Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Storage

  • Added Outscale storage driver (GITHUB-730) [Javier M. Mellid]

  • New driver for Aliyun OSS Storage Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Loadbalancer

  • New driver for Aliyun SLB Loadbalancer Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

DNS

  • Added NearlyFreeSpeech.net (NSFN) driver [Ken Drayer]

  • Added Lua DNS driver [Oltjano Terpollari]

  • Added NSOne driver [Oltjano Terpollari]

Bug fixes

  • Fix a bug in the GoDaddy driver - make sure host attribute on the connection class is correctly set to the hostname. [Tomaz Muraus]

  • Fix handling of MX records in the Gandi driver. (GITHUB-718) [Ryan Lee]

Improvements

Compute

  • Introduce new list_regions class method on the base driver class. This method is to be used with provider drivers which support multiple regions and region constructor argument. It allows users to enumerate available / supported regions. [Tomaz Muraus]

  • [dimension data] added support for VMWare tools VM information inside list_nodes responses (GITHUB-734) [Jeff Dunham]

  • [ec2] added exencrypted and exkmskeyid optional parameters to the create volume method (GITHUB-729) [Viktor Ognev]

  • [dimension data] added support for managing host anti-affinity rules, added paging support to all supported calls and added support for requesting priority ordering when creating ACL rules (GITHUB-726) [Jeff Dunham]

  • Addition of Dimension Data Australia federal government region to dimension data drivers. (GITHUB-700) [Anthony Shaw]

  • [openstack] when creating floating IPs, added pool_id as an optional argument (GITHUB-725) [marko-p]

  • [google compute] Added setMachineType method to allow for changing sizes of instances (GITHUB-721) [Eric Johnson]

  • [google compute] allow bypassing image search in standard project list (GITHUB-713) [Max Illfelder]

  • Add support for requesting a MKS token for accessing the remote console in VMware vCloud driver (GITHUB-706) [Juan Font Alonso]

  • Add support in VMware vCloud driver for v5.5 API, with snapshot support (GITHUB-658) [Juan Font Alonso]

  • Added support for adding a family to an image on Google Compute Driver (GITHUB-704) [Max Illfelder]

  • Fix to set default signature version for AWS Seoul region to v4, removed non-supported size (hs1.xlarge) (GITHUB-684) [Geunwoo Shin]

  • Support filtering by location in list_nodes for dimension data compute driver fix lack of paging support (GITHUB-691) [Jeff Dunham]

  • Support for filtering by IPv4, IPv6, network, network domain, VLAN in Dimension data driver. (GITHUB-694) [Jeff Dunham]

  • Added Node.created_at which, on supported drivers, contains the datetime the node was first started. (GITHUB-698) [Allard Hoeve] [Rick van de Loo]

Storage

  • Improvements to Google Auth for Storage and Compute and MIME bug fix (LIBCLOUD-800, GITHUB-689) [Scott Crunkleton]

  • Implement get_container, get_object and upload_object_via_stream methods in the Backblaze B2 storage driver.

    Note: Backblaze API doesn't upload streaming uploads so when using upload_object_via_stream whole file is read and buffered in memory. (GITHUB-696) [Jay jshridha]

Backup

  • Dimension Data - added additional testing, fixed bug on client response naming, added support for adding backup clients to a backup enabled node. (GITHUB-692, GITHUB-693, GITHUB-695) [Jeff Dunham]

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.0.0-rc2

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.0.0-rc2

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don't forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We would like to thank the following community members for their contribution to this release:

  • Jeff Dunham
  • Max Illfelder
  • Ken Dreyer
  • Sam Song
  • Oltjano Terpollari
  • Javier M. Mellid

Experimental support for the requests package

Background

I've just pushed a branch of the latest version of libcloud using the popular requests package by Kenneth Reitz instead of our home-rolled HTTP client library.

This article is for both users and developers of libcloud. If you want to give feedback, please join the developer mailing list.

Why?

  • requests is the defacto standard - it would be in the standard library but agreed against to allow it to develop faster https://github.com/kennethreitz/requests/issues/2424
  • it works with python 2.6->3.5
  • Our SSL experience has a lot to be desired for Windows users, having to download the CA cert package and setting environment variables just to get SSL working
  • Developers can use requests_mock for deeper integration testing
  • less code to maintain
  • the role of libcloud is for cloud abstraction, we provide no value in writing and maintaining our own HTTP client library

Benefits of requests

There are a number of benefits to having a requests package

  • The client library code is smaller, leaner and simpler.
  • Requests has built in decompression support, we no longer need to support this
  • Requests has built in RAW download, upload support, helping with our storage drivers

Implications of the change

  • There are no longer 2 classes (LibcloudHTTPSConnection and LibcloudHTTPConnection) to be provided to each driver, they are now 1 class - LibcloudConnection. You probably won't notice this because it is a property of the Connection class, but if you are developing or extending functionality then it is implicated.
  • Unit tests will look slightly different (see below)
  • This change broke 4200 unit tests (out of 6340)! I've since fixed them all since they were coupled to the original implementation, but now I don't know if all of tests are valid.

Testing with requests

Unit tests that were written like this:

class DigitalOceanTests(LibcloudTestCase):

      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_classes = \ 
           (None, DigitalOceanMockHttp)
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Because of the change have been modified to (I updated all of them - so this is just for future reference)

class DigitalOceanTests(LibcloudTestCase):

      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_class = DigitalOceanMockHttp
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Check it out!

The package is on my personal apache site, you can download it and install it in a virtualenv for testing.

pip install -e http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/apache-libcloud-1.0.0-rc2-requests.zip@feature#egg=apache-libcloud

The hashes are my apache space

Have a look at the PR and the change set for a list of changes

What might break?

What I'm really looking for is for users of Libcloud to take 15 minutes, an existing (working) libcloud script, install this package in a virtualenv and just validate that there are no regression bugs with this change.

I'm particularly sceptical about the storage drivers.

Once we have enough community feedback, we will propose a vote to merge this into trunk for future release.

Credit

Credit to dz0ny on IRC for contributing some of the requests patch.

New compute drivers and deprecated drivers in 1.0

With Libcloud 1.0.0 around the corner, it's time to have a spring clean of the compute drivers. Granted, it's not spring everywhere -actually I'm writing from Sydney, Australia where it's definitely summer.

Looking at the 52 providers in the 0.21.0 release, I have identified 5 providers that are no longer available or open.

Handling deprecated drivers

For 1.0.0, we need a clean and user-friendly way of handling deprecated drivers as well as keeping the repository clean from legacy code.

The most obvious implementation is that calls to get_driver(Provider.NINEFOLD) as an example will return a user error message saying this provider is no longer supported with a link to a new article and an alternative solution.

Currently, users trying to instantiate a HPE public cloud driver for example will get a connection error, which is not user friendly.

New compute drivers in 1.0.0-pre2

The upcoming release, so currently available in trunk contains some new compute drivers.

Full change log can be found at here.