Blog

Libcloud 1.0.0-rc2 released

By Anthony Shaw on Apr 11, 2016

We are pleased to announce the release of Libcloud 1.0.0-rc2.

This the second pre-release in the 1.0.0 series which means it brings many new features, improvements, bug-fixes, and DNS drivers.

Release highlights

This includes:

  • Deprecated drivers that were no longer available such as Ninefold, IBM SCE more details
  • The Amazon EC2 driver has been changed to use region codes instead of separate drivers for each region.
  • Introduce new list_regions class method on the base driver class
  • Support for Dimension Data backup
  • Added NSOne, LuaDNS, NearlyFreeSpeech.NET DNS drivers
  • Added Aliyun compute, load balancer and storage drivers
  • Added Outscale storage driver

Full change log can be found at here.

Important breaking change- Amazon EC2 driver changes

The Amazon EC2 API was updated to consolidate the regional-based drivers into a single driver with a region argument in the constructor.

Amazon Instances should now be instantiated using the following syntax:

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

cls = get_driver(Provider.EC2, region='us-east-i1')
driver = cls('access key', 'secret key')

This brings the Amazon API inline with the other drivers, makes it easier to maintain and switch between regions.

Bug fixes

General

  • Fix a bug with consuming stdout and stderr in the paramiko SSH client which would manifest itself under very rare condition when a consumed chunk only contained a single byte or part of a multi byte UTF-8 character. [Lakshmi Kannan, Tomaz Muraus]

  • Increase default chunk size from 1024 to 4096 bytes in the paramiko SSH client. This results in smaller number of receive calls on the average. [Tomaz Muraus]

  • Fix to Dimension Data API address for Middle-East and Africa (GITHUB-700) [Anthony Shaw]

  • Throw a more user-friendly exception on “No address associated with hostname”. (GITHUB-711, GITHUB-714, LIBCLOUD-803) [Tomaz Muraus, Scott Crunkleton]

  • Remove deprecated provider constants with the region in the name and related driver classes (e.g. EC2_US_EAST, etc.).

    Those drivers have moved to single provider constant + region constructor argument model. [Tomaz Muraus]

New or deprecated drivers

Compute

  • Deprecated IBM SCE, HP Helion, OpSource, Ninefold and CloudFrames drivers, removed driver code and tests. (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • Introduced error messages (libcloud.compute.deprecated) for deprecated drivers (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • New Compute drivers- BSNL, Indosat, Med-1, NTT-America, Internet Solutions (GITHUB-700) [Anthony Shaw]

  • New driver for Aliyun Elastic Compute Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Storage

  • Added Outscale storage driver (GITHUB-730) [Javier M. Mellid]

  • New driver for Aliyun OSS Storage Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Loadbalancer

  • New driver for Aliyun SLB Loadbalancer Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

DNS

  • Added NearlyFreeSpeech.net (NSFN) driver [Ken Drayer]

  • Added Lua DNS driver [Oltjano Terpollari]

  • Added NSOne driver [Oltjano Terpollari]

Bug fixes

  • Fix a bug in the GoDaddy driver - make sure host attribute on the connection class is correctly set to the hostname. [Tomaz Muraus]

  • Fix handling of MX records in the Gandi driver. (GITHUB-718) [Ryan Lee]

Improvements

Compute

  • Introduce new list_regions class method on the base driver class. This method is to be used with provider drivers which support multiple regions and region constructor argument. It allows users to enumerate available / supported regions. [Tomaz Muraus]

  • [dimension data] added support for VMWare tools VM information inside list_nodes responses (GITHUB-734) [Jeff Dunham]

  • [ec2] added ex_encrypted and ex_kms_key_id optional parameters to the create volume method (GITHUB-729) [Viktor Ognev]

  • [dimension data] added support for managing host anti-affinity rules, added paging support to all supported calls and added support for requesting priority ordering when creating ACL rules (GITHUB-726) [Jeff Dunham]

  • Addition of Dimension Data Australia federal government region to dimension data drivers. (GITHUB-700) [Anthony Shaw]

  • [openstack] when creating floating IPs, added pool_id as an optional argument (GITHUB-725) [marko-p]

  • [google compute] Added setMachineType method to allow for changing sizes of instances (GITHUB-721) [Eric Johnson]

  • [google compute] allow bypassing image search in standard project list (GITHUB-713) [Max Illfelder]

  • Add support for requesting a MKS token for accessing the remote console in VMware vCloud driver (GITHUB-706) [Juan Font Alonso]

  • Add support in VMware vCloud driver for v5.5 API, with snapshot support (GITHUB-658) [Juan Font Alonso]

  • Added support for adding a family to an image on Google Compute Driver (GITHUB-704) [Max Illfelder]

  • Fix to set default signature version for AWS Seoul region to v4, removed non-supported size (hs1.xlarge) (GITHUB-684) [Geunwoo Shin]

  • Support filtering by location in list_nodes for dimension data compute driver fix lack of paging support (GITHUB-691) [Jeff Dunham]

  • Support for filtering by IPv4, IPv6, network, network domain, VLAN in Dimension data driver. (GITHUB-694) [Jeff Dunham]

  • Added Node.created_at which, on supported drivers, contains the datetime the node was first started. (GITHUB-698) [Allard Hoeve] [Rick van de Loo]

Storage

  • Improvements to Google Auth for Storage and Compute and MIME bug fix (LIBCLOUD-800, GITHUB-689) [Scott Crunkleton]

  • Implement get_container, get_object and upload_object_via_stream methods in the Backblaze B2 storage driver.

    Note: Backblaze API doesn’t upload streaming uploads so when using upload_object_via_stream whole file is read and buffered in memory. (GITHUB-696) [Jay jshridha]

Backup

  • Dimension Data - added additional testing, fixed bug on client response naming, added support for adding backup clients to a backup enabled node. (GITHUB-692, GITHUB-693, GITHUB-695) [Jeff Dunham]

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.0.0-rc2

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.0.0-rc2

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We would like to thank the following community members for their contribution to this release:

  • Jeff Dunham
  • Max Illfelder
  • Ken Dreyer
  • Sam Song
  • Oltjano Terpollari
  • Javier M. Mellid

Tags: ,

Experimental support for the requests package

By Anthony Shaw on Apr 06, 2016

Background

I’ve just pushed a branch of the latest version of libcloud using the popular requests package by Kenneth Reitz instead of our home-rolled HTTP client library.

This article is for both users and developers of libcloud. If you want to give feedback, please join the developer mailing list.

Why?

  • requests is the defacto standard - it would be in the standard library but agreed against to allow it to develop faster https://github.com/kennethreitz/requests/issues/2424
  • it works with python 2.6->3.5
  • Our SSL experience has a lot to be desired for Windows users, having to download the CA cert package and setting environment variables just to get SSL working
  • Developers can use requests_mock for deeper integration testing
  • less code to maintain
  • the role of libcloud is for cloud abstraction, we provide no value in writing and maintaining our own HTTP client library

Benefits of requests

There are a number of benefits to having a requests package

  • The client library code is smaller, leaner and simpler.
  • Requests has built in decompression support, we no longer need to support this
  • Requests has built in RAW download, upload support, helping with our storage drivers

Implications of the change

  • There are no longer 2 classes (LibcloudHTTPSConnection and LibcloudHTTPConnection) to be provided to each driver, they are now 1 class - LibcloudConnection. You probably won’t notice this because it is a property of the Connection class, but if you are developing or extending functionality then it is implicated.
  • Unit tests will look slightly different (see below)
  • This change broke 4200 unit tests (out of 6340)! I’ve since fixed them all since they were coupled to the original implementation, but now I don’t know if all of tests are valid.

Testing with requests

Unit tests that were written like this:

class DigitalOceanTests(LibcloudTestCase):
  		  
      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_classes = \	
           (None, DigitalOceanMockHttp)
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Because of the change have been modified to (I updated all of them - so this is just for future reference)

class DigitalOceanTests(LibcloudTestCase):
  		  
      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_class = DigitalOceanMockHttp
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Check it out!

The package is on my personal apache site, you can download it and install it in a virtualenv for testing.

pip install -e http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/apache-libcloud-1.0.0-rc2-requests.zip@feature#egg=apache-libcloud

The hashes are my apache space

Have a look at the PR and the change set for a list of changes

What might break?

What I’m really looking for is for users of Libcloud to take 15 minutes, an existing (working) libcloud script, install this package in a virtualenv and just validate that there are no regression bugs with this change.

I’m particularly sceptical about the storage drivers.

Once we have enough community feedback, we will propose a vote to merge this into trunk for future release.

Credit

Credit to dz0ny on IRC for contributing some of the requests patch.

Tags: , ,

New compute drivers and deprecated drivers in 1.0

By Anthony Shaw on Feb 16, 2016

With Libcloud 1.0.0 around the corner, it’s time to have a spring clean of the compute drivers. Granted, it’s not spring everywhere -actually I’m writing from Sydney, Australia where it’s definitely summer.

Looking at the 52 providers in the 0.21.0 release, I have identified 5 providers that are no longer available or open.

Handling deprecated drivers

For 1.0.0, we need a clean and user-friendly way of handling deprecated drivers as well as keeping the repository clean from legacy code.

The most obvious implementation is that calls to get_driver(Provider.NINEFOLD) as an example will return a user error message saying this provider is no longer supported with a link to a new article and an alternative solution.

Currently, users trying to instantiate a HPE public cloud driver for example will get a connection error, which is not user friendly.

New compute drivers in 1.0.0-pre2

The upcoming release, so currently available in trunk contains some new compute drivers.

Full change log can be found at here.

Tags: ,

Using the container abstraction API in 1.0.0-pre1

By Anthony Shaw on Feb 05, 2016

Background

Containers are the talk of the town, you can’t escape an event or meetup without someone talking about containers. The lessons we learnt with compute abstraction are applying widely with containers in 2016. APIs are not consistent between clouds, designs are not standardised and yet, users are trying to consume multiple services.

We introduced Container-as-a-Service support in 1.0.0-pre1, a community pre-release with the intention of sparking feedback from the open-source community about the design and the implementation of 4 example drivers :

  • Docker
  • Joyent Triton
  • Amazon EC2 Container Service
  • Google Kubernetes

In this tutorial we’re going to explore how to do this:

Deploying containers across platforms.

Pulling images from the Docker hub, deploying to Docker, Kubernetes and Amazon ECS then auditing them with a single query.

Getting Started with 1.0.0-pre1

First off, let’s install the new packages, you probably want to do this within a virtualenv if you’re using Apache Libcloud for other projects.

So run these commands at a Linux Shell to create a virtualenv called ‘containers’ and install the pre-release packages into that environment.

   virtualenv containers
   cd containers
   source bin/activate
   pip install apache-libcloud==1.0.0-pre1

Now you can start using this package with a test script, let’s create one called containers.py

   touch containers.py

Using your favourite text editor, update that file to import the 1.0.0-pre1 libraries and the factory methods for instantiating containers.

   from libcloud.container.providers import get_driver
   from libcloud.container.types import Provider

get_driver is a factory method as with all libcloud APIs, you call this method with the Provider that you want to instantiate. Our options are:

  • Provider.DOCKER - Standalone Docker API
  • Provider.KUBERNETES - Kubernetes Cluster endpoint
  • Provider.JOYENT - Joyent Triton Public API
  • Provider.ECS - Amazon EC2 Container Service

Calling get_driver will return a reference to the driver class that you requested. You can then instantiate that class into an object using the contructor. This is always a set of parameters for setting the host or region, the authentication and any other options.

   driver = get_driver(Provider.DOCKER)

Now we can call our driver and get an instance of it called docker_driver and use that to deploy a container. For Docker you need the pem files on the server, the host (IP or FQDN) and the port.

   docker_driver = driver(host='https://198.61.239.128', port=4243,
             key_file='key.pem', cert_file='cert.pem')

Docker requires that images are available in the image database before they can be deployed as containers. With Kubernetes and Amazon ECS this step is not required as when you deploy a container it carries out that download for you.

   image = driver.install_image('tomcat:8.0')

Now that Docker has the version 8.0 image of Apache Tomcat, you can deploy this as a container called my_tomcat_container. Tomcat runs on TCP/8080 by default so we want to bind that port for our container using an optional parameter port_bindings

   bindings = { "22/tcp": [{ "HostPort": "11022" }] }
   container = driver.deploy_container('my_tomcat_container', image, port_bindings=bindings)

This will have deployed the container and started it up for you, you can disable the automatic startup by using start=False as a keyword argument. You can now call upon this container and run methods, restart, start, stop and destroy.

For example, to blow away that test container:

   container.destroy()

Crossing the streams; calling Kubernetes and Amazon EC2 Container Service

With Docker we saw that we needed to “pull” the image before we deployed it. Kubernetes and Amazon ECS don’t have that requirement, but as a safeguard you can query the Docker Hub API using a utility class provided

   from libcloud.container.utils.docker import HubClient
   hub = HubClient()
   image = hub.get_image('tomcat', '8.0')

Now image can be used to deploy to any driver instance that you create. Let’s try that against Kubernetes and ECS.

Amazon ECS

Before you run this example, you will need an API key and the permissions for that key to have the AmazonEC2ContainerServiceFullAccess role. ap-southeast-2 is my nearest region, but you can swap this out for any of the Amazon public regions that have the ECS service available.

   e_cls = get_driver(Provider.ECS)
   ecs = e_cls(access_id='SDHFISJDIFJSIDFJ',
               secret='THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H',
               region='ap-southeast-2')

ECS and Kubernetes both support some form of grouping or clustering for your containers. This is available as create_cluster, list_cluster.

   cluster = ecs.create_cluster('default')
   container = ecs.deploy_container(
            cluster=cluster,
            name='hello-world',
            image=image,
            start=False,
            ex_container_port=8080, ex_host_port=8080)

This will have deployed a task definition in Amazon ECS with a single container inside, with a cluster called ‘main’ and deployed the tomcat:8.0 image from the Docker hub to that region.

Check out the ECS Documentation for more details.

Kubernetes

Kubernetes authentication is currently only implemented for None (off) and Basic HTTP authentication. Let’s use the basic HTTP authentication method to connect.


k_cls = get_driver(Provider.KUBERNETES)

kubernetes = k_cls(key='my_username',
                   secret='THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H',
                   host='126.32.21.4')
cluster2 = kubernetes.create_cluster('default')
container2 = kubernetes.deploy_container(
            cluster=cluster,
            name='hello-world',
            image=image,
            start=False)

Wrapping it up

Now, let’s wrap that all up by doing a list comprehension across the 3 drivers to get a list of all containers and print their ID’s and Names. Then delete them.

containers = [conn.list_containers() for conn in [docker, ecs, kubernetes]]
for container in containers:
    print("%s : %s" % (container.id, container.name))
    container.destroy()

About the Author

Anthony Shaw is on the PMC for Apache Libcloud, you can follow Anthony on Twitter at @anthonypjshaw.

Tags: , ,

Libcloud 1.0.0-pre1 released

By Anthony Shaw on Jan 26, 2016

We are pleased to announce the release of Libcloud 1.0.0-pre1.

This is a first pre-release in the 1.0.0 series which means it brings many new features, improvements, bug-fixes, and DNS drivers.

Release highlights

A full blog post on the new features in 1.0.0 can be found here

This includes:

Full change log can be found at here.

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.0.0-pre1

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.0.0-pre1

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don’t forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

Tags: ,

« Previous Page Next Page »

Archive