Blog

Help Wanted

Help wanted - a summary of community contributions we really need your help with

Libcloud has a fantastic community, we value our community contributions and having a wide and varied experience and skillset.

Recently people have been asking what else they can do to help that isn't creating another driver, there's still plenty to do, here is a summary of some contributions that would be really valuable and don't nessecarily require advanced Python experience.

Skill set - documentation

  • Develop a 'writing a driver' from scratch tutorial that covers all the steps to developing a new DNS, Storage, Compute etc driver.
  • Improve the provider feature matrix, it's difficult to navigate and a more screen friendly format needs to be found page
  • Develop a "Libcloud in 30 minutes" tutorial that takes you through installation, getting started and loading up your cloud and using the driver.

Skill set - Web Authoring

Our website is written using Jekyll, Markdown and HTML.

Source Code

We would really value

  • Additional copy on the homepage explaining what the project does and potential use cases
  • Develop out the community pages
  • Extend the Using pages to link to the documentation site
  • Integrating video/GIF tutorials on the homepage

Skill set - Video tutorials

Complimenting our documentation with video tutorials (or GIF tutorials) on basic concepts and how to use the library would be really useful for beginners.

Skill set - Advanced Python development

Compute driver

  • The base Node class is missing standard fields to describe the CPU, Memory and Disk information for the Node. Propose, design and develop an implementation for this problem.
  • Update the Azure driver to support the ARM APIs.
  • Solve the create_node problem - (this is a big one). The driver classes are interchangable with the large and important exception of create_node. Some drivers use implicitly required kwargs, some require additional named keyword arguments, like the GCE driver requires the project name. This means any code needs to switch the argument set depending on the driver. We need a method(s) in the base driver which is consistent, and can be used across drivers. This would most likely use an Adapter or Proxy design pattern.

Container driver

  • Extend the Kubernetes driver to implement a Google Container Engine driver
  • Develop an Azure Container Service driver
  • Test and extend the Docker driver
  • Test and extend the Kubernetes driver for compatibility with the 1.2 release

Backup driver

  • Implement a driver using Amazon EBS snapshots
  • Implement an Azure Backup driver

DNS driver

  • Implement an Azure DNS driver

Skill set - Testing and experience

We have 2 major changes that need testing against real life

  • The requests implementation of our base HTTP classes, see my previous post on details. This needs thoroughly testing against

    • Compute classes, POST, GET methods
    • AWS auth token signing
    • Storage drivers using PUT and file streams
  • The container base class promoting to a "stable" state

    • Testing the docker driver against different implementations and auth mechanisms
    • Testing the Amazon ECS driver against real-world scenarios

Use case projects

There are a number of use cases that have been discussed but not implemented in a public repository. These would demonstrate the power of libcloud.

  • A storage migration utility - Our storage APIs are a great way
  • A DNS management library - Configure your DNS providers in a DSL file and have a command line utility to list records, zones and manage DNS
  • A bind export utility - Our DNS driver supports exporting a zone to bind format. Some cloud providers, such as Amazon Route53 don't offer this feature publically, so using libcloud would be a great use to allow people to export and backup data from Route53 or other services to a secondary NS slave.
  • A REST API for Libcloud - @tonybaloney has already started this using Flask RESTful supporting GET methods, this needs to extended and developed. The idea is to allow non-Python users to leverage libcloud via a RESTful API. The API will be dynamically constructed using Pythons introspection capabilities. See the project and fork this to get started

OK, I'm ready - what now?

Depending on what level of experience you are at, you can learn more about how to get started. You'll be pleased to learn that the libcloud project is beginner friendly, we accept contributions from all levels as well as abiding by the ASF code of ethics.

New to open-source?

Check out this tutorial on creating your first PR

New to Python?

Check out LearnPython.org to cut your teeth on some interactive tutorials.

Specifically to develop on libcloud you will need some tools:

  • A Python text editor like Atom, Komodo, PyCharm or emacs.
  • The Python 2.7 and 3.5 CPython interpreters installed
  • Tox for running the unit tests (easy_install tox)
  • Linting tools for checking PEP8 compliance, Flake8 (easy_install flake8) and PyLint (easy_install pylint)
  • Sphinx for building the docs

New to Apache?

Firstly, checkout The Apache Way to understand the principles of the Apache community projects.

If you're a committer on an Apache project, it means that you can commit directly to the project's repository. For instance, with Apache Libcloud committers are allowed to directly push commits into the git repository. Non-Committers can raise pull-requests on the read-only mirror on github.com/apache/libcloud, one of the committers will then use git am to merge your patch directly into the ASF git repository on your behalf. You will still get contribution recognition in the git logs.

I'm a pro, come at me

  • Read our project style and best-practice guidance on the documentation site for all of the things that are checked during the PR process.

Libcloud 1.0.0-rc2 released

We are pleased to announce the release of Libcloud 1.0.0-rc2.

This the second pre-release in the 1.0.0 series which means it brings many new features, improvements, bug-fixes, and DNS drivers.

Release highlights

This includes:

  • Deprecated drivers that were no longer available such as Ninefold, IBM SCE more details
  • The Amazon EC2 driver has been changed to use region codes instead of separate drivers for each region.
  • Introduce new list_regions class method on the base driver class
  • Support for Dimension Data backup
  • Added NSOne, LuaDNS, NearlyFreeSpeech.NET DNS drivers
  • Added Aliyun compute, load balancer and storage drivers
  • Added Outscale storage driver

Full change log can be found at here.

Important breaking change- Amazon EC2 driver changes

The Amazon EC2 API was updated to consolidate the regional-based drivers into a single driver with a region argument in the constructor.

Amazon Instances should now be instantiated using the following syntax:

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

cls = get_driver(Provider.EC2, region='us-east-i1')
driver = cls('access key', 'secret key')

This brings the Amazon API inline with the other drivers, makes it easier to maintain and switch between regions.

Bug fixes

General

  • Fix a bug with consuming stdout and stderr in the paramiko SSH client which would manifest itself under very rare condition when a consumed chunk only contained a single byte or part of a multi byte UTF-8 character. [Lakshmi Kannan, Tomaz Muraus]

  • Increase default chunk size from 1024 to 4096 bytes in the paramiko SSH client. This results in smaller number of receive calls on the average. [Tomaz Muraus]

  • Fix to Dimension Data API address for Middle-East and Africa (GITHUB-700) [Anthony Shaw]

  • Throw a more user-friendly exception on "No address associated with hostname". (GITHUB-711, GITHUB-714, LIBCLOUD-803) [Tomaz Muraus, Scott Crunkleton]

  • Remove deprecated provider constants with the region in the name and related driver classes (e.g. EC2_US_EAST, etc.).

    Those drivers have moved to single provider constant + region constructor argument model. [Tomaz Muraus]

New or deprecated drivers

Compute

  • Deprecated IBM SCE, HP Helion, OpSource, Ninefold and CloudFrames drivers, removed driver code and tests. (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • Introduced error messages (libcloud.compute.deprecated) for deprecated drivers (GITHUB-701, LIBCLOUD-801) [Anthony Shaw]

  • New Compute drivers- BSNL, Indosat, Med-1, NTT-America, Internet Solutions (GITHUB-700) [Anthony Shaw]

  • New driver for Aliyun Elastic Compute Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Storage

  • Added Outscale storage driver (GITHUB-730) [Javier M. Mellid]

  • New driver for Aliyun OSS Storage Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

Loadbalancer

  • New driver for Aliyun SLB Loadbalancer Service. (LIBCLOUD-802, GITHUB-712) [Sam Song]

DNS

  • Added NearlyFreeSpeech.net (NSFN) driver [Ken Drayer]

  • Added Lua DNS driver [Oltjano Terpollari]

  • Added NSOne driver [Oltjano Terpollari]

Bug fixes

  • Fix a bug in the GoDaddy driver - make sure host attribute on the connection class is correctly set to the hostname. [Tomaz Muraus]

  • Fix handling of MX records in the Gandi driver. (GITHUB-718) [Ryan Lee]

Improvements

Compute

  • Introduce new list_regions class method on the base driver class. This method is to be used with provider drivers which support multiple regions and region constructor argument. It allows users to enumerate available / supported regions. [Tomaz Muraus]

  • [dimension data] added support for VMWare tools VM information inside list_nodes responses (GITHUB-734) [Jeff Dunham]

  • [ec2] added exencrypted and exkmskeyid optional parameters to the create volume method (GITHUB-729) [Viktor Ognev]

  • [dimension data] added support for managing host anti-affinity rules, added paging support to all supported calls and added support for requesting priority ordering when creating ACL rules (GITHUB-726) [Jeff Dunham]

  • Addition of Dimension Data Australia federal government region to dimension data drivers. (GITHUB-700) [Anthony Shaw]

  • [openstack] when creating floating IPs, added pool_id as an optional argument (GITHUB-725) [marko-p]

  • [google compute] Added setMachineType method to allow for changing sizes of instances (GITHUB-721) [Eric Johnson]

  • [google compute] allow bypassing image search in standard project list (GITHUB-713) [Max Illfelder]

  • Add support for requesting a MKS token for accessing the remote console in VMware vCloud driver (GITHUB-706) [Juan Font Alonso]

  • Add support in VMware vCloud driver for v5.5 API, with snapshot support (GITHUB-658) [Juan Font Alonso]

  • Added support for adding a family to an image on Google Compute Driver (GITHUB-704) [Max Illfelder]

  • Fix to set default signature version for AWS Seoul region to v4, removed non-supported size (hs1.xlarge) (GITHUB-684) [Geunwoo Shin]

  • Support filtering by location in list_nodes for dimension data compute driver fix lack of paging support (GITHUB-691) [Jeff Dunham]

  • Support for filtering by IPv4, IPv6, network, network domain, VLAN in Dimension data driver. (GITHUB-694) [Jeff Dunham]

  • Added Node.created_at which, on supported drivers, contains the datetime the node was first started. (GITHUB-698) [Allard Hoeve] [Rick van de Loo]

Storage

  • Improvements to Google Auth for Storage and Compute and MIME bug fix (LIBCLOUD-800, GITHUB-689) [Scott Crunkleton]

  • Implement get_container, get_object and upload_object_via_stream methods in the Backblaze B2 storage driver.

    Note: Backblaze API doesn't upload streaming uploads so when using upload_object_via_stream whole file is read and buffered in memory. (GITHUB-696) [Jay jshridha]

Backup

  • Dimension Data - added additional testing, fixed bug on client response naming, added support for adding backup clients to a backup enabled node. (GITHUB-692, GITHUB-693, GITHUB-695) [Jeff Dunham]

Download

The release can can be downloaded from https://libcloud.apache.org/downloads.html or installed using pip:

pip install apache-libcloud==1.0.0-rc2

Upgrading

If you have installed Libcloud using pip you can also use it to upgrade it:

pip install --upgrade apache-libcloud==1.0.0-rc2

Upgrade notes

A page which describes backward incompatible or semi-incompatible changes and how to preserve the old behavior when this is possible can be found at https://libcloud.readthedocs.org/en/latest/upgrade_notes.html

Documentation

Regular and API documentation is available at https://libcloud.readthedocs.org/en/latest/

Bugs / Issues

If you find any bug or issue, please report it on our issue tracker https://issues.apache.org/jira/browse/LIBCLOUD. Don't forget to attach an example and / or test which reproduces your problem.

Thanks

Thanks to everyone who contributed and made this release possible! Full list of people who contributed to this release can be found in the CHANGES file.

We would like to thank the following community members for their contribution to this release:

  • Jeff Dunham
  • Max Illfelder
  • Ken Dreyer
  • Sam Song
  • Oltjano Terpollari
  • Javier M. Mellid

Experimental support for the requests package

Background

I've just pushed a branch of the latest version of libcloud using the popular requests package by Kenneth Reitz instead of our home-rolled HTTP client library.

This article is for both users and developers of libcloud. If you want to give feedback, please join the developer mailing list.

Why?

  • requests is the defacto standard - it would be in the standard library but agreed against to allow it to develop faster https://github.com/kennethreitz/requests/issues/2424
  • it works with python 2.6->3.5
  • Our SSL experience has a lot to be desired for Windows users, having to download the CA cert package and setting environment variables just to get SSL working
  • Developers can use requests_mock for deeper integration testing
  • less code to maintain
  • the role of libcloud is for cloud abstraction, we provide no value in writing and maintaining our own HTTP client library

Benefits of requests

There are a number of benefits to having a requests package

  • The client library code is smaller, leaner and simpler.
  • Requests has built in decompression support, we no longer need to support this
  • Requests has built in RAW download, upload support, helping with our storage drivers

Implications of the change

  • There are no longer 2 classes (LibcloudHTTPSConnection and LibcloudHTTPConnection) to be provided to each driver, they are now 1 class - LibcloudConnection. You probably won't notice this because it is a property of the Connection class, but if you are developing or extending functionality then it is implicated.
  • Unit tests will look slightly different (see below)
  • This change broke 4200 unit tests (out of 6340)! I've since fixed them all since they were coupled to the original implementation, but now I don't know if all of tests are valid.

Testing with requests

Unit tests that were written like this:

class DigitalOceanTests(LibcloudTestCase):

      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_classes = \ 
           (None, DigitalOceanMockHttp)
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Because of the change have been modified to (I updated all of them - so this is just for future reference)

class DigitalOceanTests(LibcloudTestCase):

      def setUp(self):
          DigitalOceanBaseDriver.connectionCls.conn_class = DigitalOceanMockHttp
          DigitalOceanMockHttp.type = None
          self.driver = DigitalOceanBaseDriver(*DIGITALOCEAN_v1_PARAMS)

Check it out!

The package is on my personal apache site, you can download it and install it in a virtualenv for testing.

pip install -e http://people.apache.org/~anthonyshaw/libcloud/1.0.0-rc2-requests/apache-libcloud-1.0.0-rc2-requests.zip@feature#egg=apache-libcloud

The hashes are my apache space

Have a look at the PR and the change set for a list of changes

What might break?

What I'm really looking for is for users of Libcloud to take 15 minutes, an existing (working) libcloud script, install this package in a virtualenv and just validate that there are no regression bugs with this change.

I'm particularly sceptical about the storage drivers.

Once we have enough community feedback, we will propose a vote to merge this into trunk for future release.

Credit

Credit to dz0ny on IRC for contributing some of the requests patch.

New compute drivers and deprecated drivers in 1.0

With Libcloud 1.0.0 around the corner, it's time to have a spring clean of the compute drivers. Granted, it's not spring everywhere -actually I'm writing from Sydney, Australia where it's definitely summer.

Looking at the 52 providers in the 0.21.0 release, I have identified 5 providers that are no longer available or open.

Handling deprecated drivers

For 1.0.0, we need a clean and user-friendly way of handling deprecated drivers as well as keeping the repository clean from legacy code.

The most obvious implementation is that calls to get_driver(Provider.NINEFOLD) as an example will return a user error message saying this provider is no longer supported with a link to a new article and an alternative solution.

Currently, users trying to instantiate a HPE public cloud driver for example will get a connection error, which is not user friendly.

New compute drivers in 1.0.0-pre2

The upcoming release, so currently available in trunk contains some new compute drivers.

Full change log can be found at here.

Using the container abstraction API in 1.0.0-pre1

Background

Containers are the talk of the town, you can't escape an event or meetup without someone talking about containers. The lessons we learnt with compute abstraction are applying widely with containers in 2016. APIs are not consistent between clouds, designs are not standardised and yet, users are trying to consume multiple services.

We introduced Container-as-a-Service support in 1.0.0-pre1, a community pre-release with the intention of sparking feedback from the open-source community about the design and the implementation of 4 example drivers :

  • Docker
  • Joyent Triton
  • Amazon EC2 Container Service
  • Google Kubernetes

In this tutorial we're going to explore how to do this:

Deploying containers across platforms.

Pulling images from the Docker hub, deploying to Docker, Kubernetes and Amazon ECS then auditing them with a single query.

Getting Started with 1.0.0-pre1

First off, let's install the new packages, you probably want to do this within a virtualenv if you're using Apache Libcloud for other projects.

So run these commands at a Linux Shell to create a virtualenv called 'containers' and install the pre-release packages into that environment.

   virtualenv containers
   cd containers
   source bin/activate
   pip install apache-libcloud==1.0.0-pre1

Now you can start using this package with a test script, let's create one called containers.py

   touch containers.py

Using your favourite text editor, update that file to import the 1.0.0-pre1 libraries and the factory methods for instantiating containers.

   from libcloud.container.providers import get_driver
   from libcloud.container.types import Provider

get_driver is a factory method as with all libcloud APIs, you call this method with the Provider that you want to instantiate. Our options are:

  • Provider.DOCKER - Standalone Docker API
  • Provider.KUBERNETES - Kubernetes Cluster endpoint
  • Provider.JOYENT - Joyent Triton Public API
  • Provider.ECS - Amazon EC2 Container Service

Calling get_driver will return a reference to the driver class that you requested. You can then instantiate that class into an object using the contructor. This is always a set of parameters for setting the host or region, the authentication and any other options.

   driver = get_driver(Provider.DOCKER)

Now we can call our driver and get an instance of it called docker_driver and use that to deploy a container. For Docker you need the pem files on the server, the host (IP or FQDN) and the port.

   docker_driver = driver(host='https://198.61.239.128', port=4243,
             key_file='key.pem', cert_file='cert.pem')

Docker requires that images are available in the image database before they can be deployed as containers. With Kubernetes and Amazon ECS this step is not required as when you deploy a container it carries out that download for you.

   image = driver.install_image('tomcat:8.0')

Now that Docker has the version 8.0 image of Apache Tomcat, you can deploy this as a container called my_tomcat_container. Tomcat runs on TCP/8080 by default so we want to bind that port for our container using an optional parameter port_bindings

   bindings = { "22/tcp": [{ "HostPort": "11022" }] }
   container = driver.deploy_container('my_tomcat_container', image, port_bindings=bindings)

This will have deployed the container and started it up for you, you can disable the automatic startup by using start=False as a keyword argument. You can now call upon this container and run methods, restart, start, stop and destroy.

For example, to blow away that test container:

   container.destroy()

Crossing the streams; calling Kubernetes and Amazon EC2 Container Service

With Docker we saw that we needed to "pull" the image before we deployed it. Kubernetes and Amazon ECS don't have that requirement, but as a safeguard you can query the Docker Hub API using a utility class provided

   from libcloud.container.utils.docker import HubClient
   hub = HubClient()
   image = hub.get_image('tomcat', '8.0')

Now image can be used to deploy to any driver instance that you create. Let's try that against Kubernetes and ECS.

Amazon ECS

Before you run this example, you will need an API key and the permissions for that key to have the AmazonEC2ContainerServiceFullAccess role. ap-southeast-2 is my nearest region, but you can swap this out for any of the Amazon public regions that have the ECS service available.

   e_cls = get_driver(Provider.ECS)
   ecs = e_cls(access_id='SDHFISJDIFJSIDFJ',
               secret='THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H',
               region='ap-southeast-2')

ECS and Kubernetes both support some form of grouping or clustering for your containers. This is available as create_cluster, list_cluster.

   cluster = ecs.create_cluster('default')
   container = ecs.deploy_container(
            cluster=cluster,
            name='hello-world',
            image=image,
            start=False,
            ex_container_port=8080, ex_host_port=8080)

This will have deployed a task definition in Amazon ECS with a single container inside, with a cluster called 'main' and deployed the tomcat:8.0 image from the Docker hub to that region.

Check out the ECS Documentation for more details.

Kubernetes

Kubernetes authentication is currently only implemented for None (off) and Basic HTTP authentication. Let's use the basic HTTP authentication method to connect.

k_cls = get_driver(Provider.KUBERNETES)

kubernetes = k_cls(key='my_username',
                   secret='THIS_IS)+_MY_SECRET_KEY+I6TVkv68o4H',
                   host='126.32.21.4')
cluster2 = kubernetes.create_cluster('default')
container2 = kubernetes.deploy_container(
            cluster=cluster,
            name='hello-world',
            image=image,
            start=False)

Wrapping it up

Now, let's wrap that all up by doing a list comprehension across the 3 drivers to get a list of all containers and print their ID's and Names. Then delete them.

containers = [conn.list_containers() for conn in [docker, ecs, kubernetes]]
for container in containers:
    print("%s : %s" % (container.id, container.name))
    container.destroy()

About the Author

Anthony Shaw is on the PMC for Apache Libcloud, you can follow Anthony on Twitter at @anthonypjshaw.