Havana Design Summit: Swift API Discussions

As part of the Swift technical track last week at the OpenStack design summit, we had several topics on the Swift API. Swift has a remarkably stable API. We’ve added to the API, but we haven’t removed anything or changed any existing behavior beyond some minor conformance-to-spec fixes. This means that clients written years ago still work even when talking to Swift clusters deployed just yesterday.

External API

Although it is stable, the Swift API is not without some minor warts. There are a few inconsistencies in the API and a few awkward parts that would break clients if they changed. Cleaning up these parts of the API will help client developers write cleaner Swift applications and will allow end users to more easily use cross-cloud Swift clusters.

Figuring out what will be changed in the Swift API will be a long proccess, but there are a few important baseline things that need to happen before any changes to the API can be made. First, we need a formal definition of what the Swift API is. We have never had a formal API spec. Swift has always relied on the careful attention of its contributors to ensure that existing clients don’t break. A spec won’t lessen the need for careful attention by contributors and reviewers, but it will allow client developers to know exactly what they can expect from a particular deployment. A formal API spec also allows deployers to know what must be supported to ensure support for data migration between Swift clusters. As a side-benefit, formally defining the API will expose gaps in our current docs and help us keep our docs more up-to-date.

The second thing we must do as a community is define our API for discovering the supported Swift API. Users need to be able to determine what API a particular Swift cluster supports in order to know how to talk to it. There has been a lot of work on API discoverability in other OpenStack projects, so I hope that we can use some of their techniques and lessons-learned in Swift.

Once we have these two things, an API spec and API discoverability, we can start the discussions around what needs to change in the Swift API and go about implementing the changes in the code.

I expect that all of these questions will create quite a bit of discussion in the community. As a group, we need to get feedback from deployers (of all sizes), developers, and end users. Together, we’ll be able to make improvements and find the path that is best for everyone.

Internal API

Since a Swift cluster is a set of cooperating processes running on many servers, it implies that there is an internal API too. This API is how the communication between the nodes works and how the storage nodes talk to the underlying storage volumes.

While this internal API isn’t nearly as formal or rigid as Swift’s external API, there are opportunities to improve it too. Parts of Swift’s code can be refactored to allow cleaner abstractions so that specific optimizations or alternatives can be implemented.

A while back, the concept of a Local File System (LFS) was proposed to Swift. Ultimately, the proposed patch was not merged, but the idea is a good one. The concept allows for filesystem-specific optimizations to be made. For example, an XFS module could optimize the way it walks over inodes or a ZFS module could take advantage of its ZFS-specific self-healing properties.

Other interested parties have started working on the concept for LFS rencently, specifically with the goal of better integrating Swift and GlusterFS. I’m hopeful that the patches will be successfully merged this time around, and I’m looking forward to the additional functionality the LFS feature will allow.

Next Steps

To move forward on improving both the internal and external APIs for Swift, we need community involvement for a few things:

  • Formally defining the current Swift API
  • Implementing API version discoverability into Swift
  • Completion of the LFS patch for talking to storage volumes
  • Refactoring the proxy code to abstract communication with storage servers

I’m looking for people in the Swift community to help completing these tasks. If you’re interested, drop by #openstack-swift on freenode and let’s talk!

John Dickinson

John Dickinson

Director of Technology, SwiftStack
OpenStack Swift Project Technical Lead


OpenStack, OpenStack Swift, PlanetOpenstack, SwiftStack

Havana Design Summit: Enhancing Performance with Thread Pools Configuring HAProxy as a SwiftStack Load Balancer


© 2014 SwiftStack Inc.        San Francisco, CA         contact@swiftstack.com