Even before the launch of the OpenStack Training Marketplace, SwiftStack organized the first 4 workshops in our training series. The goal of these 1-day workshops has been to bring hands-on Swift training to locations where developers, operators, and vendors want to learn more about Swift. The last one we held in San Jose CA on 10/30 was our biggest yet.
Based on feedback, we are looking to host these workshops about once a month. We will be going back for encore performances in some cities, and adding many new cities. We have dates for our next 2 workshops, both of them in the Northeast where we’ve received many requests for training:
For the holiday season, we are taking donations of $99 for the workshops, with all proceeds going to great charities with local presence and global reach. In New York, proceeds will go to the NYU Medical Center for MS research, and in DC proceeds will go to the Wounded Warrior project.
Also new for the series, we have partners sponsoring the Happy Hour after the workshop. In New York, The Ergonomic Group is sponsoring, and in DC we are happy to have RackTop Systems take part and sponsor. If you have a city you would like to see our next workshop land, would love to hear from you.
OpenStack Foundation did a really great job in getting all of the session videos posted very quickly. The Summit last week was the biggest yet for Swift content, and the videos are available to everyone to consume and enjoy, so share them with your friends.
Another highlight was the very long line that formed when Joe began signing books, captured in a panoramic photo by SwiftStack VP of EngineeringOrion Auld. It continued for 90min and would have kept going if we had more time! Request a book for yourself
Swift related videos:
In my next post, I’ll share the next 2 cities coming up in the Leveraging OpenStack Swift Workshop series for December and January (hint, GMT+5 for both ;-).
Kinetic drives are the first of their kind: key/value drives which communicate via Ethernet. The top question we’re getting asked is: What do they look like? What will the physical hardware look like with Seagate Kinetic drives?
First, let’s dispel some myths. Physically, the drives look just like any SAS or SATA drives available today. No, they do not have a RJ45 connector in the back. What you won’t see in the data center is a monster bundle of Cat 5/6 cables running up to a top of the rack switch!
Early in our development we had a PCB which plugged directly into the 3.5in drive. The traces on the PCB attached a surface mounted RJ45 jack. We just plugged the drives into the wall where cabling led back to one of the switches in the office.
After rummaging around in the lab for a power supply we were in business to begin development with a set of prototype drives. As with most revolutions, it started from humble beginnings. The pile of drives was labeled “Clay’s Lab (of awesome)” as he was leading the development effort on our end to integrate OpenStack Swift and flush-out the needed APIs with the Seagate Kinetic team.
No Really, Ethernet to a Hard Drive
Naturally while our initial prototypes were being directly plugged into a switch, that is not the typical way that the devices will be consumed. What is interesting here is that they can be plugged directly into a switch.
- Two 1GB Ethernet ports per drive
- Each drive has an independent IP address
- No bonding
- Active / Active (both interfaces can be simultaneously used)
- DHCP by default /configurable static IP assignment
- IPV6 and IPV4 compatible
So the question remains, what do these things plug into? A special connector? Nope! It’s the same pin-out as an existing SAS connector. The same 5v/12v pins that were formerly SAS are now dual Ethernet.
This means that they can be used in all kinds of configurations. At SwiftStack we are intending to use them at large-scale in a data center. However, it’s equally valid to be able to supply PoE (Power over Ethernet) to individual drives for certain applications.
Drives in a Box
This past week Jim Hugues came into the office and dropped-in with a developer box that he called the Developer Four Pack. This little unit is basically a 4-drive cluster. It is identical to what a larger version would hold, with individual power switches for each drive and an integrated low-cost switch. This is a great early demonstration of how these systems will ultimately be configured in a physical chassis.
Future Hardware Platforms
In the future these will be connected with the same drive trays that exist today, They’ll just have Ethernet connections, which are traces of copper, that go to an Ethernet switch rather than a SAS switch. The technology is “backplane ethernet” using the standard socket to connect to the drives.
How the hardware platforms will be built is likely with the switching integrated directly into the server chassis. Each drive has two Ethernet ports which will go to two different switches inside a server chassis. Those switches will provide two 10GbE ports.
Seagate has already announced many Kinetic ecosystem participants who will be providing products for this platform.
Where Does the Storage Application Live?
The storage application sits outside of the physical storage hardware. In our case with SwiftStack, we use a separate pool of compute nodes to provide OpenStack Swift services to API (HTTP) clients. The advantage here is that multiple storage boxes can be “multi-homed” if you will, to multiple compute nodes across separate switching fabrics.
The benefits include:
The entire system can be even more tolerant of failure. By separating the compute resources from the storage resources, we create much smaller failure domains and distribute the storage responsibility.
Better compute utilization. By creating a pool of compute nodes who have responsibility we can more accurately scale the capacity needed to manage the storage resources.
More to Come
Do take a look at the webinar we did with one of the Kinetic Architects at Seagate and our previous post on the Launch of Seagate Kinetic with OpenStack Swift as well as the code SwiftStack developed to run OpenStack Swift with Kinetic.
In future posts we’ll outline how we use the Kinetic API to build a storage application, specifically with OpenStack Swift, and how the devices can be managed. We are looking forward to this platform roll-out in the months ahead.
The full SwiftStack team will be in Hong Kong next week for the OpenStack Summit. Anyone who will also be there must come see us on one of 3 ways.
First, SwiftStack friends are invited to a nearby dinner on Tuesday 11/5 with a full bar & great local food from a local chef. Full dinner info and RSVP at https://swiftstackdinnerhk.eventbrite.com
Second, stop by booth C15 just inside the expo on the right, where we will have Seagate Kinetic hardware on display. Also, let me know a good time to schedule a meeting with Joe Arnold, Anders Tjernlund, John Dickinson, Orion Auld, Gary Butler, or any of the SwiftStack crew. We have a private meeting room on the show floor perfect for F2F meetings.
Third, SwiftStack customer Concur will be part of the Tuesday 9AM Keynote, specifically Dan Wilson who took this picture of the guys who deployed SwiftStack @ Concur.
Plus we have other 4 other great sessions:
Tuesday, November 5, 12:40 pm - 12:55 pm Demo Theater: Global Clusters with OpenStack Swift - Led by Joe Arnold Expo Hall (AsiaWorld-Expo)
Tuesday, November 5, 3:40 pm - 4:20 pm Panel Discussion: Software Defined Storage, including Joe Arnold SkyCity Grand Ballroom C (SkyCity Marriott Hotel)
Thursday, November 7, 9:50 am - 10:30 am Case Study: Concur Delivering a SaaS application with Swift, Joe Arnold & Dan Wilson of Concur SkyCity Grand Ballroom C (SkyCity Marriott Hotel)
Thursday, November 7, 4:30 pm - 6:00 pm Workshop: Deploying OpenStack Swift, led by John Dickinson and Martin Lanner SkyCity Grand Ballroom A&B (SkyCity Marriott Hotel)
Hope to see you in Hong Kong.
Last week many of the major contributors to Swift gathered in Austin, Texas for a few days of design, development, and review. We had a great week (including some fantastic Texas BBQ), and, I think, made some great progress for the project.
At the hackathon, we had people from Rackspace, SwiftStack, HP, Intel, Red Hat, IBM, Box, PDFComplete, and eNovance.
One of the great things about having so many contributors and users in the same place is that we could do very rapid reviews for major code patches that are being developed.
On Tuesday, we covered the refactoring work that Red Hat has been leading. The refactoring cleanly separates the Swift process code from code that is talking to the storage volumes. Initially targeted to allow clean integration for GlusterFS, this work allows new technology like Seagate’s new Kinetic drives, and also offers a place to add focused performance improvements in the future.
Wednesday afternoon we spent some time covering the proposed new “Swift Sync” patch. “Ssync” updates the existing replication mechanisms to use native Swift primitives for transport rather than rsync.
Thursday we covered several topics as a group including feature discoverability and cluster federation. HP has been working on creating a way for clusters to cooperate when one runs out of physical space. Feature discoverability is another nice addition to Swift that will help with test coverage and cross-cluster Swift clients.
In addition to these big topics, there were continual ad-hoc conversations around erasure codes, API polishing, metadata enhancements, performance improvements, and more.
We had a great few days together, and I hope we’ll be able to do it again soon.
Today Seagate announced their new Kinetic drive platform. This new hard drive platform is a revolution in drive technology. The technology also triggers a redefinition in storage architecture. Very rarely does one get to preside over a fundamental shift in the way that data is stored, and this is one of those sea changes. SwiftStack is proud to have partnered with Seagate to implement Kinetic support for OpenStack Swift, and make the developments available to the open source community. For SwiftStack customers, they will have a practical way to leverage these new devices to build large-scale storage systems with the Kinetic storage platform.
Shortly after the annoucement we hosted a webinar with James Hughes a managing principal technologist at Seagate and Ali Fenn an executive at Seagate. See the link below to view the discussion.
We also put together a post on what the Seagate Kinetic devices looks like.
If we look back on the history of data storage – drives were first connected to the mainframe. The intelligence actually lived on the mainframe. What happened during the sea change to distributed systems was storage intelligence migrated off of the server to what we know today as storage controllers, spawning a new industry. Data storage requires processing power, and while CPU is no longer scarce on servers, storage controllers have evolved to optimize data availability and resiliency. Network storage controllers have driven the current generation of storage architectures, and enabled an industry built on an increase in utilization and increased storage efficiency.
Scale-out object storage systems have virtually limitless scale properties for both data storage and system throughput across regions and zones.
However the world has changed. Application needs have pushed beyond what this single controller (HA pair) can accomplish.
What has happened with recent storage architectures is that we have needed to scale-out, not scale up. Today’s applications need storage to scale out horizontally because more and more user data is concentrating into fewer and fewer systems.
So the next generation of storage systems must scale-out horizontally. Software-defined storage meets this need, allowing for additional amounts of storage capacity by adding servers in a cluster. Each storage server contains many drives and is accompanied by CPU, RAM, networking, etc. The scale-out architecture has left traditional storage controllers high and dry when it comes to building large-scale storage systems.
With this next generation of storage systems the storage interfaces have also changed. This is because the preponderance of applications are built for mobile/tablet and web consumption. So it’s only natural that the native storage protocol of the future would be the same protocols that currently power the Internet. SwiftStack supports these protocols for your data center.
When protocols like HTTP and Ethernet are used, we can create storage systems that span not just one rack, not just one networking switch, not just one data center – but can span the across the internet (or a large-scale private corporate network). Now with the Kinetic platform, those same protocols are available all the way down to the drive.
The Seagate Kinetic platform ushers in a new era to define the new unit of measure for storage. The platform includes the following components that are radically different than the existing generation of disk drives.
The hard drives speak Ethernet rather than SATA, SAS, or fiber channel. This means that the hard drives can talk directly to other devices and other components in the system directly rather than going through intermediary devices, controllers, or other compute nodes.
Key / Value Interface:
The key / value storage interface means that clients communicate objects to the devices rather than blocks. This acknowledges that the unit of storage and the system is not bytes on disk – it is objects. Scale-out storage systems use objects. Objects make systems scale. Now objects are available at the drive level.
By speaking over Ethernet, these devices can be part of a platform that can easily scale-out to virtually any size. Simple networking infrastructure can route requests across a very large number of devices.
To learn more about the storage platform and to get a developer toolkit-
What the systems will look like:
The composition of storage systems that take advantage of Kinetic drives will be radically simplified. The storage nodes will simply be an enclosure for the drives and an everyday Ethernet switch. As a result, these enclosures will reduce system overall costs. The enclosures will consume less electricity, allow for greater density, adding to cost savings.
Now that there is scale independence at the component level, the entire storage system can be built with just enough external compute resources to satisfy the demands of the workload.
How Kinetic works with OpenStack Swift:
Today’s software-defined approach with compute and storage together in the same node is already changing storage architectures. The Kinetic platform takes it to the next level. Having the storage device speak in keys and values means that there is less impedance mismatch between Swift (which is native object) and traditional drives (which speak blocks). This results in more efficiency and utilization.
What we have done at SwiftStack is incorporate the Kinetic drives into the OpenStack Swift storage system as open source. Then we use pools of computing resources to provide the full Swift services in the cluster (auditing, replication, request routing authentication, etc.).
The source code to our current work in progress is availabe on GitHub through SwiftStack’s Preview of Swift on Kinetic.
Our role at SwiftStack is to ensure the storage system that powers the world’s largest clouds today also supports the world’s first generation of scale-out drives. SwiftStack is ensuring there are management capabilities built around the Kinetic platform from day one. That way users can incorporate this new technology to seamlessly, operate, and scale their storage clouds with ease.
The new generation of storage architecture is moving in. This sea change is an opportunity for the next generation of storage leaders to emerge. We are proud to be working with Seagate on this initiative to complete SwiftStack’s picture of software-defined storage in the data center supporting the build-out of next generation applications.
Today is the release of OpenStack Havana, and I want to share the exciting things that are going on in OpenStack Object Storage (Swift). You can find the full change log in the source repo of the project. You can download the latest release of Swift at https://launchpad.net/swift/havana/1.10.0.
During the last six months, the Swift community has finished the global clusters feature. Made up of several individual elements, global clusters solves two very common storage problems: multi-site durability and local availability.
Many storage use cases require a wide geographic separation of data to protect against single-site failure. Although rare, it’s possible for entire data centers to go offline. When that happens, you want to ensure that your data is still durably stored and highly available. By keeping data in multiple, geographically disperse locations, a global Swift cluster can give you the durability and availability you require.
This feature also enable higher performance clusters by allowing reads and writes to a local geographic region. Data requested from a particular region can be served from that region. For example, a global Swift cluster with servers in both New York and LA will be able to quickly serve East coast requests from the New York servers and West coast requests from the LA servers.
SwiftStack has previously written about Swift global clusters and the use cases they solve.
Concur will be speaking about their use of global clusters at the Hong Kong OpenStack conference. If you are joining us in Hong Kong, please come hear their story.
Beyond global clusters, Swift’s Havana release also includes other significant features. We have improved Swift’s performance by optimizing how it talks to storage volumes. Memcache connections have been optimized by keeping a pool of connections. Handoff node selection (i.e. choosing an alternate storage node in the event of failures) is now much faster. Swift now has support for conf.d style config directories. And we’ve added support for a cluster-wide crossdomain.xml file to better enable Flash apps reading content directly from a Swift cluster.
These performance improvements are the result of work from the entire community. As Stefano Mafuli mentioned is his recent blog post, Swift’s community has been growing. During the 6 months of the Havana release cycle, the top 10 contributors represent 6 different companies. 136 different contributors from (at least) 60 different companies have helped make Swift what it is today: an storage system that powers the world’s largest storage clouds.
My vision for Swift is that everyone, everywhere will use Swift every day, even if they don’t realize it. We’re seeing this vision fulfilled more every day. Global clusters, optimized performance, an active contributor community, and significant deployers like Concur make it possible.
OpenStack Object Storage (code name Swift) could be the most representative storage workload for new data center and cloud service provider architectures. Intel agrees, which is why OpenStack Swift is being used this week at the Intel Developers Forum in San Francisco to showcase the scale-out workload capabilities of the new Intel® Atom™ C2000 product family (code named “Avoton”).
With OpenStack Object Storage, the proxy services which route incoming requests to store or read data and the storage services that store the actual data are commonly run on different nodes, specifically tuned for those workloads. Leveraging the C2000 product family, new all-in-one nodes for storage workloads are possible.
At IDF13, Intel chose to run OpenStack Swift for an all-in-one 12-drive, 1U storage node (code named Canyon Run) demo in the Intel booth #835, and is also using Swift workloads as part of a Rack Scale Architecture demonstration in Intel booth #822.
Intel Engineers are leading a session “How Intel® Technology Can Unlock Capacity Storage Solutions in the Cloud” today, as well as a session on the future “OpenStack* Swift Erasure Codes: A Smart Cloud Storage Solution”.
Intel is one of the companies in the community working on Erasure Codes, for more information check out the joint announcement. Thanks!
Hi, Mario @ SwiftStack here with my first blog post since recently joining SwiftStack. My posts will focus on news, events, and solutions with customers and partners. I’ll try and keep them short, and leave the detailed technical posts that everyone loves to Joe A & John D.
Historically, the best way to learn about OpenStack Object Storage (code name Swift) has been to read Joe’s book on the subject. Request a copy today, it is a great read! So far in my first month at SwiftStack, the most common request I’ve gotten has been to bring hands-on OpenStack Swift training to different meetup groups. So that is what we are doing, scheduled full day Swift technical training and discussion workshops in cities across the USA.
Click here to see the list of dates, drill in on a date to see the detailed agenda, then register for one near you. Users, contributing vendors, and well-known members of the community will be getting together to share experiences, and actually experience the latest capabilities of the technology in the hands-on workshop. First cities are San Francisco (2 weeks from today), Seattle, Boston, San Jose, and Austin. Don’t see your favorite city near you? Let me know so we can add it to the list for future 1-day workshop events.
If you don’t have time for a workshop, another great way to see OpenStack Swift & SwiftStack in action is by joining one of our 30min live demos. To start, we are having them 2-3 times per week, Tu-Th at different times to cover a few different time zones. Click here to join. The full schedule of times can be found in the lower left of the page. Hope to see you soon.
Quick question: Have marketing ideas or feedback for SwiftStack? Let me know, I’d love your input as we continue to work on the tools and resources to make users successful with OpenStack Swift and private cloud storage for their data center. Thanks!
Today John Dickinson posted on the OpenStack mailing list a message with links to the design for erasure codes (EC for short) in Swift. This design has been built with the collaboration of Intel, Box and the input of other users of Swift. This post will go a bit deeper into the design philosophy of Swift and how EC will fit in.
Erasure codes will enable a cluster to reduce the overall storage footprint while maintaining a high degree of durability. This is a huge benefit for some use cases for Swift. But that’s not the whole story! Why didn’t Swift start with EC in the first place?
There is no free lunch
Saving space seems to be a no brainer. Store more with less? Of course! But, alas, there is a cost with saving space as well, we live in a world of tradeoffs.
The CAP Theorem says that when designing a distributed system, there is Consistency, Availability and Partition Tolerance, pick any two. Consistency means that all parts of the system see the same data at the same time. Availability is a guarantee that every request receives a response about whether it was successful or has failed. And partition tolerance means the system continues to operate despite arbitrary message loss or failure of part of the system.
Swift chooses Availability and Partition Tolerance over Consistency.
What this means is that when there is a partition (like servers not being able to talk to each other), Swift is tolerant of this partition and will still service the request. Because Swift replicates data across failure domains, any segment of the cluster will be able to serve the object and the system remains highly available.
Availability in the face of partitions is the huge benefit of using replicas. But there are also a couple of other benefits:
Latency and Reconstruction
For many of our customers, concurrency and latency rule the day. In a replica model, a read request can be served from a single storage volume. This means less network overhead, less CPU effort, and results in lower latency to the user.
Failure handling is fairly simple in a replicated system, too. When a drive fails and is replaced, any other remaining replica can copy the data directly to the new replacement drive. This simple copy-only operation has low CPU requirements and doesn’t need to coordinate connections with many servers in the cluster.
Replicas and Regions
In Swift’s current replication model, all data is fully copied multiple times throughout the cluster. At SwiftStack we recommend 3 replicas for single-site Swift clusters and 2 x 2 replicas for two, globally-distributed sites. This strategy gives the cluster fantastic durability and availability since whole copies of data are distributed across the failure domains and requests can be served from local replicas instead of being reassembled from bits across the WAN.
More than one way to slice an app
We love the replica model. It’s operationally simple, low-latency, and highly available. This pairs up nicely with the use cases of many production Swift clusters. Users expect the data to be available instantly, on any device and stored forever!
However! Why not provide replicas where there is an advantage AND erasure codes to save space when the demands on the data are less intensive?
Erasure Codes with Swift
To provide a smaller storage footprint, we will build on the existing components in Swift to provide a highly-durable storage system using erasure codes. The design goal is to be able to have erasure-coded storage in addition to replicas in Swift. This will allow a choice in how to store data and make the right tradeoffs based on the use case.
Swift currently uses a data structure called the Ring to distribute a partition space across the cluster. This partition space is core to the replication system in Swift. It allows Swift to quickly and easily synchronize each partition across the cluster. When any component in Swift needs to interact with data, a quick lookup is done locally in the Ring to determine the possible partitions for each replica.
Swift already has three rings to store different types of data in Swift. There is one for account information (Swift, after all is a multi-tenant storage system), another for containers (so that it’s convenient to organize objects under an account) and another for the object replicas. To support erasure codes, there will be an additional ring that is created to store erasure code chunks.
In a Swift cluster the proxy servers stream requests to the appropriate storage nodes. When a request is made to store an object that is erasure coded, the proxy tier will encode the object into chunks and stream those chunks to appropriate partitions.
Reading the Data
When a request is made to read an object that is erasure coded, the proxy tier will make the appropriate connections to the storage servers and decode the object data as it is sent to the client. One advantage of erasure coding is that a drive failure during a read can still result in valid data being sent to the client if there are enough other chunks to reconstruct the missing part.
Building on tested components
There are a few advantages to doing the encoding in the proxy tier. First, is it enables us to stand on the shoulders of the existing, battle-hardened mechanisms that Swift already has. Second, it concentrates the CPU constraints into the existing CPU-heavy proxy servers. In most real-world deployments, storage capacity grows faster than concurrent access requirements, so keeping the erasure coding machinery in the proxy server allows deployers to keep costs low.
We can use the existing concept of handoff locations in Swift to store parts when (not if) there are physical hardware or connectivity failures. Each node in the system knows how to dig deeper for chunks if they are not found in their primary location.
The replicator is a continuously running to ensure that each partition has in it what it is supposed to have. A hash file is created in each each partition which is a ‘hash of hashes’ of every file in the partition. That way, replica contents can be quickly compared with a ‘check your twins’ model.
With erasure codes, we can use the same strategy to ‘check your siblings’. The replicator equivalent for erasure codes, the “reconstructor”, can check neighboring chunks quickly to ensure that individual chunks are available. When there is a failure, the reconstructor can rebuild and push the chunks to the locations where they are missing.
Containers for control granularity
Containers are used in Swift similar to how buckets are used in AWS S3. They provide a good handle that allows an account to provide attributes on a collection of objects, for example setting access control lists (ACLs). Containers are a good organizing method to set a storage policy to replication or erasure code.
Use different equipment for EC Tier
The advantage of using an additional ring is to be able to store erasure coded data on different physical hardware if desired. For example, if the data is used for active archive purposes, then it may make sense to use slower, more dense hardware to further drive down costs.
The tradeoffs are pretty simple. Erasure coded objects are stored across many more servers than replicated data. This means that each read and write must establish more network connections to the storage nodes. These additional connections increase the complexity of the failure handling requirements and add latency to each request. Also, since the data must be encoded on write and decoded on read, there is CPU overhead introduced by erasure codes.
While there are tradeoffs, erasure codes in Swift will provide an option for storage savings without sacrificing the overall durability of the system. Now, deployers of Swift will be able to choose between both EC and Replicas based on which is best for their use case.