Kinetic Motion With Seagate and OpenStack Swift
Today Seagate announced their new Kinetic drive platform. This new hard drive platform is a revolution in drive technology. The technology also triggers a redefinition in storage architecture. Very rarely does one get to preside over a fundamental shift in the way that data is stored, and this is one of those sea changes. SwiftStack is proud to have partnered with Seagate to implement Kinetic support for OpenStack Swift, and make the developments available to the open source community. For SwiftStack customers, they will have a practical way to leverage these new devices to build large-scale storage systems with the Kinetic storage platform.
Shortly after the annoucement we hosted a webinar with James Hughes a managing principal technologist at Seagate and Ali Fenn an executive at Seagate. See the link below to view the discussion.
We also put together a post on what the Seagate Kinetic devices looks like.
If we look back on the history of data storage – drives were first connected to the mainframe. The intelligence actually lived on the mainframe. What happened during the sea change to distributed systems was storage intelligence migrated off of the server to what we know today as storage controllers, spawning a new industry. Data storage requires processing power, and while CPU is no longer scarce on servers, storage controllers have evolved to optimize data availability and resiliency. Network storage controllers have driven the current generation of storage architectures, and enabled an industry built on an increase in utilization and increased storage efficiency.
Scale-out object storage systems have virtually limitless scale properties for both data storage and system throughput across regions and zones.
However the world has changed. Application needs have pushed beyond what this single controller (HA pair) can accomplish.
What has happened with recent storage architectures is that we have needed to scale-out, not scale up. Today’s applications need storage to scale out horizontally because more and more user data is concentrating into fewer and fewer systems.
So the next generation of storage systems must scale-out horizontally. Software-defined storage meets this need, allowing for additional amounts of storage capacity by adding servers in a cluster. Each storage server contains many drives and is accompanied by CPU, RAM, networking, etc. The scale-out architecture has left traditional storage controllers high and dry when it comes to building large-scale storage systems.
With this next generation of storage systems the storage interfaces have also changed. This is because the preponderance of applications are built for mobile/tablet and web consumption. So it’s only natural that the native storage protocol of the future would be the same protocols that currently power the Internet. SwiftStack supports these protocols for your data center.
When protocols like HTTP and Ethernet are used, we can create storage systems that span not just one rack, not just one networking switch, not just one data center – but can span the across the internet (or a large-scale private corporate network). Now with the Kinetic platform, those same protocols are available all the way down to the drive.
The Seagate Kinetic platform ushers in a new era to define the new unit of measure for storage. The platform includes the following components that are radically different than the existing generation of disk drives.
The hard drives speak Ethernet rather than SATA, SAS, or fiber channel. This means that the hard drives can talk directly to other devices and other components in the system directly rather than going through intermediary devices, controllers, or other compute nodes.
Key / Value Interface:
The key / value storage interface means that clients communicate objects to the devices rather than blocks. This acknowledges that the unit of storage and the system is not bytes on disk – it is objects. Scale-out storage systems use objects. Objects make systems scale. Now objects are available at the drive level.
By speaking over Ethernet, these devices can be part of a platform that can easily scale-out to virtually any size. Simple networking infrastructure can route requests across a very large number of devices.
To learn more about the storage platform and to get a developer toolkit-
What the systems will look like:
The composition of storage systems that take advantage of Kinetic drives will be radically simplified. The storage nodes will simply be an enclosure for the drives and an everyday Ethernet switch. As a result, these enclosures will reduce system overall costs. The enclosures will consume less electricity, allow for greater density, adding to cost savings.
Now that there is scale independence at the component level, the entire storage system can be built with just enough external compute resources to satisfy the demands of the workload.
How Kinetic works with OpenStack Swift:
Today’s software-defined approach with compute and storage together in the same node is already changing storage architectures. The Kinetic platform takes it to the next level. Having the storage device speak in keys and values means that there is less impedance mismatch between Swift (which is native object) and traditional drives (which speak blocks). This results in more efficiency and utilization.
What we have done at SwiftStack is incorporate the Kinetic drives into the OpenStack Swift storage system as open source. Then we use pools of computing resources to provide the full Swift services in the cluster (auditing, replication, request routing authentication, etc.).
The source code to our current work in progress is availabe on GitHub through SwiftStack’s Preview of Swift on Kinetic.
Our role at SwiftStack is to ensure the storage system that powers the world’s largest clouds today also supports the world’s first generation of scale-out drives. SwiftStack is ensuring there are management capabilities built around the Kinetic platform from day one. That way users can incorporate this new technology to seamlessly, operate, and scale their storage clouds with ease.
The new generation of storage architecture is moving in. This sea change is an opportunity for the next generation of storage leaders to emerge. We are proud to be working with Seagate on this initiative to complete SwiftStack’s picture of software-defined storage in the data center supporting the build-out of next generation applications.