SwiftStack
Blog

Swift Capacity Management

If you’ve run a storage system, you know there are two ironclad rules of storage: data accumulates and size of new disks increases. In this post, we’re going to examine how an OpenStack Swift cluster operator deals with each of these and how SwiftStack makes them easy to deal with.

Rule 1: Data Accumulates

Data grows over time. If you’re running a storage system, you know that you have to keep adding capacity to keep up with user demand, and that means adding more disks.

Adding capacity: the naive approach

If you’re running OpenStack Swift, adding a new disk is a long, though straightforward, process. Let’s say you just physically added a 3TB disk with label d16 in storage node 192.168.12.104. To make Swift use it, you add it to the rings, like so:

$ swift-ring-builder account.builder add z1-192.168.12.104:6002/d16 3000
$ swift-ring-builder container.builder add z1-192.168.12.104:6001/d16 3000
$ swift-ring-builder object.builder add z1-192.168.12.104:6000/d16 3000

We use 3000 for the weight since that’s the size of the drive in GB; it’s a useful convention.

Then, rebalance the rings and copy them to all your storage nodes:

$ swift-ring-builder account.builder rebalance
$ swift-ring-builder container.builder rebalance
$ swift-ring-builder object.builder rebalance

$ scp account.ring.gz swift-node-1:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-1:/etc/swift/container.ring.gz
$ scp object.ring.gz swift-node-1:/etc/swift/account.ring.gz
$ scp account.ring.gz swift-node-2:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-2:/etc/swift/container.ring.gz
$ scp object.ring.gz swift-node-2:/etc/swift/account.ring.gz
 ...
$ scp account.ring.gz swift-node-10:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-10:/etc/swift/container.ring.gz
$ scp object.ring.gz swift-node-10:/etc/swift/account.ring.gz

Swift picks up the ring changes, and everyone’s happy. Right?

Wrong. Your cluster is now suffering from degraded performance. By adding that 3TB drive all at once, you caused one full drive’s worth of data to move around in the cluster. That’s what rebalancing does: gives each device its fair share of load. On a cluster that’s 50% full, that’s 1.5 TB of data that is now in flight to the new drive. For that to all fit through a 10Gb Ethernet port would take about 20 minutes at 100% utilization, assuming the source drives have the spare capacity to send the data, the new drive can consume it at that pace, and the network switches can support that transfer. In practice, you’re looking at hours of degraded performance from your cluster.

Adding capacity gradually

The way to keep your Swift cluster happy and growing is to add capacity gradually over time, not in big lumps. With plain Swift, you’d do this:

$ swift-ring-builder account.builder add z1-192.168.12.104:6002/d16 25
$ swift-ring-builder container.builder add z1-192.168.12.104:6001/d16 25
$ swift-ring-builder object.builder add z1-192.168.12.104:6000/d16 25

Notice that we’re using 25 for the weight, so the drive is only going to get a small part of the data set assigned to it.

As always, rebalance the rings and copy them to all your storage nodes:

$ swift-ring-builder account.builder rebalance
$ swift-ring-builder container.builder rebalance
$ swift-ring-builder object.builder rebalance

$ scp account.ring.gz swift-node-1:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-1:/etc/swift/container.ring.gz
$ scp object.ring.gz swift-node-1:/etc/swift/account.ring.gz
$ scp account.ring.gz swift-node-2:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-2:/etc/swift/container.ring.gz
$ scp object.ring.gz swift-node-2:/etc/swift/account.ring.gz
 ...

Wait a while (say, an hour) for things to settle down, then add a little more weight:

$ swift-ring-builder account.builder set_weight z1-192.168.12.104:6002/d16 50
$ swift-ring-builder container.builder set_weight z1-192.168.12.104:6001/d16 50
$ swift-ring-builder object.builder set_weight z1-192.168.12.104:6000/d16 50
$ swift-ring-builder account.builder rebalance
$ swift-ring-builder container.builder rebalance
$ swift-ring-builder object.builder rebalance

$ scp account.ring.gz ...

Then wait another hour and do it again, and again, and again. 120 hours later, your new drive will be a full member of your Swift cluster.

Adding capacity gradually with SwiftStack

The cycle of add weight, push rings, wait, and repeat cries out for some automation. Here’s how you do it with SwiftStack:

Screenshot of gradual capacity addition

Choose “Add Gradually”, click “Change”, and SwiftStack will add a bit of weight to that drive once an hour until it’s a full member of the cluster. You can see how it’s progressing…

Screenshot with progress indicator

…and a (long) while later, it’s done:

Screenshot showing completion

We feel that this is a big improvement over running dozens of commands hundreds of times each.

Rule 2: Disks Get Bigger

It used to be that 150GB disks provided the most storage per dollar and 500GB disks were the biggest you could buy; now 2TB disks are the cost sweet spot and 4TB disks are hitting the market. If you’ve got a storage cluster using older disks, you can breathe new life into it by taking out the old, slow, small disks and replacing them with new, fast, large ones. Of course, the trick is doing it while maintaining availability and performance.

Removing Capacity Gradually

We saw before that making drastic changes to the disks in your Swift cluster is a recipe for poor performance and angry users, so we’re going to skip past drastic changes and go right to the gradual solution.

First, you have to take an old drive out of service so that you can physically remove it and not affect the cluster’s operation. That’s done by gradually reducing its weight in the ring, then rebalancing, pushing, waiting a while, and repeating.

We’re goint to remove the 500GB disk labeled d5 from the storage node at 192.168.12.104 and replace it with a new 2TB drive. To start, we look at the existing builder file and see what the device’s weight is:

$ swift-ring-builder account.builder 
account.builder, build version 117
1048576 partitions, 3 replicas, 4 zones, 4 devices, 0.02 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
             0     1  192.168.12.104  6000        d2 500.00     786469    0.00 
             1     2  192.168.12.104  6000        d3 500.00     786498    0.01 
             2     3  192.168.12.104  6000        d4 500.00     786263   -0.02 
             3     4  192.168.12.104  6000        d5 500.00     786498    0.01 
            ...

Then turn its weight down a bit (say, by 25):

$ swift-ring-builder account.builder set_weight z1-192.168.12.104:6002/d5 475
$ swift-ring-builder container.builder set_weight z1-192.168.12.104:6002/d5 475
$ swift-ring-builder object.builder set_weight z1-192.168.12.104:6002/d5 475

$ swift-ring-builder account.builder rebalance
$ swift-ring-builder container.builder rebalance
$ swift-ring-builder object.builder rebalance
$ scp account.ring.gz swift-node-1:/etc/swift/account.ring.gz
$ scp container.ring.gz swift-node-1:/etc/swift/container.ring.gz
$ scp account.ring.gz swift-node-1:/etc/swift/account.ring.gz
...

Wait an hour and do it again; repeat until its weight is 0, then wait another hour (for the last bits of data to flee), take it out of the ring (as follows), then physically replace the drive.

$ swift-ring-builder account.builder remove z1-192.168.12.104/d5
$ swift-ring-builder container.builder remove z1-192.168.12.104/d5
$ swift-ring-builder object.builder remove z1-192.168.12.104/d5
$ swift-ring-builder account.builder rebalance
$ swift-ring-builder container.builder rebalance
$ swift-ring-builder object.builder rebalance

Now that your new drive is in place, add it to the ring and gradually turn its weight up, as done above.

Removing Capacity Gradually with SwiftStack

SwiftStack also makes it easy to remove drives gradually. It looks just like gradual capacity addition; just choose “Remove Gradually” and click “Change”, then sit back and let the magic happen.

Screenshot of gradual capacity removal

Sam Merritt

Sam Merritt

Senior Engineer, SwiftStack
OpenStack Swift Core Team

Categories

OpenStack, OpenStack Swift, SwiftStack

CloudStack going Apache 2 Monitoring Swift With StatsD

Comments

© 2014 SwiftStack Inc.        San Francisco, CA         contact@swiftstack.com