Monitoring Swift With StatsD
A Swift cluster is a complicated beast—a collection of many daemons across many nodes, all working together. With so many “moving parts” it’s important to be able to tell what’s going on inside the cluster. Tracking server-level metrics like CPU utilization, load, memory consumption, disk usage and utilization, etc. is necessary, but not sufficient. We need to know what the different daemons are doing on each server. What’s the volume of object replication on node8? How long is it taking? Are there errors? If so, when did they happen?
In such a complex ecosystem, it’s no surprise that there are multiple approaches to getting the answers to these kinds of questions. Let’s examine some of the existing approaches to Swift monitoring and then discuss what we do here at SwiftStack.
The Swift Recon middleware can provide general machine stats (load
average, socket stats,
/proc/meminfo contents, etc.) as well as Swift-specific
- The MD5 sum of each ring file.
- The most recent object replication time.
- Count of each type of quarantined file: account, container, or object.
- Count of “async_pendings” (deferred container updates) on disk.
Swift Recon is middleware installed in the object server’s pipeline and
takes one required option: a local cache directory. Tracking of async_pendings
requires an additional cron job per object server. Data is then accessed by
sending HTTP requests to the object server directly, or by using the
There are some good Swift cluster stats in there, but the general server metrics overlap with existing server monitoring systems and to get the Swift-specific metrics into a monitoring system, they must be polled. Swift Recon is essentially acting as a middle-man metrics collector. The process actually feeding metrics to your stats system, like collectd, gmond, etc., is probably already running on the storage node. So it could either talk to Swift Recon or just collect the metrics itself.
There’s an upcoming update to Swift Recon which broadens support to the account and container servers. The auditors, replicators, and updaters can also report statistics, but only for the most recent run.
At SwiftStack, we need to track many more aspects of the cluster’s operation beyond what Swift Recon covers, which brings us to the next tool.
Florian Hines developed the Swift-Informant middleware to get real-time visibility into Swift client requests. It sits in the proxy server’s pipeline and after each request to the proxy server, sends three metrics to a StatsD server:
- A counter increment for a metric like
- Timing data for a metric like
obj.GET.200. [The README says the metrics will look like
duration.acct.GET.200, but I don’t see the “duration” in the code. I’m not sure what Etsy’s server does, but our StatsD server turns timing metrics into 5 derivative metrics with new segments appended, so it probably works as coded. The first metric above would turn into
- A counter increase by the bytes transferred for a metric like
This is good for getting a feel for the quality of service clients are experiencing with the timing metrics, as well as getting a feel for the volume of the various permutations of request server type, command, and response code. Swift-Informant also requires no change to core Swift code since it is implemented as middleware. However, because of this, it gives you no insight into the workings of the cluster past the proxy server. If one storage node’s responsiveness degrades for some reason, you’ll only see that some of your requests are bad—either as high latency or error status codes. You won’t know exactly why or where that request tried to go. Maybe the container server in question was on a good node, but the object server was was on a different, poorly-performing node.
So we need deeper visibility into the cluster’s operation, behind the proxy servers.
Florian’s statsdlog project increments StatsD counters based on logged events. Like Swift-Informant, it is also non-intrusive, but statsdlog can track events from all Swift daemons, not just proxy-server. The daemon listens to a UDP stream of syslog messages and StatsD counters are incremented when a log line matches a regular expression. Metric names are mapped to regex match patterns in a JSON file, allowing flexible configuration of what metrics are extracted from the log stream.
Currently, only the first matching regex triggers a StatsD counter increment, and the counter is always incremented by 1. There’s no way to increment a counter by more than one or send timing data to StatsD based on the log line content. The tool could be extended to handle more metrics per line and data extraction, including timing data. But even then, there would still be a coupling between the log textual format and the log parsing regexes, which would themselves be more complex in order to support multiple matches per line and data extraction. Also, log processing introduces a delay between the triggering event and sending the data to StatsD. We would prefer to increment error counters where they occur, send timing data as soon as it is known, avoid coupling between a log string and a parsing regex, and not introduce a time delay between events and sending data to StatsD. And that brings us to the next method of gathering Swift operational metrics.
Swift StatsD Logging
StatsD was designed for application code to be deeply instrumented; metrics are
sent in real-time by the code which just noticed something or did something. The
overhead of sending a metric is extremely low: a
sendto of one UDP packet. If
that overhead is still too high, the StatsD client library can send only a random
portion of samples and StatsD will approximate the actual number when flushing
To avoid the problems inherent with middleware-based monitoring and after-the-fact log processing, we integrated the sending of StatsD metrics into Swift itself. Our submitted change set currently reports 124 metrics across 15 swift daemons and the tempauth middleware. Details of the metrics tracked are in the Admin Guide.
The sending of metrics is integrated with the logging framework. To enable,
log_statsd_host in the relevant config file. You can also specify the
port and a default sample rate. The specified default sample rate is used unless
a specific call to a statsd logging method (see the list below) overrides it.
Currently, no logging calls override the sample rate, but it’s conceivable that some
metrics may require accuracy (sample_rate == 1) while others may not.
[DEFAULT] ... log_statsd_host = 127.0.0.1 log_statsd_port = 8125 log_statsd_default_sample_rate = 1
Then the LogAdapter object returned by
get_logger(), usually stored in
self.logger, has the following new methods:
set_statsd_prefix(self, prefix)Sets the client library’s stat prefix value which gets prepended to every metric. The default prefix is the “name” of the logger (eg. “object-server”, “container-auditor”, etc.). This is currently used to turn “proxy-server” into one of “proxy-server.Account”, “proxy-server.Container”, or “proxy-server.Object” as soon as the Controller object is determined and instantiated for the request.
update_stats(self, metric, amount, sample_rate=1)Increments the supplied metric by the given amount. This is used when you need to add or subtract more that one from a counter, like incrementing “suffix.hashes” by the number of computed hashes in the object replicator.
increment(self, metric, sample_rate=1)Increments the given counter metric by one.
decrement(self, metric, sample_rate=1)Lowers the given counter metric by one.
timing(self, metric, timing_ms, sample_rate=1)Record that the given metric took the supplied number of milliseconds.
timing_since(self, metric, orig_time, sample_rate=1)Convenience method to record a timing metric whose value is “now” minus an existing timestamp.
Note that these logging methods may safely be called anywhere you have a logger object. If StatsD logging has not been configured, the methods are no-ops. This avoids messy conditional logic each place a metric is recorded. Here’s two example usages of the new logging methods:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
We wanted to use the pystatsd client library (not to be confused with a similar-looking project also hosted on GitHub), but the released version on PyPi was missing two desired features the latest version in GitHub had: the ability to configure a metrics prefix in the client object and a convenience method for sending timing data between “now” and a “start” timestamp you already have. So we just implemented a simple StatsD client library from scratch with the same interface. This has the nice fringe benefit of not introducing another external library dependency into Swift.
In this post we looked at a few ways to gather metrics for a Swift cluster. None of them are complete. Swift Recon doesn’t look at enough and overlaps with general-purpose server monitoring packages. Swift-Informant only sees the world from a client’s perspective. Statsdlog only handles counter incrementing and parses log messages after-the-fact versus incrementing counters directly when they occur. Our integrated StatsD logging doesn’t gather “gauge” metrics like “how many async_pendings are there right now.” Instead, we only increment a counter when something is quarantined or an async_pending is created. Those metrics naturally map to a gauge.
We feel a Swift cluster’s operation is best tracked with the combination of a general-purpose server monitoring system, a mechanism for polling Swift-specific gauge metrics, and deep StatsD logging instrumentation in Swift for counter and timing metrics. For polling Swift-specific gauge metrics, we’re probably best off using a plugin for a general-purpose collection system. That plugin could read data from swift-recon, or gather the Swift-specific metrics directly.
At SwiftStack, we use collectd plus some Python plugin code for server monitoring. We also embed a StatsD server in collectd so there’s one process per node funneling stats back to a Graphite cluster. With this setup, we have all the aforementioned bases covered: general purpose monitoring, Swift-specific gauge monitoring, and real-time counter and timing data directly from Swift.
Once you have all this great data, what do you do with it? Well, that’s going to require its own post. But in addition to the obvious, like graphing it, you can perform anomaly detection, trigger alerts, maintain a real-time view of entity health, avoid surprises with capacity forecasting, and more!
“Monitoring Swift With StatsD” by SwiftStack, Inc. is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.