This chapter covers the different types of metrics and what they’re useful for.
Counters are for counting things. They answers questions like:
How many requests did my app handle in the last hour?
How many times did my app rate limit a request?
How many bytes did my app upload?
Metrics analysis tools (Datadog, Graphite, Grafana, and so on) can also look counters during periods of time also known as a rate. With tools like that, you can answer questions like:
How many requests does my app handle per minute?
Reporting sizes and measures (gauge)¶
When you want to measure something over time, you use a gauge. They answer questions like:
How much memory does my app use?
How many items are in the crashmover queue?
How many threads are running?
Measuring these once isn’t interesting. Measuring these periodically and then seeing those measures go up and down over time and possibly in response to various events (sudden high load, external service going down, and so on) is very interesting.
These numbers are useful for monitoring and alerting. For example, maybe when the crashmover queue hits a specified threshold, that means the app is in trouble. That’s something you’d want to get alerted to.
Getting statistical distributions (timing, histogram)¶
When you want to measure something over time, but you want statistical information about those values, you want to use timing and histogram.
Statsd-type monitoring tools will calculate the count, average, median, 95th percentile, and max for these values over some period.
This helps you answer questions like:
What’s the median, 95%, and max HTTP request payload sizes?
What’s the median and 95% durations for executing some database query?
In these cases, knowing just the measured values isn’t helpful, but knowing the median and 95% is very helpful. You’d be able to see how those things generally are and what the extreme cases are like.
Some backends don’t support histogram. In that case, histograms are reported as timings because the two are essentially the same where “histogram” is the more general of the two.
Depending on the backend you’re using, there may be rules for the metrics stats names you’re generating.
Generally, if you follow these rules, your stats should work across all backends:
Use only ASCII letters, ASCII numbers, and periods.
The metric stats name should begin with a letter.
Keep metric stats names short.