Metrics 101. Essential terms and definitions

Table of contents

  1. Metrics are the measurements that reduce uncertainty
  2. Any metric has associated value and costs
  3. The motto of valuable metrics: High uncertainty with high cost of being wrong
  4. Predictive and descriptive metrics
  5. How to define a health metric for a community

Metrics are the measurements that reduce uncertainty

A metric is a quantitative measurement or a few measurements of something in some specific context. The purpose of any metric is to help decision-makers make better decisions. The difficulty of making a decision is that usually there is some uncertainty associated with the decision. We use metrics to reduce that uncertainty. Metrics do not need to reduce uncertainty to zero, but they must add some additional information about a phenomenon or information that simplifies understanding of the phenomenon.

Any metric has associated value and costs

The value of a metric is the difference between two decisions: one made with a metric and the other made without it. If it is equal to zero, then there is no point in having the metric, since the metric does not improve (or does not participate at all in) the decision. The costs of a metric are the costs of thinking through, implementing, and using the metric. Although the cost of metrics in online communities is kept to a minimum, metrics require interpretation and attention by a decision maker or an analyst. Any new metric you add creates yet another data point by itself and one more thing to look at. If metrics are not created wisely they add more noise to the data than there was prior to them and overwhelm the decision making person, reducing the value of having them at all.

If a metric has no value, there is no reason to have it.

The motto of valuable metrics: High uncertainty with high cost of being wrong

We can represent uncertainty as a set of possible outcomes of an event where each outcome has its own probability of happening and a value, positive or negative. Always, especially when you are just starting adding metrics to your community, look for areas where there is high uncertainty and high cost of being wrong. Those situations make metrics being of high value.

Predictive and descriptive metrics

We can divide all metrics into two categories.

  • Descriptive metrics. We use descriptive metrics to understand the current state of the community. Typical examples of metrics that describe the state are the number of monthly engaged users, pageviews, site satisfaction, etc.

  • Predictive metrics. We use predictive metrics to understand what the future of the community might look like. Predictive metrics also help when we cannot run an A/B test, so you can model several what-if scenarios with different parameters of the system and compare the outcomes. When one works on predictions they need not only to be able to predict something but also be able to define what a positive answer for a metric would look like. Any forecast is a predictive metric.

Different tasks require different metrics. Most of the tasks on the individual contributor level require metrics that describe the current state of the community, whereas tasks at the executives level require predictions. The reason is that executives use metrics to plan for the future when individual contributors use metrics to direct their current actions.

How to define a health metric for a community

We say that a system is healthy if it efficiently does what it has been designed for. In other words, with healthy metrics we measure efficiency. Efficiency of a system is a broad term and can be measured in many ways. As a result, to avoid the problem of interpreting health metrics, we want to reduce the number of the metrics to a single metric that measures the most important process on which all others depend. To determine this process, ask yourself the following questions:

  • What does health or success mean for the system?
  • If everything goes well, what will the system look like?

The answer to these questions is what needs to be measured. The resulting metric is the health metric for the system.