How to designing metrics for your community

Table of contents

  1. Identify the independent subsystems of your community project
  2. Define what metric levels are needed
  3. Define the priority of metrics within the levels

Almost everyone starts tracking their communities with just a few metrics, but the number of metrics in use quickly grows to dozens and sometimes hundreds. Many metrics seem important enough to track them constantly and use for decision making. Problems begin almost immediately when, according to one important metric, the community is doing good, but according to another, it is doing bad. In addition to these two metrics, there are a dozen other important metrics and each of them tells a very unique story. Confirmation, as well as refutation, can be found for any hypothesis one likes. It is almost impossible to interpret metrics unambiguously in such a situation.

To prevent this from happening, it is necessary to approach metric design in a special way:

  • Identify the independent subsystems
  • Create an hierarchy of metrics with different levels of detail
  • Define the priority of metrics within the levels

Identify the independent subsystems of your community project

In order for measurements to be interpretable and have practical meaning, it is necessary to isolate the object of observation. An online community is a complex system consisting of several independent subsystems. The first step to creating community metrics is to identify all independent high-level subsystems. Usually an online community consists of at least three independent parts and each part has its own set of dedicated metrics:

  • Software
  • Community
  • Contents

Here is why those are independent subsystems. One can migrate a community (the users) and content to a completely new platform without impacting the project much. One can work on new content with an existing community or create a new community around existing content. At the same time, many different communities with different content can use the exact same software. All three parts are interconnected and form a single whole, though they are not tightly connected to one another to form a system with clear feedback loops.

If you have a unique setup for your community, you can figure out which parts of your particular setup are independent and which are dependent by trying to replace each part one by one with their analogues. If you are able to replace a part without significant changes to other parts it is independent and must have its own dedicated set of metrics.

Define what metric levels are needed

People at different levels in the company need metrics with different levels of detail. Directors and department heads are usually interested in the overall health of the project and its major subsystems. Product managers, developers, and community managers usually need lower level metrics related to the specific initiatives they are working on at the moment.

One of the best ways to achieve this without duplicating or overloading the number of metrics is to represent the metrics as a hierarchy, where higher-level metrics include data from all lower levels and the lower-level each metric measures one specific element of the system. We can cover the most needed metrics at the following four levels:

  1. Health metrics of the entire project
  2. Health metrics for each independent subsystem (content, community, software, etc)
  3. Success metrics for ongoing initiatives
  4. Health indicators

At the highest level, metrics show how the whole system works in general. Going down through the hierarchy of metrics allows us to observe specific phenomena and measure their effectiveness in isolation.

Depending on the size of the company you work for and your colleagues’ needs, you can have all four levels or only one.

Define the priority of metrics within the levels

Usually we have a few specific health metrics for each subsystem that are used separately. These metrics sometimes move in different directions. The first step in overcoming the chaos is to prioritize metrics of the same level through all subsystems. To do that you need to prioritize the subsystems themselves. For the communities that are in the content business, the priority of health metrics for the subsystem level usually are like follows (from highest to lowest):

  1. Content
  2. Community
  3. Software

At the first glance the order may seem controversial because in the long run the most important thing for any community project is the community itself, not the content. At the same time, in real life the first priority is to survive. If the project is closed because there is no money to pay for the servers, it does not matter whether the community was healthy or there was much to improve. Quality content attracts new members, keeps current members engaged, and generates revenue to pay the bills. The software plays an exclusively applied role for any community. All users want to have a platform with many useful tools that work quickly and stably, but simply because you have a good website no one will use it. People join and stay in a community because they are interested in either the topic (the content) or other users (the community).

If your project is in another type of business consider prioritizing metrics that measure things that generate money, influx of new users and anything else that makes the project sustainable. Then community. Everything else goes after these.