May 5, 2025
Measuring product usage is a challenge which stumps many companies. I've experienced first hand the frustrations of product managers who had no idea how their features were performing.
In this guide I'm going to cover everything you'll need to know to accurately measure product usage. It doesn't matter if you're a one-man shop or part of a growing startup, this guide will help you implement a framework that will help you answer all your product usage questions.
Before we dive into the details it's a good idea to understand what we want to achieve.
As I see it we have two main goals.
The first goal is to implement the usage tracking framework. This framework will allow the company to accurately track product usage as it scales.
The product, R&D and operations teams will have joint responsibility on implementing, optimizing and maintaining this framework.
The usage tracking framework should be implemented as soon as possible in the company's history to help prevent data gaps as you scale and optimize your product. Each data gap is a major barrier to answering product usage questions and will add a lot of frustration to both your product and analyst teams.
Once the usage tracking framework is put into place we can start setting goals related to turning raw data into business insights. This responsibility falls on the operations teams and involves the following:
Online products are consistently changing. New features are being added and existing features are evolving frequently. In order to guarantee a consistent flow of quality, "BI-friendly" data you'll need to implement the usage tracking framework.
To implement this framework, follow the steps below.
I recommend that you go and read my detailed guide on how to run a data audit for your startup. The audit will help you map out the gaps that you have in your tracking infrastructure, and the amount of work needed to start filling in these gaps.
Create a Google Spreadsheet which lists all the gaps you have and work with R&D to help you enrich your spreadsheet with extra info that will be helpful when you present the gaps to the leadership. The more info you have the easier it will be to estimate the work needed to fill in the gaps.
Running the data audit will also help you better understand R&D's current process for implementing product usage tracking.
In order to accurately measure product usage you will need to get your hands on 2 types of data, dimensional and event-based.
Dimensional data is information about our users and other objects that are important to the business. This information is constantly changing and always represents the most up to date version of itself.
Below is an example of 3 rows in a typical dimension table. In this example it happens to be a "users" table.
The biggest challenge with dimensional data is that it's only a snap shot in time, it is not historical.
Let's say we save the date our users turn on feature X. We would see something like this:
activated_feature_X_at
23/10/2018 01:03:15 AM
At first glance there doesn't seem to be any issues with this but remember, this is the last time the user activated this feature. What if the user turned this feature on and off 5 times over the last 6 months? We would have no way to identify when the user activated the feature for the first time.
What if we save both the first and last time this action happened? This is better than the example above but it still prevents us from answering questions which we may want to answer later. An example would be, how long does it take a user to preform this action 10 times, or how many times does a user activate and deactivate features within the first 6 months after creation?
The best kind of data you can save for the purpose of measuring usage is event-based data.
Event-based data is information that is recorded for specific actions that we deem importnat. Event data has a few key components which are important to understand.
Most companies first get a taste of event-based data when they start using tools like Mixpanel, or Kissmetrics. These tools help companies record events which are then used to populate out-of-the-box reports. Another service I highly recommend is Segment. There are a few nuanced issues with these tools but the biggest issue I've found is cost. Both Segment and Mixpanel have raised their prices in the last 12 months and even though they offer great free plans, it is a big ask for most early stage companies to move to a paid tier.
Let's now look at a few examples of events.
Event Example #1 - Signed Up
The "Signed Up" event is a classic example of an event which most businesses send.
One obvious question behind this event is why would we send it when we are already saving this information in our database for when a user is created.
There are a few reasons why I'd recommend sending this event. The first is because typically the first reporting a business will do on its' events will be through a tool like Mixpanel. For my startup, Feedio, I use Mixpanel and can easily track the number of signups to the service through their segmentation report. The alternative would be for me to run queries on our database which is more time consuming and carries unnecessary risk. A second reason is to remove the need to join our event data with another data source to get a key user dimension, signup date. Since I'll have the milestone among our event data, I can use it for cohort analyses or funnels within Mixpanel or whichever tool I'm using to record events.
Event Example #2 - Feature Enabled
Feature enabled is a critical event and should be released along with any new feature. The event's twin is "Feature Disabled", which will be sent when a user disables a feature.
Good job, you've got a list of events being recorded and now it's time to start analyzing product usage. At this stage you have 3 options.
Option #1 - Out-of-the-box reporting
If you're using a tool like Mixpanel you'll be able to answer a wide range of questions with their built in reporting. My favorite reports are the segmentation, funnel and cohort reports. There are a few nuanced challenges with these reports but all-in-all they are great.
Option #2 - Export and Excel / Tableau
The next option you have is to export your event data via one of the built-in reports, the tools API, or through a support ticket, and then use Excel, Tableau or some other data visualization tool to run the analysis.
This method is very manual, time consuming and frustrating but it will allow you to do some more advanced analyses which won't be possible with the built-in reports.
Note that you may need to run numerous exports and blend the data before you'll be able to start your analysis.
Option #3 - BI Infra
The third option is to invest in some BI infrastructure and move the event data into your own database. You have a few options here including building your own custom ETLs, using a tool like Stitch to move the data, or to use Segment which offers data pipelines as part of its' services.
This approach is the most costly but in the long-run you will be able to get a far higher return from your data infrastructure than the two other approaches.
The usage tracking process is an internal process for guaranteeing that no product change goes live without tracking being a part of the change.
The funnel has a few steps.
The idea behind this process is to put the responsibility of collecting data in the hands of multiple individuals. This makes it much less likely that changes are made to the product and relevant data is not collected.
At this stage you should have a good idea of the data you need to record, the data gaps you have, and the company's approach to data collection.
At this point you want to organize all this information so it can be communicated to key decision makers in the organization. Once you're done organizing everything, schedule a meeting and share the following:
From my experience most early stage companies have dedicated very little resources and focus on collecting the data needed to accurately measure product usage. You will most likely find a long list of gaps that will need to be filled before an analyst can start answering even the most basic questions.
It will be your job to convince the leadership that scarce resources should be dedicated to this need. The best way to convince them is to explain the risk to the business in not being able to answer key questions on product usage. Also explain how frustrating it is that the analysts can't support the VP of product and his team with product-related insights.
Make sure you have a well thought out plan on the exact gaps you want to fill and their order of importance. Start with gaps which prevent you from answering mission critical questions (more on that below).
Convince the VP of Product to fight on your behalf and to make sacrifices so that everyone can get more insightful quicker.
Let's assume that you've been smart with your data collection and invested in your BI infrastructure from the early days. You have more data on your users than you know what to do with it. Good job, we can now get to the fun part.
The next step in our process is to come up with good product usage questions that we'd like to answer. This might sound simple enough but where do we start?
There is no perfect methodology for prioritizing questions but it does help to put your questions into one of the buckets I've listed below.
Another way to prioritize questions is to ask yourself, "how many people will be impacted by the answer to the question"? The more people that are directly affected the higher the priority.
The best tip I can give you is to use your common sense and focus first on mission critical and medium impact questions before spending time on low impact questions.
Let's assume we were working as a product analyst at Zapier and our goal was to measure product usage in order to direct company strategy.
If you've never heard of Zapier, they are a B2B company that makes it easy for businesses to connect different tools to each other to create awesome automation flows. I'm a big fan and you should definitely check them out.
Let's look at an example of a mission critical product usage question that Zapier would want to answer.
Since Zapier relies on businesses converting from their free tier to paid, and then remaining as a paid user, I'd start with the "wow" funnel.
I define the "wow" funnel as the funnel that the user goes through in order to get to that "wow" moment. This is the bare minimum we expect from the product. From that point users should continue to see value and then upgrade.
I've included a visual below of how I see Zapier's "wow" funnel.
The first step in Zapier's "wow" funnel involves the user signing up. This is our starting point. The next step is for the user to create their first zap. A zap in Zapier is a connection between two tools, like for example Gmail and Google Sheets. Once a zap is created Zapier will automatically run the zap every 15 minutes (on the free plan, less on the paid plans) and if relevant, a task is completed.
Once that first task is completed the user experiences the power of the service and will be pleasantly surprised, hence the name, "wow" moment.
So how would I track this funnel as an analyst?
I'd focus on 3 key events which I've listed below.
Once I've identified these events I can build out a funnel. I personally would use Tableau to do this work, assuming I could get my hands on the raw event data in a standard relational database. If not then I'd build out a funnel report in Mixpanel or look at goal conversions in Google Analytics.
Once I had access to the data I'd aim to build cohort views in either table or bar chart form.
The first view would include the raw numbers. This is important in order to determine if we have enough data to work with. Since Zapier is a mature company I'd imagine they have more than enough data to work with.
The second view would show the conversion rate along the funnel in relation to the first step. We can see from my example that in February 2018 56% of users reached the "wow" moment.
The last view would show the conversion rate between the different steps. This is critical to identify which step of the funnel is the most problematic. In our example we see that the big drop off happens between the first and second steps. If our focus was on optimizing this funnel, step 2 is where we'd spend our time.
The most accurate way to run such an analysis would be with a time constraint. You'd first need to determine the time constraint by running a separate analysis. In this analysis you'd take users which got to the end of the funnel and then group them by how long they took to reach each step in the funnel. I'd then recommend doing a running sum of the percentage of total to determine the 80% percentile.
You want to end up with an insight like "80% of users which reach the end of the funnel, reach that point within 120 hours (5 days)". A good tip is to use hours to normalize the data (some users signup later in the day).
So now that you have your time constraint you can plug that into your analysis. This makes the analysis more accurate since you're now comparing apples to apples. Before the time constraint users that are on the fringes in terms of time to convert will skew your numbers in earlier cohorts.
The biggest issue with time constraints is that it results in delays in determining the "health" of recent cohorts since it can take time for certain things to happen. This is just the nature of analytics and something everyone needs to live with.
There are a number of tools which are relevant when talking about measuring product usage.
I could write an entire post dedicated to this subject so I'm just going to list the tools I'd recommend exploring.
Well think about it this way. What would the cost be to the business if you can't make an informed product decision and end up making the wrong choice? Is that a risk you can live with?
My advice would be to complete step #1 in the usage tracking framework, complete a data audit. Once the audit is complete you'll have a better idea where you stand and the amount of work that is needed to get a minimal set up in place.
No you don't but I'd recommend having a dedicated individual who is responsible for this process.
For very early stage companies this may be the CTO or a senior developer. For later stage companies it could be the responsibility of a product manager or a business analyst. Go with the analyst if you have one.
I'd start by hiring a very strong senior business analyst. This individual will act as your project manager and team leader. Once his onboarding is complete let him read this blog post and get to work on the data audit.
If possible, shift some R&D resources around to start filling in the gaps the analyst has mapped out.
Next I'd signup with Tableau and get your analyst to start learning the tool. I'd look at Segment for my tracking and either Redshift or BigQuery as the tech for my data warehouse.
I'd bring on an outsourcing firm which can build my ETLs, hire an in-house data engineer or consider using a tool like Stitch. If you can do it in-house from day 1.
Depending on the size and type of business I'd start hiring strong business analysts.
If setting up your BI infra is a priority this is where you should be within 6 - 8 months:
Yes and no. In the early stages of the company there aren't any issues with using a service like Mixpanel or Google Analytics which will provide self-service reporting.
The issue with this approach is that it's very limiting and you won't be able to answer more complex questions.
Your managers will demand all kinds of things which you won't be able to deliver without having direct access to the raw data. You want complete freedom so that you can run on demand ad hock analyses and build custom data visualizations.
Time is a dimension which is often overlooked. Dedicate time to understand the time component in your business, especially when it comes to your core funnels.
Some examples to consider:
The list goes on and on.
Retention and usage behavior are also critical product metrics you should investigate.
There are a number of ways you can improve on this front.
Below are a few ideas:
I believe a company should try and know as much as it can about the inner workings of it's operations. This of course comes with a price. Common sense should be your guide and I'd recommend prioritizing your knowledge gaps and work according to that list. If you're in a fast moving company you may never get to the low priority items on that list which is perfectly fine.
I honestly believe that a company which does not adopt the usage tracking framework is putting itself at serious risk. If I had to pick one section of this post it would be to implement that framework.
If that is too much see what needs to happen so you can hire a senior business analyst who can take ownership of this project.
There are many aspects involved in measuring product usage. I hope this guide has helped you better understand the steps that are needed to implement a solid framework that will support the business.
Let the usage tracking framework be your guide and do your best to complete this project sooner rather than later. The longer you go without measuring your product usage, the tougher it will be to dig yourself out of your hole.
Feel free to comment below with any suggestions on how I can improve this guide or if you have any questions on any of the advice that I've shared.
Thanks for reading.
Join 1k+ analysts who get our weekly newsletter.
