Do you know exactly how many people are signing up, using your product and returning month after month?
Having worked with tech businesses over the last few years, I’ve noticed that data and reporting beyond acquisition is often overlooked. Many businesses become preoccupied with encouraging people to register for their application, but few concentrate on engagement.
Allow me to explain why the failure to track full funnel growth is an issue. A popular framework for tracking funnels is Pirate Metrics, popularised by 500 Startup’s Dave McClure. Pirate Metrics has 5 stages:
In normal marketing reporting the only part of this funnel we tend to measure is Acquisition. But the problem is, it’s only when they reach the monetisation phase that they become something tangible on your Profit and Loss (P&L) statement, and it’s only when they reach the referral stage that they turn into a seriously useful asset when it comes to the health and growth of your business.
To see how this works in practise, here is how pirate metrics can be applied to Uber’s business model:
Between each stage, every business will see a natural drop-off. Not every user who signs up to your product will end up paying for it, let alone inviting other users. But the efficiency that someone goes from being acquired to being monetised will have a huge impact on your business. And unless you know the drop offs at each stage, and why they’re happening, it’s impossible to make an educated decision on which part of your business you should focus on - or if it’s even a viable business.
The rate at which these people progress through each stage has a direct impact on your business and the revenue they bring to it. The less drop off between each stage, the more efficient you are at converting sales and marketing expenses into profit. However, if you can’t identify the amount of drop off, and the reason behind their lack of progression, then there’s no way you can make the decisions needed to drive your business onward.
Typically, any reasonable growth report will include metrics on spend and acquisition.
Here’s an example of two ride-hailing businesses:
Based on the data above, it’s safe to say that Company A spends less money acquiring users than Company B, but we can’t yet identify which one is the best business. Why? Because business isn’t only about spending money - it’s about making money. So let’s include some revenue numbers.
Now it’s a different story. Sure, Company B pays more to acquire users - but they’re worth twice as much. They’ve made a £300 profit (spend - revenue), whilst Company A has only made a profit of just £150.
If you were going to invest in one of these businesses, which would it be? Company B, right?
Before you decide, understand this - we still can’t make an informed decision when it comes to deciding which company is the best at turning their spend into profit. This is because most digital products aren’t transactional. They usually realise their revenue over many months, or even years in the case of enterprise software.
Let’s factor retention into the equation:
How about now? Over the course of that cohort of user’s lifetime, Company A would make £600 more than company B.
Without sufficient data that accurately illustrates company performance, it’s impossible to pass judgement when it comes to deciding if a business is performing well, or when selecting which parts of the business need to improve.
Whilst Company B would definitely benefit from fixing the efficiency of their marketing spend, it’s their retention that’s making them fall behind their competition. Without this data, 99% of managers would recommend improving their acquisition simply because it’s easier to measure and that’s always how it’s been done.
When the team at Broadstone first came to us, it was evident they didn’t just want a marketing agency that would charge them for a fixed number of hours worked on the account each month - they wanted one that could help them achieve their goals while being completely transparent in doing so.
Broadstone Engage are looking to disrupt a £6 billion market by bringing an age-old profession into the mobile era. Their marketplace helps security guards find, apply for and secure paid, on-demand roles. We started working with Broadstone in December 2017, a month before their launch.
Initially, a few things were obvious. Their founding team possessed very strong domain knowledge and held existing relationships that connected them to those with the vacancies. Their development team was talented and committed, and their community team is top class.
But what they didn’t have was the data that enabled them to make quick, effective decisions about the growth of their product.
In this post, I’ll share how we helped Broadstone build a data function from the ground up.
To begin with, we needed to associate an action Broadstone user takes to each stage in the funnel:
Acquisition - When someone creates an account, they’re acquired. This is pretty self-explanatory, but whilst some businesses will track downloads only (a game monetised through ads, for example) Broadstone requires user accounts.
Metrics we track for this stage are:
By Channel: Volume, spend, CPA.
Activation - When someone submits a job application, they’re activated. Applying to jobs was chosen because it means the user has uncovered the core benefit of Broadstone - locating a relevant job that matches their desired compensation, location and various other factors. This match needs to sync closely to the user’s goal, so ‘completing a profile’ simply isn’t enough. We want more users matching with relevant jobs, not just more users with bigger profiles.
Metrics we track for this stage are:
# of people applying to jobs
% of total users applying to jobs
% of users who created an account, but didn’t reply to a job
Monetisation - When someone completes a shift, they’re monetised. Broadstone earns a commission for every hour a user works through a Broadstone partner.
Metrics we track for this stage are:
# of people who completed a shift
% of people who completed a shift
Avg. # of hours per shift
Total Revenue
Retention - Each consecutive month after users have applied to a job, they’re retained. Broadstone doesn’t only rely on usage to be valuable - so simply logging into the app isn’t enough. Whilst this definition may change or be expanded to include the consecutive months in which they’ve worked a job, this definition gives us a more actionable result.
Metrics we track for this stage are:
Overall 3-month retention
Retention by Cohort
Referral - Simply, when they successfully invite another user, they’re referring.
Metrics we track for this stage are:
# of new users from invites
# of users who clicked send an invite
% of users who clicked send an invite
(We can’t track the amount of people who actually sent an invite as it’s completed in the native OS environment, either iOS or Android).
There are two main sources from which we use to collect and retrieve data from.
Mixpanel
An analytics app that allows us to track in-app (and some external) events and create reports with segmentations based on values associated with the event or with the user. Because Mixpanel associates events with user profiles, it helps us do a few important things. For example, it enables us to see how many users did something, and not just how many times an event happened. It also allows us to see who the users were and communicate to those users by way of automated emails, push notifications and SMS messages.
We can build reports quickly in Mixpanel without requiring the assistance of the engineering team (outside of implementing the events, which is a one-off job).
The database
Each month, we create a static report that gives Broadstone an overview of how well their product is performing. Because the definition of these numbers never changes and we don’t segment them, we use the database as our single source of truth.
Overall Growth Model
The overall growth model is a spreadsheet that we use to track our core metrics month on month and year on year. This growth model also allows us to make projections into what performance will look like if we improve in certain areas.
The downsides of collecting data for a month on month growth model in a spreadsheet is that the data isn’t cohorted. By this, I mean that it doesn’t show how many people who signed up then viewed AND THEN worked a shift. It shows how many people signed up, and how many people worked a job, and how many people worked a shift.
This is why we rely on tools like Mixpanel, where data can be joined together based on the user’s cohort.
Mixpanel
Mixpanel itself has brilliant reporting features, and it allows us to quickly create reports pertaining to many different values and segments on the fly. Cohort reports are a particular strong point when it comes to Mixpanel, and it would be difficult to recreate using SQL without another third-party data visualization or BI tool.
Monthly Growth Report
The monthly growth report is where we take all of the data and turn it into insights. In my eyes, it’s not my client’s job to understand the nuances of every data point - they just want to know what they need to do to grow their business quickly and effectively. In this growth report we provide an update on the impact of the previous month, and recommendations and insights supported by data.
Whilst I don’t want to lessen the importance of reading the rest of this blog, I know all you’re interested in is seeing what the benefit of this work is.
Broadstone uses a matching algorithm based on how well matched a user is for the vacancy. Because of this, Broadstone needs to build a profile of the user before they apply to a job.
Using Mixpanel’s funnel feature, we were able to identify a large drop off in people signing up and then applying to jobs.
Now we knew there was a bottleneck, the next step is to figure out why people weren’t progressing.
Thankfully, Mixpanel isn’t just about numbers. Every number in Mixpanel corresponds to a person, so they don’t only tell you that someone did something - they tell you who that someone was.
Because of this, we were able to create a segment of users who had signed up in the last 30 days, but hadn’t yet applied to a job. The next step was to email them to ask a simple question. “Why?”
Again - Mixpanel makes this a breeze. Using messages, we are able to create segments based on behaviour or profile attributes.
We gathered the feedback from these users and realised that people were losing motivation to complete a lengthy onboarding process before they even knew if any jobs were available for them.
With this in mind, we set to work redesigning the onboarding process so that every piece of information was only gathered if it fulfilled one requirement: matching users with roles.
Initially, lengthy contact information and other miscellaneous data was collected. Now, we only collect data that’s vital when it comes to matching people with appropriate jobs.
Once users had found a role that was right for them, we completed the data collection process as part of the application. This meant that people could see the value in completing a task, so they were more likely to do it.
The results?
A 3.5x uplift in people signing up -> applying to jobs.
If there’s one thing that will kill your ability to gain traction, especially as a marketplace business, it’s a bad first impression. Broadstone’s users were downloading the application and expecting to meet a feed of perfectly matched jobs. Instead, they had to go through a lengthy sign-up process that didn’t add anything to the task they had employed Broadstone to achieve.
A vital part of growth in many businesses is organic referrals - people talking about your solution without you giving them an incentive to do so. But, with a bad first impression, the only discussion that’s going to occur will be a negative one.
By matching more people with jobs in a timelier manner, you effectively increase the likelihood of a good first impression - which will absolutely have a positive impact on your referability and your retention.
Let’s go back to the growth model we outlined above. Broadstone makes money when people complete jobs, so the more people applying to jobs, and the more people working jobs, the more money Broadstone makes. Due to client confidentiality, I can’t disclose Broadstone’s exact user economics, but I can create a demonstrative model based on what I’ve already shared.
According to Broadstone’s existing onboarding success rate, if they spend £10 to acquire every signed-up user and 5% turned into activated users, they’d spend £200 for every user that had the ability to apply to jobs immediately and get matched with jobs in the future. Both of these are vital towards Broadstone earning revenue.
However, by improving the rate at which we activate users, we can see that Broadstone’s marketing spend has become five times more efficient. Now, instead of spending £200 per activated user, they’re spending £40.
The major benefit of making improvements to your post-acquisition journey, is that they provide an asymmetric upside. Every future cohort who goes through that journey now has a 3.5 times greater chance of becoming an active user.
Imagine that you have sand in a bucket full of holes, and a 100 metre dash to get at least ¾ of that bucket to the end line. The less growth-minded person might decide that the best way to do that is to set off as fast as possible, only to realise they’ve got to fill their bucket back up every 10 metres before they can continue. However, the growth minded person would realise that a little work upfront would result in moving forward and losing less sand, while their competitors would be frantically scrambling to fill up a bucket that’ll never get them to the finish line.
