Lean Startup Metrics: AARRR!!!

I wrote a bit about Lean Startup Metrics last year after attending a Lean Coffee TO session.

One particularly useful framework is "Startup Metrics for Pirates" from Dave McClure.

His popular "AARRR" model suggests segmenting your metrics into:

• Acquisition
• Activation
• Retention
• Referral
• Revenue

I decided to crunch about 13 months of data from my project management software startup PMRobot.

Here's what I found:

My Analysis

1. An 13% signup conversion rate seems awesome! -- until you look at activations.
2. The first column shows very clearly how many people you need to put into the funnel at the top to get revenue out at the bottom. It's often more than you think!
3. Breaking down the step to step rates is critical -- it lets you see at a glance where you're losing people.
4. We still have a lot of work to do helping users activate. We've been simplifying the interface and adding inline help. What else would you suggest?
5. Our retention is not as good as I would hope. Would you recommend exit interviews to find out why the left?

What do you think? What else can we learn from this data?

A few side notes about the chart:

• Conversions from step to step read diagonally -- ie. Activation -> Retention is 12.82%.
• The first column is the absolute conversion rate from all new visitors.
• The numbers are a bit misleading since signups are per user and activation/retention/revenue is per organization (which could include many users)
• This includes data only for customers acquired through the website.
• Therefore, it excludes my two largest revenue-generating customers, since they were converted by face-to-face meetings.
• "Activation" is an in-app metric corresponding roughly to "actually using the product for something real" -- as opposed to just putting in test data.
• "Retention" indicates long-term frequent users -- the ones that are "hooked" :)
• "Revenue" are customers that are currently on monthly credit card billing.

---

About the author: Jason Hanley is a software developer, project manager, world traveler, and a private pilot. He spends his time trying to make project management for software consulting companies easier, and just plain better.

Lean Coffee: Pirate Metrics and Customer Feedback

Great little Lean Coffee session this morning at Big Bang. My notes:

• How do you decide what to build?
• AARRR!
• Acquisition: Signups
• Activation:
• Subjective: 1 post, 2 posts, a series of events?
• Retention
• How many coming back every X days, X months
• "Active users"
• Referral
• Revenue
• Example: How many signups per blog posting?
• Focus on one metric at a time -- for instance "how do we get people to activate?"
• Activation phase: Need to focus on particular group/niche
• What is that "painful" thing people need to do to set up
• Switching costs
• Revenue: What makes the highest profit?
• Sometimes your metrics are wrong
• For instance, are your activations _really_ activations?
• Metrics: Are you tracking the "offline" parts of the experience?
• Retention: Continued activations
• It's not just one feature -- it's how they all fit together
• Everyone has their one different "one thing"
• You can divide into different users groups, and each one can have their own activation
• Example metric for Basecamp: "User has a conversation"
• You often have to sell to your customer multiple times
• Activation metric should always be testing your _new_ users, not mixed up with existing users
• What set of features provide the most value: to customer, to business
• Which feature is causing us the most problems?
• Start measuring first -- then set goals

SUPER BONUS!!! Live screenshots from PMRobot's custom dashboard.

You can have a look at the type of metrics we currently track.

Click to zoom in.

Recent signups: Note that contact information is right there so we can email for feedback quickly

"Active" users: Currently defined as created more than 7 days ago and logged in within past 7 days

More active users: We use highlighting and bold to show which ones are further along in the process

Organization dashboard: Shows a small subset of "key" accounts

Lean Coffee: Vizualize.Me Case Study

The summer has been busy, and I didn't think I'd be able to get back into Lean Coffee before September, but this session's topic was too good to pass up.

Eugene Woo from Vizualize.Me presented an amazing case study for the group.

Here are a some key notes from the discussion:

• Won StartupWeekend, 2000 signups on the first day
• Thousands of signups per day -- now over 100,000
• After a month, decided to pursue it full-time
• Did a "press push" about winning StatupWeekend, etc. but didn't really take
• Made an Ashton Kutcher sample and pushed to blogs
• Didn't really get much on the blogs
• But major press picked up on it somehow
• Led to articles on FastCompany and Mashable
• "You don't learn when you're building"
• No mockups, built a working prototype quickly
• Important for learning -- wouldn't learn the same things with a mockup
• Customer development can be depressing, but is also exciting
• Priorities are not clear, but at least you know the problems
• 1-3 months away from "true" MVP
• So far 100% equity, seeking a seed round
• 10 or so competitors have emerged
• So far they have a huge branding head start
• 50% conversion rate from signup list, 15-20% fill rate for Wufoo survey
• Lots of wrong assumptions and learning: eg. Thought people wouldn't need to edit inline
• At least 5 phone calls per week, lots of email and surveys
• Use MixPanel for data collection
• Started out collecting everything, but now focusing on a few key metrics at a time
• Biggest potential competition: If LinkedIn builds similar capability
• Jobs market is ripe for disruption
• Best source of leads/signups: LaunchRock, Twitter

Thanks again Eugene for sharing, and looking forward to see Vizualize.me continue to grow and evolve!

Lean Coffee: Customer Segments

Today's Lean Coffee was hosted by Mark at BNOTIONS.

Some great points on customer segments:

• Reference: Geoffrey Moore - Crossing the chasm
• Get your early adopters to become evangelists, help you cross the gap
• They give feedback and a slightly different perspective
• Invest in retention more than acquisition --> referrals
• Early majority requires training/support/documentation than early adopter
• Early adopter term applies across many fields -- projects, organizations, etc.
• Influencer -- Applies to music, cars, etc.
• Identify people that have an active need, ask them questions
• When they have a clearly identified problem, and the pain is high enough, they're much more open to new stuff
• Segment your customers by talking to them and identifying similarities
• How do you identify evangelists?
• If they're supporting other users or otherwise being vocal (in a positive way)
• Stats: Who is the most effective at bringing in users? Often the same people that use it the most
• When you identify evangelists, give them stuff, help them help you
• The biggest skeptics can become the biggest evangelists
• Personal touch helps engagement -- send a note, give a phone call

That's it for today! What were your takeaways from the session?

Lean Metrics

Chris Eben hosted this morning's Thursday session of Lean Coffee Toronto at The Working Group.

We chatted about Metrics, including:

• What are metrics? Defining the things that are most important to your business
• Don't base decisions on metrics until you know they're the right metrics
• Start at revenue and work backwards to find out the actions that lead to revenue
• Be careful about which numbers you're basing decisions on:
• Does your sales funnel indicate whether your pricing is wrong?
• ...or if you're trying to sell to the wrong people?
• Try to see where the bottleneck is -- eg: What are they key indicators of dropoff?
• Don't make decisions on the "marketing/vanity" metrics
• Another thing aggregate metrics often don't tell you:
• One user might be 10x more valuable than another
• How do you know which ones?
• "Stop listening to" and "start watching" your customers
• Product-market fit: 40% of your users would care if you disappeared
• Can't apply metrics from one customer segment to another
• ie. early adopters vs. majority
• Track at least one key metric in each part of the AARRR model
• Practical example:
• If you notice that conversions suck after 1 month, could be "honeymoon period"
• Trim eval period back to 10 days and improve conversions
• One of the best tools: Another set of eyes, second opinion
• OKCupid demographics data set overviews -- generally useless but entertaining
• Need to change your key metrics you have as you move into new business phases

We also spoke briefly about my recent software project management survey, how survey design is difficult, and how we could spend an entire session talking about surveys. I'd be happy to plan and moderate such a session, but would need a location.

Inside The Lean Startup

This evening I attended a great panel discussion event at the MaRS Discovery District entitled "Inside The Lean Startup".

There were some great points brought up by the panel of @ashmaurya, @skanwar, @leilaboujnane, and @davidcrow of StartupNorth etc.

My takeaways:

• What do people misunderstand about Lean?
• not a step-by-step guide -- it's a set of philosophies
• not "cheap" or "bootstrapping" -- process efficiency
• Potential mistakes:
• Relying on the wrong set of tactics for the stage they're at
• Over complication in systems (ie. continuous integration)
• Should be about changing behaviour -- getting out and talking to customers
• All about efficiency -- get the most for the least
• Key Metric: Money -- Are people paying you for your product?
• Key Question: How disappointed would you be if this product would be taken away tomorrow? 
• 40%+ indicates not a fluke
• Key Points:
• Reduce assumptions
• Identify riskiest parts first and evaluate
• Don't afraid to be embarased
• Find the shortest route to get in front of customer and get feedback
• Lean applies to: unknown problems / unknown solutions
• Hypothesis test: pull the plug and see if anyone cares!
• When do you start charging? From day 1. ("free" is a customer acquisition tactic, not a business model)
• Starting out: Is this a problem worth solving?
• Make sure a hypothesis is falsifiable -- the scientific method
• "Life is too short to build products nobody cares about." -- Ash
• Adding features:
• Unused features are waste
• Creates technical debt
• Start with "no"
• How do you get feedback?
• Initial hypothesis: pain is so great that they want to be involved in the process
• Find your early adopters -- as visionary as you are (rare breed, hard to find)
• People are always open to telling you about their problems ("tell me about your pain")
• If you're not getting feedback, find out why -- not reachable? don't like forms, method of communication, type of question

And some miscellaneous notes:

• Key Resources:
• 4 steps to the epiphany
• http://www.custdev.com/
• Startup Metrics for Pirates
• Upcoming: http://runningleanhq.com/

• Toronto Events:
• http://to.hackdays.ca/
• http://democamp.com/
• http://www.meetup.com/leancoffeeto/

Valuable stuff. Looking forward to more events like this one!

Customer Development

Another really interesting Lean Coffee Toronto this morning.

Here are some of the notes I took from the conversation:

• Always have checks and balances in place
• Verification: The best way to determine product-market fit
• Dave McClure metrics for startups (The "AARRR" presentation)
• Four Steps to the Epiphany (book by Steve Blank) (.pdf file)
• Canadian startups tend to skip this step or do it too late
• Use a rapid prototype instead of just an idea
• Don't redesign the whole thing just based on your first customer's feedback
•  Know when to say no
• Customers may not know what they want
• When first starting have the "seven conversations"
• Measure. Validate.
• Look for lovers and haters -- anything that evokes a strong response should be investigated further

Some personal thoughts on the subject:

• You can learn a lot from "negative" feedback
• If someone tells you they won't use your product, ask, "Why not?"
• You may have a good idea or product, but lack the ability to explain it well enough
• The most successful products are the ones people actually understand
• It can take a long time and many iterations to figure out your "pitch"
• Don't stop working on it until you can tell people "get it" -- for better or for worse
• If they are indifferent, you have a problem (mentioned by Andrew near the end of the meetup)

Great conversation again everyone! An hour just isn't long enough. We always seem to be just getting into the good stuff when wrapping up :)

How to fail successfully

Today I attended Lean Coffee Toronto's 15th Meetup. What a great group of people and amazing discussions.

We talked about the concept of "failing successfully". Here are a few notes I took:

• Don't put all your eggs in one basket
• "Parallel prototyping" helps -- better results with the same overall effort
• Make sure you learn lessons for future projects
• Admit your mistakes frankly
• When do you stop?
• Sometimes it's very hard to let go of an idea
• Always get external feedback -- advisory boards, etc.
• Example of successful failure: the production of Iron Giant
• Metrics and measurement:
• Define success and define failure
• Did you make money? Did you build your brand?
• Tie your metrics back to goals
• Always do a port-mortem

... and some thoughts of my own on the topic:

• WGMGD -- What gets measured gets done
• My biggest take from a Services Marketing class I took a while back
• Don't be "stealthy"
• Get your product out there as early as possible and get feedback
• "First to market" and "stolen ideas" are myths
• "Failure is always an option"
• Mythbusters and Adam Savage are awesome :)

Thanks again everyone for a great meetup!