Lean Startup Metrics: AARRR!!!

I wrote a bit about Lean Startup Metrics last year after attending a Lean Coffee TO session.

One particularly useful framework is "Startup Metrics for Pirates" from Dave McClure.

His popular "AARRR" model suggests segmenting your metrics into:

• Acquisition
• Activation
• Retention
• Referral
• Revenue

I decided to crunch about 13 months of data from my project management software startup PMRobot.

Here's what I found:

My Analysis

1. An 13% signup conversion rate seems awesome! -- until you look at activations.
2. The first column shows very clearly how many people you need to put into the funnel at the top to get revenue out at the bottom. It's often more than you think!
3. Breaking down the step to step rates is critical -- it lets you see at a glance where you're losing people.
4. We still have a lot of work to do helping users activate. We've been simplifying the interface and adding inline help. What else would you suggest?
5. Our retention is not as good as I would hope. Would you recommend exit interviews to find out why the left?

What do you think? What else can we learn from this data?

A few side notes about the chart:

• Conversions from step to step read diagonally -- ie. Activation -> Retention is 12.82%.
• The first column is the absolute conversion rate from all new visitors.
• The numbers are a bit misleading since signups are per user and activation/retention/revenue is per organization (which could include many users)
• This includes data only for customers acquired through the website.
• Therefore, it excludes my two largest revenue-generating customers, since they were converted by face-to-face meetings.
• "Activation" is an in-app metric corresponding roughly to "actually using the product for something real" -- as opposed to just putting in test data.
• "Retention" indicates long-term frequent users -- the ones that are "hooked" :)
• "Revenue" are customers that are currently on monthly credit card billing.

---

About the author: Jason Hanley is a software developer, project manager, world traveler, and a private pilot. He spends his time trying to make project management for software consulting companies easier, and just plain better.

Lean Coffee: Pirate Metrics and Customer Feedback

Great little Lean Coffee session this morning at Big Bang. My notes:

• How do you decide what to build?
• AARRR!
• Acquisition: Signups
• Activation:
• Subjective: 1 post, 2 posts, a series of events?
• Retention
• How many coming back every X days, X months
• "Active users"
• Referral
• Revenue
• Example: How many signups per blog posting?
• Focus on one metric at a time -- for instance "how do we get people to activate?"
• Activation phase: Need to focus on particular group/niche
• What is that "painful" thing people need to do to set up
• Switching costs
• Revenue: What makes the highest profit?
• Sometimes your metrics are wrong
• For instance, are your activations _really_ activations?
• Metrics: Are you tracking the "offline" parts of the experience?
• Retention: Continued activations
• It's not just one feature -- it's how they all fit together
• Everyone has their one different "one thing"
• You can divide into different users groups, and each one can have their own activation
• Example metric for Basecamp: "User has a conversation"
• You often have to sell to your customer multiple times
• Activation metric should always be testing your _new_ users, not mixed up with existing users
• What set of features provide the most value: to customer, to business
• Which feature is causing us the most problems?
• Start measuring first -- then set goals

SUPER BONUS!!! Live screenshots from PMRobot's custom dashboard.

You can have a look at the type of metrics we currently track.

Click to zoom in.

Recent signups: Note that contact information is right there so we can email for feedback quickly

"Active" users: Currently defined as created more than 7 days ago and logged in within past 7 days

More active users: We use highlighting and bold to show which ones are further along in the process

Organization dashboard: Shows a small subset of "key" accounts

Lean Coffee: Vizualize.Me Case Study

The summer has been busy, and I didn't think I'd be able to get back into Lean Coffee before September, but this session's topic was too good to pass up.

Eugene Woo from Vizualize.Me presented an amazing case study for the group.

Here are a some key notes from the discussion:

• Won StartupWeekend, 2000 signups on the first day
• Thousands of signups per day -- now over 100,000
• After a month, decided to pursue it full-time
• Did a "press push" about winning StatupWeekend, etc. but didn't really take
• Made an Ashton Kutcher sample and pushed to blogs
• Didn't really get much on the blogs
• But major press picked up on it somehow
• Led to articles on FastCompany and Mashable
• "You don't learn when you're building"
• No mockups, built a working prototype quickly
• Important for learning -- wouldn't learn the same things with a mockup
• Customer development can be depressing, but is also exciting
• Priorities are not clear, but at least you know the problems
• 1-3 months away from "true" MVP
• So far 100% equity, seeking a seed round
• 10 or so competitors have emerged
• So far they have a huge branding head start
• 50% conversion rate from signup list, 15-20% fill rate for Wufoo survey
• Lots of wrong assumptions and learning: eg. Thought people wouldn't need to edit inline
• At least 5 phone calls per week, lots of email and surveys
• Use MixPanel for data collection
• Started out collecting everything, but now focusing on a few key metrics at a time
• Biggest potential competition: If LinkedIn builds similar capability
• Jobs market is ripe for disruption
• Best source of leads/signups: LaunchRock, Twitter

Thanks again Eugene for sharing, and looking forward to see Vizualize.me continue to grow and evolve!

PMRobot Project Management Survey #1 Results

Back in March, I put together a survey with two very simple statements:

1. When managing a project, I often feel like a glorified secretary

The results for this question surprised me. Only about half of the 32 respondents felt this way.

2. Communication between clients and programmers is poor or non-existent

This question turned out more like I expected. Most people agreed that client-programmer communication could be improved.

There were also some great insights from the free form comments:

• Programmers: "Often they are not engaged early enough in the project.  Therefore they are missing information that has already been covered."
• "If your programmers just aren't client-safe, then you are doomed to be, or to rely upon, 'Systems Analysts' of the sort you see in the movie 'Office Space'."
• "It's important to engage with clients and programmers. When you wait for them to come to you and don't engage proactively, then communication suffers."

Thanks again to all who responded!

I've already used this feedback to help prioritize the communication aspect of PMRobot over the automation features.

P.S. I'm running a new survey about existing tools. I'd love to hear your feedback at http://bit.ly/ifX4SV

Lean Coffee: How to launch a product

This morning we chatted about how to launch a product the lean way. A few key observations:

• What's the definition of "launch" for Lean?
• Multiple "launches" during customer development
• You need to refocus you team from development to sales and support
• Infrastructure: As you move from beta, backup and failover systems become more important
• Looking for early adopters
• Do beta/soft launches first to smaller groups
• Don't see a lot of "big bang" launches anymore -- more "constant" betas
• However, "hard" launches are still common in enterprise (CDs, media, etc.)
• Slowly going away and moving to SAAS (software as a service)
• Product Lifecycle Curve (from Crossing the Chasm)
• Techniques:
• Target influential bloggers in the field
• LaunchRock (a bit spammy, but works)
• Need organic growth -- make it easy for people to do the marketing for you
• It's a different story launching something that people need to actually spend money on
• Difficult to cut through the noise and get people to pay attention
• The personal approach:
• "Bribe" early adopters and influencers and give a personal touch (example: Hashable)
• An email from the founder makes people feel important
• Personal connections can't be faked
• As you grow, build the personal connections into the culture ("customer development team")
• Big launch with press release, etc. less personal -- not as approachable
• Your current users are more valuable than new ones
• Are you launching a product, or launching a company (with a business model)?
• Don't assume it's going to "go viral" -- Design for if it doesn't
• Categories of influencers: How well do you know your customers?
• If you want to get coverage, have something truly interesting, novel to say (example Gmail "goggles")
• Key influencers (especially well-known ones) can make all the difference
• Frequent, iterative launches: Find "excuses" to get heard, and stay consistent
• It takes many times of people hearing something before they actually remember

This is great advice as I personally iterate towards progressively larger "launches" of our project management software, PMRobot.

These Lean Coffee sessions are always a great way to start the day and get thinking about "big picture" stuff that you might not otherwise.

Thanks to Jeremy for hosting, and everyone else for attending and contributing!

Lean Coffee: Project vs. Product Management

This morning's Lean Coffee TO was hosted by Jeremy at Jar Creative. We had an interesting discussion (and side topics) related to project and product management, including:

• Difference: Projects are finite (at least they're supposed to be :)
• Products: The end isn't always clearly defined
• Is Waterfall methodology still is use? Yes. (unfortunately)
• Product management is a bridge between the customers and development team
• Product Management overlaps with Customer Development
• Projects: Divide them up by features / user stories
• Time-boxing: Pick a fixed date
• Product management is about making decisions
• What do you do in the later phases when your product is launched?
• Sometimes need to go back and do a full adjustment
• Often when you release people don't use it the way you expect
• Difference between strategy and tactics: long term goals with iterations as a tactic
• Helpful: The ability to turn feautures on and off for individual users/groups
• eg: Google, Facebook, Big Bang (Woople)
• Sometimes discussing features takes longer than just implmenting it and getting immediate feedback
• How does technology affect product development? (ie. mobile, tablets, etc.)
• The technology decision should come from product management / customer development

Great seeing everyone, and looking forward to next Tuesday's session with Ali from Well.ca.

Lean Marketing

I attended the brand shiny new Tuesday session of Lean Coffee Toronto today at the awesome BNOTIONS headquarters at Yonge and Bloor.

My summary follows:

• Thursday's session of "How to spend $5,000" apparently wasn't the right approach to the discussion
• (good thing we got a second try :)
• Different parts: Researching vs. Positioning vs. Building Demand
• Can you describe the product in a few short sentences? (ie. the length of a Google Ad)
• Very important to have consistent messaging
• Creating the UVP (unique value proposition) is the hardest part.
• Technique: "It's like X for Y"
• Example: It's like Dropbox for Development Environments
• Pros: You can use all of the marketing efforts from X for free!
• Cons: Helps describe the What, but not necessarily the Why
• 6 steps: (iterative)
• Problem
• Values
• Market
• Ideal consumer
• Competition
• Positioning (USP)
• Technique: 3rd-party re-explanation:
• Explain to someone, then have that person explain it to someone else new
• Tip: Don't try to deliver too many different concepts in your message
• How far along do you need to get to get proper feedback? Idea? Mockups? Prototype?
• Answer: All of the above. Start with an idea and move up from there as you get validation
• Pivoting: Do you change the market or the product? (or both?)

And some Meetup-related stuff:

• Next meetup: Same topic both days again
• Perhaps some attempt to:
• Post reading material prior to meetup (and have people actually read it)
• Keep the discussion on topic via the moderator (a difficult task, to be sure)
• Look at the San Fran Group for topic ideas and literature
• Check out http://dooo.sh/it/ from Big Bang (blog post)

Please comment on this post or email any corrections, additions or updates.

Until next time!

Structuring Experiments: Product within a service company

Some notes on today's Lean Coffee Toronto case study, hosted superbly by Jet Cooper.

• Rocketr: 0 to launch in 99 days
• Andrew joined the firm after some long-term travel
• They had 3-month and 10-year plans, but nothing in-between
• Unique: Created a separate entity external to Jet Cooper for Rocketr
• Corporation
• Board of Directors
• Legal Agreements
• Monetary Investment (both from Jet Cooper and people involved)
• Hardest part: Defining the MVP
• Artificial constraints: Having a hard launch date helped keep focus
• Talking to people early on was extremely valuable
• Every week they had potential customer using it
• First 30 days were basically lost due to a substantial change in methodology and approach
• But, this lead to valuable lessons learned for Jet Cooper as a whole
• Now focused on "Agile Design"
• Tools: Paper, Balsamiq, HTML (all the standard stuff)
• Product vision: "Open source" your early thinking about projects
• Focus and forget about the stuff that doesn't matter
• If it's truly important, it will be brought up again

Jason's thoughts:

• The "Agile Design" lessons sound extremely valuable
• Would love to see these organized and documented publicly (perhaps in Rocketr? :)
• Is the separate entity and investment really necessary?
• How much overhead does the legal/corporate stuff add in the early stages?
• There was mention that the board of directors often took up a lot of time
• The "throw stuff out" advise is good (sounds very Tim Ferris or 37signals although not 100% sure of the source)
• Could this approach be "lean-ified" a bit?

Really great to hear these cases studies. Looking forward to next week!

Developing a product within a service company

For this week's Lean Coffee Toronto, we discussed the ins and outs of developing a product inside of a service company.

A few interesting discussion points from my notes:

• Products often come from satisfying an internal need
• Working on products have many intangible benefits, including:
• Increased motivation
• "Bragging rights" for developers
• How do you know when to "unbind" the product from the service company? No clear answers:
• When the product needs investment in marketing, support, enhancements, etc.?
• When the market is established?
• How do you know when to "kill" the idea?
• Treat it the same way as you would any investment
• If it is being used and generating value internally, this question isn't relevant
• Ask yourself: Is the product a hobby, or a business?

I found this discussion extremely valuable as we begin to productize our own internal system: http://softwarepms.com/

Thanks to everyone who attended, and especially to Andrew for facilitating, and Jet Cooper for hosting.