Lean Startup Metrics: AARRR!!!

I wrote a bit about Lean Startup Metrics last year after attending a Lean Coffee TO session.

One particularly useful framework is "Startup Metrics for Pirates" from Dave McClure.

His popular "AARRR" model suggests segmenting your metrics into:

• Acquisition
• Activation
• Retention
• Referral
• Revenue

I decided to crunch about 13 months of data from my project management software startup PMRobot.

Here's what I found:

My Analysis

1. An 13% signup conversion rate seems awesome! -- until you look at activations.
2. The first column shows very clearly how many people you need to put into the funnel at the top to get revenue out at the bottom. It's often more than you think!
3. Breaking down the step to step rates is critical -- it lets you see at a glance where you're losing people.
4. We still have a lot of work to do helping users activate. We've been simplifying the interface and adding inline help. What else would you suggest?
5. Our retention is not as good as I would hope. Would you recommend exit interviews to find out why the left?

What do you think? What else can we learn from this data?

A few side notes about the chart:

• Conversions from step to step read diagonally -- ie. Activation -> Retention is 12.82%.
• The first column is the absolute conversion rate from all new visitors.
• The numbers are a bit misleading since signups are per user and activation/retention/revenue is per organization (which could include many users)
• This includes data only for customers acquired through the website.
• Therefore, it excludes my two largest revenue-generating customers, since they were converted by face-to-face meetings.
• "Activation" is an in-app metric corresponding roughly to "actually using the product for something real" -- as opposed to just putting in test data.
• "Retention" indicates long-term frequent users -- the ones that are "hooked" :)
• "Revenue" are customers that are currently on monthly credit card billing.

---

About the author: Jason Hanley is a software developer, project manager, world traveler, and a private pilot. He spends his time trying to make project management for software consulting companies easier, and just plain better.

Lean Coffee: Pirate Metrics and Customer Feedback

Great little Lean Coffee session this morning at Big Bang. My notes:

• How do you decide what to build?
• AARRR!
• Acquisition: Signups
• Activation:
• Subjective: 1 post, 2 posts, a series of events?
• Retention
• How many coming back every X days, X months
• "Active users"
• Referral
• Revenue
• Example: How many signups per blog posting?
• Focus on one metric at a time -- for instance "how do we get people to activate?"
• Activation phase: Need to focus on particular group/niche
• What is that "painful" thing people need to do to set up
• Switching costs
• Revenue: What makes the highest profit?
• Sometimes your metrics are wrong
• For instance, are your activations _really_ activations?
• Metrics: Are you tracking the "offline" parts of the experience?
• Retention: Continued activations
• It's not just one feature -- it's how they all fit together
• Everyone has their one different "one thing"
• You can divide into different users groups, and each one can have their own activation
• Example metric for Basecamp: "User has a conversation"
• You often have to sell to your customer multiple times
• Activation metric should always be testing your _new_ users, not mixed up with existing users
• What set of features provide the most value: to customer, to business
• Which feature is causing us the most problems?
• Start measuring first -- then set goals

SUPER BONUS!!! Live screenshots from PMRobot's custom dashboard.

You can have a look at the type of metrics we currently track.

Click to zoom in.

Recent signups: Note that contact information is right there so we can email for feedback quickly

"Active" users: Currently defined as created more than 7 days ago and logged in within past 7 days

More active users: We use highlighting and bold to show which ones are further along in the process

Organization dashboard: Shows a small subset of "key" accounts

Lean Coffee: Git Money!

Some brief notes from this morning's "Git Money" Lean Coffee TO session at Big Bang:

• No, this was not about Git :)
• How do you validate whether someone will pay, before you even build the product?
• Example: Coffee Shop owners
• Biggest problem: How to get feedback from their customers
• Made up rough wireframes and brought them to coffee shops
• Asked for $20, promised working software by a certain date
• In return, they got $200 of service
• Reduced the discount rate as more customers signed up
• Social proof made it an easier sale
• Would this approach work for more expensive products?
• Potentially -- Companies sell unfinished software all the time
• Start with a discount, and reduce that discount over time
• What is the product is really complicated?
• Customers may ask to see more before handing over money
• In that case, you may need to look to a different type of "visionary" customer
• What if your product has an existing competitor?
• All products have some type of competition (ie. the "do nothing" alternative, or other channels that might not be "direct" competition)
• How much do you need?
• Depends on how good a sales person you are
• Might be able to see just based on a conversation
• Otherwise, might need wireframes, working prototype, etc.
• Future discussion topic: Customer Objections

Insane Apache FastCGI settings

We recently moved PMRobot to a new server, configured very similar to the old one with WHM/CPanel.

The only major difference was a newer version of the Apache Web Server -- 2.2.21 vs 2.2.16.

Unfortunately, not long after the switch, reports started coming in that most attachments were not uploading. Only very small ones (less than 100K) seemed to work.

Since there were no recent changes to our attachment code, I started digging through logs to discover the source of the problem.

None of the usual error logs reported seemed to contain anything useful.

Finally I worked my down down to the root Apache error log and discovered this little gem:
[warn] [client ...] mod_fcgid: HTTP request length 137881 (so far) exceeds MaxRequestLen (131072), referer:

A nondescript little warning that thankfully led me to this post which solved the problem.

This person also seemed to have had the same problem. 

The icing on the cake was the little warning at http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxrequestlen, which reads:

Warning: Before 2.3.6, this defaulted to 1GB. Most users of earlier versions should use this directive to set a more reasonable limit.

My question is WHY!?! Why did they reduce a 1GB default limit to 128K? Surely someone must have realized that havoc that might wreak, no?

In any case, if you encounter this problem, simply add the following to your Apache config:

<IfModule mod_fcgid.c>
 FcgidMaxRequestLen 1000000000
</IfModule>

and you should be good to go again.

Lean Coffee: Are ideas overrated?

Had a great LeanCoffeeTO this morning at BNOTIONS with a lively discussion roughly summarized by the following points:

• Startups have become very popularized
• There's a trailer for an American Idol-type startup TV show
• Problem with Lean: Doesn't really allow for "blue sky" thinking?
• Ideas need to come at the right time
• Only see a limited view at any given time snapshot
• Is starting a "startup" different than starting a "business"?
• Startups need to have a "grand vision"?
• Treating Lean as "gospel" is dangerous
• Process needs to be flexible
• Lean and its process makes a great base for discussion
• Very different environment for businesses now (vs late 90s/early 2000s)
• Faster pace, easier technology
• Value of ideas: Can turn any idea into "something"
• Idea vs. vision
• Idea is just a "spark" that can lead to a full vision
• Twitter has evolved
• Started by a failed podcasting company
• Interviewing is hard
• Lean process reduces risk
• However -- Reducing the risk is not the same as getting to success?
• Starting a startup not "sensible", the sensible thing is to get a job
• Would benefit Ontario if health care software providers followed more Lean principles?

So the discussion took a few interesting turns along the way, but overall a very valuable session.

Great to see everyone and really looking forward to the upcoming one year celebration events.

Lean Coffee: Raising your first round

I think I've been spending a few too many late nights coding PMRobot UI enhancements, as I had some trouble actually finding today's Lean Coffee Meetup.

I was running late due to the TTC, and the Meetup Android app hadn't updated the recently-changed location. So I wandered through an empty building trying to figure out where I had gone wrong.

Luckily somebody (who had obviously had more sleep) pointed me to the open door I had just walked past, and I was able to join in and take down some notes about raising investment:

• The Canadian Angel investing community is not as established as in the US
• Tech startups are getting more popular -- getting harder to find developers here
• Canada is "a couple steps behind" but heading in the right direction
• Investors are looking to invest immediately and get their returns in 5 years
• Currently near the end of a 5-year cycle -- little money available
• Something to check out: Angelist
• Canadian companies get government matching and SRED, so big financial advantage
• It's not beneficial to hold back on communicating the details of an idea
• What is the right time to approach investors? Probably not the idea stage
• Government grants, funding, etc. can work at the concept stage
• Having solid market research can help with grants
• There are grant writing companies who will help you apply for a cut of the cash
• Board of advisors
• Not easy to establish
• What's in it for them?
• Angels place a large value on your board
• Angels tend to be on the team more than the idea

Glad I finally found everyone this morning. Next time I may need a pre-LeanCoffee coffee :)

See you then!

Lean Coffee: Vizualize.Me Case Study

The summer has been busy, and I didn't think I'd be able to get back into Lean Coffee before September, but this session's topic was too good to pass up.

Eugene Woo from Vizualize.Me presented an amazing case study for the group.

Here are a some key notes from the discussion:

• Won StartupWeekend, 2000 signups on the first day
• Thousands of signups per day -- now over 100,000
• After a month, decided to pursue it full-time
• Did a "press push" about winning StatupWeekend, etc. but didn't really take
• Made an Ashton Kutcher sample and pushed to blogs
• Didn't really get much on the blogs
• But major press picked up on it somehow
• Led to articles on FastCompany and Mashable
• "You don't learn when you're building"
• No mockups, built a working prototype quickly
• Important for learning -- wouldn't learn the same things with a mockup
• Customer development can be depressing, but is also exciting
• Priorities are not clear, but at least you know the problems
• 1-3 months away from "true" MVP
• So far 100% equity, seeking a seed round
• 10 or so competitors have emerged
• So far they have a huge branding head start
• 50% conversion rate from signup list, 15-20% fill rate for Wufoo survey
• Lots of wrong assumptions and learning: eg. Thought people wouldn't need to edit inline
• At least 5 phone calls per week, lots of email and surveys
• Use MixPanel for data collection
• Started out collecting everything, but now focusing on a few key metrics at a time
• Biggest potential competition: If LinkedIn builds similar capability
• Jobs market is ripe for disruption
• Best source of leads/signups: LaunchRock, Twitter

Thanks again Eugene for sharing, and looking forward to see Vizualize.me continue to grow and evolve!