Lean Coffee: Pirate Metrics and Customer Feedback

Great little Lean Coffee session this morning at Big Bang. My notes:

• How do you decide what to build?
• AARRR!
• Acquisition: Signups
• Activation:
• Subjective: 1 post, 2 posts, a series of events?
• Retention
• How many coming back every X days, X months
• "Active users"
• Referral
• Revenue
• Example: How many signups per blog posting?
• Focus on one metric at a time -- for instance "how do we get people to activate?"
• Activation phase: Need to focus on particular group/niche
• What is that "painful" thing people need to do to set up
• Switching costs
• Revenue: What makes the highest profit?
• Sometimes your metrics are wrong
• For instance, are your activations _really_ activations?
• Metrics: Are you tracking the "offline" parts of the experience?
• Retention: Continued activations
• It's not just one feature -- it's how they all fit together
• Everyone has their one different "one thing"
• You can divide into different users groups, and each one can have their own activation
• Example metric for Basecamp: "User has a conversation"
• You often have to sell to your customer multiple times
• Activation metric should always be testing your _new_ users, not mixed up with existing users
• What set of features provide the most value: to customer, to business
• Which feature is causing us the most problems?
• Start measuring first -- then set goals

SUPER BONUS!!! Live screenshots from PMRobot's custom dashboard.

You can have a look at the type of metrics we currently track.

Click to zoom in.

Recent signups: Note that contact information is right there so we can email for feedback quickly

"Active" users: Currently defined as created more than 7 days ago and logged in within past 7 days

More active users: We use highlighting and bold to show which ones are further along in the process

Organization dashboard: Shows a small subset of "key" accounts

Lean Problem Interviewing

Lean Coffee was hosted today by mad scientists in white lab coats (ie. the Big Bang Technology crew).

We talked about the Problem Interview and various tips & techniques, such as:

• Keep a "perimeter" -- don't go too far outside the topic
• Who/Where/What/When/Why questions (from journalism)
• Why validate?
• Are there any customers?
• Assumption: Your plan A won't work
• No validation = no repeatable sales process
• Beware of confirmation bias: Tendency to seek the answers you want to hear
• Problem interview vs. Solution interview (very different)
• Remember: Can pivot around the business model, solve different problems
• Do you need a script? Helps keep things on track - vs. map or outline
• But more open-ended discussion lets you discover a lot more
• Customers are like kids -- easy to lead them, and have them agree with things they don't actually want
• Should be a conversation, not a formal interview
• They'll tell you the symptoms -- you need to determine the problem
• Keep doing the interviews until you know what the answers are going to be
• Problem: Looking for what's "broken"
• Important: What people say vs. what they do is often very different
• Methodology doesn't prevent you from still missing the mark on a "must have" problem
• Failed products lead to the next one -- always learning
• Answers change in different contexts -- anonymously vs in a group (observer effect)
• Maximize the comfort level to get honest answers
• Warm them up: Ask about themselves, make environment safe
• Technique: Parrot back responses
• Lets them confirm, helps them feel more comfortable
• Discover people's motives

Another great session!

Next week I'm going to be in Orlando on business (I know, poor me) so hopefully someone else can take a few notes.

Lean Metrics

Chris Eben hosted this morning's Thursday session of Lean Coffee Toronto at The Working Group.

We chatted about Metrics, including:

• What are metrics? Defining the things that are most important to your business
• Don't base decisions on metrics until you know they're the right metrics
• Start at revenue and work backwards to find out the actions that lead to revenue
• Be careful about which numbers you're basing decisions on:
• Does your sales funnel indicate whether your pricing is wrong?
• ...or if you're trying to sell to the wrong people?
• Try to see where the bottleneck is -- eg: What are they key indicators of dropoff?
• Don't make decisions on the "marketing/vanity" metrics
• Another thing aggregate metrics often don't tell you:
• One user might be 10x more valuable than another
• How do you know which ones?
• "Stop listening to" and "start watching" your customers
• Product-market fit: 40% of your users would care if you disappeared
• Can't apply metrics from one customer segment to another
• ie. early adopters vs. majority
• Track at least one key metric in each part of the AARRR model
• Practical example:
• If you notice that conversions suck after 1 month, could be "honeymoon period"
• Trim eval period back to 10 days and improve conversions
• One of the best tools: Another set of eyes, second opinion
• OKCupid demographics data set overviews -- generally useless but entertaining
• Need to change your key metrics you have as you move into new business phases

We also spoke briefly about my recent software project management survey, how survey design is difficult, and how we could spend an entire session talking about surveys. I'd be happy to plan and moderate such a session, but would need a location.

To Document or Not To Document

As Lean Coffee TO's unofficial "documenter", I found our discussion on documentation very enlightening.

There seemed to be two opposing viewpoints -- oddly enough, from two physical sides of the room.

My bullet point summary follows:

• Why are people afraid of documentation?
• Makes it "real"
• Not enough time
• Documentation makes delegation possible
• Must be kept up to date
• Cost: overhead for every document you make
• Sometimes you'll write something and never look at it again
• Can documentation limit creativity? (depends on the type)
• Very important for remote teams
• Also important for new hires
• Organizing documentation is very difficult
• It should grow organically
• Document according to risk level
• Surgeons that document their process kill less people
• Helps "be kind to your future self"
• Example: Captain Picard and his log (had to be there to get this one -- awesome example :)
• People have many ways of learning -- writing helps think about problems differently
• Our brains are not designed for storage
• Lean aspect: Wait until there's real pain before documenting

And my own thoughts on the matter:

• I personally hate writing documentation, but...
• Our memories are more like goldfish than we think -- we forget stuff quick
• I document Lean Coffee out of necessity -- It's the only way I can justify the time invested to get some long-term learning that I can refer back to
• Documentation leads to Automation (and automation is very efficient/lean)
• Templates -- I only mentioned this briefly, but think it's ultra-important:
• Saves massive amounts of time with certain processes
• Again, leads to automation -- with a good, up-to-date template, I can create a proposal in 10 minutes that might have otherwise taken 3 hours
• You need to keep stuff in one place (or at least link the multiple sources together)
• Everything to do with my company is either in Google Docs or our project management system, and they both link to each other
• On the personal side of things, my girlfriend and I keep all important info in shared Google Docs and Google Calendars
• I've found that numbered lists or bullet points are better than paragraphs and sentences (since that's how we tend to think)
• Finally, I think that organizing documentation is far more important than simply creating it
• Perhaps you'd use some of your previous notes if they were actually somewhere you could find them :)

Structuring Experiments: Product within a service company

Some notes on today's Lean Coffee Toronto case study, hosted superbly by Jet Cooper.

• Rocketr: 0 to launch in 99 days
• Andrew joined the firm after some long-term travel
• They had 3-month and 10-year plans, but nothing in-between
• Unique: Created a separate entity external to Jet Cooper for Rocketr
• Corporation
• Board of Directors
• Legal Agreements
• Monetary Investment (both from Jet Cooper and people involved)
• Hardest part: Defining the MVP
• Artificial constraints: Having a hard launch date helped keep focus
• Talking to people early on was extremely valuable
• Every week they had potential customer using it
• First 30 days were basically lost due to a substantial change in methodology and approach
• But, this lead to valuable lessons learned for Jet Cooper as a whole
• Now focused on "Agile Design"
• Tools: Paper, Balsamiq, HTML (all the standard stuff)
• Product vision: "Open source" your early thinking about projects
• Focus and forget about the stuff that doesn't matter
• If it's truly important, it will be brought up again

Jason's thoughts:

• The "Agile Design" lessons sound extremely valuable
• Would love to see these organized and documented publicly (perhaps in Rocketr? :)
• Is the separate entity and investment really necessary?
• How much overhead does the legal/corporate stuff add in the early stages?
• There was mention that the board of directors often took up a lot of time
• The "throw stuff out" advise is good (sounds very Tim Ferris or 37signals although not 100% sure of the source)
• Could this approach be "lean-ified" a bit?

Really great to hear these cases studies. Looking forward to next week!