Question Everything vs. Intuition

One of the interesting features of the Steve Blank’s Customer Development model or Eric Ries‘ Lean Startup model is the idea that you subject all assumptions to some kind of validation which is usually some form of direct or indirect customer feedback.

If you think a customer’s are all frustrated with the current state of acme widgets and that your idea for a new widget will solve all their problems then go out and find some potential customers and find out for sure. If you think a drop shadow on the sign-up button on your website will drive people to click on it more then make a version with and without it and see which version gets clicked more.

You can quickly imagine the result of following this methodology to its extreme. You will literally question everything.

Say you have a usability expert who has worked on designing widgets for 20 years and follows all the trends and research. She has designed several successful widgets that have made their companies hundreds of millions of dollars. You hire her to design your new widget. She’s going to take all of her previous knowledge and extrapolate how best to design it. In other words, she’s going to guess. A good guess sure, but a guess nonetheless. Under these circumstances, I think far more leeway should be given to her in terms of questioning her assumptions. However, there are people without such obvious domain experience who also have great intuition.

One of the great things about Customer Development is that it forces you to write down your hypothesis and before you go out and verify that it is correct. It’s helpful for a number of reasons including making sure you don’t repeatedly make certain types of assumptions. If you always believe more women will use your product than men and you work that into your assumptions and you’re proven consistently wrong, then hopefully you’ll learn and make better assumptions later.

After a few months of verifying assumptions, you can actually go back and come up with metrics. How often were you right? How often were you wrong? Are there certain areas you were consistently right about? Are there areas that you are consistently wrong about?

How can you integrate these metrics into the decision making processes?

One idea that comes to mind is that you use it to help prioritize what you validate or even if you validate. I imagine there being a long list of things to validate. They need to be ordered in some way. It seems logical to order things by their risk to the company and that risk can be partially mitigated when the assumption is made by someone who is consistently right. When the risk to the company is exceptionally low and the assumption is time sensitive, it will naturally fall off the list.

It may turn out that formal tracking and application of intuitiveness is simply too cumbersome. People’s time is in short supply in startups and it does seem like a big process without some software to help out. Still, it is a fascinating metric. I wish I knew how good my intuition was.

Customer Development + Scrum + TDD + XP = ?

I’ve been obsessed with Steve Blank’s Customer Development lately talking about it to everyone who will listen to get their impressions of it. I’ve also been reading pretty much everything I can find on it which isn’t that much.

Eric Ries has done a good pass at trying to merge Agile with Customer Development and adding in a bunch of XP and post-XP processes like continuous integration and continuous deployment. The idea being that if you can tighten up the engineering cycle and tighten up the customer feedback cycle can only help by letting you know of potential problems early and often.

However, I must say that I wish things were just a little more explicit and well defined. Reading Steve Blank’s The Four Steps to the Epiphany, I found myself constantly asking myself both why and how. To be fair, the book is really a set of lecture notes in book form and it is not surprising that it skips over certain topics.

So if one were to write a practical guide to Customer Development complete with Agile product management practices, XP engineering practices, what exactly would it look like? In Eric Ries’ version, the two processes seem to be only loosely coupled. The iterative cycle of the customer development team happens in parallel with the iterative cycle of the product development team with occasional synchronization points. However looking at Customer Development, my inclination is to make it an integral part of the Agile process.

My preferred flavor of Agile is Scrum mixed with XP. I like a lot of the engineering practices of XP like pair programming and test driven development, but I like the more scalable project management features of Scrum. Like other Agile methodologies, Scrum is all about small incremental changes followed by a learning phase, a re-prioritization of features which leads to a (hopefully) minor course correction. The learning phase is traditionally a set of meetings where people present what they completed and receive feedback from stakeholders on what they like and what they don’t.

My initial urge is to make Customer Development a Scrum process that drives the Product Development Scrum process. The product the Customer Development team delivers would be the features and priorities for a viable product based on customer feedback.

However something about this bothers me. I don’t like dependencies and I think it might require far too many changes to the Scrum process to really be consider Scrum anymore. Plus, what are the benefits to merging it in?

The second idea is to sit a Customer Development team member on each Product Development team making highly cross-functional teams. The Customer Development person would have somewhat dual responsibilities in that they would also contribute towards re-prioritizing the feature lists. There are some problems with this option too. After all, the Customer Development person has responsibilities that don’t directly relate to the feature backlogs. However it does have the benefit of being able to have someone there to do multivariate testing and get customer feedback for the features the team is implementing.

Then there is the question of iteration length. I’m not sure Scrum is really compatible with ultra-short intervals that you would get from continuous deployment and there is something to be said for longer development cycles. For one, simply interpreting the customer feedback for a product that changes hourly is difficult and I’m not sure purely relying on the numbers you get back from multivariate / split testing as a way to determine feature prioritization is a sane policy.

Even with all these questions of process worked out, there is still the question of actual technology. Adding continuous deployment requires software that can deploy it to a server, determine if it is working properly and remove it if necessary. Doing split testing or multivariate testing requires any number of pieces of software, some of which you may have to build yourself especially if you’re doing something other than testing web site features.

If you could package up all the software required into a bundle, what would it contain for each of the major languages used out there?

I feel like it is going to be a lot of fun finding answers to all these questions.

Customer Development

I met with Andrew Chen yesterday and had a great talking about all manner of things from online advertising to fireworks. One thing we discussed that really piqued my interest though was the subject of Customer Development.

For the last couple weeks, I’ve come across several instances of a product development style that I’d never encountered before. What I saw were very flat startups with no product developers, no engineering managers and almost an intentional lack of anything resembling a detailed vision.

Instead of designing the product first and building it, these companies design by experimentation. They put out multiple versions of the same product and see which one does the best. It’s simple enough. Google has had a system up for a while now called the Website Optimizer that essentially lets you do this and from what I understand, they employ the methodology extensively at Google. What these companies were doing though is different. It seems they are actually using a tight customer feedback loop to go beyond optimizing the visual look of their products and using it instead essentially figure out new features to build in the first place.

I’ve been reading everything I can get my hands on on the subject. Some parts of of it appeal to me greatly. At Napster, I often tested changes on a small subset of our users, but I always thought that this was a bad thing. Over the years I’ve come to realize that users will put up with a lot if they like the service enough. Plus, I think if you manage the process in the right way and let the users feel like they are actively participating in making a product better, a few might even grow more attached to it.

I’m still skeptical of some things. For instance, without someone who understands the entire product, you might end up building it using a poor underlying technology which will lead to having to rewrite the entire product eventually. Statistics for successful rewrites aren’t pretty. I’m also a bit wary of using the Agile methodology of very small incremental changes without someone who understands the big picture since it can take you down a dead-end pretty quickly.

You can find more about Customer Development from Steve Blank and his book Four Steps to the Epiphany and from Eric Ries’ blog Lessons Learned.

Overall though I can’t say I’ve seen a more exciting methodology in a while. I can’t wait to experiment with it.

P.S. - If anyone has any more resources to share, I’d love to hear about them!

Busy

I’m sorry I haven’t written anything in a while. I’ve been busy working on Heavy Electrons and it has really eaten up all my time. Things are moving along now pretty well and I’m hoping to have some new features up soon.

I should probably post more about Heavy Electrons too. I’ll have to do that.

Back to work…

Scaling Internet Applications

So you built a wildly successful service and now the users are pounding at the gate to get in. You pat yourselves on the back, draft up a business plan to present to the VCs and start dreaming about how you’re going to spend all that money. 6pm rolls around and the users are still flooding in. In fact, more are flooding in than every before. By 7pm the web servers are starting to get slow. It seems the slower they get the more users want to come on. By 8pm, the service has ground to a halt.

You spend all night tracking down problems, rejiggering your MySQL configuration, getting a friend to host another server for you. By 6am everything seems calm and the service is running smoothly again. A day goes by with no problems. A week. You’re thinking about a launch party. Then someone posts about it on Techcrunch and the whole thing comes grinding to a halt again.

Congratulations. You are a success.

The scenario is common with startups. So common that invite-only betas were created to help limit the number of users to what a service could handle (and as a marketing tactic, but that’s another story…).

Why is it so common? Well it is impractical for a small startup to fully optimize their software for millions of potential users. Not only is functionality not written in stone yet, but you have no idea how your software is going to be used. Plus there is the time and experience involved, both of which are precious in an early-stage startup. Finally, there are so many “experts” on scaling software giving advice on the Internet that scaling seems easy, certainly nothing to worry about.

If you ask people how to optimize a web server, you will receive literally hundreds of answers. Use linux, use lighthttpd, tweak your tcp settings, up your MTU, buy more machines, install more memory, buy faster CPUs, use an in-kernel webserver, split your static and dynamic sites, use a CDN, use python, use perl, use ruby, host it on amazon… With so many options to choose, how could anyone not think scaling was easy?

Let’s talk about the dirty little secret related to software scaling. Every application is different. What works for one system may not work for another. Just because LiveJournal use memcached to offload the database doesn’t mean you should. Just because Facebook uses massive numbers of MySQL servers doesn’t mean you should. Just because the Twitter architect recommended denormalizing the database doesn’t mean you should.

For the most part, none of these guys claimed to be The One True Way. They simply wrote about what worked for them. However there are almost always a situation where a particular method of scaling won’t work or will actually making things worse.

There are a few things that every early-stage startup should do to prepare for and manage scaling problems:

Track the usage of your applications

Before you launch, make sure you have at least basic logging and analysis setup. Not only is the data useful for business purposes, but will help you prioritize features to optimize.

Emergencies aren’t over after the system is back up

The absolutely worst thing you could do after a major system failure is go back to work like nothing happened. Failures always come in sets. You do not want to end up in a constant fire fighting situation where there is no time to think.

Emergency fixes tend to patch the symptom and not the problem. Time should be devoted to figuring out exactly what went wrong and why.

Stress test your system

Using that wonderful usage data you’ve collected, figure out where the next breaking point will occur. If you’ve had an outage, try to reproduce it. Plot out your growth. How much longer will it be before the system reaches capacity?

Call in an expert

You probably don’t need someone who built a stock exchange working full time for you. However, having someone who has faced the same problems you’re likely to face spend even a single day looking at your architecture, usage patterns and system can save you a lot of pain later on.

They say that’s too many users is a good problem to have. I agree. Just be sure you can cope with your success.

Quantifying interestingness

The Internet has become tedious.

There is so much information published these days, both directly and indirectly, by people that it has become a chore to sift through it all to find the truly interesting bits. The vast majority of the twitters, pownces, Facebook feed stories, LinkedIn Network Updates and are mundane at best.

If there was nothing worthwhile in these services, I could just ignore them altogether. Unfortunately, buried deep within each one are unique insights, interesting tidbits of information on friend’s lives and the occasional juicy bit of gossip. The problem is that the signal-to-noise ratio on these services is so ridiculously high that it has become work to actually find the good bits.

Quantifying interestingness is hard - extremely hard. Essentially you are trying to guess if someone will be interested in something before they have been exposed to it. The techniques used by services such as Amazon, Digg and others don’t lend themselves particularly well to large numbers of transitory messages with small audiences. This is especially true for Twitter where messages are often sent by SMS to users immediately after they are written.

However the services themselves, either by design or by accident, have contributed greatly to the amount of noise. Facebook seems to generate a feed story for seemingly every ordinary action a user takes. Twitter’s choice of a 140 character message doesn’t leave room for well thought out ideas and their system of broadcasting out replies to everyone means seeing a lot of messages without any context because you aren’t following all parties.

To Facebook’s credit, they do attempts to filter news feed articles based on what they think users will read. A news feed page contains only a small portion of all friend feed stories. Unfortunately it does very little to stem the flood of updates I see every day that waste my time. The filter controls aren’t fine grained enough to allow me to, for instance, only show messages where I know all the parties involved. This alone would significantly improve my experience.

If the amount of noise on these services gets much higher, I’m going to have to seriously rethink how I use these services. If that means getting my news second hand from friends the old way, well so be it.

Is San Francisco part of Central California?

According to websites I’ve visited recently, San Francisco and most of the Bay Area is in Central California. Welcome to CenCal!

I grew up in both the Bay Area and Los Angeles area and I was brought up to believe that San Francisco is in Northern California. As a kid I didn’t think much about Central California. It was just some place in “the middle.” I knew Southern California started around Santa Barbara and Northern California ended somewhere around Monterey.

For the last couple years I’ve visited a lot of websites with snow reports, surf reports and occasionally marine weather and I’ve noticed something quite peculiar. They refer to this area as Central California.

Surfline seems to think that Central California starts in San Francisco and extends all the way down to Northern Santa Barbara. Northern California officially starts across the Golden Gate in Marin.

The Surfrider Foundation says Northern California is everything North of San Francisco.

Even Wikipedia’s article on Pacifica (5 miles south of San Francisco) says it is part of Central California.

Surely these people must be wrong. I can’t have grown up in Central California. That is crazy. Or is it?

The graphic is an image of California divided into thirds based on latitude. It isn’t absolutely perfectly accurate, but since my entire world is crashing down before me, you’ll have to forgive the quick job.

If you look at the graphic, San Francisco does indeed fall in Central California. Worse, it isn’t even the Northern most part. Who knew that Marin, Napa and Sonoma were in Central California?

Of course there is more than one way to divide a map. What if you divide it based on population?

The map to the left rough. I only used county population figures which means that the real borders are a little bit different due to uneven population distribution across counties. The data I used is from a couple years ago. At the time California had a population of 37,172,015, so each colored section represents about 12.4 million people.

I like this one better. All of Santa Clara, Santa Cruz and part of Monterey counties are included. It does go a bit further south than what I think of as Northern California. Southern California is a mess though. All of a sudden LA is split in two and Santa Barbara is part of Central California.

I could probably munge things a lot more. Maybe pull some political data or incoming information. The thing is that the separation of California is really more of a feeling.

This is my imaginary map of how I view California. Yosemite is ours. Santa Cruz is ours. Most of the Sierra Nevada mountain range is ours.

Southern California gets Santa Barbara and San Bernardino. I thought to include Death Valley, but then I don’t think I remember it as really part of Southern California.

I’m not sure what I’d do if I lost the NorCal label. CenCal t-shirts? No. I don’t think I could get used to that.

Are social networks social?

I don’t know when it happened. Recently, I’ve stopped using the Internet and started viewing it.

For years I was extremely active on IRC, newsgroups, mailing lists, blogs and review sites. You can actually find posts by a preteen me on Usenet archives from 1992. I may not have met a lot of the people I interacted with, but I felt like I was part of a community and as a member of the community, I contributed.

Lately however I use the Internet to either look up facts or be entertained. I’ve stopped using it to connect with people. I don’t feel like I’m part of any community anymore.

We are in the age of Social Networking. Aren’t social networks supposed to be social? How is it that I don’t feel anymore attached to Facebook than I do to my address book?

I guess my real question is: what do social networks actually do?

The truth is that social networks let me portray a shallow persona of how I want to be seen to people portraying their own shadow persona of how they want to be seen. People there aren’t real. They don’t write about their real feelings or express their real self because the audience consists not only of their closest friends, but also their co-workers, acquaintances, distant family, ex-bfs/gfs, etc.

Social networks offer little more than watered down look at who people really are. This won’t help strengthen your bonds with existing friends and it certainly won’t help finding new friends.

That’s not to say social networks don’t have any good uses. If you want people you meet to be able to find you and contact you, they work great. I just don’t think they work well for anything particularly social.

When music was cool

I think the biggest sin that the major labels have committed was making music lame.

How is that possible? How can music, a fundamental building block of cool be lame?

Maybe I’m biased. Actually there is no maybe. I am biased. I was at Napster. I tried to deal with them at SNOCAP. I never understood the phrase “soul sucking” until I dealt with the major music labels. They are run by lawyers who will spend 6 months on a contract to extract every last dime out of you even if their own salaries far exceed what was gained. I mean they really are run by lawyers. Of course they are litigious! Of course they’d think suing their customers would solve their problems - strike fear into the hearts of those evil file sharers everywhere!

Thankfully the music industry is evolving. The major labels won’t die. Saying otherwise is denying how they came into existence in the first place. They will simply buy smaller, more successful indie labels until eventually they replace themselves from the inside. It’ll take a while, but short of firing their entire legal staff or hiring an executive staff that knows how to order them around (as opposed to be ordered around by them), it is the quickest way

The death of DRM is a good sign. I hope SNOCAP had something to do with that. In the end though, they still haven’t found the right price point. They are still focused on selling music. It isn’t all their fault. That is what they own. They haven’t built a secondary market like the movie industry. They let the artists own it.

Maybe tomorrow I won’t think music is lame. There is always hope.

I wish there was a small Macbook

I haven’t bought a new Apple laptop to replace my ancient 12″ iBook. I’ve been patiently waiting for Apple to put out a new 12″ laptop. It doesn’t seem like they’re going to.

I bought my girlfriend a 13″ Macbook. It is not small. It is significantly bigger than my 12″. If you look at the raw numbers, the 13″ Macbook is actually smaller than the iBook, but that’s only if you count the thickness. I don’t care about the thickness!

I like thin phones because I put them in my front pocket and don’t like what it does to my pants. I do not carry my laptop in my pocket. I put it in a laptop case which is padded and makes everything thicker. I put it in that case with the power cord and a dozen other things that increase the thickness of what I’m carrying.

The thickness of my laptop does not matter. What does matter is the width and height. I carry my laptop around. I balance it on my lap while in parks. I put it in front of me on crowded airline flights. Plus, a smaller width and height allows me to carry a significantly smaller looking laptop bag which more people see than my laptop itself.

Another thing I actually like about the Macbook over the Macbook Pro and Macbook Air is the casing. I owned an original Mac Titanium. A year in and I had a warped casing, snapped the clamps that held the LCD display in place, dented it in several places and otherwise made a pretty sad looking laptop.

I like the polycarbonate casing of the Macbook. It is strong and I am hard on laptops. My five year old iBook doesn’t have so much as a scratch on it even though it has been through many countries, a couple sailboats, fallen from several high places and traveled quite a bit without any type of protective casing.

So Apple, please make a smaller laptop. I will thank you. The Japanese will thank you. More importantly, I’ll finally be able to buy a new laptop.