Testing for Value

Companies need a way to validate their ideas - and reduce the cost of adapting them when validation says that their value isn't so good in their current state. Accepting that most products and business strategies evolve, this validation needs to be efficient, effective and ongoing. Ideas are business value for companies who rely on innovation. For a start-up with a new product this process is make or break.

I recently read Eric Ries' "The Lean Startup" closely followed by Marty Cagan's "Inspired: How to Create Products Customers Love". Both interesting books, and both talking about minimum viable products and measuring response to products in order to guide innovation and direction.



The two books are, however, quite different to each other in style - Ries leaving me with lots of thinking to do (I don't mind this) while Cagan gave more direct advice. The lean startup advocates testing ideas in the marketplace to learn from them, evolve them and measure their value, using a minimum viable product (with spoofing functionality as an option) to get information for a minimum investment of resources. Value isn't just money, but behaviour and growth - Ries calls it innovation accounting. Cagan talks about putting prototyping first and measuring response to prototypes in order to get real information from the design phase before investing in the full product, as changes to concept are expensive to put through production. So, advocating rather different things despite the apparent overlap, with Ries talking about high risk entrepreneurial products while Cagan's approach feels a bit more about established companies and/or product areas.

I'm involved in a project that ran with prototypes first and another (at an earlier stage) that seems to be going more down Ries' approach (and I've seen a few that deployed a vision with little of either) - but a few, incomplete, pieces of anecdata don't qualify me to say one is better than the other! So, I'll ask some open questions that my reading prompted:
Is Cagan's approach more design centric and so more appropriate for products where feel is key? Is there a class of product where feel isn't important? No, probably not (though looking at the mother-ship's choice of finance software I see that not everyone thinks this way).
How much specification is needed before going to test with a product? Ries notes that the usual way to prototype with an interaction designer tends to result in a big deliverable, not agile iterative learning. Maybe changing this is the path to combining both.
Cagan discusses his reasoning: the need to put user experience design before (in time) engineering. The lightweight nature of hi-fi prototypes allowing faster exploration of what works for users than can be achieved with releasable product - and if the prototype doesn't work then it needs rethinking, engineering won't help. Engineering of code then uses the prototype as a spec. So, what place does this leave for product as experiment? I think there are a couple of cases:
First, for testing the things that a pure UX prototype can't test. I think the smoke and mirrors of wizard of oz can only go a limited way with systems enabling social connection or being used in an ongoing, highly mobile, in the background of our lives way.
Second, for testing value. Can you reliably prototype people putting their hands in their pockets and paying? I suspect not.
Third, are there cases where the minimum viable product can be engineered for less cost than the prototyping process times the risk? There is often a failure to imagine what is truly minimal - I know I naturally engineer stuff.
Fourth, where the engineering artefacts are the key to the product. The UX might require its own testing but could be fairly generic for a product where the guts provides the value difference. Experiments might replace UX prototyping here, depending on how well the real world as it affects this product can be simulated. A networking product I'm involved with falls into this category.

Later Cagan specifically addresses the early sprint as prototype question, and finds three good reasons not to. However, with constant testing of small changes subsequent development from an initial idea might work this way. This hits the willingness of users to experience constant (or gratuitous, Apple take note!) change. They aren't. But can changes be made so small that they barely register, particularly if much of the change is behind he scenes? If you can do continuous deployment with properly small changes and A/B testing on real traffic then you may be testing changes that prototyping can barely measure. So, can prototypes and big-change management then be pushed back for all but the biggest step changes in features and over-arching concepts?

I've spent a while on engineering lately. Both these books remind me to go back to the user experience and explore a product that tests whether people will want, maybe even love, the idea I've got. I think the key unknowns for me are in users' heads, not in the user interface. My UI is a clunky prototype for now, but even with some polish it won't be the big idea. For me, Ries' measurements of test deployments to learn about users and the value in the business are an appealing approach. As business and product become less minimal and better understood then I think Cagan's ideas find a better fit - but I'd still want to get out there and test for value in the wild sooner rather than later.

This now leaves me with the problem of picking the right metrics to do innovation accounting with. What are my precursors to value and growth and how to I measure them?...