It’s a no-brainer to want to “optimize.” Who wouldn’t? But for what are you optimizing? Are you measuring a single success metric that delineates success from failure? What drives the threshold for success? Why is said threshold where it is? Is your single success metric aligned with critical business outcomes? These are the things you need to have a firm handle on before contemplating “how” you’re planning to improve.
Data should be the primary driver behind all of this.
Single Success Metric
A single success metric may be a combination of several individual parameters. For example, take the NFL passer rating. It takes into account metrics such as passing attempts, completions, yards, touchdowns, and interceptions. Improving one of those metrics at the detriment of another isn’t necessarily a good thing, but maybe it is, depending on the weighting of each.
Consider the idea of creating a new user experience where you’re measuring the conversion rate, revenue of the current transaction, customer lifetime value, and cost of inquiries to your customer service center regarding said experience. Let’s say you’re experimenting with two different design variations against a control group, both affecting the metrics mentioned above a little differently:
- Conversion rate: up 5%↑
- Revenue of current transaction: up 3%↑
- Customer lifetime value estimation: down 2%↓
- Cost of customer service: down 3%↓
- Conversion rate: up 2%↑
- Revenue of current transaction: down 1%↓
- Customer lifetime value estimation: up 7%↑
- Cost of customer service: up 1%↑
Given the above measurements, would you keep the control or extend variation 1 or 2 to everyone? It’s impossible to say without understanding the lifts and drags in terms of current and future revenue. Are you playing the long game or the short game? Do you need cash now (revenue of current transaction) or are you investing in your future (customer lifetime value)? On the surface, it might sound good to reduce the cost of customer service per transaction. Let’s say the cost of each customer service touchpoint is $20, but if you’re losing revenue in place of saving costs in customer service, it’s likely not worth it.
Point being, usually no single metric in isolation will suffice, but a single success metric or “score” that represents a weighted combination of a group metrics helps the product owner make smarter decisions.
Why are you or your organization in business? The answer to that question will help define your business KPI’s. You are not in business to reduce customer service costs, that is not a business objective. If so, you could just shut down your customer service center and call it a success (while revenue tanks). Likewise, you are not in business merely to convert potential customers. If so, you could sell at 50% under cost, and your conversion rate would skyrocket while you quickly go into the red and out of business.
Problem Solution Mapping
Clearhead embraces the notion of thoroughly understanding the problem before giving any thought to hypothesizing about a solution. It seems like one of those “duh” things, but it’s sometimes people’s nature to be quick to offer solutions before completely understanding what problem they’re solving. You never want to have a solution looking for a problem. The following video touches upon the truth that many ideas are destined to waste time and money, and often times companies don’t know if the idea worked, or if they could have come up with a better product or experience for their customers. Why do so many ideas fail, and how can we pick ideas that will succeed?
Ideas fail when they don’t solve problems. The more problems you solve with your design, the more value you create. But ideas that solve problems are hard to find, so instead of starting with ideas, begin by identifying clear business goals. Then, using data and clear evidence, focus on the largest problems blocking these goals (i.e. problems most worth solving).
Clearhead’s Problem Solution Mapping goes as follows:
- Establish business KPIs and goals
- Identify business problems impeding said goals
- Use data to analyze and prioritize problems
- Ideate solution hypotheses
- Develop optimization roadmap
Metrics That Matter
Optimizely put out a good article about aligning your optimization program with goals that are important to your business. The last thing you want to do is spend time and resources building inefficient and unfocused tests that do not make a real difference. One way to construct impactful tests that affect the bottom line is by leveraging a goal tree that maps metrics that are critical to your business’ success.
Watch this video about aligning experiments with business goals.
Begin at the top of the tree. What does your company do to generate revenue? How does each metric break down into a smaller set of metrics? Once you identify the KPIs your program will target directly, you’ll be able to hypothesize approaches and tactics that make an impact. Use strategies and tactics to create meaningful experiments and campaigns.
It all starts with goals and problems, not ideas first.