Optimization to Innovation: Building a powerhouse experimentation program

Client Direct Energy
Industry Energy Services

Direct Energy is one of the largest providers of power and gas in the United States. They are a leading energy and energy-related service provider with almost 6 million commercial and residential customers across North America. As a company, Direct Energy has to contend with fierce competition from rival providers in their various geographic locations.

Contents

Overview

When Direct Energy came to hawkinsCoding, they already understood the value of iterative testing to improve competitive differentiation from their visitors. With an inhouse experimentation team that had delivered positive results to date, Direct Energy decided it was time to take their experimentation program to the next level. The three key areas where Direct Energy wanted to improve their optimization program were:

Resource Constraints: Direct Energy wanted to do more experiments but were limited by the availability of internal staff.

Repeatable Framework: Direct Energy wanted more structure to deliver on and show results.

CRO Education: Knowledge transfer from industry experts like hawkinsCoding to help elevate their experimentation program.

Throughout this study, we will highlight some of the biggest “A-Ha” moments in our partnership with Direct Energy by taking you through a journey on what a maturing experimentation program goes through over time, and the evolving methodology behind its growth. Our mission, since the outset, has been to understand the unique visitor behavior by leveraging our proven methodology of statistically valid testing and CRO program optimization.

Evolution of the plans page through classic CRO

The beginnings of our experimentation partnership with Direct Energy largely focused on the plans page.

The plans page was a place where users could shop and select an electricity plan for purchase, comparing and contrasting their choices. Similarly, to shopping for a wireless plan, shoppers could visit the plans page and pick a plan that was suitable to their individual needs.

Experimentation around this area of the website was the large part of our focus. Testing was primarily focused on optimization, with experiments that iterated off each other and built on their predecessors’ insights.

One of the major benefits of conversion rate optimization is that—done properly—it is a much better alternative to a traditional website redesign. Rather than investing months in a redesign and hoping for a positive outcome, continuous optimization enables you to iteratively redesign a page, funnel, or website, generating uplift and / or learning with every test. That learning piece is one of the most important, and overlooked, parts of CRO.
In one of the first experiments we ran, we wanted to gather information on two key questions:

  1. Do customers like more, or less choice? We decided to test this by limiting their available selections from 4 to 3 available plans.
  2. Will ordering the plans by price make it easier for users to better comprehend their selections and make a choice?
Variation A insights

This first variant on the plans page uncovered one thing about energy shoppers: having fewer choices provided a worse customer experience. In experimentation, insights are often gained when an experimental variation has a negative impact on performance. This can reaffirm insights and inform strategy before any major changes to the shopping experience are undertaken. By uncovering that Direct Energy customers may prefer choice, future experiments and site designs can be built understanding this concept to avoid making any future changes that can have a negative impact on revenue.

UI clarity and simplicity matter – all plans clarity redesign

When designing a user experience, it is incredibly important to keep in mind that users will be accessing your site on mobile, laptops and tablets as well. A UI that avoids clutter, is simplistic and informative to the shopper can strengthen your user experience and help boost conversion.


For this experiment, the control UI appeared like the below. The CTAs were placed below the KWH rate, rather than in line with the plan row creating a lot of wasted empty space. This “wasted space” created a very cluttered and condensed plan description section to the right. Conversion hypothesized that this potentially unclear layout was confusing to the visitors and could be having a negative effect on conversion rates.

Our hypothesis was proven correct; a “Clarity Redesign” of the plans page “all plans” view saw a substantial increase in conversions by creating a more simplistic experience that reduced clutter by changing the placement of the CTA’s on the page. The overall UI improvements highlighted the power behind creating an experience that focused on a higher processing fluency with a more simplistic and 
user-friendly UI.

Removing the “popular plans” – expanding choice

Back in the first example, we uncovered that this organization’s shoppers MAY prefer choice. hawkinsCoding wanted to validate this insight by running a test that eliminated the popular plans tab entirely, showing shoppers all their options at once. The insights from the previous losing experiment helped inform a new strategy for the plan selection, and lead to a new hypothesis.


When you stumble upon a principle that has high elasticity with your audience, you may have unlocked a core customer insight. This type of insight can have a massive impact throughout the entirety of your customer journey, and you’ll likely want to validate it in multiple places.


We developed two experimentation hypotheses:



Hypothesis A: Our first variation had a collection of simple and basic improvements to the UI of the popular plans tab (plans layout themselves) building upon the “clarity” and “processing fluency” insights from the first test. We made the featured plan and promotion clearer and more prominent, and we made the existence of the tabbed structure of the popular and all plans views more apparent as well.

The site redesign

Why redesign? Understanding Direct Energy’s decision

After 3 years of optimization and continued improvement and lift through experimentation, Direct Energy had reached a point in 2019 where they had decided that their brand was due for a bit of a refresh. Brand refreshes are sometimes needed to take large leaps forward in customer acquisition and overall brand competitiveness in the marketplace. Part of this refresh meant that a website redesign was on the horizon. Our partnership and years of incrementalism and learning through experimentation leading up to this point had armed the team responsible for the redesign with a substantial amount of insight about the user and their preferences of the experience to make really informed design decisions and start from a place of “data-drive redesign”. An effective CRO program, like the one developed at Direct Energy, can help inform the next large, bold, iteration of experimentation: the website redesign.

Our work to date gave the redesign team a better understanding of:

  • The power of choice
  • Site Filters – when to include and exclude them
  • Copy and Terminology Usage
  • Balancing processing fluency with new ideas

The customer insights we were able to discover together through the inception of their CRO program helped inform this redesign with learnings from all the experiments conducted to this point. The redesign is, in itself, a large experiment. Direct Energy acknowledged the risk of an experiment of this size and have committed to measuring its impact on the business.


With the support of hawkinsCoding, the team ensured the redesigned elements made use of key experiment findings and tweaked and tested them as they developed the redesigned pages. And through this thoughtful approach, were able to launch a redesigned digital experience with no negative impact to their key customer journey metrics – a difficult feat in the world of redesigns.

Mature experimentation: The user experience at Direct Energy

Direct Energy now wanted to expand on more innovative experiments to build new features and functionality to better understand how people really shopped for energy plans – the next phase of the experimentation program helped them answer questions like “Do people think about an energy plan in monthly bill estimates or Kwh? What impact would a bill estimator have on the user experience?” In order to answer these types of questions, we built innovative feature experiments like bill estimators.

A culture of experimentation and decision making at Direct Energy

Experimentation at Direct Energy has grown from simply having an outside partner agency run tests to improve the conversion rates of their digital properties, to expanding the understanding of experimentation as a methodology of making more informed decisions across the organization.

Conclusion

Direct Energy’s journey from the inception of a CRO program towards an evolutionary site redesign is one that is filled with critical customer insights that will inform user experience and shopping strategy for years to come.


Our partnership with Direct Energy was able to grow beyond the initial three objectives of their organization. We were able to help them develop a program that aided in testing prioritization, scaled their experimentation practice with our team of CRO experts, and transferred many of our best practices to the team at Direct Energy through knowledge transfer. As we look ahead towards new and innovative experiments that utilize mixed methodologies, hawkinsCoding and Direct Energy will build on a cutting-edge program that will help develop initiatives for years to come.

Direct Energy’s journey from the inception of a CRO program towards an evolutionary site redesign is one that is filled with critical customer insights that will inform user experience and shopping strategy for years to come.


Our partnership with Direct Energy was able to grow beyond the initial three objectives of their organization. We were able to help them develop a program that aided in testing prioritization, scaled their experimentation practice with our team of CRO experts, and transferred many of our best practices to the team at Direct Energy through knowledge transfer. As we look ahead towards new and innovative experiments that utilize mixed methodologies, Conversion and Direct Energy will build on a cutting-edge program that will help develop initiatives for years to come.

Direct Energy’s journey from the inception of a CRO program towards an evolutionary site redesign is one that is filled with critical customer insights that will inform user experience and shopping strategy for years to come.


Our partnership with Direct Energy was able to grow beyond the initial three objectives of their organization. We were able to help them develop a program that aided in testing prioritization, scaled their experimentation practice with our team of CRO experts, and transferred many of our best practices to the team at Direct Energy through knowledge transfer. As we look ahead towards new and innovative experiments that utilize mixed methodologies, Conversion and Direct Energy will build on a cutting-edge program that will help develop initiatives for years to come.

Visit site