Automated Testing

Automating the site testing flow for merchants and agency developers

Role

Sole UX/UI Designer

Duration

1 Week Sprint

Tools

Figma

Zoom

Miro

Pen and Paper

Research Methods

User Interviews

Usability Tests

Comparative and Competitive Analysis

Introducing Automated Testing

Automated Testing is a feature that allows users to create and set up various checkout flows they would like to test to ensure that their eCommerce stores are constantly running smoothly.

Site tests need to happen often, and they are generally manual and tedious tasks. Automated testing solves the need performing for manually performing each test - which are many.

Shifting Gears into Automated Testing

During ongoing discovery research on the eComm site build phase, it quickly became apparent that the maintenance phase is where the real tedious activity lay for users - less so that during the build phase. 

During the research sessions, one particular task participants mentioned fell under both, build and maintenance - site testing. I found it to be quite intriguing. I quickly connected with one of our researchers and the dev team to learn more about that process. I discovered that it was a highly tedious, lengthy and repetitive task that affected all our eComm users. 

Delving Deeper - What makes site testing tedious?

I collected all the data from the user research in which users mentioned testing, spoke to devs who have experience testing sites themselves, and proceeded to look at existing processes for when these tests are conducted and why. 

I learned that testing an eCommerce site is essential to make sure that everything is working. There is an abundance of aspects that users need to test, such as cart, checkout, payments, coupons etc. Each test is performed manually by users.

Examples of when users complete tests:

Examples of tests users complete:

• When building a site
• New coupon
• Plugin update
• A sale
• High site traffic 
• Connecting a new payment provider
  

• Place order with Stripe while Logged in
• Checkout with 2 items in cart with a coupon
• Sale items appear in cart with the discount and checkout using PayPal
• Student discount applied while logged in
• Square checkout as guest with coupon applied

A glimpse into a manual site testing flow

Checking out with one item using a coupon - no payment:

Pain Points

• Many different aspects need to tested
• Each test is performed manually 
• There are many scenarios for the same test
• Tests need to be completed quite often

Who are we designing for?

Aya is a merchant who owns a highly successful loose leaf tea business. As a sole owner, she has many tasks that she needs to tend to herself, such as maintaining strong communication with clients and suppliers, fullfulling client orders, managing the wearhouse, etc.

One of the tasks that takes a big chunk of her time is site testing, as most of her orders come in through her website. She has to test her sites manually which takes up a lot of her time and keeps her from completing other tasks that would help grow her business.

Aya needs a way to ensure her site is working and users are able to check out and place orders seamlessly without the major time commitment.

Adam is an agency developer. He is incharge of maintaining 12 eComm stores for his clients. His tasks include ensuring that all the right plugins are installed and up to date, the site speed and performance is up to par, orders are coming through, etc.

One of his most time consuming and tedious tasks is site testing, as he has multiple sites to test, and multiple flows and subtasks within each flow that he needs to test.

He needs a way to make his site testing processes more efficient so his time can be used in more effective ways.

Looking at competitors

To my surprise, there weren’t many competitors out there. There were a few sites and plugins that automated a couple aspects of testing, such as Stripe, Nexcess and Cypress, but none were holistic. This meant that users still needed to complete most tests manually. In addition, these sites and plugins had massive learning curves. Setting up the tests was a tedious task in and of itself. 

Problem Statement

Users need a way to test their sites in an effective way, making it less tedious and time consuming

Working collaboratively

I collected all my findings and presented them to my PM. He quickly saw opportunity in this initiative.

After getting the go ahead from my PM, I began to bring the dev team into the conversation further. The issues I was aiming to solve would require a heavy engineering load, and feasibility is certainly something that comes into play. As such, I wanted to make sure that the devs were included in the conversation and the solutioning process early. 

I led 2 workshops with the developers, our researcher, the PM where we went over the following:

We explored the concepts of:

• Automating the tests
• Allowing users to set up multiple tests
• Creating a space to test more that one aspect per test
• Having the tests be scheduled to run

Sketching and Concept Testing

At this stage, research was warranted to see if automated testing resonates with users and if it provides value. We needed to validate our findings, concepts and solutions and see if we were moving in the right direction. 

This was still a concept, and as such, we proceeded to conduct a Concept test using low-fidelity sketches. 

Research Findings

We were able to learn that this feature resonates quite well for the users.

• 8/8 users - merchants and agency devs alike - expressed high interest in the concept of automated testing
  
• Users mentioned that such a feature could save them up to hundreds of hours
• Users want to run the tests when they choose to, instead of having them be scheduled to run
• Users mentioned the need to customize each test

Automated Testing After Iteration

Automated Testing User Flow

Setting up the test:

Running test:

Automated Tests Home Page

Users are able to see their tests, surface information about each one, such as last run date and result, a link to each test's report details and the ability to run the test at their own discretion

Add test - Cart Set Up

Users are given the freedom to customize the tests they would like to run. After creating a name for the test, and users are given the option to select the item(s) they would like to be added to cart as part of the test. Users add the "item path" to ensure it gets added successfully during the test run. 

Add test - Checkout Set Up

When it comes to checking out, there are two decisions users make. If they would like to check out as a logged in user or as a guest and whether they would like a coupon to be applied or not.

If they chose to be logged in, they would input the username and password they choose to use.

If they choose to apply a coupon, they are presented with a drop down menu that shows all the coupons that are available on their site and they may select the one they would like applied.

Add test - Payments Set Up

Users are given the option to include payments in their test. The payment providers they are able to test are ones that are already connected to their site. If they choose to have it be included, they may simply select the payment provider they wish to conduct the test using. 

Next Steps

I took all the findings from the concept test, and began to move up in fidelity. The usability tests of the final Hi-Fi screens are currently underway.