Case Study – In-product feedback

In-product feedback

Collaborating with and mentoring a junior designer towards outcomes

Table of Contents

1) The Context

2) The Work

3) The Process

4) The Results

Once Upon a Time on a Design Team

Billy Gilmore is a designer on the team I lead

notes: Billy is a member of the design team that I lead. He's got great energy, a good design sense, but not a ton of experience yet. Let me tell you a story about how we worked together to deliver product feedback.

text: Billy Gilmore is a designer on the team I lead

The Business Context

notes: I'd like to start with the business context. Vmware collects product feedback across products to guage UX across the suite. The story begins with internal stakeholders being unhappy with the existing IPF, and our platform team is tasked to collaborate with the Customer Experience team to improve the IPF tool and bring it onto the platform.

notes: You can see why, it's interruptive which is very problematic when viewed in the complexity of VMware products (lot of knobs in VMware)

notes: From the design side, we weren't really happy with it either.

notes: Billy and I had never worked together, so I wanted to let him help define how to proceed.

notes: Before I knew it, he was off and running

notes: A few days later in our 1:1, the truth came out. Billy was upset. He had come up with a great design based on competitive research, but he couldn't get buy in from our key stakeholder

notes: Seeing that a blocker was developing and Billy was getting frustrated, I offered to help out.

notes: Billy had defined a design pattern that resonated with users on a humanistic, emotional level, and was based on academic research on creating the best feedback experiences.

notes: It had fewer options than the existing product feedback, which according to his research would result in a better experience. However, according to the director of customer experience, it presented a data compatibility problem (5 choices vs 7 choices)

notes: So how could we move this forward together?

notes: How could we get buy-in from the director of customer experience, the key stakeholder who has owned this area across VMware for many years? I met with her, hoping to understand her goals and see how we could gain alignment. Her perspective made sense, her goals are to provide a historical perspective of product performance. How would VMware products be able to compare customer satisfaction over time if we changed the scale?

notes: Ultimately, we were able to align around structuring our solution to meet both user and business needs.

notes: After discussing all her concerns in detail, we both agreed that helping the business get more feedback would be better for our internal stakeholders. So, if engagement would increase with a re-design, we would have enough reasoning to switch the scale.

notes: We aligned around setting up an A/B test to compare engagement with a 7-point numeric scale versus a 5-point emotion-based scale

After completing a task, a customer effort score is delivered asking the user about the ease of completing it.

Product teams can then use that information to understand the impact of design changes, new features, or technical improvements.

The interaction design across both versions of the A/B test were similar.

Product teams can define additional questions to surface within the feedback widget.

But can add no more than 3 questions

Users can choose to turn off additional feedback requests or can ask VMware to send the request later.

Another enhancement we brought (informed by existing data) was to hide the free-form entry of information, enabling the user to choose to take this action as necessary.

We made this decision based on existing data suggesting that most customers (over 80%) submitted feedback without comments.

The Work

Solutions to identified pain points

After completing a task, a customer effort score is delivered asking the user about the ease of completing it.

Product teams can then use that information to understand the impact of design changes, new features, or technical improvements.

The interaction design across both versions of the A/B test were similar.

Product teams can define additional questions to surface within the feedback widget.

But can add no more than 3 questions

notes: (informed by Billy's earlier research into optimal feedback experiences)

Users can choose to turn off additional feedback requests or can ask VMware to send the request later.

notes: (This pattern fits into the broader notifications system design that I will share momentarily)

Another enhancement we brought (informed by existing data) was to hide the free-form entry of information, enabling the user to choose to take this action as necessary.

We made this decision based on existing data suggesting that most customers (over 80%) submitted feedback without comments.

Define

Design

Discover

Deliver

Specifically, I compiled …

Internal Research
Competitive Research
Market Research

I then…

Explored Design Solutions
Whiteboarded with Colleagues
Created Hi-Fidelity Prototypes

Specifically, I …

Collaborated with the Research Team
Conducted 1:1 Usability Testing
Synthesized the Findings for Stakeholders

When finalizing the project I …

Collaborated with Developers to Implement Interactions
Documented Guidelines for Other Teams
Monitored Performance Metrics on Release

The Process

The process I follow, and how I used it to deliver this project

notes: https://nickbewley.com/case-studies/notifications/

The Results

What came out of the work

notes: I allowed Billy to help me understand how best to add value to his process

notes: And hopped in as necessary to help drive broader alignment across the company.

notes: Although I did collaborate to help out with much of the hands-on work, I let Billy take the spotlight. For me, as a design leader, the success of the team is most important, and I try to understand how best to plug-in to their process to add value.

notes: With my guidance, Billy developed a set of principles based on competitive and academic research.

notes: He then distilled those principles into recommendations to inform how teams could leverage the solution.

notes: Then drove alignment across the different product teams, using the competitive, academic, and user research to illustrate the benefits of the solution.

notes: Based on feedback from users and internal stakeholders, we defined future-facing, visionary design solutions

notes: Like the ability for users to follow up with a VMware team member to provide more details on their feedback.

notes: I helped to drive alignment and insure the solution's compatibility with the overall notifications system which I had defined prior.

notes: The initial results indicate that we get more engagement with the emotion-based scale, and we are gathering more data towards statistical significance.

Thank you!

I appreciate you taking the time to review my work