Terminology matters

How to enhance user experience by aligning names with expectations

One of our main goals for the development and optimization of Dynatrace is to create a state-of-the-art user experience for those using our software. This is done on one side by adding functionality to the benefit of our clients, but also by improving already existing workflows. That is why we have launched a renaming project for a particular menu item, which name is not aligned to users’ expectations.

The starting point of our renaming project

We recently invested some time analyzing the Diagnostic Tools section of Dynatrace and conducting customer research on how to improve it. We concluded that some of our analysis views, like the CPU analysis, were tools that many customers needed but remained overlooked as a capability of our platform.

Furthermore, market research also showed that this product capability could give us an edge: our CPU profiling is on 24/7, covers method hotspots, thread and memory allocation information, and allows users to analyze end-to-end traces down to the code level. And all of this is offered out-of-the-box, with no further configuration or payment needed. So, the team agreed that we needed to improve the naming of this tool to better showcase it on the platform.

Finding the right terminology is never easy. The key challenge is to speak the language of the target audience, but at the same time include the positioning and advantage of a tool in the name. The new naming was determined in a 4-step process.

5 steps to a new name

Step1: Gathering ideas

We started out with a purposeful brainstorming session, which included representatives from various roles and departments. This is how we built the basis for further discussions.

Step 2: Addressing needs

As a next step, we pointed out the key advantages and characteristics of these tool and whom we are addressing with them:

• Low overhead
• End-to-end tracking
• Down to code level
• Real-time analysis (always on)
• Target audience developers and operations
• Mapping the conclusions of our brainstorming session to these characteristics helped us choose relevant names and eliminate the rest.

Brainstorming results

Step 3: Real-life check

After making sure the length and descriptions of the suggestions fit into our product and don’t use Dynatrace-specific language, we conducted a focus group to start targeted discussions.

Step 4: The focus group

This group included the target audience, as well as colleagues from the marketing, sales and experience departments. Renaming shouldn’t be decided in a silo, so we collaborated with the focus group on positioning the proposed names.

The outcome of all these efforts resulted in the following changes:

• The main menu point labeled “Diagnostic tools” shall be renamed to “Resource Profiling”.
• The menu point “CPU Analysis” to “Continuous CPU profiling”.
• The “Resource Profiling” landing page will be adapted accordingly.

The renaming process

Step 5: Implementing A/B testing

The next in our process is to test whether the change improves the user experience. We decided to implement A/B testing to see if the new terminology lets users find the tool more easily and fulfills our assumptions and hypothesis.

Before vs. after

Since users need to be able to rely on patterns and consistency to improve their work efficiency, we decided to test the change with a smaller number of customers.

We focused on environments with specific criteria:

• Creation date after July 2019
• Minimum of 80 unique users
• Paying accounts (because in the trial phase, a product is often not used regularly and the behavior of working with it often differs from a purchased product)
• .NET or Java footprint (as diagnostic tools are especially useful for these)
• Currently lower usage of diagnostic tools (< 500 visits) to not disturb frequent users

The environments will be divided into group A and group B with an equal number of active users in both. Group A will work with the current terminology and structure, whereas group B will encounter different names as well as a restructured overview page of analysis views, with the option to share their feedback and ideas for enhancements.

Monitoring testing results and what we expect

Once the feature flag is enabled, we will start monitoring the usage of the platform by all participating clients that have the new feature and those without.

We created a dashboard to monitor certain metrics, including:

• Pageviews = how many users are accessing diagnostic tools vs. resource profiling
• Drilldowns = how many users are going the path of “diagnostic tools” -> “CPU analysis” vs. how many are going the route of “resource profiling” -> “continuous CPU profiling”
• # of Clicks = how many users click on “continuous CPU profiling” vs. “CPU analysis”

Our hypothesis is that we will see a higher page view for “resource profiling” than for “diagnostic tools”. If this happens, we will relabel the menu item, but if this doesn’t happen, we will have to conduct further research on whether the name we used is correct or whether renaming in the end doesn’t solve the problem.

We will conduct A/B testing for 2 months and then, based on the data we gather, we will decide whether to roll this out to all environments.

Potential pitfalls and advantages of renaming

Although our goal is to improve the user experience of our platform, renaming items in a user interface could cause issues that lead to unwanted consequences.

Imagine that you use a tool every day and suddenly a button is replaced or named differently and you can’t find it. Although the intention behind the renaming may be sound, confusion or even frustration on the part of the user is inevitable at first (at least until they become familiar with the new name).

We all constantly learn — sometimes on purpose, but most of the time, subconsciously. A complex digital platform may seem to be intuitive, but “intuitive” often means nothing more than relying on common patterns and navigation paradigms that are widely known or easy to learn. When behavior, naming, and “look and feel” are used consistently throughout a whole industry, they become industry standard and, eventually, a user expectation.

This is an advantage we have with regards to our change, because we aim to change the name of our menu item with something that is more widely used in the field.

If it’s challenging for some users to discover the “CPU Analysis” feature due to the current naming, we have a great opportunity to come up with a more meaningful and expressive name. Our hypothesis is, if we use the language of the addressed target audience, users will better adopt the feature and it will ultimately help them in optimizing their resource usage.

This is not to say that you should not change any names at all once you have decided upon them, rather be sure that the changes you make have a well-thought reasoning behind them. Furthermore, make sure to test and track the user’s reactions to the change to further improve upon it.

In short

Our preliminary research has shown that the “CPU analysis” feature of Dynatrace is very useful for users but not utilized to its full potential. Our hypothesis is that the labeling of the item is not addressing the correct target audience, so we are conducting A/B testing with a group of participants to learn whether we should rename the menu item or not.

If you’d like to learn what the results of our research will be, follow our Dynatrace Engineering blog where we will post a follow-up article with our findings as soon as the testing phase is over.

Terminology matters was originally published in Dynatrace Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.