Tips from the trenches of performance engineering

Engineer Mike Kobush shares his lessons learned from over a decade of working in performance engineering.

Mike Kobush and I’s relationship dates back to 2012 when we met during a seminar on performance engineering I was holding on a Sunday afternoon in sunny San Diego. I later found out that Mike was considering skipping out on me on that day to take advantage of the great southern California weather, but I’m glad he decided to stay because our passion for performance engineering turned into a great friendship over the years. So, I was very happy that he agreed on coming on the podcast to talk about what he learned during his time as a performance engineer.

This blog post will be a summary of the key points he mentioned — but if you want to listen to the full podcast to learn more, just click on this link: How to become a performance engineer to please our instant society with Mike Kobush

From a degree in biology with a minor in chemistry to QA tester and finally performance engineer at NAIC, Mike has a lot of experience under his belt. Here are some of his best tips to get started and improve, if you want to follow a similar path.

Don’t be afraid of automating yourself out of your role

One of Mike’s work philosophies (taken from his mentor, Nestor) is to “always try to automate yourself out of your current role into the next”. You could just stay in your role, continue testing everything manually, and not do anything to potentially rock the boat. But you could also decide to challenge yourself, learn how to automate, and try it out.

Not only does this have the added benefit of making you more valuable as an employee — since you’re learning new skills and technologies and it shows you have the drive to make changes — but automation also frees up time that can be invested in other things (like fishing).

The bottom line is also that if you don’t start automating and expanding your knowledge, you could be out of a job overnight anyway. (Just think of the young generation of script kiddies that easily have the skills to automate things you never dared to try).

Learn something new every day

Mike recommends setting yourself the goal of learning something new every day. If you learn something new today, you’re already better than yesterday.

But this does not have to be solely related to your work or automation. Even something like coaching your child’s soccer team could teach you a lesson that can be related to your work. Other areas of your life should not be underestimated. When you learn something new in one area, it could be applied somewhere else too.

Take advantage of observability tools

Performance engineering is all about improving every single piece of software and the infrastructure it’s built upon, to greatly improve its performance.

But how can you know which parts to improve if you don’t have an overview of every single part of the software? Especially with today’s cloud architectures? APM and modern Observability platforms enable performance engineers to get this overview that was impossible before.

Furthermore, automating the analysis of data from the software and infrastructure saves you hours of work. Instead of spending 2–3 hours analyzing the data, then 2–3 hours fixing or improving something, now you can build dashboards and look at every single part and how it connects — all within 30 seconds. And when you make a change, you can run a script and have a new, updated dashboard in 30 seconds.

In the process of moving to the cloud, Mike’s team found out easily that multiple n+1 query issues slowed down performance. Without observability platforms, they never would have learned that their software needed more efficient query processes and response times.

See the big picture: performance engineering has a real-world impact

It might be easy to forget that what performance engineers do is not only make software better and faster. But their work directly impacts millions of users every day, especially since the advent of smartphones.

People call it the “Google effect” — this tendency that we have towards wanting instant gratification — but Mike’s point of view is that smartphones exacerbated this issue. And society is so dependent on smartphones, even more than computers, that if something goes wrong or the phone is not performing, it can even ruin your life.

The performance engineering industry has to keep up with the instant results we all love and have grown accustomed to.

Learn more in the podcast episode

These are just some of the topics we discussed with Mike — there are plenty more in the podcast episode itself, where he shared more about his career and issues he came across now that NAIC is migrating from on-premises to the cloud. Here’s again the link, if you are interested: How to become a performance engineer to please our instant society with Mike Kobush

Thanks again to Mike for coming on the podcast and I look forward to continuing working with you! And especially looking forward to our talk at Perform 2022 in Las Vegas. 😉

Tips from the trenches of performance engineering was originally published in Dynatrace Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.