Guiding Principles for Building a Performance Engineering-Driven Delivery Model

While recently attending a Dynatrace User Group in Hartford, I had the opportunity to sit in on a great presentation from a leading US insurance company as they explained their 3 year APM journey. I see a lot of these success stories, but this one was especially impressive. To see how they have refined their internal processes, successes and performance best practices to ensure delivery of high quality, high performing and highly scalable applications over these years.

The performance engineering group within the large US insurance company were the ones that started adopting application performance management (APM) in the company. Overtime they built an APM maturity model that helped them achieve successful adoption of APM across all application owners. Below is the diagrammatic view of their model:

As the diagram shows, this is an iterative model that they built from their successes and failures. For example, once they showed their service value to application owners, they started building delivery processes for their services. As their delivery processes became streamlined, they started driving awareness of their services to larger groups within the company. As the demand for their services increased, they focused on hiring strong technical personnel within their team who could not only deliver the services efficiently but also brought technical thought leadership to application owners.

The top 5 takeaways from their presentation were:

• Demonstrate Value: As a performance engineering group, it is very important to demonstrate value of the processes and the tools that they use to a bigger audience including architects, developers and IT operations. For instance, this group used Dynatrace tools to find scalability issues of an application during the load testing cycle. Equipped with this information, they engaged the architects and the developers of the application and brainstormed with them about the architectural choices that were made in the application. This joint working engagement not only allowed them to show value of the performance engineering team but also enabled them to engage with the architecture team early on in the development cycle to eliminate any scalability issues and hence significantly reducing the cost.
• Highlight “Wins”: The charter of the performance engineering team is to work across all application teams to inculcate performance management as part of the development process. This company is very much aware that the cost of fixing an issue increases multi-fold if it is not addressed early on in the development cycle. When they started the APM initiative, it was not necessarily easy. In order to show value of their services, they used past “wins” of fixing performance issues like scalability and slow response time in one application to showcase the importance of their services to other application owners, developers and architects. The end result is that they are at a stage where the demand for their services is outpacing the supply that in turn is a great place to be.
• APM is a journey: When asked about the time taken for them to reach their performance management goals, I was told that APM is a journey and does not necessarily have a finite end. They are continuously working to not only onboard more applications but also continuously improving their methodology to make it better. As shown in their model above, as the group grew and gained maturity in APM, they used their experience to take the operating model to the next level.
• Active record keeping: After successfully demonstrating value over and over again across multiple application groups, the performance engineering group has taken initiative to ensure that application performance is part of the application lifecycle. The team is now keeping an active record of performance defects that are found in design, testing and production phase of every application. The intention of this record keeping is twofold: a) To track within the team to ensure that the trend of performance issues identification lean more towards design phase and less in production b) To evangelize their team’s process and Dynatrace tools to other application and business owners.
• Influence business requirements: It is not uncommon for business owners to keep adding more functionality into an application especially if the application is a web application. Typically business owners keep adding more components into a web page without knowing the impact of these additions on performance. The team used their performance data to:
  • Set realistic expectation with the business owners about page response time
  • Help design page layouts to improve performance without compromising the functionality
  • Help define expected users & transaction volumes
  • Calculate expected page/transaction timings

During this user group, I got an opportunity to talk to other customers that are either embarking on their APM journey or deeply involved in it. One of the common messages that came from across the board was that it is absolutely important to promote and educate about APM capabilities as the enterprise matures in integrating APM in the software development lifecycle.

Conclusion

The APM journey does not have a finite end. On the contrary, it is an iterative process that continuously requires improvement and innovation. Using the past “wins” is one of the ways to gain support from application owners and spread the APM culture across the enterprise. If you have a similar experience in your journey, I would love to hear from you. If you had a different experience I would love to hear that too.