Humana: Digitally Transforming Health and Well-being with Pivotal Cloud Foundry and Dynatrace

Digital disruption is making established players in every industry rethink and change their business models. Digital has quickly become the primary channel of customer interaction for even the most conservative industries, requiring them to push past the boundaries of their traditional offerings in order to satisfy the needs of today’s customers. The healthcare industry is no different. Even as a company that was heralded as a technology leader in their space, Humana recognized the need to evolve and is taking the cultural and technological steps required to stay on top.  Presenting at Perform 2018, Bob Johnson, Director of IT for Humana, described what digital transformation looks like at Humana and the role partners like Pivotal and Dynatrace are playing in their journey.

Digital Transformation at Humana

Humana believes technology can play a critical role in health and wellness.  To underscore their belief, Humana invested in making technological innovation a core competence of the company.  To do this, they looked to Silicon Valley and more specifically Pivotal Labs.  Humana developers paired with Pivotal developers to help transform their development culture and implement agile software engineering methods. A key output of this process was the creation of their Digital Experience Center (DEC), which serves as a technology incubator for the company and is responsible for the development of Humana’s digital products.

To learn more about the history and mission of the Humana DEC, you can go here.

As part of their move to agile development, the Humana DEC chose Pivotal Cloud Foundry (PCF) because it would help them, “Accelerate their move to DevOps” by providing a platform designed to align with agile development practices. PCF abstracts away the complexities of the infrastructure and makes it easier for software engineers focus on developing new application and improvements to existing applications. One command (“cf push”) and new code is deployed into production.

Humana’s Monitoring Strategy

Humana has been a long-time customer of Dynatrace and has followed the maturation of our product line. As APM veterans, Humana has a defined monitoring strategy that consists of four parts:

1. Invest in APM, improve Operations with user experience monitoring (UEM), application monitoring, data center real user monitoring (DCRUM)

Continuous training and investment in monitoring to improve the end-user experience and performance.

2. Train development on monitoring capabilities with APM specialist certification program

Train development on Dynatrace so they have the same benefits and are using the same solutions as the Operations.

3. Establish a physical space for monitoring collaboration with Operations and Development (aka the “Death Star”)

The “Death Star” is Humana’s IT Command Center dedicated to application performance and end-user experience management. It is considered a critical component of the DEC and the emphasis on teamwork.  According to Bob, “It is core to what our success has been over the past few years in terms of driving down impacted user minutes and improving availability.”  To emphasize its importance, it is located adjacent to executive leadership’s offices and is exercised in situations when major incidents occur.

4. Continually showcase new capability and visibility to leadership

Humana is always looking to improve.  When Dynatrace launched with full automation, AI, and full stack monitoring, Humana embraced these changes and took it in to explore what was possible.

 “We have a very powerful engineering platform in Pivotal Cloud Foundry and a very powerful monitoring platform in Dynatrace, and pairing them together has been tremendously beneficial to Humana.” – Bob Johnson, Director of IT

Dynatrace as a First-Class Citizen via BOSH

To be fair, Bob never used the term “first-class citizen.”  This term came to mind as I listened to how he talked about the importance of monitoring at Humana and where monitoring seemed to fall in the chain of priorities.  Then, on one slide he had the famous haiku written by Onsi Fakhouri, Senior VP of Cloud Research and Development for Pivotal, encapsulating the goal of PCF:

Here is my source code

Run it in the cloud

I do not care how

– @onsijoe May 12, 2015

And on the following slide, Bob had his own haiku:

Oh and by the way

Monitor it for me too

Ops stuff is so cool

This is when it struck me that monitoring, and specifically Dynatrace, is considered a first-class citizen at Humana.  Dynatrace makes it easy and automatic to monitor PCF (and everything else).  The Dynatrace BOSH add-on for PCF builds monitoring into the PCF platform.  It lights up the entire platform and every application running on it. To visualize it, he used this slide:

While visuals are great, real-world example are the best. Here is one that Bob shared:

Nancy’s Story

Humana uses Nancy, a lightweight framework for building HTTP-based services on .NET.  A member-facing application was experiencing performance issues and was crashing during Humana’s sales/enrollment period.  Dynatrace was showing 500 Internal Service Errors with a specific microservice, and the errors specifically pointed to the Nancy framework.

The operations team was new to using Dynatrace and were more familiar with the Dynatrace AppMon solution.  From their experience, when they received a 500 Error, based on being a .NET shop and SOA architecture, it should be a dependent web service.  Because of this, they originally discounted the fact that Dynatrace was pointing to Nancy as the problem.

They decided the best approach (and the one they were most used to using) was to troubleshoot it with their existing playbook.  They proceeded to comb through logs on a service-by-service basis.  This took time and they were unable to find the problem, leading to further delay in the restoration of normal service.  Finally, they decided that they should look into the Nancy issue Dynatrace identified.  When they focused on that, they were able to identify the problem as not having the latest version of one of the PCF buildpacks.  As soon as they updated the buildpack, the application returned to normal.

Bob tells this story because he thinks it is a good example of the struggle engineers may have with highly automated technology solutions.  According to Bob, “It’s easy, and as engineers, you don’t always like easy.”

Looking Ahead

Bob wrapped up the presentation talking about the future of monitoring with Dynatrace at Humana. Here are a few highlights:

• The ubiquity of monitoring – He talked about the concept of the “110 plug” and the idea that monitoring should be ubiquitous. You don’t think about electricity in your office or home, it’s easy to use and it is everywhere. They want the same for monitoring.
• Tool rationalization/consolidation– As Dynatrace has evolved to be more than just APM, but a digital performance monitoring and analytics platform, Humana is looking at where it makes sense to eliminate isolated, siloed tools used in other parts of the stack in favor of Dynatrace.
• Shifting Left – Integrating DevOps continuous delivery tools with Dynatrace to help them identify issues quickly regardless of the platform they are developing on.

For more details, here is a video of the full session: