Site logo

Not Playing – Spotify

Adapting Code and Culture: How New Relic Went Open Source

Published on
12 mins read
Written by
hero

This post was originally published on New Relic's blog. I wrote this as companion piece to a presentation I am doing at All Things Open 2020.


On April 30 of 2020, New Relic’s Chief Product Officer, Bill Staples, announced internally that we were about to make a major change. Even though we had invested hundreds of thousands of engineering hours towards developing the most comprehensive instrumentation available through our agents, integrations, and SDKs, we were now going to open source it. All of it. And we needed to do it in only three months.

The future of New Relic is open instrumentation

Our Instrumentation team’s mission is simple: “Enable every engineer in the world to gather complete telemetry about their digital systems.” It’s straightforward in language and grand in scope. We share this same grand goal with OpenTelemetry, a key project of the Cloud Native Computing Foundation (CNCF). In fact, we’ve been a key contributor to the OpenTelemetry project for over a year, but seeing the growing support for it, especially from some of the top cloud providers, such as Microsoft and Google (#2 and #3 contributors respectively), helped us realize the benefit of moving away from vendor-specific instrumentation towards an industry-backed open standard. Supporting a consistent method of instrumentation will only lead to more instrumentation and more data and bring the team closer to accomplishing its mission.

OpenTelemetry is the future of instrumentation and the future of New Relic.

We’re still developing and supporting our current agents, but, as these CNCF standards mature and their instrumentation becomes generally available, more of our instrumentation for applications, services, and infrastructure will implement the OpenTelemetry and Prometheus standards and be built downstream from their projects.

It bears repeating: All of our future instrumentation will be open source and based on these open standards when they are mature.

Lighting an open source fire

Alt Text

The Instrumentation team’s leadership, Ramon Guiu, VP of Product Management, and Greg Unrein, Group Vice President of Engineering, decided this wouldn’t be a gradual change for New Relic—no, we were going to move fast with determination. Greg and Ramon lit a fire under the engineering organization with a three-month deadline. By July 30th, when we would launch our completely reimagined New Relic One, all of our most popular proprietary agents, SDKs, and integrations would live in the open source domain, with more following as they become open source ready.

Obviously, we didn’t complete this transformation overnight. A lot went into open sourcing our code. We had to do an enormous amount of code preparation. We had to make sure we weren’t using any third-party, non-open source libraries. We had to prep our code for public consumption, as code commenting and documentation standards had evolved over time. We had to redefine the way we comment and communicate via code. Our Instrumentation team took all of this on with an aggressive deadline. It wasn’t an impossible request, but it was a difficult one.

We also understand that making the leap from a closed, proprietary culture to an open source culture can make or break a business. It can be the difference between staying innovative or becoming irrelevant. We knew we’d have to change a lot to adopt an open source culture. We’d have to rethink the way we handle issues and feature requests. We’d have to re-establish the way we deal with customer escalations. Security within and around our products would have had to change, as would the way our engineers track their work and write their code. We’d even have to change the way we hire engineers.

We knew headed into this, that pretty much everything about the way we built, maintained, and supported our instrumentation had to change.

The Open by Default project

To ensure our success through this transformation, we spun up a team led by three Relics:

  • Sharr Creeden, Director of Software Engineering on our Instrumentation team. Sharr is responsible for the engineering team that builds our instrumentation.
  • Melissa Klein, Open Source Program Manager from our Open Source Program Office. Before joining New Relic, Melissa was the Open Source Program Manager at IBM for 15 years. Her experience is crucial.
  • Ben Evans, Principal Engineer and JVM Technologies Architect. Ben has been involved in the production of new Java standards for many years, and he’s been working with open source projects, frequently in leadership positions, for over 20 years. He’s the author of the O’Reilly book, Java In A Nutshell.

They named this project, “Open by Default.”

The Open by Default project had four guiding principles to shift our Instrumentation team’s focus from building proprietary software to creating open source software:

  1. Open by default: Any new software the Instrumentation team builds will be open source by default. The amount of work the team does in non-open, source repos will, over time, decrease to a minimum. This will require a major shift in thinking that goes beyond code to include transitioning into workflows where project governance and design decisions will take place in the open.
  2. Go where the engineers are: If a community already exists, New Relic should contribute to that community, not pull that community to our projects. Contributing code is not the hard part of open source software—community building and maintenance take time, patience, and commitment. Starting a successful new community is difficult, and it can take years to see returns on your investment. We won’t waste our time and resources on building communities that already exist.
  3. Do the work: While we expect that New Relic will contribute in a leadership capacity to many important open source and open standards projects, we recognize that in many cases our contribution will come from showing up and doing the work—and being seen doing it—rather than angling for leadership roles before contributing.
  4. Our contributions are visible: Our web properties should be top hits for engineers searching for docs and other materials relating to the open source projects we contribute to. We must invest resources, inside and outside of Engineering, to ensure that our audience comes to us for information about the open source projects we contribute to and does not read about our work, unacknowledged, on competitors’ websites. Customers and prospects should know that they can find open source at New Relic.

The Open by Default transformation guidebook

In addition to these guiding principles, the Open by Default team published a guidebook, a library of policy and practice proposals collaboratively built across the Instrumentation team. The guidebook defined the future state of how they would work in the open and align everyone on the team around those policies and practices.

The primary function of the guidebook is to communicate information efficiently and provide mechanisms for collaboration and feedback. The policies in the guidebook include enhancements to:

  • Issue lifecycle management: Historically, we defined the issue lifecycle process on a per-team basis, so different agents had slightly different processes. Open by Default standardized that process across teams. One notable change is teams moved from private instances of Jira to GitHub Issues, which made issue management more open source friendly.
  • Feature request process: Our existing feature request process had three entry points: 1) our product roadmap software, 2) account teams, and 3) support channels. The Open by Default team decided that the most open source-friendly way to handle our feature request process was to stay consistent with how we handle issues and use GitHub Issues. But instead of forcing all feature requests to come through GitHub Issues, the Open by Default team decided to make GitHub Issues the home for feature requests and the place where they would be managed, but to also sync issues with our roadmap software, so that customers and account teams familiar with it wouldn’t be disrupted.
  • Public roadmaps: Product roadmaps resided in the same product roadmap software as our feature request process, but, since we were moving a lot of processes to GitHub Issues, the Open by Default team wanted to stay consistent and move roadmaps to GitHub Projects. Doing this amidst all of the other work our engineering teams had to do and on the timeline, they had to work with wasn’t feasible. So they split the transition into two phases: 1) Including a ROADMAP.md file based off of a template at the top-level of each repository as an MVP, 2) move roadmaps to GitHub Projects in the future.
  • Security process: We have a robust security reporting process that we intend to keep, but, with the move to GitHub Issues, we saw an opportunity for security issues to come in through a publicly visible channel. The Open by Default team decided that security issues will not be deleted and will remain publicly visible in GitHub Issues and will be closed (but still searchable and visible) after they are resolved. For highly sensitive, embargo-level security issues that need to be removed from public view, there is a non-default workflow involving tracking in private JIRA.
  • Customer escalation process: We updated our existing customer escalation process to adapt to the new channels and audience we get with our move to GitHub Issues. We created templates for issues and feature requests to enable our Global Technical Support team, as well as self-directed users who submit their own tickets. Templates for common replies and routing were developed for engineers who triage issues.
  • Engineering work tracking: Since we moved our issue and feature request processes to GitHub Issues, and because OpenTelemetry uses GitHub Issues as well, it only made sense to transition work tracking to GitHub Issues too.
  • Engineering management practices: We updated the engineering career ladder and performance analysis process to better align with our engineers’ new work process. Planning and timeline expectations were also updated to better align with the open source workflow.
  • Single-track development: The Open by Default team decided that all of our open source products would engage in single-track development by default—no maintaining multiple source code repositories for the same project—except in cases involving embargo-level security issues or work that contains sensitive, confidential, or personally identifiable information.
  • Hiring profile: Before the Open by Default project, we hadn’t emphasized open source development, so we needed to update our hiring profile to be better suited for attracting and hiring open source engineers. The Open by Default team revised the attributes and traits we’d look for in a new hire and revised the code challenge to allow candidates to demonstrate an ability to work with open source processes.
  • Moderation guidelines for the Explorers Hub: The Explorers Hub is our forum where customers, especially developers, can learn “how do I” build on New Relic. Users can post questions and answers, give feedback on our open source projects, and contribute to other solutions. For our open source project categories on the Explorers Hub, a process was defined that detailed expectations for moderator response to new topics, communication norms for any questions that required engineering support, and the response cadence for engineering teams.

Were we successful?

To say the initial push to open source our instrumentation was successful would be an understatement. Our engineers not only delivered all of the planned projects by the July 30 deadline, but they delivered more than planned earlier than planned. All of our most popular instrumentation along with several other projects were ready for open source well in advance of July 30, so we pushed up the announcement to July 22.

Check it out: Don’t miss InfoQ’s Q&A with Sharr, Ben, and Greg about open sourcing our instrumentation.

To measure the success of the Open by Default project beyond the initial launch and announcement, the team decided to look at metrics around issue resolution, which is indicative of our engineers’ adoption of the new processes, as well as community engagement. To satisfy this, at least initially, they incorporated metrics from the GitHub’s Open Source Guide on Open Source Metrics. The plan going forward is to baseline and work toward improving every area of metrics covered in the guide: discovery, usage, retention, and maintainer activity.

By these measures, was our Open by Default project successful? We don’t know yet. We’re still setting the baselines. Having these metrics defined beforehand, however, helps us identify when something isn’t working with our policies and processes, so we can take steps to remediate the problem, whether that’s by revising our policies and processes, providing additional training to our engineers, or taking problems to project leaders to address with their teams.

If New Relic can do it, so can you

The world runs on software, and we truly believe that to build a more perfect internet and a better world, we need to build better software—together. That starts with each of us open sourcing what we can. If it isn’t a competitive advantage, you should open source it. It’s not simple. It can be difficult, but if we can do it, you can, too. Learn from our story, and do it better than we did. Then share your story, so it’s easier for the next company looking to make this bold move. In fact, this post was inspired by Adobe’s story about adopting an open source culture. If we all commit to adopting such practices and sharing our learnings along the way, it’s not an exaggeration to say that releasing open source software can soon be a standard process for all of us, all of the time.

Visit New Relic Open Source to learn more about our open source projects and contributions and start contributing today.