Songbird: The Journey to Containerize a .NET Framework Monolith

The first time I’d ever heard about containers was in 2018, when a coworker showed us a presentation on what they were and how they worked. My only thoughts were “cool, that’s just overcomplicated virtual machines (VMs). Anyways…” It’s embarrassing how naïve and wrong I was.

I was still new to software development and this topic was outside of what I cared about, namely bug fixes and some smaller product features I was working on at the time. Who knew that 4 years later we’d hear whispers in the wind about containers, and I would be leading the charge on making that happen.

But Joe, why spend all this time and effort containerizing a monolith that is already stable and reliable?

Well dear reader, our reasons may not be the same as yours, but we chose to do this for modernization, scalability, and reduced infrastructure costs. Our application running on .NET Framework and relying on Active Directory (AD) wasn’t keeping up with the needs of customers, who wanted OAuth, cross platform support, and more scalability. Containerized web applications can scale as needed, and the infrastructure costs of adding more containers is a fraction of what it costs to set up more Windows VMs and keeping around an entire AD environment.

We had a challenge in front of us: Our flagship certificate lifecycle automation solution, Keyfactor Command, was a .NET Framework monolith at the time, which heavily relied on AD and Internet Information Services (IIS). It also used a Microsoft Software Installer (MSI) and Windows only GUI for installation/configuration. Taking all of that, wrapping it in a Windows container and calling it a day wasn’t a realistic option.

While Windows containers are a thing, they’ve never been industry standard, and we didn’t want to embed ourselves even more into the Windows ecosystem. Windows Containers would’ve worked as a temporary step along the way, but as a coworker once so wisely put it: “There is nothing more permanent than a temporary solution.” It was time to take cross platform seriously.

We set ourselves a few lofty goals:

1. The Keyfactor Command platform can be bootstrapped and run in a Kubernetes container environment.
2. Related components (the ‘worker bees’), including the Orchestrator and CA Gateways, would also containerized, so the entire product suite can be run in the same environment.
3. Maintain backwards compatibility on all product features.
   1. AD authentication still works for customers currently using it
   2. Windows-based installations continue to function going forward and without disruptions

Great! But how do we get there? As we set out on our journey to containerization, our laundry list of to-dos continued to grow….

First the app needs to be able to run cross platform, so we needed to convert from .NET Framework to .NET 8. Part of that migration would involve no longer relying on IIS and AD for authentication, so add OAuth and OIDC support to the list. We also had to implement the .NET runtime conversion and OAuth on all product components, not just the platform itself. Did I mention you need to replace everything under the hood that relies on Windows with something else entirely? Bottom line, this is as close to a rewrite as you can get without just doing a rewrite. Nevertheless, we gave the effort the codename “Songbird” and got to work.

At the time we started Songbird, we were already well on our way with another effort that we were calling “AD Independence” internally. This covered breaking apart our tight coupling with AD and eventually adding OAuth support to the product as the end goal. Great, so that’s in progress, and by happenstance the .NET conversion ended up being a requirement to complete AD Independence. So, first things first, time to tackle the .NET conversion.

For those not as familiar with .NET, converting from Framework to .NET is a rollercoaster ride of trivialities and complete rewrites. While 95% of your code base will survive the transition 1 for 1, the app startup code in particular needs a complete rewrite. Then you must do that for each application. The Command platform is a combination of 7 applications that all work together: Five being ASP.NET websites, one a Windows Service, and another the console configuration application.

Converting the ASP.NET websites was certainly the most challenging part of this, as the entire ASP.NET pipeline from receiving a request to the point where our controller layer receives is entirely different. Not only did the app start code need redoing, but entire chunks of basic web applications, such as authorization, request logging, and error handling, needed to be redone to better fit the new ASP.NET mold.

Another wrench to throw in the .NET conversion mix here is our Configuration Wizard. Today this is a WPF-based application, which isn’t cross platform even with a .NET runtime. We would need to convert to another UI framework, like MAUI or Avalonia, but what would we get out of the deal? Being able to install Command onto a Linux box and using something like NGINX as an IIS replacement was not on the list of end goals, so we ended up scrapping this idea.

Instead, we moved as much of our core configuration into a common multi-targeted library and stood up a new console app that mimics the configuration capabilities of a wizard. This new app, the Database upgrade tool, would be the bootstrapper for the database in the container deployment case. You can’t use a GUI from a container anyways :).

Now that everything is converted over to .NET, onto the authentication (AuthN) and authorization (AuthZ) problem. Previously, we relied on IIS to perform the authentication via Basic or Kerberos auth, and then ASP.NET provided a user identity to use. This topic deserves its own post to dig deeper into, but in essence we had to use the ASP.NET middleware to allow both IIS AD-based auth to continue to function and add in OAuth auth code and client credential flows otherwise. Additionally, instead of mapping application roles and permissions to AD users and groups, the security model needed to be reworked to be claims based.

It took us a long time, but we’re finally here and we’re ready to start putting the pieces together. To not overcomplicate our builds, we chose to make our dockerfiles as slim as possible, and copy already-built binaries into the images, instead of building the entirety of the source code again in the container images.

Once we had the images, they needed to be run somewhere. Early in development, this was local instances of Docker Compose with a ton of custom configuration, along with a reverse proxy. We quickly realized that this wasn’t going to scale or be very supportable in production as there were simply too many configuration settings and containers that needed to line up for everything to work as expected. We’d also previously set a goal that the application would be run in Kubernetes (K8s).

We decided to take another approach and go up a layer in the container ecosystem to orchestrate with Kubernetes (K8s) and Helm. Using Helm, we could define a Chart with one common set of configuration values that does all the work of hooking everything up correctly for you. We briefly considered writing a K8s operator, which effectively does the same thing, but decided against it as this required more code that needed to be maintained, whereas Helm is flat configuration templates that closely mirror what gets deployed to K8s.

We spent a lot of time making this Chart as easy to use as possible. Since one of our requirements was maintaining backward compatibility, Keyfactor Command still needed to be installable via our MSI on Windows, and we wanted to keep the learning curve low for using this alternative installation method. Even though the Helm Chart will likely get consumed in production by automated pipelines, it was important to us that internal users that weren’t very container savvy could be pointed at a cluster, given a values file that they need to fill out the important bits on, and then run a `helm install`, and 2 minutes later they can log into their Command instance and start using it. At the same time, while the default case on the chart is as simple as possible, we’ve provided enough configuration to allow the application to be customized to any need.

Our final challenge concerned a limitation that our database must be created and configured before the applications themselves can start successfully. In the old Windows environments database configuration is performed as part of the installation process, along with the associated web services and IIS configuration. K8s is very declarative, you tell it to spin up some applications and it does so. Do you see the problem? If we spin up the webservices and a job to configure the database all at the same time, something is bound to fail. Since we’re trying to make this as easy to use as possible, our only option was to bake this into the Chart which is more difficult than it may appear.

Helm, like K8s, also tries to function as declaratively as possible. But we do have a few options to achieve this startup flow:

1. Deploy everything together, use Init containers to poll the database and wait for a configuration job to complete, then start the webservices.
2. Use Helm hooks to weigh the deployments such that the configuration job runs before the webservices exist, and once complete, deploy the webservices.

We spent a significant amount of time considering these options, as each had their pros and cons and there was no clearly better option. Option two ended up winning out due to concerns of race conditions that may be possible with option one. Hooks themselves have a few downsides, namely if there is a failure in the installation Helm will not clean up the hook resources and you must do it by hand. Not really an issue in production, but during development testing this was a constant pain point that has nearly driven us back to the init containers option many times.

And that’s it. Now that we had the playbook down, we just had to apply a similar concept to the other components. Say hello to the modern cloud, and goodbye to ye olde Windows servers. With this completed we now have Keyfactor Command version 24.4 with containerization available for customers and our lofty goals set earlier finally completed.

In the end, we were able to achieve our goal of creating a robust containerized deployment of Command using Kubernetes and Helm, while still maintaining a Windows MSI based installation. Although it was a lot of work involving numerous complex steps, the result successfully meets all our requirements. We learned a lot along the way and now see an ocean of possibilities to make our application more cloud native and even better for customers.

This was our first Keyfactor development blog, and if you enjoyed it, we’d love to hear your feedback so we can take you behind the scenes of other projects in the future. Connect with me on LinkedIn and send me a message if you enjoyed what you read and stay tuned for similar development blogs in the future.