In the world of web infrastructure, what starts as a simple goal can often lead you down a fascinating rabbit hole of history, philosophy, and clever engineering. This is the story of our journey to build a simple, single-purpose, open-source CDN for changelog.com and the one major hurdle that stood in our way: Varnish, our HTTP caching layer of choice, doesn't support TLS backends.
How This Started
Our project - pipely.tech - is a minimal Varnish configuration deployed globally across 16 Fly.io regions. The goal was simplicity. However, we hit a wall when we realized the open-source version of Varnish can't connect to an origin server over HTTPS; it only speaks plain HTTP. This was a showstopper.
Enter Nabeel Sulieman, a shipit.show guest, who had previously introduced us to KCert, a simpler alternative to cert-manager. We knew if anyone could help us solve this TLS conundrum, it was him. After a couple of false starts, we finally recorded the final solution. As Nabeel aptly put it: Third time is the charm.
What Makes TLS & SSL Interesting for Nabeel?
We started by asking Nabeel what draws him to the often-complex world of TLS and SSL. For him, it's a fascination with the core concepts of encryption and a drive to simplify the tools around it. He finds the idea of asymmetric public/private keys inherently interesting and believes that the existing tooling is often harder to use than it needs to be.
This drive for simplicity is what led him to create projects like KCert. He later evolved this thinking by building his own custom reverse proxy, first with .NET's YARP (Yet Another Reverse Proxy) and now being rewritten in Go. His goal was to combine the reverse proxy and certificate management into a single, simple tool that did exactly what he needed, without the complexity of tools like nginx-ingress and cert-manager.
Disabling Issues & Pull Requests (in GitHub repositories)
Our conversation took a brief detour into the philosophy of open-source maintenance. We discussed the radical idea of a project being "open source, but not open contribution," where the author disables issues and pull requests to maintain a singular vision. We discovered that the database project SQLite follows this model. It's a powerful reminder that for some projects, the best way for others to contribute is to fork the code and adapt it to their own needs, preserving the original author's focus.
What is Pipely?
Pipely is our open-source CDN project, which started as a mere 20-line Varnish configuration. The current task on our roadmap was clear: "Support TLS origins." Nabeel, having heard about our struggle on a podcast, initially suspected the limitation was a commercial tactic to push users toward Varnish Enterprise.
He first built a proof-of-concept proxy using .NET, which worked but consumed about 20MB of RAM. Seeking a more lightweight solution, he rewrote it in Go. The result is tls-exterminator, a tiny, 55-line utility that does one thing: it listens for HTTP requests on one port and forwards them to a secure HTTPS endpoint on another. At just 1-2MB of RAM, it was the perfect, lightweight solution.
Why No SSL? (in Varnish)
But why does this limitation exist in the first place? We dug into Varnish's history and found our answer in a post titled Why no SSL? by its creator. The official documentation points to a tool called Hitch, but it's designed to terminate incoming TLS—the exact opposite of our needs.
Who is Poul-Henning?
The author of that post is Poul-Henning Kamp (PHK), the lead architect and primary developer of Varnish. He's a legendary figure in the open-source world, also known for his extensive work on FreeBSD, including creating FreeBSD Jails.
In his 2011 post, PHK laid out his reasoning:
- Complexity: SSL libraries like OpenSSL are massive, complex, and historically a source of security nightmares.
- Performance: TLS termination is CPU-intensive. Building it into Varnish would compromise the highly optimized, high-speed data path that is Varnish's entire reason for being.
- Philosophy: Running a separate, dedicated process for TLS is the right architectural choice. It keeps Varnish focused on its core competency: being an incredibly fast HTTP cache.
The Bikeshed
In a delightful twist, we discovered that Poul-Henning Kamp is the very person who coined the term "bikeshedding" in a 1999 email, describing the tendency of committees to argue over trivial details while ignoring complex ones. We had a good laugh about bikeshedding the very topic his post was about. His follow-up post after the Heartbleed vulnerability—a resounding "I told you so"—only solidified our decision. We would follow his advice and use a separate process.
Pipely Pull Request #8 & Dagger Instead of Docker
With the philosophy settled, we turned to implementation. Nabeel submitted a pull request to Pipely that integrated tls-exterminator. This is where the tooling conversation got interesting.
Nabeel's initial testing setup for tls-exterminator involved multiple Dockerfiles and a docker-compose file. It was cumbersome and fragile. This led him to Dagger, a programmable CI/CD engine that lets you build automation pipelines in code.
Instead of shell scripts and YAML, Dagger allowed him to define his entire integration test—spinning up a test server, the tls-exterminator proxy, and the test runner itself—in a self-contained, reproducible Go function. It elegantly solved the orchestration problem that makes multi-service testing so difficult.
The Pipely Dagger Module
We embraced this approach, building out a Dagger module for the entire Pipely project. Our Dagger pipeline now programmatically assembles the final production container:
- It starts with a base Varnish container.
- It uses Go to fetch and install specific versions of
tls-exterminatorandgoreman(a utility for running multiple processes). - It generates a
Procfileon the fly, which tellsgoremanhow to start both Varnish andtls-exterminator.
This "infrastructure as code" approach gives us a powerful, repeatable, and platform-agnostic way to build, test, and package our application. It's the perfect modern solution to a problem rooted in decades of software design philosophy.
Let's Benchmark! What's the Cost?
With a clear implementation path using Dagger, one critical question remained: what is the performance cost? We were introducing an extra hop—Varnish talking to the tls-exterminator proxy, which then talks to the origin. Every millisecond of latency counts.
To measure the overhead, we set up a simple benchmark. We ran our new, combined container locally and used hurl, our command-line testing tool, to measure the response times.
The test was straightforward:
- Baseline: A direct request to the HTTPS origin.
- Proxied: A request to the local Varnish instance, which would then go through
tls-exterminatorto the same origin.
The results were exactly what we had hoped for. The added latency from the tls-exterminator sidecar was consistently in the low single-digit milliseconds. It was a negligible, almost immeasurable cost for the functionality we were gaining. This benchmark was the final validation we needed. The solution wasn't just elegant and well-architected; it was also incredibly fast. We had successfully "exterminated" the TLS problem without sacrificing performance 🙌
What Happens Next? From PR to Production
With the proof-of-concept validated and the performance deemed excellent, we laid out the final steps to get this into production. The path forward was now clear and simple:
- Merge the Pull Request
- Update the Varnish Config
- Publish and Deploy
Wrap-up
This journey was a perfect example of how a seemingly simple technical hurdle can lead to a deeper understanding of software architecture. We started with a limitation in Varnish, which led us to uncover the core design philosophy of its creator. This, in turn, guided us toward a solution that was not only effective but also aligned with sound engineering principles: a small, single-purpose sidecar.
Thanks to a great collaboration and the power of modern tools like Dagger, we were able to build, test, and validate this solution in a clean, repeatable, and high-performance package. Now, with a clear path forward, we're ready to roll out an open-source CDN: Pipely.
