Enhancing Deployment Safety at GitHub with eBPF: Breaking Circular Dependencies

The Challenge of Circular Dependencies

GitHub, like many tech companies, runs its own production services on its own platform. This means that the very code that powers github.com is hosted on github.com. While this practice allows GitHub to act as its own biggest customer and test changes internally before rolling them out to users, it creates a unique challenge: a circular dependency. If github.com were to experience an outage, developers would lose access to the source code needed to fix it.

To mitigate this, GitHub maintains a mirror of its code for emergency fixes and pre-built assets for rollbacks. But this only solves the first layer of the problem. Deploy scripts themselves can introduce new circular dependencies—for instance, by downloading binaries from GitHub or relying on internal services that in turn depend on GitHub. This is where eBPF (extended Berkeley Packet Filter) comes into play.

Types of Circular Dependencies

Consider a scenario: a MySQL outage prevents GitHub from serving release data from repositories. To resolve the incident, engineers need to apply a configuration change to the affected MySQL nodes via a deploy script. Let’s examine the different circular dependencies that could surface.

Direct Dependencies

The deploy script attempts to pull the latest release of an open-source tool from GitHub. Because GitHub is down and cannot serve the release data, the script fails to complete. This is a straightforward, direct circular dependency.

Hidden Dependencies

The script uses a servicing tool already present on the machine’s disk. However, when that tool runs, it checks GitHub for an update. If it cannot reach GitHub (due to the outage), the tool may crash or hang, depending on how it handles the update check failure. The dependency is hidden because it’s not obvious from the script itself.

Transient Dependencies

The deploy script makes an API call to another internal service (e.g., a migrations service). That service, in turn, tries to fetch the latest release of an open-source tool from GitHub. The failure propagates back to the deploy script, creating a transient—and often hard-to-diagnose—circular dependency.

How eBPF Helps Break the Cycle

Historically, the responsibility for avoiding circular dependencies fell on each team owning stateful hosts. They had to manually review deployment scripts and identify potential dependency loops. In practice, many dependencies go unnoticed because they are hidden or transient.

While designing its new host-based deployment system, GitHub evaluated novel approaches to prevent deployment code from creating circular dependencies. The solution they found was eBPF. With eBPF, GitHub can selectively monitor and block network calls or system operations that would create a circular dependency—all without modifying the deployment scripts or the tools they use.

Selective Monitoring and Blocking

eBPF runs sandboxed programs inside the Linux kernel, allowing GitHub to attach hooks to system calls related to networking, file I/O, or process execution. For example, they can write an eBPF program that intercepts any attempt by a deploy script to connect to GitHub’s servers. If the connection is detected, the program can either log it for analysis or block it outright, depending on the policy.

This approach is lightweight and efficient because eBPF programs run in kernel context. They can inspect every system call without introducing significant overhead. Moreover, eBPF is already supported in modern Linux kernels, making it a practical choice for GitHub’s infrastructure.

Putting eBPF into Practice

GitHub’s deployment system uses eBPF to define a whitelist of allowed external connections for each deploy script. Any connection attempt outside that list is blocked. This ensures that even if a script or a tool has an unexpected dependency, it won’t be able to reach GitHub (or any other service) in a way that could cause a circular dependency during an incident.

The eBPF programs are loaded at the start of a deploy and unloaded when the deploy completes. This dynamic nature means the protection is only active when needed, reducing the attack surface and performance impact.

Getting Started with eBPF

If you’re interested in applying similar techniques, here are a few steps to begin:

• Learn the basics: Understand how eBPF programs are written in C and compiled into bytecode. Tools like bcc (BPF Compiler Collection) or libbpf can help simplify the process.
• Identify key system calls: Focus on syscalls like connect(), sendto(), or execve() that are relevant to your deployment security needs.
• Use existing frameworks: Consider leveraging open-source eBPF frameworks such as Cilium or Falco for higher-level abstractions.
• Test in a safe environment: Before deploying eBPF programs in production, thoroughly test them in a sandboxed environment to ensure they don’t break legitimate operations.

By adopting eBPF, GitHub has turned a complex circular dependency problem into a manageable, kernel-enforced safety net. The same approach can be applied by any organization that runs critical infrastructure and wants to ensure their deployment processes remain robust even during incidents.

For more details, check out the section on circular dependencies or our eBPF implementation.