The Challenge of Self-Hosting at Scale
GitHub, like many tech companies, runs its own infrastructure. One unique aspect is that GitHub hosts its own source code on the very platform it builds—github.com. This dogfooding approach allows internal testing before releasing features to users. However, it introduces a critical risk: if github.com goes down, the company loses access to its own code repository. This creates a circular dependency, where the tool needed to fix an outage itself becomes unavailable.
To mitigate this, GitHub maintains a mirror of its code for emergency fixes and pre-built assets for rollbacks. Yet, even with these safeguards, other subtle circular dependencies can creep into deployment scripts. For instance, a deployment script might inadvertently rely on an internal service or download binaries from GitHub itself—operations that fail during an outage. This is where eBPF (extended Berkeley Packet Filter) comes into play.
Understanding Circular Dependencies in Deployments
Imagine a scenario: a MySQL outage disrupts GitHub's ability to serve release data from repositories. To resolve the incident, engineers need to apply a configuration change to the affected MySQL nodes via a deploy script. At this point, several types of circular dependencies can block recovery.
Direct Dependencies
A direct dependency occurs when the deploy script itself requires GitHub services. For example, the MySQL deploy script might attempt to pull the latest release of an open-source tool from GitHub. If GitHub cannot serve the release data due to the outage, the script fails—exacerbating the very problem it was meant to fix.
Hidden Dependencies
Hidden dependencies are less obvious. The deploy script may use a servicing tool already present on the machine's disk. However, when the tool runs, it might check GitHub for updates. If it cannot reach GitHub (because of the outage), the script could hang or fail, depending on how the tool handles the connectivity error.
Transient Dependencies
Transient dependencies involve chain reactions. The MySQL deploy script might call an internal API, such as a migrations service, which in turn tries to fetch the latest release of a tool from GitHub. The failure propagates back to the deploy script, crippling the entire incident response.
How eBPF Prevents Deployment Circular Dependencies
Previously, the responsibility rested on individual teams to audit their deployment scripts and identify circular dependencies. This manual approach was error-prone and difficult to enforce consistently. GitHub's new host-based deployment system leverages eBPF to automate detection and blocking of problematic calls.
eBPF allows GitHub to run sandboxed programs within the Linux kernel without modifying kernel source code or loading kernel modules. By attaching eBPF programs to system calls, GitHub can monitor and selectively block network requests made by deployment scripts. For example, eBPF can intercept DNS lookups or HTTP requests to internal services or GitHub and reject them if they would create a circular dependency.
This approach provides a robust and transparent safety net. The deployment system defines a policy of allowed destinations (e.g., internal artifact stores or pre-approved mirrors). Any script that attempts to reach an external or disallowed service is halted immediately, preventing hidden and transient dependencies from triggering failures.
Getting Started with eBPF for Deployment Safety
GitHub's implementation serves as a blueprint for other organizations. To write your own eBPF programs for similar purposes, follow these high-level steps:
- Define the policy: Identify which network calls are permissible during deployments (e.g., local package managers, internal registries).
- Write eBPF hooks: Use the bpf() syscall or libraries like libbpf to attach programs to system calls such as connect() or sendto().
- Filter based on context: Check process metadata (PID, cgroup) to ensure only deployment scripts are restricted.
- Block or allow: Return 0 to allow the call or a negative value to deny it, with appropriate logging for debugging.
A simple example in C:
SEC("kprobe/sys_connect") int kprobe__sys_connect(struct pt_regs *ctx) { /* check IP address and process context */ if (is_deployment_script(current) && is_forbidden_ip(ip)) { return -EPERM; } return 0; }This snippet prevents deployment scripts from connecting to forbidden IPs, such as GitHub's servers during an outage.
Conclusion: A Safer Path Forward
By integrating eBPF into its deployment system, GitHub has transformed a manual, error-prone review process into an automated safeguard. The tool proactively stops direct, hidden, and transient circular dependencies before they can disrupt incident response. This not only improves uptime but also reduces cognitive load on engineers during high-pressure events.
As the eBPF ecosystem matures, more organizations can adopt similar techniques to enforce deployment safety. Whether you're running a small infrastructure or a platform like GitHub, eBPF offers a powerful, low-overhead mechanism to break circular dependencies—keeping your systems resilient when they need it most.