Introduction
Maintaining a platform like GitHub requires strong defense mechanisms—rate limits, traffic controls, and layered protections—to ensure availability during abuse or attacks. However, these safeguards can inadvertently outlive their purpose, blocking legitimate users. This guide walks you through how to identify, review, and remove outdated protections, drawing from GitHub's experience where emergency-response rules persisted too long. By following these steps, you'll prevent false positives and maintain a healthy user experience.
What You Need
- Access to your platform's monitoring and alerting systems (e.g., observability dashboards)
- Log data for rate-limit violations and blocked requests
- A list of current defense rules (rate limits, fingerprinting patterns, business logic rules)
- User feedback channels (social media, support tickets, forums)
- Cross-functional team (engineering, security, product)
- Documentation of past incidents and emergency responses
Step-by-Step Guide
Step 1: Establish Observability for Defenses
Before you can fix outdated protections, you need to see them. Implement comprehensive observability that tracks not just feature performance but also defense mechanisms. Monitor rate-limit hits, block rates, and false-positive metrics. For example, GitHub observed that only 0.5–0.9% of matching fingerprint requests were actually blocked—but those blocks were 100% for matched criteria. Use these numbers to set baselines and trigger alerts when false positives exceed a threshold (e.g., 0.003–0.004% of total traffic).
Step 2: Collect User Reports of Abnormal Blocks
Gather feedback from users encountering unexpected errors—like “Too many requests” during normal browsing. Scrutinize social media, support tickets, and community forums. GitHub users reported low-volume browsing getting blocked after clicking links from other services or apps. Document each report with timestamps, user agent, and the request pattern that triggered the block. This step is crucial because users often highlight where your defenses are misfiring.
Step 3: Analyze Blocked Request Patterns
Investigate the root cause by comparing blocked requests against your current defense rules. Look for rules added during past incidents that might have been left in place. In GitHub's case, they found protection rules from past abuse incidents that used composite signals—combinations of fingerprinting techniques and business logic. These signals, while effective against abuse, also matched some legitimate logged-out users. Identify which rules are based on outdated threat patterns.
Step 4: Quantify False-Positive Impact
Calculate the rate of false positives relative to total traffic. GitHub found that false positives represented about 0.003–0.004% of all requests—small but unacceptable. Use your observability data to compute this percentage. Also determine the proportion of users affected and the severity of disruption (e.g., blocks during normal browsing vs. during heavy usage). This quantifiable data helps prioritize cleanup efforts.
Step 5: Review Historical Incident Responses
Audit all defense rules added as emergency responses. Emergency fixes often use broader controls to stop active abuse quickly, but they may not be suitable long-term. GitHub's rules were created based on patterns strongly associated with abusive traffic at the time. However, those same patterns later matched legitimate requests. Create a timeline of each incident and its associated defense rule, then evaluate whether the threat pattern still exists or has evolved.
Step 6: Remove or Refine Outdated Protections
Based on the review, decide which rules to remove outright or refine. For rules that still serve a purpose but cause false positives, adjust the thresholds or combine them with additional signals to reduce false matches. GitHub removed the rules that were causing false positives after discovering they were no longer needed. Document the change and communicate it to your team. Use Tips below for best practices on removal.
Step 7: Improve Rule Design to Minimize False Positives
When creating new protections, ensure they are as targeted as possible. Avoid using generic fingerprinting alone; layer business logic that reflects current legitimate usage patterns. For example, only block requests that match both a high-confidence abuse pattern AND an abnormal rate—not just one. GitHub's composite approach did well (only 0.5–0.9% of matching fingerprints led to blocks), but the business logic needed updating. Build in automatic expiration or review triggers for emergency rules.
Step 8: Establish a Regular Review Cycle
Set up periodic audits (e.g., quarterly) of all defense rules. Include cross-functional stakeholders: security, engineering, and customer support. During each review, answer: Is this rule still necessary? Is it causing false positives? Has the threat landscape changed? Use a ticketing or change-management system to track reviews and approvals. GitHub's experience shows that even small false-positive percentages (0.003–0.004%) can disrupt real users, so no rule is too minor to review.
Step 9: Communicate Changes to Users
If your cleanup affects user-facing errors, proactively apologize and explain. GitHub recognized that they should have caught and removed protections sooner and apologized for the disruption. When you remove rules, notify affected users via blog posts or release notes. Transparency builds trust, especially when users experienced incorrect blocks. Include details on what was fixed and how you're preventing recurrence.
Step 10: Monitor After Cleanup
After removing outdated protections, continue monitoring false positives and overall system health. Watch for any new abuse patterns that might exploit the removed rules—then update your defenses accordingly. Compare post-cleanup metrics to pre-cleanup baselines (e.g., GitHub saw false positives drop). Use this data to fine-tune your observability thresholds for future alerts.
Tips
- Treat emergency rules with expiration dates – Add temporary rules with a built-in sunset period (e.g., 30 days). Review before renewal.
- Segment user groups – Apply stricter controls only to high-risk traffic (e.g., logged-out sessions) and less restrictive rules for authenticated users.
- Use canary testing – Roll out new protections to a small subset of traffic first to measure false-positive rates.
- Keep documentation updated – Maintain clear records of why each rule was added, the incident it addressed, and the expected false-positive rate.
- Leverage user feedback loops – Encourage users to report blocks easily (e.g., “Report an issue” button on error pages). Their reports are often the first sign of a problem.
- Automate alerts for anomaly spikes – Set alerts for sudden increases in block rates or false-positive metrics, so you catch regressions quickly.
- Test composite signals regularly – Since composite signals can evolve, revalidate fingerprinting combinations against current legitimate traffic patterns.