7 Key Insights into Scaling AI Code Review with Specialized Agents

In the fast-paced world of software development, code review is both a critical quality gate and a notorious bottleneck. Merge requests can linger in queues for hours, with reviewers context-switching and leaving nitpicks, leading to delays and frustration. At Cloudflare, we faced this challenge across thousands of repositories. Our solution? Instead of a monolithic AI reviewer, we built a CI-native orchestration system with up to seven specialized agents—each focusing on a distinct aspect like security, performance, or code quality. Here are seven essential things you need to know about how we scaled AI code review.

1. The Bottleneck Problem: Why Traditional Code Review Fails at Scale

Code review is indispensable for catching bugs and sharing knowledge, but it often becomes a major drag on engineering throughput. A typical flow: an engineer submits a merge request, it lands in a queue, a reviewer later context-switches to read the diff, leaves a handful of nitpicks about variable naming, the author responds, and the cycle repeats. At Cloudflare, the median wait time for a first review was measured in hours—a huge inefficiency across tens of thousands of merge requests. This motivated us to explore AI-driven solutions that could provide rapid, accurate feedback without overwhelming human reviewers.

2. Initial AI Experiments: The Pitfalls of Off-the-Shelf Tools

Our first attempt was to adopt existing AI code review tools. Many worked reasonably well and offered customization, but they lacked the flexibility needed for an organization of Cloudflare’s size. The common theme was insufficient configurability—we couldn’t tailor them to our specific codebase conventions, security requirements, or internal engineering standards. This led us to try a more DIY approach: taking a git diff, feeding it into a loosely constructed prompt, and asking a large language model (LLM) to find bugs. The results were noisy—vague suggestions, hallucinated syntax errors, and redundant advice like “consider adding error handling” on functions that already had it. Clearly, a naive summarization approach wasn’t enough for complex codebases.

3. The Key Insight: Specialized Agents Beat a Monolithic Model

Instead of building a single, monolithic code review agent with a massive generic prompt, we pivoted to a modular architecture. We built an orchestration system around OpenCode, an open-source coding agent. When an engineer opens a merge request, it triggers a coordinated team of AI specialists—up to seven distinct reviewers covering security, performance, code quality, documentation, release management, and compliance with our internal Engineering Codex. Each specialist has a focused prompt and context, minimizing hallucinations and noise. A coordinator agent then deduplicates findings, judges severity, and composes a single structured review comment.

4. Architecture Deep Dive: Plugins All the Way

Our system relies on a plugin-based architecture that decouples the core orchestration from version control, model providers, and review rules. Hardcoding a single VCS or LLM would be rigid at our scale. Instead, we use a plugin system that allows swapping components—GitHub, GitLab, and Bitbucket integrations, multiple LLM backends (OpenAI, Anthropic, etc.), and custom review rules per repository. This flexibility ensures the system can evolve as our infrastructure grows. The coordinator agent (also a plugin) manages the workflow: it triggers specialists, collects outputs, filters duplicates, and produces a final comment with actionable items.

5. Real-World Results: Thousands of Merge Requests and Counting

We’ve been running this system internally across tens of thousands of merge requests. The results are impressive: it approves clean code swiftly, flags genuine bugs with high accuracy, and actively blocks merges when it detects serious problems like security vulnerabilities. Engineers trust its feedback because it’s precise and relevant. The system has significantly reduced median review times from hours to minutes, freeing human reviewers to focus on architectural and design discussions. This initiative is part of our broader Code Orange: Fail Small strategy to enhance engineering resiliency.

6. Overcoming the Hurdles: Deduplication and Severity Judgment

One major challenge with multiple agents is handling overlapping suggestions. A performance specialist might note a slow query that the security agent also flags as a potential injection point. The coordinator agent deduplicates these by comparing context and message similarity. It also judges severity using a priority matrix: critical issues (e.g., hardcoded credentials) block merges; moderate ones (e.g., inefficient loops) are flagged as warnings; and minor suggestions (e.g., style tweaks) are grouped separately. This structure prevents noise and ensures engineers know what truly matters.

7. Integration into CI/CD: Putting LLMs in the Critical Path

Putting LLMs directly in the CI/CD pipeline is risky—they can be slow or unpredictable. Our system runs as a CI step triggered by merge request events. We set strict timeouts (e.g., 2 minutes per specialist) and fallback plans: if the LLM fails to respond, the review is skipped without blocking the pipeline. We also use caching and incremental diffs to keep costs low. By integrating seamlessly into developers’ existing workflows, the AI review feels like a helpful colleague rather than a bureaucratic hurdle.

Building a scalable AI code review system isn’t about a single magic model; it’s about orchestrating specialized agents that complement each other. Cloudflare’s approach—using a plugin architecture, deduplication logic, and severity judgment—has transformed a bottleneck into a productivity booster. Whether you’re a small startup or a large enterprise, these lessons can help you harness AI for faster, more accurate code reviews.