Scaling Access Governance Scans from 2M to 45M Records per Day
A production-oriented breakdown of parallel batch processing, scan orchestration, and the engineering trade-offs behind a 22x throughput improvement.
Jan 12, 2026 1 min read Distributed Systems
Context
Access Governance scans need to process large permission and metadata surfaces across enterprise storage systems. A scan that works for a small tenant can become too slow once it has to traverse millions of records every day across SMB, OneDrive, and Microsoft APIs.
System Shape
scan scheduler -> protocol connector -> batch planner -> parallel workers -> normalized records -> governance storeDesign Notes
- Split scanning into explicit batches so workers can make progress independently.
- Keep protocol-specific logic isolated from the normalized governance model.
- Track throughput by source, tenant, protocol, batch size, retry rate, and scan age.
- Treat rate limits and partial failures as expected scheduling inputs rather than exceptional cases.
- Keep the output idempotent so retries do not corrupt downstream access state.
Trade-offs
- More parallelism improves throughput, but it can increase pressure on external APIs.
- Larger batches reduce scheduling overhead, but they make retries more expensive.
- Source-specific tuning is useful, but too many source-specific paths can make the platform harder to operate.