Designing the Actual Access Audit-Log Pipeline

The Problem

Access events are noisy, bursty, and business critical. The Actual Access pipeline needs to ingest real-time audit logs from SharePoint, Google Drive, and NetApp, then provide a dependable data stream for downstream ransomware detection and automated response features.

Pipeline Shape

source audit logs -> ingestion boundary -> normalization -> enrichment -> detection features -> response workflows

Design Checklist

• Preserve enough source context to debug mismatches between providers.
• Normalize identities, file references, timestamps, and action types into a stable model.
• Separate transient provider failures from malformed or unsupported events.
• Track lag, retry age, per-source volume, parsing failures, and downstream freshness.
• Keep the pipeline replayable so detection logic can evolve without losing historical context.

Reliability Notes

High-volume security telemetry needs boring failure behavior. The system should degrade by delaying, retrying, or quarantining specific work units rather than blocking the entire stream.