01
Executive Summary
Resolved
Production log visibility was fully restored. A Datadog exclusion filter on the main index had been scoped more broadly than intended; it was disabled, and log ingestion returned to normal.
As part of bringing Datadog observability into production for travel.padi.com, an exclusion filter was added to the main log index to sample out routine, non-sensitive logs and keep ingestion volume and cost down.
Its query was not scoped tightly enough to that subset, so for about eight hours it dropped info-level logs across the whole main index — the bulk of log volume. Warning and error logs were not part of the exclusion and kept flowing throughout.
The impact was limited to informational logging. The application stayed up and end users were unaffected, and because warning and error logs were never dropped, error alerting and investigation kept working — the effect was reduced visibility of routine logs, not a site outage or a loss of error visibility. The PADI engineering team traced the problem to the misconfigured filter, disabled it, and log ingestion returned to normal immediately.
02
Customer and Business Impact
- For about 8 hours, info-level logs — the bulk of log volume — were excluded from the
main index on travel.padi.com production.
- Warning and error logs were never excluded and kept flowing, so error alerting and investigation were unaffected.
- Observability impact only — the application stayed available, with no disruption to the site, bookings or authentication.
- The dropped info-level logs were not indexed and are not recoverable; no warning or error logs, application data or transactions were affected.
- Exact start/end timestamps are being confirmed against Datadog ingestion metrics.
03
Answers to Stakeholder Questions
- ChainWhat led to this?
We were setting up production log management in Datadog and added an exclusion filter to the main index to sample out routine, non-sensitive logs and control ingestion volume and cost.
- FailureWhat went wrong?
The query was not scoped to the intended routine-log subset, so it dropped info-level logs across the whole main index. Warning and error logs were not excluded and were unaffected.
- DetectionHow was it found?
The sharp drop in log volume showed on the Datadog dashboard and was traced to the misconfigured exclusion filter on the main index.
- ResolutionHow was it fixed?
The filter was disabled, and log ingestion into the main index returned to normal immediately.
- PreventionWhat are we doing now?
Exclusion filters will be scoped with explicit, tested queries and validated before production; index and exclusion changes go through peer review; and dashboard, alerting and ingestion changes will be discussed with the PADI engineering team beforehand and applied via infrastructure as code rather than directly in the Datadog console.
04
Incident Timeline
~08:00 — Log volume on the main index drops sharply after a production change to the exclusion filters, as info-level logs begin to be excluded.08:00–16:00 — Info-level logs stay excluded while warning and error logs continue to be indexed; the site and end-user functionality run normally throughout, with no customer-facing impact.On detection — The drop is spotted on the Datadog dashboard and traced to the misconfigured exclusion filter on the main index.On resolution — The filter is disabled; log ingestion resumes immediately.~16:00 — Indexed log volume is confirmed back to normal.
Times reflect the Datadog log-volume dashboard as displayed (UTC−07:00). Exact change, detection and resolution timestamps are being confirmed against Datadog audit logs and will be finalised in the review.
05
Root Cause and Contributing Factors
Primary root cause: an exclusion filter on the main log index was given a query that was not scoped to the intended routine, non-sensitive subset, so it excluded info-level logs across the whole index — the bulk of log volume. Warning and error logs were not covered and continued to be indexed.
Systemic contributing factor: the change was made directly in production — in the Datadog console rather than through infrastructure as code — without first validating what the query would match.
- A query meant to sample a narrow subset of routine info logs instead matched info-level logs across the whole index.
- The filter was not tested against a limited scope or non-production index before being enabled on
main.
- Exclusion filters drop data before indexing, so the effect is invisible until log volume is inspected.
06
Detection and Resolution
How was this identified, given nothing failed for users or services?
An exclusion filter drops matching logs before they are indexed, so the platform kept serving traffic normally and nothing failed from a user or service perspective — the only symptom was reduced log volume in Datadog.
The drop was spotted on the log-volume dashboard and traced to the misconfigured filter on the main index; disabling it restored normal ingestion immediately, with no further action needed on the application side.
07
What Went Well and What Did Not
- WorkedNo customer impact at any point — the site, bookings and authentication ran normally throughout.
- WorkedWarning and error logs were never dropped, so error alerting and investigation kept working throughout.
- WorkedOnce identified, the fix was a single low-risk action — disabling the filter — that restored ingestion immediately.
- GapThe filter change was made directly in production, in the Datadog console rather than as code, without first validating what the query would match.
08
Corrective and Preventive Actions
- CompletedDisabled the misconfigured filter on the
main index; log ingestion restored and confirmed flowing.
- PlannedScope every exclusion filter with an explicit, tested query and validate it against a limited scope or non-production index before applying to production.
- PlannedRoute Datadog index and exclusion-filter changes through peer review rather than applying them directly.
- PlannedAgree dashboard, alerting and log-ingestion changes with the PADI engineering team beforehand, and apply them via infrastructure as code (IaC), not directly in the Datadog console.
- PlannedScope log-configuration permissions so index changes are deliberate and auditable.
- PlannedDocument standard exclusion-filter patterns for the PADI Datadog indexes.
09
Lessons Learned
- Exclusion filters drop data silently; too broad a query can wipe out whole tiers of logs — here, all info-level logs — without any service failing.
- Observability configuration changes are production changes and need the same validation and review as application code.
- Such changes should go through infrastructure as code, agreed with the PADI engineering team in advance, rather than ad hoc in the console.
- Log-index changes should be validated against a limited scope before reaching a live index.
10
Guardrails Being Added
Changes to Datadog exclusion filters, dashboards, alerting and log ingestion will be scoped with tested queries, discussed with the PADI engineering team beforehand, and applied through infrastructure as code rather than directly in the Datadog console — so a filter can no longer silently drop production logs.