Mark Sayson

Model a deletion graph, but don't execute it

Sun, 27 Sep 2026 04:30:00 +0000

Modeling deletion as a dependency graph sounds good in theory: identify which systems are upstream of others and determine the order in which to delete customer data.

Trying to enforce that ordering in real-world workflows quickly becomes a nightmare. It adds unnecessary complexity and creates points of failure throughout the dependency graph.

However, mapping the graph is still useful: it exposes failure modes, highlights where deletion-aware access controls are required, and helps prioritize next steps for your deletion program.

The naive solution

Imagine a system where customer personal data flows through several services.

Customer Accounts shares personal data with Ordering and Notifications. Ordering also passes personal data to Search and Notifications, while Analytics consumes data from Ordering, Search, and Notifications.

When a customer asks us to delete their data, what order should be followed for deletion?

We could follow the flow of data:

Delete the Customer Account
Then delete from Ordering
Then delete from Search and Notifications
Finally from Analytics after all its upstreams have deleted the customer.

It sounds reasonable, and it stops personal data from flowing back into downstream systems after they’ve fulfilled the deletion requests.

But there’s an important assumption hiding here: data-flow dependencies aren’t deletion-order dependencies. Sometimes they share direction; sometimes deletion must run opposite to the data flow.

Analytics consumes data from multiple upstream systems, but does it actually have to wait for all of them to finish deleting before it can start? The graph shows the relationships exist, but doesn’t show which ones require ordering.

And even if we knew, enforcing the order across a live graph runs into two problems: keeping the graph accurate, and keeping one failure from blocking everything else.

The graph is never stable

In a large organization, the graph becomes stale almost immediately. Teams add data sources and consumers, split services, and introduce caches and derived data sets. None of these changes necessarily show up in the deletion workflow.

Deprecations are particularly problematic. If a decommissioned service is still in the graph, an ordered workflow can wait indefinitely for a completion signal that will never arrive, silently blocking every downstream deletion.

Strictly ordered execution needs every one of those changes to be reflected in the graph immediately. Automation helps, but the inventory then becomes another system that must be maintained, validated, and trusted.

A map that’s 90% accurate still produces a useful list of remediation work. An orchestrator that’s 90% accurate can stall every request that routes through the stale 10%.

One failure shouldn’t block everything else

Suppose Customer Accounts has a deletion problem. Under the naive plan, Ordering can’t start until Customer Accounts succeeds, Search can’t start until Ordering succeeds, and so on.

One team’s outage shouldn’t block deletion for hundreds of other services.

A better model is to treat deletion as asynchronous. The request propagates to every system, and each does its own work. Some delete immediately. Others take longer, fail temporarily, and retry.

Deletion doesn’t need to behave like a single distributed transaction where every system commits together. Systems should make progress independently where they safely can.

When strict deletion ordering is actually required

Ordering is required when deleting in the wrong order would break an invariant: a rule the system enforces to stay consistent, such as “every order must reference an existing customer.” Two patterns come up most often.

Referential integrity

Within a single database, this is familiar: if orders reference a customer and there’s no cascading delete, the orders must be removed first.

The same pattern exists across services. Suppose Ordering holds a hard reference to each customer account, and Customer Accounts refuses to delete an account while orders still reference it. Data flows from Customer Accounts to Ordering, but Ordering must delete its records before the account becomes eligible for deletion.

Represent this explicitly as an ordering constraint. Ordering deletes asynchronously, and the account becomes eligible for deletion once those references are gone.

The ordering exists because of the invariant, not because of the data flow.

In-flight work

A different class of ordering constraint occurs when in-flight work requires the source state to remain available while it executes.

The required sequence might be:

Stop new jobs or orders
Complete or cancel in-flight work
Delete source data

Without that ordering, a queued job could execute after the source has been deleted and recreate data that was supposed to be removed, or fail because the referenced state no longer exists.

Model the constraints, enforce them locally

A graph used for orchestration should identify the smallest set of dependencies that require deletion ordering.

In the example, customer data reaches every system, but the only real constraint may be that Ordering deletes before Customer Accounts.

Every edge that does impose ordering should record why: the invariant that breaks if deletion happens in the wrong order.

When strict ordering is required, enforce it as close as possible to the system that owns the invariant. That may be cascading deletes in a database, or a service that refuses to delete a parent while dependents exist.

A central orchestrator can coordinate genuine cross-system ordering constraints, but it shouldn’t turn every dependency into a global workflow.

Enforce ordering where the invariant lives. Keep everything else asynchronous.

Prevent deleted data from coming back

Asynchronous deletion introduces race conditions, which I cover in more detail in “Deleted and back again: mitigating unintended data resurrection”.

Search deletes a customer’s data, but Ordering may not have processed the deletion yet, so it continues working normally and sends that data back to Search, which stores it again.

Strict ordering could help prevent this by making Search wait for Ordering. The more robust fix is to make Ordering deletion-aware. Once a customer is subject to deletion, Ordering can filter its exports and reject API requests for that customer.

Downstream systems shouldn’t have to wait for upstream systems to delete. Upstream systems should stop sending data that’s subject to deletion.

That changes the question from “What order should these systems delete in?” to “What guarantees does each system need while deletion is in progress?”

Asynchronous doesn’t mean fire-and-forget

Once deletion is asynchronous, the same request can be finished in one system and pending in another. That’s expected, but the program needs to distinguish deletion that’s in progress and on track for its deadline from deletion that’s stalled, failed, or unverified. That requires every system to report where each request stands:

Accepted: the system received the request
Processed: the system ran its deletion
Verified: there is evidence of expected deletion or de-identification state

This distinction matters because a service responding “success” to a request isn’t necessarily enough evidence for an audit. For privacy deletion programs, I’d consider a system verified when we have evidence that personal data is no longer operationally available except where there are documented retention exemptions.

Use the graph to find the work

Most privacy programs already have most of this graph through records of processing activities. What they usually lack is what deletion depends on: which flows impose ordering, where data can flow back after deletion, which retention exceptions apply to each system, and how each system proves it deleted.

For each dependency, ask:

Question	Gap it reveals	Typical fix
Does this dependency impose a deletion invariant?	Ordering constraints	Enforce locally, or model an explicit constraint
Can data flow after deletion begins?	Resurrection paths	Deletion-aware transfers and durable deletion state
Can queued or in-flight work outlive the source data?	In-flight races	Drain queues or reject stale work
How do we know deletion completed?	Verification gaps	Per-system accepted, processed, and verified states
Who owns the dependency?	Operational gaps	Assign owners; build shared tooling for common gaps
Is data shared with a processor or third party?	Recipient gaps	Propagate deletion requests and track confirmation

Summary

Make deletion asynchronous by default.
Add ordering only where a concrete correctness constraint requires it, and enforce that ordering as close to the constraint as possible.
Make systems robust to deletion happening somewhere else at the same time.
Model the deletion graph to find engineering work. Don’t execute it.

Posts in this series

Why data deletion is still an unsolved infrastructure problem
Why deletion means different things in different systems
Gaps in data deletion verification and auditability
Deletion is not always deletion: retention exceptions and competing obligations
Deleted and back again: mitigating unintended data resurrection
(Current post) Model a deletion graph, but don’t execute it

Deleted and back again: mitigating unintended data resurrection

Mon, 31 Aug 2026 04:30:00 +0000

Kathy submits a personal data deletion request. Ordering, marketing, and personalization systems remove her records, and yet, a week later, her data is back as if nothing had happened.

In complex data architectures, keeping deleted personal data deleted is a persistent engineering challenge. Systems re-ingest, recompute, replay, and repopulate data long after a deletion request has been processed.

A single missed dataset can re-propagate a user’s personal data across an organization’s interconnected systems, undermining the best attempts at achieving deletion.

What is data resurrection?

Data resurrection is the unintended reintroduction of personal data into a system after that data has been deleted or should no longer be processed.

This is distinct from:

Re-identification, where anonymized data becomes identifiable through inference or aggregation.
Retention exceptions, where data is intentionally preserved for legal or operational reasons.

Common sources of unintended data resurrection

Source 1: Upstream systems not onboarded to deletion

Upstream systems that still hold personal data continue emitting events, syncing records, or serving APIs that downstream systems consume. Even if downstream systems correctly delete a customer’s personal data upon request, they will re-ingest whatever their upstream dependencies continue to publish.

Source 2: Out-of-order deletion

Many deletion workflows have a hidden assumption that upstream systems delete first and downstream systems follow. When this ordering isn’t enforced, downstream systems delete and re-ingest a customer’s personal data from upstream systems that haven’t deleted yet.

Even if deletion requests are sent to upstream systems first, workflows are often asynchronous. Some services delete immediately, while others batch deletions into daily or weekly jobs during periods of low customer traffic.

Source 3: Soft deletion across system boundaries

When an upstream system soft-deletes a record by setting a “deleted” flag and continues propagating it, the record carries its deletion state in a field that downstream consumers must be aware of. Consumers may not act on that flag, especially if it was added after the integration was built. The result is that soft-deleted records can still be treated as active downstream.

Source 4: Event replay and cached data

Systems that replay events, cache data, or ingest historical data snapshots from upstream services can inadvertently maintain or reintroduce personal data that was deleted from primary data stores.

Replay and sync workflows need to propagate deletion markers or tombstones. Caches and search indexes also need an explicit deletion path.

Source 5: Restored backups

Disaster recovery workflows restore data from backups that predate the deletion request. Unless backup recovery and data sharing workflows are deletion-aware, restoring a single upstream backup can repopulate dozens of downstream systems with previously deleted personal data.

Organizations often do not remove customer data from backups, but recovery processes must account for deletion requests since the backup was taken. Centralized resurrection detection and re-deletion workflows can reduce service-specific recovery logic, at the cost of temporarily propagating resurrected data before it’s caught.

Source 6: Silent deletion failures

When systems report “success” after receiving or processing a deletion request but silently fail to delete or anonymize all records, they can continue to be sources of unredacted personal data long after a deletion request.

Silent failures can stem from schema drift or deletion logic that does not cover all secondary or derived datasets.

Controls to mitigate data resurrection

It’s tempting to say that every system should simply implement deletion and that orchestrators should enforce a global deletion order across the dependency graph.

In practice, this is not feasible in large organizations. You cannot guarantee deletion order across a large number of systems with different semantics, asynchronous workflows, independent scheduling, bidirectional data flows, and varied failure modes.

The practical goal is not perfect ordering of deletion. It is eventual deletion: every system should converge to its expected deletion state within the applicable deletion window, and the organization should detect those that do not.

Control 1: Automatic detection and deletion replay

Periodic checks for deleted identifiers can help find data that was missed or later introduced, and trigger a notification to onboard to deletion workflows or replay deletion if the service was already onboarded.

However, replaying deletion requests isn’t a silver bullet. Continually polling for re-emergence of deleted data and retriggering deletion requests creates churn, increased service load, and does not scale with high numbers of accumulated deletion requests. When the underlying resurrection pathways remain open, deletion replays simply repeat the same work indefinitely.

Control 2: Deletion-aware access controls

Deletion-aware access controls prevent services from accessing or propagating data for customers whose data has been deleted. A platform team should build these once for reuse, rather than each service implementing its own.

The simplest implementation is a query layer that checks a central deletion registry before returning a record, so that data subject to deletion is blocked from processing. Where low latency is critical, services can maintain a locally replicated set of deleted identifiers with a defined maximum delay between a deletion decision and its availability to the service. The tradeoff is between enforcement latency and query-time overhead.

These controls create deletion boundaries. Even if some upstream systems have not yet onboarded to deletion, data propagation stops at the boundary of any system that enforces deletion-aware access controls.

Deletion-aware controls do not eliminate the need for deletion replay, since data can still reach systems that do not enforce these controls before it reaches a deletion boundary. Those systems remain in scope for detection and remediation, but the boundary limits how far the resurrected data can propagate.

Control 3: Automatic expiry of data

Some systems only need recent data to operate. These can automatically delete records after a defined time period shorter than the applicable deletion window, using time-to-live (TTL) or object expiry lifecycles.

These services may not need explicit deletion onboarding if they can demonstrate that data is automatically removed within the deletion window.

Control 4: Data minimization

Where possible, the simplest option is often to stop processing personal data in the first place.

Given the choice to invest weeks of effort on deletion workflows, or spend a day dropping fields and stale datasets that are no longer needed, many teams may pick the latter, reducing compliance risk and storage costs in the process.

A deletion resilience maturity model

The following maturity model is a conceptual framework for thinking about deletion resilience. Levels are defined by who owns deletion logic, and how effectively resurrection is caught and mitigated.

Level	How deletion is implemented	What resurrection looks like
1. Service-owned	Each system builds its own deletion logic, often inconsistently. Service owners self-identify when they need to onboard.	Common, and usually undetected. Upstream systems may not delete, downstream systems re-ingest. Deletion replays and manual clean-up are frequent and expensive, or the risk is accepted.
2. Platform-assisted	Deletion is treated as privacy infrastructure rather than a cost each service absorbs. Shared tooling offers hard deletion, TTL expiry, and deletion-aware access controls with common semantics.	Still occurs, but contained and easier to remediate. Deletion replays run periodically across services where resurrection is detected.
3. Systemically resilient	Controls are built into infrastructure by default, with centralized deletion state, deletion-aware boundaries, and minimal service team context needed to implement or validate.	Rare, localized, and quickly corrected. Automated detection and reconciliation cover replay and recovery paths.

Many organizations can have characteristics of both Levels 1 and 2: tooling may exist, while onboarding costs enough that a long tail never fully completes it, leaving resurrection a systemic risk. Moving from Level 1 to Level 2 is mostly a tooling problem. Level 3 requires deletion state to be treated as an organization-level architectural concern rather than solely as service-local logic.

Deletion resilience is a systems property, not a per-service outcome

A successful deletion response does not prove that personal data has stopped being processed across systems with deletion expectations.

Organizations need to understand how personal data subject to deletion can continue to propagate. They should enforce containment controls in systems with legitimate retention exceptions, and build verification mechanisms that test both successful deletion and reintroduction of previously deleted data.

Evidence should show which systems are subject to deletion, which have compensating measures such as automatic expiry, and which retain data under approved exceptions. Retained data should be blocked from use for unapproved purposes.

Deletion resilience is not achieved by making every system delete perfectly or by enforcing a single global deletion order.

It comes from making deletion state durable, containing deleted data at system boundaries, allowing deletion state to converge within the applicable window, and continuously detecting and correcting resurrection.

Posts in this series

Why data deletion is still an unsolved infrastructure problem
Why deletion means different things in different systems
Gaps in data deletion verification and auditability
Deletion is not always deletion: retention exceptions and competing obligations
(Current post) Deleted and back again: mitigating unintended data resurrection
Model a deletion graph, but don’t execute it

Deletion is not always deletion: retention exceptions and competing obligations

Sun, 21 Jun 2026 04:30:00 +0000

One of the more surprising aspects of privacy engineering is how many different deletion and retention requirements can apply to the same customer data. A single deletion request may result in immediate deletion in some systems, multi-year retention in others, and event-driven retention in cases such as legal holds.

In other words, deletion is not always deletion. While the intent of a customer’s deletion request may be straightforward, legal, financial, fraud prevention, security, and operational obligations often require organizations to retain specific categories of data. Understanding when data should be deleted, when it must be retained, and how retained data must be restricted is one of the more challenging aspects of building effective deletion programs.

For many organizations, a single deletion request can produce different outcomes across systems depending on the purpose of processing personal data, applicable retention requirements, and the justification for continued retention or processing.

Use Case	Example Retention Policy	Potential Justification
Marketing and advertising	Delete upon deletion request	None for marketing use
Customer accounts	Retain 30 days - 18 months after account closure	Legitimate interest (account recovery, service quality, dispute resolution)
Security audit logs	Retain 1-2 years after log timestamp	Legitimate interest (security); legal obligation in some jurisdictions
Analytical datasets	Anonymize upon deletion request	No longer personal data if anonymization is effective
Financial transaction records	Retain 3-7 years after account closure	Legal obligation (tax, accounting regulations)
Legal hold records	Retain until hold lifted	Legal obligation (litigation, regulatory investigation)
ML/AI training datasets	Exclude deleted/suppressed records; retrain periodically from upstream sources	Legitimate interest (model improvement), subject to data minimization

Additional considerations:

Already trained ML/AI models may retain implicit knowledge about personal data subject to deletion requests.
Deletion obligations extend to third-party data processors, and organizations must propagate deletion requests to them. However, it may be difficult or impossible to prove deletion by external vendors.
There is rarely a clean way to wipe single records from data archives or rolling backups. A common pattern is to configure lifecycle rules to automatically delete data over an age threshold. Preventing resurrection of personal data is a larger problem we will discuss in a future article.

Retention does not allow unrestricted processing

Retention does not provide a blanket justification for continued use of personal data.

Even when there is a legal basis to retain records for purposes such as financial reporting or fraud prevention, other systems may be obligated to stop processing the same personal data.

This is where many deletion programs break down. Services storing personal data often share that data with other systems through APIs, data exports, and event streams. Once a deletion request has been processed, organizations need to ensure that downstream systems are not continuing to use retained data for purposes that no longer have a valid legal basis.

Organizations often address this through segregated datasets, filtered exports, or other technical controls that enforce purpose-limited access to retained personal data.

For example, financial systems may need continued access to records associated with deleted customers, while marketing and personalization systems should only have access to records of active, consenting customers.

Scheduling deletion after retention requirements expire

In practice, many systems define retention periods but lack reliable mechanisms to act when those periods end, causing data to persist indefinitely by default.

To operationalize retention, organizations need to translate policy into explicit deletion triggers tied to either time or business events.

Common patterns include:

Time-based expiry (eg. 7 years after account closure, 18 months after ticket resolution)
Event-based expiry (eg. account closure, transaction finalization, legal hold removal)
System lifecycle rules that automatically purge data over a certain age

At a minimum, retention must be treated as an executable rule, not just a policy definition. Without this, retention expiration becomes unreliable and enforcement depends on ad-hoc cleanup rather than predictable system behavior.

Demonstrating compliance

Once deletion and retention behavior is defined and implemented across systems, an equally difficult problem is proving it actually happened. In practice, many compliance gaps stem from the inability to consistently verify outcomes across distributed systems.

Demonstrating compliance requires more than a single “deletion successful” response. Different systems produce different evidence, and that evidence needs to be traceable back to the original request and retention requirement. In many organizations, that evidence is generated by different teams, stored in different systems, and retained for different periods of time.

At a minimum, organizations need to maintain:

Request traceability: a link between the deletion request and every system that was expected to act on it.
Execution evidence: logs or records showing what action each system took (delete, suppress, restrict, or retain).
Retention justification: explicit recording of why data was retained where deletion did not occur (eg. legal obligation, audit requirement, or active legal hold).
Expiry verification: proof that time-based or event-based retention rules were eventually enforced.

This evidence is often distributed across APIs, data pipelines, and storage systems that were not designed with coordinated auditability in mind. As a result, compliance verification tends to rely on aggregation layers rather than individual system logs.

In more mature implementations, organizations treat deletion requests as auditable workflows rather than point-in-time operations. Each step — propagation, enforcement, and retention exception handling — is recorded in a way that can be reconstructed later for audit, regulatory inquiry, or internal validation.

Even when audit trails demonstrate that systems acted on a deletion request and reported success, this does not prove that personal data is absent or no longer subject to processing. Proving absence is a substantially harder problem because it requires validating that personal data cannot be found in any systems where it should have been removed, while distinguishing those systems from others where retention is justified.

Ultimately, compliance in deletion programs is less about proving that data was deleted everywhere at once, and more about demonstrating that each system behaved consistently with its defined retention and processing obligations over time.

Posts in this series

Why data deletion is still an unsolved infrastructure problem
Why deletion means different things in different systems
Gaps in data deletion verification and auditability
(Current post) Deletion is not always deletion: retention exceptions and competing obligations
Deleted and back again: mitigating unintended data resurrection
Model a deletion graph, but don’t execute it

Gaps in data deletion verification and auditability

Sat, 13 Jun 2026 04:30:00 +0000

All systems reported “Deletion succeeded” after Kathy submitted a data deletion request on the company website last month. Yet she is still receiving personalized marketing emails, internal services are still passing around her data, and her order history still appears in business analytics reports.

Something has clearly gone wrong.

A deletion request is often treated as complete once a workflow reports success, but that signal doesn’t mean data has been removed from all infrastructure or excluded from all processing activities.

Most systems can confirm they executed a deletion request. Very few can confirm whether all personal data associated with a customer was actually removed everywhere it existed and remained absent over time.

That gap between “operation completed” and “data is fully removed from processing” is where most deletion guarantees fail in practice.

What was tracked as success

In distributed systems, a deletion request typically passes through an orchestrated workflow:

a central service receives the request
tasks are fanned out to onboarded backend systems
each system executes local deletion logic
success/failure signals are reported back to the orchestrator

At the end of this flow, the system reports success, but with a narrow interpretation:

the request was accepted
all tasks were executed or acknowledged
no immediate errors were returned

This doesn’t mean that the customer’s data has been removed from processing across all systems.

Deletion is treated as an operation, not a system-wide, verifiable state.

An idealized system model

A fully verifiable deletion program would require:

A complete and continuously updated inventory of systems storing or processing personal data
Lifecycle state of each system’s deletion capabilities and onboarding status
End-to-end visibility into deletion requests across all systems
Backfill of prior deletion requests for onboarded systems
System-wide prevention of reintroduction of deleted identities

Why this model does not exist in practice

Each component is reasonable in isolation, and many organizations already have partial implementations: system catalogs, deletion workflows, onboarding processes, and audit logs. However, end-to-end provability remains unsolved, with structural gaps in environments where infrastructure is continuously evolving.

1. System inventories drift from reality

System inventories are frequently incomplete and not automatically maintained, especially those specific to privacy expectations. New services appear without formal registration with privacy workflows, new data flows emerge through API integrations and ad-hoc data pipelines, and third-party integrations bypass internal catalogs.

As a result, system coverage both lags reality and becomes stale even for existing services. Teams may accurately attest to their current state and start processing new types of customer data tomorrow.

2. Deletion capabilities are inconsistently deployed

Deletion support is typically added incrementally across services, often focused on primary data storage while excluding derived datasets and downstream data pipelines.

This creates a persistent mismatch between documented and actual enforcement.

3. Deletion state is local, not system-wide

Even where deletion workflows exist, state is local. A system can confirm it executed a deletion request, but has no visibility into whether upstream and downstream systems have done the same.

This is a direct architectural outcome of how distributed systems are designed for local correctness and independent development.

4. Deletion history is fragmented

Backfilling prior deletion requests after onboarding assumes a replayable deletion request record and a complete history of request fulfillment across systems.

Without this history, newly onboarded systems may have no reliable mechanism to identify customers whose data should already have been deleted.

The deletion request record is often easier to centralize than fulfillment evidence, which is often fragmented across systems due to inconsistent logging, deletion semantics, and retention policies.

5. Deletion does not prevent future processing

Preventing reintroduction of deleted data is not centrally enforceable. It must hold across all platforms, pipelines, warehouses, event systems, and API integrations, each with different semantics for processing personal data.

A customer deleted today can reappear tomorrow through historical imports, replicated datasets, replayed event streams, or integration with systems not onboarded to deletion.

Typical verification approaches

Sampling-based checks
Per-system logs that may differ in format, granularity, and completeness
Best-effort reconciliation reports
Manual audits during compliance review

None provide systemic guarantees. More fundamentally, proving the absence of data is often harder than proving that a deletion workflow executed successfully.

As a result, organizations often rely on process evidence rather than proof of outcome. Auditors and regulators may be shown that deletion workflows executed successfully, even when there is limited visibility into whether personal data was removed from all systems that process it.

When a customer finds evidence that their data was not actually deleted, this can trigger regulatory scrutiny, corrective actions, and potentially significant financial penalties.

Toward a real verification model

Automated inventories and compliance tracking

We need continuous discovery of systems and datasets containing personal data, and unified deletion state tracking. This is an incremental process that may initially only cover a few technologies where automation is easiest.

Many organizations may rely on periodic manual processes where service owners attest to new personal data processing use cases and follow guidance on privacy expectations.

Lowering compliance effort through automation and tooling

Compliance outcomes improve when we have automatic tracking and accountability for deletion expectations, and low-effort mechanisms for onboarding datasets to service-appropriate deletion mechanisms.

This is not one-size-fits-all, as each system may have different deletion semantics, such as hard deletion, retention but suppression from downstream processing, scheduled archival and eventual deletion, or anonymization.

Organizations can achieve more consistent deletion outcomes when reusable tooling and patterns exist for common deletion strategies.

Standardizing proof of deletion

Global deletion state requires thorough, continuous detection and tracking of all datasets that store personal data, and per-system-and-request deletion statuses.

Verification also requires clearly defined deletion semantics for each system, since the expected outcome may be hard deletion, anonymization, suppression from downstream processing, or retention under a legal exception.

Standardized traceability of per-system request fulfillment and deletion semantics provides a stronger foundation for demonstrating deletion outcomes.

Enforcing exclusion of data from processing

There is no simple, universal solution to enforce that data subject to deletion is never processed outside of legally permitted exceptions.

However, we can get closer to the expected state when all systems that store personal data subscribe to and backfill deletion requests, and when systems that retain personal data that is subject to deletion implement access controls to suppress use by downstream services.

Closing thoughts

Privacy tooling often focuses on answering where personal data lives, and whether a deletion request was executed across the owning services. A larger challenge is determining whether personal data remains absent from all systems where it should no longer be processed.

As architectures evolve, new systems, integrations, and data flows continually expand the scope of deletion obligations. Verification must therefore be continuous rather than point-in-time.

Without system-wide visibility into deletion state and enforcement outcomes, organizations are left validating workflow execution rather than deletion itself, leaving them unable to determine whether customers like Kathy are still being processed.

Posts in this series

Why data deletion is still an unsolved infrastructure problem
Why deletion means different things in different systems
(Current post) Gaps in data deletion verification and auditability
Deletion is not always deletion: retention exceptions and competing obligations
Deleted and back again: mitigating unintended data resurrection
Model a deletion graph, but don’t execute it

Why deletion means different things in different systems

Thu, 04 Jun 2026 04:30:00 +0000

A customer submits a data deletion request. The deletion workflow executes across dozens of systems, and every system reports success.

Yet weeks later, the customer’s records still exist in a data warehouse, a search index, database backups, and a fraud investigation system.

At first glance, this looks like a deletion failure. However, every system may have behaved exactly as designed. The problem is that “delete” meant something different in each environment, and the organization may struggle to explain the discrepancy during an audit.

Deletion is not a single operation

Privacy requirements are typically expressed as a desired outcome, such as removing personal data that should no longer be retained or processed.

Translating this into technical requirements is difficult because organizations store data across systems designed for different purposes, such as:

Transaction processing
Search
Analytics and machine learning
Logging and observability
Archival storage

A transactional database may physically remove records or mark them as inactive to preserve referential integrity. A search index may remove documents asynchronously and update indexes over time. A data warehouse may rewrite partitions or de-identify records containing personal data. An event stream may represent deletion through tombstone events and downstream compaction.

As a result, the same deletion request can produce different outcomes across systems, depending on how each system interprets what deletion should do.

Common deletion patterns

These outcomes typically fall into a small number of recurring deletion patterns.

Hard deletion: Data is physically removed from the system.

Most aligned with intuitive expectation of deletion
Conceptually simple
Can introduce referential integrity or recovery challenges

Soft deletion: Data remains in the system but is marked as inactive or deleted.

Common in transactional systems
Supports recovery
Data still physically exists

Soft deletes can become problematic if the personal data is still readily available to internal teams.

Tombstoning: A deletion marker replaces the original record to signal removal.

Common in event streams and distributed databases
Supports synchronization across eventually consistent systems
Original data may persist until compaction processes run

Anonymization or de-identification: Identifiers are removed or transformed so that records can no longer be linked back to the user.

Original identity no longer recoverable
Data may remain useful for analytics and reporting

Organizations may use anonymization or de-identification when full deletion is not required or feasible. However, preserving analytical utility while removing all identifying information can be difficult. Re-identification risks increase when joining datasets across systems.

Key destruction (crypto-shredding): Encryption keys are deleted so that data can no longer be decrypted.

Common in cloud architectures
Encrypted data remains stored
Access is prevented via inability to decrypt the data rather than through physical deletion
Potentially acceptable only where resulting data is beyond recovery

Suppression: Data is retained but excluded from operational processing.

Common in marketing and advertising
Typically implemented via fine-grained access controls

Comparison of deletion patterns and their desired outcomes

Deletion pattern	Original data retained?	Recoverable?	Desired outcome
Hard delete	No	No	Data should no longer exist
Soft delete	Yes	Yes	Data may need to be restored
Tombstone	Temporarily	Sometimes	Deletion must be communicated to other systems
Anonymization	Partially	No	Identity should be removed while preserving analytical value
Key destruction	Yes	No	Data should become permanently inaccessible
Suppression	Yes	Yes	Data must be retained but not actively used

Why inconsistent deletion semantics create operational problems

The existence of multiple deletion models is not inherently problematic.

The problem arises when organizations assume they are equivalent.

Example 1: Mismatched hard vs soft deletion expectation

Privacy team expects deletion.
System performs soft delete.
Downstream systems continue to process soft-deleted data due to inconsistent enforcement of deletion semantics.
Result: Data remains accessible internally, against intended policy.

Example 2: Suppression inconsistently applied across data use cases

Customer requests deletion.
Marketing platform suppresses customer data from email campaigns.
Personal data remains accessible to other operational systems.
Result: Teams disagree on whether preventing their system’s use of data is equivalent to deleting personal data.

Example 3: Anonymized data re-identified in derived datasets

Privacy team expects customer personal data to be removed.
Analytics platform de-identifies records rather than deleting.
Data scientists continue querying aggregated datasets.
Privacy reviewers discover that individuals become re-identified when datasets are joined.
Result: Teams disagree on cross-system anonymization requirements.

Example 4: Data reappearance via event replays

Source system deletes data.
Replaying historical pipeline events rehydrates AKA recreates deleted records.
Systems lack mechanisms to prevent or remove rehydrated data after replay.
Result: Systems differ on whether deletion should prevent historical state from being reintroduced, resulting in an enforcement gap.

Key insight: Many deletion failures are semantic mismatches rather than execution failures.

Why this matters for automation

Organizations often want standardized workflows and automated orchestration and reporting.

However, in order for a deletion platform to answer “Did deletion succeed?”, it must first answer: “What was the expected outcome for this system?”

Instead of treating deletion as a collection of service-specific scripts, organizations may need a way to declare:

What deletion means for each system
Which deletion model applies
How success is verified
What evidence should be collected

The challenge is explicitly defining deletion semantics, ensuring each system enforces and verifies the intended deletion mechanism, and validating that personal data has been removed across the organization.

Posts in this series

Why data deletion is still an unsolved infrastructure problem
(Current post) Why deletion means different things in different systems
Gaps in data deletion verification and auditability
Deletion is not always deletion: retention exceptions and competing obligations
Deleted and back again: mitigating unintended data resurrection
Model a deletion graph, but don’t execute it

Why data deletion is still an unsolved infrastructure problem

Sun, 31 May 2026 04:30:00 +0000

A customer submits a data deletion request that propagates across backend services. One service only partially deletes the customer’s records due to a bug. Another misses the request because it was down for maintenance, and several analytics datasets were never integrated into the deletion process.

Months later, the customer continues receiving highly personalized marketing emails from the company. She submits a data access request and discovers that the company still has her name, address, phone number, purchase history, and detailed behavioral data from years of interactions with their services.

What began as a routine privacy request has now become a regulatory problem. A complaint to a data protection authority leads to an investigation, revealing an inability to demonstrate consistent deletion across the company’s systems. The company now faces corrective action and potential penalties.

Many of these failures were invisible to the company until the compliance team had to prove that deletion had actually occurred everywhere.

A typical deletion request path

A single request initiated from a customer-facing application may propagate across multiple services, storage systems, analytical platforms, and downstream consumers. Each system may implement deletion differently, operate on different schedules, or fail independently.

No single team owns the entire lifecycle. Application teams own services, platform teams operate infrastructure, analytics teams maintain data pipelines and data warehouses, and additional systems produce or consume derived datasets.

The problem becomes significantly harder at enterprise scale. Large organizations may operate hundreds of independently managed services and thousands of datasets while continuously adding new data flows and third-party integrations.

Over time, data lineage becomes more difficult to track, inventories become harder to maintain, and deletion workflows diverge across teams. What begins as a straightforward compliance requirement gradually becomes a coordination problem spanning large portions of an organization’s infrastructure.

The hidden complexity behind a “simple” deletion request

From a user’s perspective, deleting personal data sounds simple.

Inside a modern organization, the same request can require teams to:

Locate all systems containing relevant customer data
Resolve identifiers across different schemas and services
Execute system-specific deletion logic
Verify completion across distributed components
Handle retries and partial failures
Produce audit evidence
Prevent downstream recreation of deleted data

Customer data rarely exists in a single form. The same individual may appear as a primary key in a transactional database, an attribute in an analytics platform, or within large object-store datasets shared across many users. Deletion often requires system-specific logic tailored to how each platform stores and processes data.

The core difficulty is coordinating consistent execution across systems with different deletion semantics, retention policies, operational constraints, and failure modes.

The execution gap in privacy tooling

The privacy technology ecosystem has matured significantly. Organizations now have access to tools for privacy request management, governance, and data discovery.

These tools address important parts of the problem, but they do not solve execution.

Governance platforms help define what should happen. Cloud-native capabilities help execute deletion within individual systems. Between those layers is the challenge of executing and validating deletion across production systems.

Organizations often fill this gap with scripts, service-specific workflows, and operational procedures that become increasingly difficult to maintain, validate, and audit as independently managed services proliferate and coordination requirements compound across teams.

Why right-to-erasure is uniquely challenging

Among privacy requirements, right-to-erasure (“the right to be forgotten”) places some of the greatest demands on an organization’s architecture.

Unlike many privacy obligations, deletion cannot be satisfied through policy alone. Organizations must coordinate actions across all systems where customer data resides and demonstrate that those actions completed as intended.

This exposes architectural inconsistencies that might otherwise be missed:

Systems that interpret deletion differently
Incomplete or unverifiable audit trails
Service-specific workflows that are difficult to maintain
Fragile operational dependencies
Manual processes that don’t scale

Right-to-erasure is therefore more than a privacy requirement. It is a stress test of an organization’s ability to manage data consistently across its infrastructure.

Privacy as an architectural capability

The industry has largely approached privacy as a governance problem.

However, governance alone does not execute deletion, validate outcomes, recover from failures, or provide consistent guarantees across distributed systems.

Privacy obligations increasingly depend on infrastructure capabilities that most organizations were not built with.

That means treating privacy as a first-class architectural concern rather than a collection of scripts and workflows. Organizations need a way to define, execute, and verify deletion consistently across systems.

What comes next

If deletion is fundamentally an infrastructure problem, the next step is understanding how and why it fails in practice at scale.

In upcoming posts, we’ll explore common failure modes in deletion programs, including inconsistent deletion semantics, coverage gaps as architectures evolve, resurrection of previously deleted data, and gaps in verification and auditability.

After that, we’ll examine the structural properties of deletion programs that allow organizations to consistently orchestrate, execute, and verify deletion across distributed systems.

Posts in this series

(Current post) Why data deletion is still an unsolved infrastructure problem
Why deletion means different things in different systems
Gaps in data deletion verification and auditability
Deletion is not always deletion: retention exceptions and competing obligations
Deleted and back again: mitigating unintended data resurrection
Model a deletion graph, but don’t execute it

Polymorphic JSON deserialization with Java sealed interfaces and Jackson

Sat, 16 May 2026 04:30:00 +0000

When building a JSON-based API that accepts multiple request types, you need a strategy for mapping incoming payloads to the appropriate data model. Java sealed interfaces combined with Jackson’s polymorphic type annotations provide a clean way to support multiple strongly typed backend data models.

Problem

Consider an API that receives data processing requests. Each request provides a requestType field that determines which fields are relevant.

All requests share the same lifecycle and processing steps, but each request type corresponds to a different data model with different required and optional fields.

For example:

Account deletion: {"requestId": "request-1234", "requestType": "deleteAccount", "accountId": "1234"}
Scoped data deletion: {"requestId": "request-1234", "requestType": "deleteScopedData", "accountId": "1234", "scope": {"deletePurchaseHistory": true, "deleteSubscriptions": false, "deletePreferences": false}}

Approach: Sealed interface + Jackson annotations

Step 1: Define sealed interface with annotations mapping request types to implementations

import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;

/**
 * Polymorphic deletion request interface that routes to request-type-specific data models.
 */
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "requestType")
@JsonSubTypes({
    @JsonSubTypes.Type(value = AccountDeletionRequest.class, name = "deleteAccount"),
    @JsonSubTypes.Type(value = ScopedDataDeletionRequest.class, name = "deleteScopedData")
})
public sealed interface DeletionRequest
        permits AccountDeletionRequest, ScopedDataDeletionRequest {
    // Define common method declarations and static methods here.
}

Java interfaces allow us to keep core business logic independent of child implementation details, making the system easier to test and extend. Sealed interfaces, available in Java 17+, restrict which classes are allowed to implement the interface via a permits clause.

When combined with Jackson @JsonTypeInfo and @JsonSubTypes annotations which wire the JSON discriminator field (here, requestType) to the appropriate implementation, this enables a polymorphic contract with automatic, compile-time-checked JSON routing to strongly typed implementations.

Step 2: Define concrete implementations as classes or records

In the following implementations of DeletionRequest, we set @JsonIgnoreProperties(ignoreUnknown = true) to ignore unmentioned attributes when deserializing JSON strings matching known request types.

This allows us to maintain strong typing for attributes relevant to this service while being agnostic of other upstream attributes our service doesn’t use.

You can drop @JsonIgnoreProperties if you prefer to enforce the complete possible schema of input JSON strings, which will throw a runtime exception when receiving any new attribute you haven’t defined.

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

/**
 * Data model for account deletion requests.
 */
@JsonIgnoreProperties(ignoreUnknown = true)
public record AccountDeletionRequest(
    String accountId,
    String requestId
) implements DeletionRequest {
}

/**
 * Data model for scoped data deletion requests
 */
@JsonIgnoreProperties(ignoreUnknown = true)
public record ScopedDataDeletionRequest(
    String accountId,
    String requestId,
    DeletionScope scope
) implements DeletionRequest {
}

/**
 * Data model for data deletion scope.
 *
 * Used when user requests deleting specific categories of data.
 */
public record DeletionScope(
    boolean deletePreferences,
    boolean deletePurchaseHistory,
    boolean deleteSubscriptions
) {}

Step 3: Deserialize input JSON strings to strongly typed data models

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.exc.InvalidTypeIdException;

// Note: Prefer to instantiate ObjectMapper once as static final field or singleton and reuse across classes.
// ObjectMapper is thread-safe, and it's expensive to create new instances.
final ObjectMapper mapper = new ObjectMapper();

final DeletionRequest deletionRequest;
try {
    deletionRequest = mapper.readValue(inputJson, DeletionRequest.class);
} catch (final InvalidTypeIdException e) {
    // Replace BadRequestException in catch statements with locally relevant class for 400 Bad Request exceptions
    throw new BadRequestException("Unsupported request type", e);
} catch (final JsonProcessingException e) {
    throw new BadRequestException("Invalid request payload", e);
}

With Java 21+, the compiler enforces that every permitted subtype in switch statements on sealed interface instances are handled, removing a class of potential runtime errors.

When to use a polymorphic API

A single endpoint accepting multiple JSON schemas can make sense when all request types share the same lifecycle and infrastructure (authorization, queues, auditing) and only diverge in execution logic.

Prefer separate endpoints when request types have different authorization rules, rate limits, or lifecycle rules, or when you need clean OpenAPI/Swagger/Smithy documentation, which don’t work well with polymorphic schemas.

Polymorphic APIs are more difficult for clients to consume and understand compared to stand-alone APIs, so they are a better fit for internal APIs between services your team controls.

When to use sealed interfaces to implement polymorphic APIs

Sealed interfaces and Jackson polymorphic type annotations are a good fit for implementing polymorphic schemas when all the following are true:

Your JSON payloads share a discriminator field but have different schemas per type.
You want a closed set of permitted subtypes.
You want subtypes to be strongly typed, each potentially requiring its own validation logic.

They are a poor fit when either of the following is true, in which case abstract classes may be a better replacement for sealed interfaces:

Other teams need to be able to add subtypes without modifying your code (sealed interface subtypes must be defined in the same module).
You cannot use Java 17+, the first Long Term Support version supporting sealed interfaces.

Pros and cons vs common alternatives

Strategy	Pros	Cons
Sealed interface + Jackson polymorphism	Supports strongly typed subtypes with different required and allowed fields Easily add subtypes with minimal code changes Compile-time rejection of unsupported subtypes Clean pattern matching without casting	Requires all subtypes to be defined in same module Requires Java 17+
Single class with nullable fields	Simple Works pre-Java 17	Runtime null checks everywhere No compile-time safety Unclear which fields apply to each type More messy to enforce validations on subtypes
Inheritance with abstract class	Familiar OOP pattern Subclasses can live anywhere Works pre-Java 17	Anyone can subclass, no compile-time restriction on subtypes Compiler does not enforce exhaustive handling of subtypes
`@JsonAnySetter` with `Map<String, Object>`	Maximum flexibility Allows for unknown schemas Works pre-Java 17	Zero type safety, accepts all inputs Validation is manual and error-prone
Enum and factory method	Explicit type registry (as with sealed interface) More flexible, full control over deserialization Works pre-Java 17	More boilerplate code than sealed interface + annotations No compile-time validation of correct/complete routing

Key takeaways

sealed provides a closed type hierarchy so the compiler knows every possible implementation.
Jackson’s @JsonTypeInfo simplifies routing deserialization to the appropriate type based on a discriminator field.
Java records minimize boilerplate for immutable, validated data models.

In practice, this pattern has reduced the effort to support new data processing types from roughly a week per type to a few hours. The sealed interface and Jackson annotations handle the routing, leaving only type-specific models and logic to implement.

Enabling ad hoc runs of queue-based ECS services

Sat, 04 Apr 2026 00:00:00 +0000

Background

A common ECS use case is to continually poll a queue for new events to process, with auto-scaling based on CPU or memory utilization, or the size of the queue backlog. These services can be tricky to build reliable integration tests for, as queue backlogs may result in long delays before a new message is processed.

A simple way to enable efficient integration tests for these services is to support an optional environment variable that signals the service to immediately process the message without long-polling the queue, as if it were the content of a single SQS message body.

Example code

public static void main(String[] args) {
    String jsonRequest = System.getenv("ADHOC_REQUEST");
    if (jsonRequest != null && !jsonRequest.isEmpty()) {
        // Execute the single JSON request
        executeRequestFromJson(jsonRequest);
    } else {
        // Continually poll a queue to process pending requests
        longPollQueue();
    }
}

static void executeRequestFromJson(String jsonRequest) {
    MyParsedRequest request = parseRequestFromJson(jsonRequest);
    executeRequest(request);
}

static void longPollQueue() {
    while (true) {
        Optional<MyQueueMessage> message = getMessageFromQueue();
        if (message.isPresent()) {
            MyParsedRequest request = parseRequest(message.get());
            executeRequest(request);
            acknowledgeSqsMessage(message.get());
        }
    }
}

Example execution via AWS CLI

After setting variables for the AWS region and ECS cluster, service, and task definition ARNs, you can run a variation of the following to run a new ECS task immediately, without waiting for a new message to be processed in a queue that may have a large backlog of pending messages.

YOUR_ECS_CONTAINER_NAME should be replaced by the name field value in the task definition’s containerDefinitions.

The ECS container overrides allow you to provide specific environment variable overrides while maintaining other existing environment variables from the ECS task definition.

networkConfig=$(aws ecs describe-services \
    --region $awsRegion \
    --cluster $ecsClusterArn \
    --services $ecsServiceArn \
    --query 'services[0].networkConfiguration' \
    --output json)
aws ecs run-task \
    --region $awsRegion \
    --cluster $ecsClusterArn \
    --task-definition $ecsTaskDefinitionArn \
    --launch-type FARGATE \
    --network-configuration $networkConfig \
    --overrides '{
        "containerOverrides": [{
            "name": "YOUR_ECS_CONTAINER_NAME",
            "environment": [{
                "name": "ADHOC_REQUEST",
                "value": "{\"yourRequestKey\":\"yourRequestValue\"}"
            }]
        }]
    }'

Depending on the integ test environment, you can also use the ECS RunTask API in a similar way as the above CLI command.

After implementing this pattern in your service code, your integration tests can run ad hoc ECS tasks with specific test requests and validate the expected output - for example, periodically calling another API to validate a record has been successfully created or updated after a wait time, with a maximum number of retries.

Since the task processes only the injected request, tests can use a much shorter max duration than they would need if waiting for a large queue backlog to drain.

Strategies for querying periodic S3 data snapshots

Sat, 31 May 2025 00:00:00 +0000

Background

A common AWS analytics use case is making aggregate queries across multiple data sets stored in S3. For example, one partner team may store product metadata, while another team stores purchase order metadata, and we may want to join these data sets to determine which products are most popular across each marketplace.

In this post I’ll cover a few options for syncing data to S3 and retrieving data snapshots for use in aggregate queries, and specifically discuss the use case where we need to maintain access patterns to complete, recent data snapshots without partial unavailability during data syncs.

A few options for syncing data to S3

1. Syncing each record to a unique, stable file path

Pros:

Provides a stable “latest” data set that always represents all records.
Enables O(1) look-ups of specific records if data consumers query by S3 key (“filepath”).
Low storage costs since only a single copy of each record is stored.

Cons:

Poor performance and higher costs for aggregate queries at scale. Retrieving tens of thousands of small files is much slower than retrieving a few multi-MB files.
Limits design choices for data providers and may increase complexity of their implementation (eg. AWS Glue defaults to writing aggregated partial result files).
No historical data is retained unless explicitly backed up elsewhere.

This is best suited to when data consumers retrieve specific records, only need their latest state, and don’t make aggregated queries across a large number of records.

2. Appending all events as new data without overwriting prior events

Pros:

Provides a complete history of events, allowing for time-series analysis.
If partition data by time, supports efficient queries of specific time periods.

Cons:

Increase storage costs as accumulate historical data.
Increased complexity, latency, and cost to retrieve the latest version of each record.
If data isn’t partitioned in a way that aligns with query use cases, may have very inefficient scans.
If upstream workflows fail, may not have a way to recover data, and have missing records.

This is best suited to queries of time-partitioned events, rather than needing the latest state of records.

3. Overwriting one or more files that include multiple records

Pros:

Consumers always access the most up-to-date version of the dataset.
More efficient aggregate queries than retrieving thousands of single-record files.
Low storage costs since only a single copy of each record is stored.

Cons:

Risk of data loss if a failure occurs during the overwrite process.
Data consumers querying during a data sync may receive incomplete or duplicate data.
No historical data is retained unless explicitly backed up elsewhere.

This can work for aggregated queries of the latest records, but if we can’t accept the risk of corrupt/partial/duplicate data during sync failures or read/write race conditions, we may prefer writing to separate snapshot partitions.

4. Periodically writing complete data to time-based partitions

Pros:

Consumers always access a complete dataset when querying a completed/past partition.
More efficient aggregate queries than retrieving thousands of single-record files.
Maintain historical data for as long as prior time partitions are kept in storage.

Cons:

More complex to identify the latest complete partition.
Data consumers querying the most recent partition during a data sync may receive incomplete data. Need some type of completeness signal to mitigate.
Increase storage costs as accumulate historical data.

We can mitigate storage costs by setting a lifecycle rule to automatically delete S3 files past a certain age, eg. auto-delete files over 2 weeks old, if we only need to query recent snapshots.

Case study: Aggregate queries where completeness is important

For this post, we’ll consider the case where we want to optimize for aggregated queries against hundreds of thousands of records, with filter criteria applied across all records. Our business requirements are that data consumers must always have access to complete and accurate data (no duplicates or missing records), but data does not need to be real-time as long as it’s up to date within a few hours.

In this case, single-record files and time-series events are not a good fit due to increased latency to query latest state across this number of records. We may prefer time-based partitions with complete data written to a new partition every N hours, eg. hourly, to mitigate partial/duplicate data issues during concurrent read/write operations.

A few options for querying time-partitioned snapshots

A common challenge for time-partitioned partitions is identifying the latest complete partition, especially if upstream data syncs are not 100% successful.

1. Retrieve a specific snapshot relative to the current time

Assuming we have hourly snapshots that are partitioned by snapshot_date and snapshot_hour, where snapshot_date is formatted as "2025-01-31" and snapshot_hour is formatted as "01" or "23" to support consistent string comparisons, the following Athena SQL query retrieves data from the last hour’s time partition:

SELECT * FROM upstream_dataset
WHERE snapshot_date = CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%Y-%m-%d') AS VARCHAR)
AND snapshot_hour = CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%H') AS VARCHAR);

Pros:

Simple, easy to understand.
Very efficient since only querying data from a single partition, and it’s O(1) to locate that partition.

Cons:

May fail to get any data if there were upstream sync issues for the selected partition. This is a blocker for us.
May get partial data if query the most recent partition during an ongoing sync.
May need to query older and more out-of-date partitions to avoid the above race condition.

2. Retrieve most recent snapshot

We can improve on the prior option by using an initial query to identify the most recent snapshot partition that has data.

Since it could be expensive to search across all partitions, we can set a maximum look-back period, for example, the last 7 days, to allow for occasional upstream failures that may take a few days to resolve.

Example Athena SQL query:

WITH MostRecentSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN MostRecentSnapshotPartition
    ON upstream_dataset.snapshot_date = MostRecentSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = MostRecentSnapshotPartition.snapshot_hour;

Pros:

Relatively simple and easy to understand.
Guaranteed to get most recent snapshot with data within the given look-back period.

Cons:

May get partial data if query the most recent partition during an ongoing sync.

3. Retrieve the second most recent snapshot

If we need to mitigate race conditions where the latest partition may have partial data, we could always query for the second most recent partition.

We can maintain the same maximum look-back period to limit the query search space for performance reasons, while accounting for a few days of upstream failures.

Example Athena SQL query:

WITH RecentSnapshotPartitions AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    -- Get the two most recent partitions with data
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 2
),
RankedSnapshotPartitions AS (
    SELECT
        snapshot_date,
        snapshot_hour,
        -- Set a row number so we can select the second most recent partition
        ROW_NUMBER() OVER (ORDER BY snapshot_date DESC, snapshot_hour DESC) AS recency_rank
    FROM RecentSnapshotPartitions
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN RankedSnapshotPartitions
    ON upstream_dataset.snapshot_date = RankedSnapshotPartitions.snapshot_date
    AND upstream_dataset.snapshot_hour = RankedSnapshotPartitions.snapshot_hour
    AND RankedSnapshotPartitions.recency_rank = 2;

Pros:

If the second most recent partition is always complete, this guarantees complete data.

Cons:

More complex and difficult to understand than other options.
Less efficient than other options.

4. Retrieve the most recent snapshot older than the period between data syncs

If we need to account for temporary upstream failures and cannot accept race conditions where the most recent partition may sometimes be complete, we can simplify from Option 1 by querying for the most recent partition older than the time between syncs, eg. the most recent partition more than 1 hour old.

WITH RecentCompleteSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Get snapshots from more than 1 hour ago, to avoid querying a partition with an ongoing sync if have race conditions near the start of an hour
    WHERE snapshot_date <= CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%Y-%m-%d' AS VARCHAR))
    AND snapshot_hour < CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%H') AS VARCHAR)
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    AND snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN RecentCompleteSnapshotPartition
    ON upstream_dataset.snapshot_date = RecentCompleteSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = RecentCompleteSnapshotPartition.snapshot_hour;

Pros:

If populated partitions that are older than the period between syncs are always complete, guarantees complete data.
Simpler and more efficient than Option 3, while covering the flaws of Options 1-2.

Cons:

Slightly more complex query than Options 1-2.

5. Send data completeness events to trigger downstream queries

If we can request that our data provider pushes a data completeness signal file such as an empty file named _SUCCESS to the latest S3 directory after completing a data sync, we can set up S3 PutObject notifications that filter for this specific filename and trigger downstream workflows whenever this signal is received.

This is ideal for event-based workflows that only need to listen for a single trigger, while it may not be sufficient if there are multiple upstream datasets that we need to query that have different schedules and may not all be complete at the same time.

Pros:

Guarantees completeness of the given data set at the time we receive the event.
Allows downstream workflows to run with the most recent data possible, as soon as data is received.

Cons:

More complex to support multiple upstream data sets, best for single-dependency workflows.
May not work with workflows that are strictly schedule-based and cannot be easily triggered by an event.

6. Leverage AWS Glue Catalog, trigger Glue Crawler from Glue job event

If we populate our S3 bucket with an AWS Glue job, we can set up a Glue Catalog and a Glue Crawler that updates the catalog with an abstracted representation of the source data and its available partitions for downstream services such as AWS Athena to query.

If we set up the Glue Catalog Table as the data source for our downstream queries, we will only query partitions that it has crawled.

We can set up a Glue trigger to automatically run the Crawler after the upstream Glue job that populates new S3 partitions has succeeded, or after some event is received via EventBridge, such as creation of a _SUCCESS file, to automatically make new partitions available only after their sync jobs have completed.

We could then use the Athena SQL query from Option 2 but with the Glue Table as our source, to simplify querying the most recent complete partition.

WITH MostRecentSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN MostRecentSnapshotPartition
    ON upstream_dataset.snapshot_date = MostRecentSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = MostRecentSnapshotPartition.snapshot_hour;

Pros:

Abstracts logic for how to identify the most recent partition from data consumers.
Enables always querying the most recent complete partition.
Avoids the incomplete/duplicate data issues from Options 1-2 and the query complexity from Options 3-4.

Cons:

Requires additional infrastructure set-up and hardware costs for Glue resources and event-based triggers.
May add several minutes of data latency from events -> Glue trigger -> Glue crawler -> Glue catalog update, compared to directly querying S3 for the most recent partition more than N hours old.

7. Leverage AWS Glue Catalog, use AWS Lambda to update a “latest” table after completeness events

If we have multiple upstream data sets which rules out Option 5, and we’re willing to invest a few developer weeks to optimize and simplify queries for data consumers, we could create a custom Lambda function that programmatically updates Glue resources to point to a “latest” S3 partition whenever a data sync completes.

We could require data providers to write an empty _SUCCESS file to the new partition after a successful sync. We can then set up a S3 PutObject notification that filters for _SUCCESS files, and make this a trigger to our Lambda function.

The Lambda function will then:

Query the S3 directory for files in the new partition and scan them to infer their partitions and table schema.
Query Glue’s UpdateTable API to update the Glue Catalog Table that represents the latest version of the snapshot-based data set, to have the latest data schema and point to the new partition.

Data consumers can then simplify their queries to the following, without needing to select snapshot partition attributes:

SELECT * FROM upstream_dataset;

Pros:

Significantly simplifies queries for data consumers.
Enables always querying the most recent complete partition.
Avoids the incomplete/duplicate data issues from Options 1-2 and the query complexity from Options 3-6.

Cons:

Adds multiple weeks of developer effort to replicate AWS Glue’s functionality for inferring schemas and updating Glue Catalog Tables.
Increases backend architecture and code complexity, with more points of failures that need to be maintained and supported.

If this works, the end result may be great from the data consumer side, but in many cases this developer time is more valuable spent on other problems, and we’re willing to accept either slightly more complex but well-documented queries for data consumers, or occasional race conditions where recent snapshots may be incomplete if we happen to be reading at the same time that a data sync is occurring.

Case study: Retrieving recent data when completeness is critical and developer time is limited

Take the case where we are limited to a few days of developer time to set up associated infrastructure, while we have the business requirements that:

Data consumers must always be able to query complete and accurate data that is no more than a few hours old. Missing or duplicate data due to read/write race conditions is not acceptable.
Data consumers must be able to aggregate data from multiple upstream sources while satisfying the above conditions for completeness without partial data if querying during data syncs.

In this scenario, Option 6 may be an acceptable tradeoff, where we leverage existing AWS Glue functionality to automatically surface new partitions and schemas after receiving a data sync completion signal, and point data consumers to the Glue Catalog Table to query partitions that have completed syncs.

Data consumers will then need to be aware of snapshot partition attributes to select data from the most recent partition, but they can simply query for the most recent partition since we will only surface partitions after receiving a data sync completion signal.

Granting AWS Organization member accounts access to Cost Explorer

Sat, 10 May 2025 00:00:00 +0000

Introduction

By default, adding accounts to an AWS Organization results in consolidated billing and cost management in the Organization management account, and Organization member accounts lose access to Cost Explorer, Billing, and other cost management services unless access is explicitly enabled from the Organization management account.

This post walks through how to allow member accounts to access cost management services, to enable each team to review and manage their AWS spending.

Enabling Cost Explorer and Cost Optimization Hub

Cost Explorer allows you to analyze AWS service costs and usage changes over time, and is free to access through the AWS UI. You can filter and group costs and usage across multiple dimensions including AWS service, resource, usage type, region, and tag.

After logging into the Organization management account using the root user, navigate to the Cost Explorer service to automatically enable it from the current time onwards. You will need to do this once per account in your Organization, logging in as the root user for each account.

Cost Optimization Hub is a free service that provides cost optimization recommendations across multiple services. To enable member accounts access, we’ll need to enable the services from the AWS Organizations management account, and explicitly add permissions to the Permission Sets granting them access.

To enable Cost Optimization Hub, navigate to the Cost Optimization Hub landing page, scroll down, select “Enable Cost Optimization Hub for this account and all member accounts”, and click “Enable”.

As indicated by the prompt, to fully benefit from this service you’ll also need to opt into the free AWS Compute Optimizer service to import service rightsizing recommendations. Follow the link to navigate to AWS Compute Optimizer, and click “Get started”.

Select to opt in all member accounts, and click “Opt in”.

Enabling access for member accounts

On the top right hand of the AWS console, select your account name to open the account dropdown, and click “Account”, or directly navigate to https://us-east-1.console.aws.amazon.com/billing/home?region=us-east-1#/account.

Scroll down to “IAM user and role access to Billing information” section and click “Edit”. Select “Activate IAM Access” and click “Update”.

Navigate to “Billing and Cost Management” > “Cost Management Preferences”.

Under the “General” tab, enable the options below, then select “Save preferences” at the bottom of the page and confirm changes.

Linked account access
Linked account refunds and credits
Linked account discounts

Under the “Cost Explorer” tab, enable “Granular data” with “Resource-level data at daily granularity”, and select “All services”, then click “Save preferences” at the bottom of the page and confirm changes.

Under the “Cost Optimization Hub” tab, under “Organization and member account settings”, select “Enable Cost Optimization Hub for all member accounts” and “Allow member account discount visibility”, then click “Save preferences” and confirm changes.

Accessing Cost Explorer from member accounts

At this point, team members logging into Organization member accounts to view Cost Explorer will still see access denied errors across all widgets, even if their IAM permissions grant access to all Cost Explorer actions.

When clicking on one of the errors, we can see the following message:

You don’t have permission to [Cost Explorer:]. To request access, copy the following text and send it to your AWS administrator. Learn more about troubleshooting access denied errors

User: [REDACTED_ACCOUNT_ID]

Service: [Cost Explorer]

Name: [AccessDeniedException]

HTTP status code: [400]

Context: [IAM user access not activated]

Request ID: [REDACTED_REQUEST_ID]

To resolve this error, we need to:

Log into the Organization member account as the root user.
Visit AWS Cost Explorer once to enable this service if we haven’t done this before.
Navigate to Account settings via the top right hand account dropdown > “Account”.
Scroll down to “IAM user and role access to Billing information” section and click “Edit”.
Select “Activate IAM Access” and click “Update”.

Non-root users with billing permissions should now have access to Cost Explorer, while the page may initially show zero costs and only show correct spending for the current time going forward.

After waiting for a few days, I noticed that the default view still shows zero spending, while after updating the time period to the last 7 days and updating the granularity to daily, I see correct costs for the past few days up to the date when Cost Explorer was enabled from the Organization management account.

We can now add any filters or groupings we like, such as grouping costs by usage type to see what specify AWS service usages are contributing to our daily spending.

Granting cross-account user groups access to Cost Explorer

We need to explicitly grant user groups access to Cost Explorer, otherwise they will be denied by default.

Users with full read permissions across all AWS services will be able to view Cost Explorer after the earlier steps in this post, but we may want to allow additional user roles access as well.

See my last post’s “Creating cross-account role-based permission groups” section for how to create a Permission Set and assign user groups these permissions on specific linked accounts.

To manage your Permission Sets, navigate to “IAM Identity Center” > “Multi-account permissions” > “Permission sets”.

Assuming you have created a Permission Set that does not have explicit permissions to Cost Explorer, and you want that group to have read access to all cost management services, select that Permission Set.

In this case, I have a SupportUser Permission Set where I want my support team to be able to view cost and usage data for their assigned accounts.

While this Permission Set only has the SupportUser policy assigned, users logging in with this role will be denied access to all Cost Explorer widgets with the error:

User: [REDACTED_ACCOUNT_ID]

Service: [Cost Explorer]

Name: [AccessDeniedException]

HTTP status code: [400]

Context: [User: arn:aws:sts::REDACTED_ACCOUNT_ID:assumed-role/REDACTED_ROLE_ID is not authorized to perform: ce:GetCostAndUsage on resource: arn:aws:ce:us-east-1:REDACTED_ACCOUNT_ID:/GetCostAndUsage because no identity-based policy allows the ce:GetCostAndUsage action]

Under the Permission Set’s “Permissions” > “Inline policy” section, select “Edit”.

Enter the following IAM policy document, or the scoped permissions you want to grant:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CostManagementViewAccess",
            "Effect": "Allow",
            "Action": [
                "account:GetAccountInformation",
                "aws-portal:ViewBilling",
                "billing:Get*",
                "billing:List*",
                "budgets:ViewBudget",
                "budgets:Describe*",
                "ce:Describe*",
                "ce:Get*",
                "ce:List*",
                "consolidatedbilling:Get*",
                "consolidatedbilling:List*",
                "cur:Describe*",
                "cur:Get*",
                "freetier:Get*",
                "sustainability:GetCarbonFootprintSummary"
            ],
            "Resource": "*"
        }
    ]
}

Scroll to the bottom of the page and save your changes.

Validate that users assigned this Permission Set on a given account now have read access to Billing and Cost Management pages.

Posts in this series

Using AWS Organizations to standardize security controls across AWS accounts
(Current post) Granting AWS Organization member accounts access to Cost Explorer

References

AWS Organizations User Guide: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html

AWS Cost Explorer Access User Guide: https://docs.aws.amazon.com/cost-management/latest/userguide/ce-access.html

AWS IAM Identity Center Permission Sets User Guide: https://docs.aws.amazon.com/singlesignon/latest/userguide/permissionsetsconcept.html