Mark Sayson

Why deletion means different things in different systems

Thu, 04 Jun 2026 04:30:00 +0000

A customer submits a data deletion request. The deletion workflow executes across dozens of systems, and every system reports success.

Yet weeks later, the customer’s records still exist in a data warehouse, a search index, database backups, and a fraud investigation system.

At first glance, this looks like a deletion failure. However, every system may have behaved exactly as designed. The problem is that “delete” meant something different in each environment, and the organization may struggle to explain the discrepancy during an audit.

Deletion is not a single operation

Privacy requirements are typically expressed as a desired outcome, such as removing personal data that should no longer be retained or processed.

Translating this into technical requirements is difficult because organizations store data across systems designed for different purposes, such as:

Transaction processing
Search
Analytics and machine learning
Logging and observability
Archival storage

A transactional database may physically remove records or mark them as inactive to preserve referential integrity. A search index may remove documents asynchronously and update indexes over time. A data warehouse may rewrite partitions or de-identify records containing personal data. An event stream may represent deletion through tombstone events and downstream compaction.

As a result, the same deletion request can produce different outcomes across systems, depending on how each system interprets what deletion should do.

Common deletion patterns

These outcomes typically fall into a small number of recurring deletion patterns.

Hard deletion: Data is physically removed from the system.

Most aligned with intuitive expectation of deletion
Conceptually simple
Can introduce referential integrity or recovery challenges

Soft deletion: Data remains in the system but is marked as inactive or deleted.

Common in transactional systems
Supports recovery
Data still physically exists

Soft deletes can become problematic if the personal data is still readily available to internal teams.

Tombstoning: A deletion marker replaces the original record to signal removal.

Common in event streams and distributed databases
Supports synchronization across eventually consistent systems
Original data may persist until compaction processes run

Anonymization or de-identification: Identifiers are removed or transformed so that records can no longer be linked back to the user.

Original identity no longer recoverable
Data may remain useful for analytics and reporting

Organizations may use anonymization or de-identification when full deletion is not required or feasible. However, preserving analytical utility while removing all identifying information can be difficult. Re-identification risks increase when joining datasets across systems.

Key destruction: Encryption keys are deleted so that data can no longer be decrypted.

Common in cloud architectures
Encrypted data remains stored
Access is prevented via inability to decrypt the data rather than through physical deletion

Suppression: Data is retained but excluded from operational processing.

Common in marketing and advertising
Typically implemented via fine-grained access controls

Comparison of deletion patterns and their desired outcomes

Deletion pattern	Original data retained?	Recoverable?	Desired outcome
Hard delete	No	No	Data should no longer exist
Soft delete	Yes	Yes	Data may need to be restored
Tombstone	Temporarily	Sometimes	Deletion must be communicated to other systems
Anonymization	Partially	No	Identity should be removed while preserving analytical value
Key destruction	Yes	No	Data should become permanently inaccessible
Suppression	Yes	Yes	Data must be retained but not actively used

Why inconsistent deletion semantics create operational problems

The existence of multiple deletion models is not inherently problematic.

The problem arises when organizations assume they are equivalent.

Example 1: Mismatched hard vs soft deletion expectation

Privacy team expects deletion.
System performs soft delete.
Downstream systems continue to process soft-deleted data due to inconsistent enforcement of deletion semantics.
Result: Data remains accessible internally, against intended policy.

Example 2: Suppression inconsistently applied across data use cases

Customer requests deletion.
Marketing platform suppresses customer data from email campaigns.
Personal data remains accessible to other operational systems.
Result: Teams disagree on whether preventing their system’s use of data is equivalent to deleting personal data.

Example 3: Anonymized data re-identified in derived datasets

Privacy team expects customer personally identifiable information (PII) to be removed.
Analytics platform de-identifies records rather than deleting.
Data scientists continue querying aggregated datasets.
Privacy reviewers discover that individuals become re-identified when datasets are joined.
Result: Teams disagree on cross-system anonymization requirements.

Example 4: Data reappearance via event replays

Source system deletes data.
Replaying historical pipeline events rehydrates AKA recreates deleted records.
Systems lack mechanisms to prevent or remove rehydrated data after replay.
Result: Systems differ on whether deletion should prevent historical state from being reintroduced, resulting in an enforcement gap.

Key insight: Many deletion failures are semantic mismatches rather than execution failures.

Why this matters for automation

Organizations often want standardized workflows and automated orchestration and reporting.

However, in order for a deletion platform to answer “Did deletion succeed?”, it must first answer: “What was the expected outcome for this system?”

Instead of treating deletion as a collection of service-specific scripts, organizations may need a way to declare:

What deletion means for each system
Which deletion model applies
How success is verified
What evidence should be collected

The challenge is explicitly defining deletion semantics, ensuring each system enforces and verifies the intended deletion mechanism, and validating that personal data has been removed across the organization.

Why data deletion is still an unsolved infrastructure problem

Sun, 31 May 2026 04:30:00 +0000

A customer submits a data deletion request that propagates across backend services. One service only partially deletes the customer’s records due to a bug. Another misses the request because it was down for maintenance, and several analytics datasets were never integrated into the deletion process.

Months later, the customer continues receiving highly personalized marketing emails from the company. She submits a data access request and discovers that the company still has her name, address, phone number, purchase history, and detailed behavioral data from years of interactions with their services.

What began as a routine privacy request has now become a regulatory problem. A complaint to a data protection authority leads to an investigation, revealing an inability to demonstrate consistent deletion across the company’s systems. The company now faces corrective action and potential penalties.

Many of these failures were invisible to the company until the compliance team had to prove that deletion had actually occurred everywhere.

A typical deletion request path

A single request initiated from a customer-facing application may propagate across multiple services, storage systems, analytical platforms, and downstream consumers. Each system may implement deletion differently, operate on different schedules, or fail independently.

No single team owns the entire lifecycle. Application teams own services, platform teams operate infrastructure, analytics teams maintain data pipelines and data warehouses, and additional systems produce or consume derived datasets.

The problem becomes significantly harder at enterprise scale. Large organizations may operate hundreds of independently managed services and thousands of datasets while continuously adding new data flows and third-party integrations.

Over time, data lineage becomes more difficult to track, inventories become harder to maintain, and deletion workflows diverge across teams. What begins as a straightforward compliance requirement gradually becomes a coordination problem spanning large portions of an organization’s infrastructure.

The hidden complexity behind a “simple” deletion request

From a user’s perspective, deleting personal data sounds simple.

Inside a modern organization, the same request can require teams to:

Locate all systems containing relevant customer data
Resolve identifiers across different schemas and services
Execute system-specific deletion logic
Verify completion across distributed components
Handle retries and partial failures
Produce audit evidence
Prevent downstream recreation of deleted data

Customer data rarely exists in a single form. The same individual may appear as a primary key in a transactional database, an attribute in an analytics platform, or within large object-store datasets shared across many users. Deletion often requires system-specific logic tailored to how each platform stores and processes data.

The core difficulty is coordinating consistent execution across systems with different deletion semantics, retention policies, operational constraints, and failure modes.

The execution gap in privacy tooling

The privacy technology ecosystem has matured significantly. Organizations now have access to tools for privacy request management, governance, and data discovery.

These tools address important parts of the problem, but they do not solve execution.

Governance platforms help define what should happen. Cloud-native capabilities help execute deletion within individual systems. Between those layers is the challenge of executing and validating deletion across production systems.

Organizations often fill this gap with scripts, service-specific workflows, and operational procedures that become increasingly difficult to maintain, validate, and audit as independently managed services proliferate and coordination requirements compound across teams.

Why right-to-erasure is uniquely challenging

Among privacy requirements, right-to-erasure (“the right to be forgotten”) places some of the greatest demands on an organization’s architecture.

Unlike many privacy obligations, deletion cannot be satisfied through policy alone. Organizations must coordinate actions across all systems where customer data resides and demonstrate that those actions completed as intended.

This exposes architectural inconsistencies that might otherwise be missed:

Systems that interpret deletion differently
Incomplete or unverifiable audit trails
Service-specific workflows that are difficult to maintain
Fragile operational dependencies
Manual processes that don’t scale

Right-to-erasure is therefore more than a privacy requirement. It is a stress test of an organization’s ability to manage data consistently across its infrastructure.

Privacy as an architectural capability

The industry has largely approached privacy as a governance problem.

However, governance alone does not execute deletion, validate outcomes, recover from failures, or provide consistent guarantees across distributed systems.

Privacy obligations increasingly depend on infrastructure capabilities that most organizations were not built with.

That means treating privacy as a first-class architectural concern rather than a collection of scripts and workflows. Organizations need a way to define, execute, and verify deletion consistently across systems.

What comes next

If deletion is fundamentally an infrastructure problem, the next step is understanding how and why it fails in practice at scale.

In upcoming posts, we’ll explore common failure modes in deletion programs, including inconsistent deletion semantics, coverage gaps as architectures evolve, rehydration of previously deleted data, and gaps in verification and auditability.

After that, we’ll examine the structural properties of deletion programs that allow organizations to consistently orchestrate, execute, and verify deletion across distributed systems.

Polymorphic JSON deserialization with Java sealed interfaces and Jackson

Sat, 16 May 2026 04:30:00 +0000

When building a JSON-based API that accepts multiple request types, you need a strategy for mapping incoming payloads to the appropriate data model. Java sealed interfaces combined with Jackson’s polymorphic type annotations provide a clean way to support multiple strongly typed backend data models.

Problem

Consider an API that receives data processing requests. Each request provides a requestType field that determines which fields are relevant.

All requests share the same lifecycle and processing steps, but each request type corresponds to a different data model with different required and optional fields.

For example:

Account deletion: {"requestId": "request-1234", "requestType": "deleteAccount", "accountId": "1234"}
Scoped data deletion: {"requestId": "request-1234", "requestType": "deleteScopedData", "accountId": "1234", "scope": {"deletePurchaseHistory": true, "deleteSubscriptions": false, "deletePreferences": false}}

Approach: Sealed interface + Jackson annotations

Step 1: Define sealed interface with annotations mapping request types to implementations

import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;

/**
 * Polymorphic deletion request interface that routes to request-type-specific data models.
 */
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "requestType")
@JsonSubTypes({
    @JsonSubTypes.Type(value = AccountDeletionRequest.class, name = "deleteAccount"),
    @JsonSubTypes.Type(value = ScopedDataDeletionRequest.class, name = "deleteScopedData")
})
public sealed interface DeletionRequest
        permits AccountDeletionRequest, ScopedDataDeletionRequest {
    // Define common method declarations and static methods here.
}

Java interfaces allow us to keep core business logic independent of child implementation details, making the system easier to test and extend. Sealed interfaces, available in Java 17+, restrict which classes are allowed to implement the interface via a permits clause.

When combined with Jackson @JsonTypeInfo and @JsonSubTypes annotations which wire the JSON discriminator field (here, requestType) to the appropriate implementation, this enables a polymorphic contract with automatic, compile-time-checked JSON routing to strongly typed implementations.

Step 2: Define concrete implementations as classes or records

In the following implementations of DeletionRequest, we set @JsonIgnoreProperties(ignoreUnknown = true) to ignore unmentioned attributes when deserializing JSON strings matching known request types.

This allows us to maintain strong typing for attributes relevant to this service while being agnostic of other upstream attributes our service doesn’t use.

You can drop @JsonIgnoreProperties if you prefer to enforce the complete possible schema of input JSON strings, which will throw a runtime exception when receiving any new attribute you haven’t defined.

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

/**
 * Data model for account deletion requests.
 */
@JsonIgnoreProperties(ignoreUnknown = true)
public record AccountDeletionRequest(
    String accountId,
    String requestId
) implements DeletionRequest {
}

/**
 * Data model for scoped data deletion requests
 */
@JsonIgnoreProperties(ignoreUnknown = true)
public record ScopedDataDeletionRequest(
    String accountId,
    String requestId,
    DeletionScope scope
) implements DeletionRequest {
}

/**
 * Data model for data deletion scope.
 *
 * Used when user requests deleting specific categories of data.
 */
public record DeletionScope(
    boolean deletePreferences,
    boolean deletePurchaseHistory,
    boolean deleteSubscriptions
) {}

Step 3: Deserialize input JSON strings to strongly typed data models

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.exc.InvalidTypeIdException;

// Note: Prefer to instantiate ObjectMapper once as static final field or singleton and reuse across classes.
// ObjectMapper is thread-safe, and it's expensive to create new instances.
final ObjectMapper mapper = new ObjectMapper();

final DeletionRequest deletionRequest;
try {
    deletionRequest = mapper.readValue(inputJson, DeletionRequest.class);
} catch (final InvalidTypeIdException e) {
    // Replace BadRequestException in catch statements with locally relevant class for 400 Bad Request exceptions
    throw new BadRequestException("Unsupported request type", e);
} catch (final JsonProcessingException e) {
    throw new BadRequestException("Invalid request payload", e);
}

With Java 21+, the compiler enforces that every permitted subtype in switch statements on sealed interface instances are handled, removing a class of potential runtime errors.

When to use a polymorphic API

A single endpoint accepting multiple JSON schemas can make sense when all request types share the same lifecycle and infrastructure (authorization, queues, auditing) and only diverge in execution logic.

Prefer separate endpoints when request types have different authorization rules, rate limits, or lifecycle rules, or when you need clean OpenAPI/Swagger/Smithy documentation, which don’t work well with polymorphic schemas.

Polymorphic APIs are more difficult for clients to consume and understand compared to stand-alone APIs, so they are a better fit for internal APIs between services your team controls.

When to use sealed interfaces to implement polymorphic APIs

Sealed interfaces and Jackson polymorphic type annotations are a good fit for implementing polymorphic schemas when all the following are true:

Your JSON payloads share a discriminator field but have different schemas per type.
You want a closed set of permitted subtypes.
You want subtypes to be strongly typed, each potentially requiring its own validation logic.

They are a poor fit when either of the following is true, in which case abstract classes may be a better replacement for sealed interfaces:

Other teams need to be able to add subtypes without modifying your code (sealed interface subtypes must be defined in the same module).
You cannot use Java 17+, the first Long Term Support version supporting sealed interfaces.

Pros and cons vs common alternatives

Strategy	Pros	Cons
Sealed interface + Jackson polymorphism	Supports strongly typed subtypes with different required and allowed fields Easily add subtypes with minimal code changes Compile-time rejection of unsupported subtypes Clean pattern matching without casting	Requires all subtypes to be defined in same module Requires Java 17+
Single class with nullable fields	Simple Works pre-Java 17	Runtime null checks everywhere No compile-time safety Unclear which fields apply to each type More messy to enforce validations on subtypes
Inheritance with abstract class	Familiar OOP pattern Subclasses can live anywhere Works pre-Java 17	Anyone can subclass, no compile-time restriction on subtypes Compiler does not enforce exhaustive handling of subtypes
`@JsonAnySetter` with `Map<String, Object>`	Maximum flexibility Allows for unknown schemas Works pre-Java 17	Zero type safety, accepts all inputs Validation is manual and error-prone
Enum and factory method	Explicit type registry (as with sealed interface) More flexible, full control over deserialization Works pre-Java 17	More boilerplate code than sealed interface + annotations No compile-time validation of correct/complete routing

Key takeaways

sealed provides a closed type hierarchy so the compiler knows every possible implementation.
Jackson’s @JsonTypeInfo simplifies routing deserialization to the appropriate type based on a discriminator field.
Java records minimize boilerplate for immutable, validated data models.

In practice, this pattern has reduced the effort to support new data processing types from roughly a week per type to a few hours. The sealed interface and Jackson annotations handle the routing, leaving only type-specific models and logic to implement.

Enabling ad hoc runs of queue-based ECS services

Sat, 04 Apr 2026 00:00:00 +0000

Background

A common ECS use case is to continually poll a queue for new events to process, with auto-scaling based on CPU or memory utilization, or the size of the queue backlog. These services can be tricky to build reliable integration tests for, as queue backlogs may result in long delays before a new message is processed.

A simple way to enable efficient integration tests for these services is to support an optional environment variable that signals the service to immediately process the message without long-polling the queue, as if it were the content of a single SQS message body.

Example code

public static void main(String[] args) {
    String jsonRequest = System.getenv("ADHOC_REQUEST");
    if (jsonRequest != null && !jsonRequest.isEmpty()) {
        // Execute the single JSON request
        executeRequestFromJson(jsonRequest);
    } else {
        // Continually poll a queue to process pending requests
        longPollQueue();
    }
}

static void executeRequestFromJson(String jsonRequest) {
    MyParsedRequest request = parseRequestFromJson(jsonRequest);
    executeRequest(request);
}

static void longPollQueue() {
    while (true) {
        Optional<MyQueueMessage> message = getMessageFromQueue();
        if (message.isPresent()) {
            MyParsedRequest request = parseRequest(message.get());
            executeRequest(request);
            acknowledgeSqsMessage(message.get());
        }
    }
}

Example execution via AWS CLI

After setting variables for the AWS region and ECS cluster, service, and task definition ARNs, you can run a variation of the following to run a new ECS task immediately, without waiting for a new message to be processed in a queue that may have a large backlog of pending messages.

YOUR_ECS_CONTAINER_NAME should be replaced by the name field value in the task definition’s containerDefinitions.

The ECS container overrides allow you to provide specific environment variable overrides while maintaining other existing environment variables from the ECS task definition.

networkConfig=$(aws ecs describe-services \
    --region $awsRegion \
    --cluster $ecsClusterArn \
    --services $ecsServiceArn \
    --query 'services[0].networkConfiguration' \
    --output json)
aws ecs run-task \
    --region $awsRegion \
    --cluster $ecsClusterArn \
    --task-definition $ecsTaskDefinitionArn \
    --launch-type FARGATE \
    --network-configuration $networkConfig \
    --overrides '{
        "containerOverrides": [{
            "name": "YOUR_ECS_CONTAINER_NAME",
            "environment": [{
                "name": "ADHOC_REQUEST",
                "value": "{\"yourRequestKey\":\"yourRequestValue\"}"
            }]
        }]
    }'

Depending on the integ test environment, you can also use the ECS RunTask API in a similar way as the above CLI command.

After implementing this pattern in your service code, your integration tests can run ad hoc ECS tasks with specific test requests and validate the expected output - for example, periodically calling another API to validate a record has been successfully created or updated after a wait time, with a maximum number of retries.

Since the task processes only the injected request, tests can use a much shorter max duration than they would need if waiting for a large queue backlog to drain.

Strategies for querying periodic S3 data snapshots

Sat, 31 May 2025 00:00:00 +0000

Background

A common AWS analytics use case is making aggregate queries across multiple data sets stored in S3. For example, one partner team may store product metadata, while another team stores purchase order metadata, and we may want to join these data sets to determine which products are most popular across each marketplace.

In this post I’ll cover a few options for syncing data to S3 and retrieving data snapshots for use in aggregate queries, and specifically discuss the use case where we need to maintain access patterns to complete, recent data snapshots without partial unavailability during data syncs.

A few options for syncing data to S3

1. Syncing each record to a unique, stable file path

Pros:

Provides a stable “latest” data set that always represents all records.
Enables O(1) look-ups of specific records if data consumers query by S3 key (“filepath”).
Low storage costs since only a single copy of each record is stored.

Cons:

Poor performance and higher costs for aggregate queries at scale. Retrieving tens of thousands of small files is much slower than retrieving a few multi-MB files.
Limits design choices for data providers and may increase complexity of their implementation (eg. AWS Glue defaults to writing aggregated partial result files).
No historical data is retained unless explicitly backed up elsewhere.

This is best suited to when data consumers retrieve specific records, only need their latest state, and don’t make aggregated queries across a large number of records.

2. Appending all events as new data without overwriting prior events

Pros:

Provides a complete history of events, allowing for time-series analysis.
If partition data by time, supports efficient queries of specific time periods.

Cons:

Increase storage costs as accumulate historical data.
Increased complexity, latency, and cost to retrieve the latest version of each record.
If data isn’t partitioned in a way that aligns with query use cases, may have very inefficient scans.
If upstream workflows fail, may not have a way to recover data, and have missing records.

This is best suited to queries of time-partitioned events, rather than needing the latest state of records.

3. Overwriting one or more files that include multiple records

Pros:

Consumers always access the most up-to-date version of the dataset.
More efficient aggregate queries than retrieving thousands of single-record files.
Low storage costs since only a single copy of each record is stored.

Cons:

Risk of data loss if a failure occurs during the overwrite process.
Data consumers querying during a data sync may receive incomplete or duplicate data.
No historical data is retained unless explicitly backed up elsewhere.

This can work for aggregated queries of the latest records, but if we can’t accept the risk of corrupt/partial/duplicate data during sync failures or read/write race conditions, we may prefer writing to separate snapshot partitions.

4. Periodically writing complete data to time-based partitions

Pros:

Consumers always access a complete dataset when querying a completed/past partition.
More efficient aggregate queries than retrieving thousands of single-record files.
Maintain historical data for as long as prior time partitions are kept in storage.

Cons:

More complex to identify the latest complete partition.
Data consumers querying the most recent partition during a data sync may receive incomplete data. Need some type of completeness signal to mitigate.
Increase storage costs as accumulate historical data.

We can mitigate storage costs by setting a lifecycle rule to automatically delete S3 files past a certain age, eg. auto-delete files over 2 weeks old, if we only need to query recent snapshots.

Case study: Aggregate queries where completeness is important

For this post, we’ll consider the case where we want to optimize for aggregated queries against hundreds of thousands of records, with filter criteria applied across all records. Our business requirements are that data consumers must always have access to complete and accurate data (no duplicates or missing records), but data does not need to be real-time as long as it’s up to date within a few hours.

In this case, single-record files and time-series events are not a good fit due to increased latency to query latest state across this number of records. We may prefer time-based partitions with complete data written to a new partition every N hours, eg. hourly, to mitigate partial/duplicate data issues during concurrent read/write operations.

A few options for querying time-partitioned snapshots

A common challenge for time-partitioned partitions is identifying the latest complete partition, especially if upstream data syncs are not 100% successful.

1. Retrieve a specific snapshot relative to the current time

Assuming we have hourly snapshots that are partitioned by snapshot_date and snapshot_hour, where snapshot_date is formatted as "2025-01-31" and snapshot_hour is formatted as "01" or "23" to support consistent string comparisons, the following Athena SQL query retrieves data from the last hour’s time partition:

SELECT * FROM upstream_dataset
WHERE snapshot_date = CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%Y-%m-%d') AS VARCHAR)
AND snapshot_hour = CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%H') AS VARCHAR);

Pros:

Simple, easy to understand.
Very efficient since only querying data from a single partition, and it’s O(1) to locate that partition.

Cons:

May fail to get any data if there were upstream sync issues for the selected partition. This is a blocker for us.
May get partial data if query the most recent partition during an ongoing sync.
May need to query older and more out-of-date partitions to avoid the above race condition.

2. Retrieve most recent snapshot

We can improve on the prior option by using an initial query to identify the most recent snapshot partition that has data.

Since it could be expensive to search across all partitions, we can set a maximum look-back period, for example, the last 7 days, to allow for occasional upstream failures that may take a few days to resolve.

Example Athena SQL query:

WITH MostRecentSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN MostRecentSnapshotPartition
    ON upstream_dataset.snapshot_date = MostRecentSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = MostRecentSnapshotPartition.snapshot_hour;

Pros:

Relatively simple and easy to understand.
Guaranteed to get most recent snapshot with data within the given look-back period.

Cons:

May get partial data if query the most recent partition during an ongoing sync.

3. Retrieve the second most recent snapshot

If we need to mitigate race conditions where the latest partition may have partial data, we could always query for the second most recent partition.

We can maintain the same maximum look-back period to limit the query search space for performance reasons, while accounting for a few days of upstream failures.

Example Athena SQL query:

WITH RecentSnapshotPartitions AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    -- Get the two most recent partitions with data
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 2
),
RankedSnapshotPartitions AS (
    SELECT
        snapshot_date,
        snapshot_hour,
        -- Set a row number so we can select the second most recent partition
        ROW_NUMBER() OVER (ORDER BY snapshot_date DESC, snapshot_hour DESC) AS recency_rank
    FROM RecentSnapshotPartitions
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN RankedSnapshotPartitions
    ON upstream_dataset.snapshot_date = RankedSnapshotPartitions.snapshot_date
    AND upstream_dataset.snapshot_hour = RankedSnapshotPartitions.snapshot_hour
    AND RankedSnapshotPartitions.recency_rank = 2;

Pros:

If the second most recent partition is always complete, this guarantees complete data.

Cons:

More complex and difficult to understand than other options.
Less efficient than other options.

4. Retrieve the most recent snapshot older than the period between data syncs

If we need to account for temporary upstream failures and cannot accept race conditions where the most recent partition may sometimes be complete, we can simplify from Option 1 by querying for the most recent partition older than the time between syncs, eg. the most recent partition more than 1 hour old.

WITH RecentCompleteSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Get snapshots from more than 1 hour ago, to avoid querying a partition with an ongoing sync if have race conditions near the start of an hour
    WHERE snapshot_date <= CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%Y-%m-%d' AS VARCHAR))
    AND snapshot_hour < CAST(DATE_FORMAT(CURRENT_TIMESTAMP - INTERVAL '1' HOUR, '%H') AS VARCHAR)
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    AND snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN RecentCompleteSnapshotPartition
    ON upstream_dataset.snapshot_date = RecentCompleteSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = RecentCompleteSnapshotPartition.snapshot_hour;

Pros:

If populated partitions that are older than the period between syncs are always complete, guarantees complete data.
Simpler and more efficient than Option 3, while covering the flaws of Options 1-2.

Cons:

Slightly more complex query than Options 1-2.

5. Send data completeness events to trigger downstream queries

If we can request that our data provider pushes a data completeness signal file such as an empty file named _SUCCESS to the latest S3 directory after completing a data sync, we can set up S3 PutObject notifications that filter for this specific filename and trigger downstream workflows whenever this signal is received.

This is ideal for event-based workflows that only need to listen for a single trigger, while it may not be sufficient if there are multiple upstream datasets that we need to query that have different schedules and may not all be complete at the same time.

Pros:

Guarantees completeness of the given data set at the time we receive the event.
Allows downstream workflows to run with the most recent data possible, as soon as data is received.

Cons:

More complex to support multiple upstream data sets, best for single-dependency workflows.
May not work with workflows that are strictly schedule-based and cannot be easily triggered by an event.

6. Leverage AWS Glue Catalog, trigger Glue Crawler from Glue job event

If we populate our S3 bucket with an AWS Glue job, we can set up a Glue Catalog and a Glue Crawler that updates the catalog with an abstracted representation of the source data and its available partitions for downstream services such as AWS Athena to query.

If we set up the Glue Catalog Table as the data source for our downstream queries, we will only query partitions that it has crawled.

We can set up a Glue trigger to automatically run the Crawler after the upstream Glue job that populates new S3 partitions has succeeded, or after some event is received via EventBridge, such as creation of a _SUCCESS file, to automatically make new partitions available only after their sync jobs have completed.

We could then use the Athena SQL query from Option 2 but with the Glue Table as our source, to simplify querying the most recent complete partition.

WITH MostRecentSnapshotPartition AS (
    SELECT DISTINCT snapshot_date, snapshot_hour
    FROM upstream_dataset
    -- Set a maximum look-back period to reduce query search space while allowing a few days of upstream failures
    WHERE snapshot_date >= CAST(CURRENT_DATE - INTERVAL '7' DAY AS VARCHAR)
    ORDER BY snapshot_date DESC, snapshot_hour DESC
    LIMIT 1
)
SELECT upstream_dataset.*
FROM upstream_dataset
INNER JOIN MostRecentSnapshotPartition
    ON upstream_dataset.snapshot_date = MostRecentSnapshotPartition.snapshot_date
    AND upstream_dataset.snapshot_hour = MostRecentSnapshotPartition.snapshot_hour;

Pros:

Abstracts logic for how to identify the most recent partition from data consumers.
Enables always querying the most recent complete partition.
Avoids the incomplete/duplicate data issues from Options 1-2 and the query complexity from Options 3-4.

Cons:

Requires additional infrastructure set-up and hardware costs for Glue resources and event-based triggers.
May add several minutes of data latency from events -> Glue trigger -> Glue crawler -> Glue catalog update, compared to directly querying S3 for the most recent partition more than N hours old.

7. Leverage AWS Glue Catalog, use AWS Lambda to update a “latest” table after completeness events

If we have multiple upstream data sets which rules out Option 5, and we’re willing to invest a few developer weeks to optimize and simplify queries for data consumers, we could create a custom Lambda function that programmatically updates Glue resources to point to a “latest” S3 partition whenever a data sync completes.

We could require data providers to write an empty _SUCCESS file to the new partition after a successful sync. We can then set up a S3 PutObject notification that filters for _SUCCESS files, and make this a trigger to our Lambda function.

The Lambda function will then:

Query the S3 directory for files in the new partition and scan them to infer their partitions and table schema.
Query Glue’s UpdateTable API to update the Glue Catalog Table that represents the latest version of the snapshot-based data set, to have the latest data schema and point to the new partition.

Data consumers can then simplify their queries to the following, without needing to select snapshot partition attributes:

SELECT * FROM upstream_dataset;

Pros:

Significantly simplifies queries for data consumers.
Enables always querying the most recent complete partition.
Avoids the incomplete/duplicate data issues from Options 1-2 and the query complexity from Options 3-6.

Cons:

Adds multiple weeks of developer effort to replicate AWS Glue’s functionality for inferring schemas and updating Glue Catalog Tables.
Increases backend architecture and code complexity, with more points of failures that need to be maintained and supported.

If this works, the end result may be great from the data consumer side, but in many cases this developer time is more valuable spent on other problems, and we’re willing to accept either slightly more complex but well-documented queries for data consumers, or occasional race conditions where recent snapshots may be incomplete if we happen to be reading at the same time that a data sync is occurring.

Case study: Retrieving recent data when completeness is critical and developer time is limited

Take the case where we are limited to a few days of developer time to set up associated infrastructure, while we have the business requirements that:

Data consumers must always be able to query complete and accurate data that is no more than a few hours old. Missing or duplicate data due to read/write race conditions is not acceptable.
Data consumers must be able to aggregate data from multiple upstream sources while satisfying the above conditions for completeness without partial data if querying during data syncs.

In this scenario, Option 6 may be an acceptable tradeoff, where we leverage existing AWS Glue functionality to automatically surface new partitions and schemas after receiving a data sync completion signal, and point data consumers to the Glue Catalog Table to query partitions that have completed syncs.

Data consumers will then need to be aware of snapshot partition attributes to select data from the most recent partition, but they can simply query for the most recent partition since we will only surface partitions after receiving a data sync completion signal.

Granting AWS Organization member accounts access to Cost Explorer

Sat, 10 May 2025 00:00:00 +0000

Introduction

By default, adding accounts to an AWS Organization results in consolidated billing and cost management in the Organization management account, and Organization member accounts lose access to Cost Explorer, Billing, and other cost management services unless access is explicitly enabled from the Organization management account.

This post walks through how to allow member accounts to access cost management services, to enable each team to review and manage their AWS spending.

Enabling Cost Explorer and Cost Optimization Hub

Cost Explorer allows you to analyze AWS service costs and usage changes over time, and is free to access through the AWS UI. You can filter and group costs and usage across multiple dimensions including AWS service, resource, usage type, region, and tag.

After logging into the Organization management account using the root user, navigate to the Cost Explorer service to automatically enable it from the current time onwards. You will need to do this once per account in your Organization, logging in as the root user for each account.

Cost Optimization Hub is a free service that provides cost optimization recommendations across multiple services. To enable member accounts access, we’ll need to enable the services from the AWS Organizations management account, and explicitly add permissions to the Permission Sets granting them access.

To enable Cost Optimization Hub, navigate to the Cost Optimization Hub landing page, scroll down, select “Enable Cost Optimization Hub for this account and all member accounts”, and click “Enable”.

As indicated by the prompt, to fully benefit from this service you’ll also need to opt into the free AWS Compute Optimizer service to import service rightsizing recommendations. Follow the link to navigate to AWS Compute Optimizer, and click “Get started”.

Select to opt in all member accounts, and click “Opt in”.

Enabling access for member accounts

On the top right hand of the AWS console, select your account name to open the account dropdown, and click “Account”, or directly navigate to https://us-east-1.console.aws.amazon.com/billing/home?region=us-east-1#/account.

Scroll down to “IAM user and role access to Billing information” section and click “Edit”. Select “Activate IAM Access” and click “Update”.

Navigate to “Billing and Cost Management” > “Cost Management Preferences”.

Under the “General” tab, enable the options below, then select “Save preferences” at the bottom of the page and confirm changes.

Linked account access
Linked account refunds and credits
Linked account discounts

Under the “Cost Explorer” tab, enable “Granular data” with “Resource-level data at daily granularity”, and select “All services”, then click “Save preferences” at the bottom of the page and confirm changes.

Under the “Cost Optimization Hub” tab, under “Organization and member account settings”, select “Enable Cost Optimization Hub for all member accounts” and “Allow member account discount visibility”, then click “Save preferences” and confirm changes.

Accessing Cost Explorer from member accounts

At this point, team members logging into Organization member accounts to view Cost Explorer will still see access denied errors across all widgets, even if their IAM permissions grant access to all Cost Explorer actions.

When clicking on one of the errors, we can see the following message:

You don’t have permission to [Cost Explorer:]. To request access, copy the following text and send it to your AWS administrator. Learn more about troubleshooting access denied errors

User: [REDACTED_ACCOUNT_ID]

Service: [Cost Explorer]

Name: [AccessDeniedException]

HTTP status code: [400]

Context: [IAM user access not activated]

Request ID: [REDACTED_REQUEST_ID]

To resolve this error, we need to:

Log into the Organization member account as the root user.
Visit AWS Cost Explorer once to enable this service if we haven’t done this before.
Navigate to Account settings via the top right hand account dropdown > “Account”.
Scroll down to “IAM user and role access to Billing information” section and click “Edit”.
Select “Activate IAM Access” and click “Update”.

Non-root users with billing permissions should now have access to Cost Explorer, while the page may initially show zero costs and only show correct spending for the current time going forward.

After waiting for a few days, I noticed that the default view still shows zero spending, while after updating the time period to the last 7 days and updating the granularity to daily, I see correct costs for the past few days up to the date when Cost Explorer was enabled from the Organization management account.

We can now add any filters or groupings we like, such as grouping costs by usage type to see what specify AWS service usages are contributing to our daily spending.

Granting cross-account user groups access to Cost Explorer

We need to explicitly grant user groups access to Cost Explorer, otherwise they will be denied by default.

Users with full read permissions across all AWS services will be able to view Cost Explorer after the earlier steps in this post, but we may want to allow additional user roles access as well.

See my last post’s “Creating cross-account role-based permission groups” section for how to create a Permission Set and assign user groups these permissions on specific linked accounts.

To manage your Permission Sets, navigate to “IAM Identity Center” > “Multi-account permissions” > “Permission sets”.

Assuming you have created a Permission Set that does not have explicit permissions to Cost Explorer, and you want that group to have read access to all cost management services, select that Permission Set.

In this case, I have a SupportUser Permission Set where I want my support team to be able to view cost and usage data for their assigned accounts.

While this Permission Set only has the SupportUser policy assigned, users logging in with this role will be denied access to all Cost Explorer widgets with the error:

User: [REDACTED_ACCOUNT_ID]

Service: [Cost Explorer]

Name: [AccessDeniedException]

HTTP status code: [400]

Context: [User: arn:aws:sts::REDACTED_ACCOUNT_ID:assumed-role/REDACTED_ROLE_ID is not authorized to perform: ce:GetCostAndUsage on resource: arn:aws:ce:us-east-1:REDACTED_ACCOUNT_ID:/GetCostAndUsage because no identity-based policy allows the ce:GetCostAndUsage action]

Under the Permission Set’s “Permissions” > “Inline policy” section, select “Edit”.

Enter the following IAM policy document, or the scoped permissions you want to grant:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CostManagementViewAccess",
            "Effect": "Allow",
            "Action": [
                "account:GetAccountInformation",
                "aws-portal:ViewBilling",
                "billing:Get*",
                "billing:List*",
                "budgets:ViewBudget",
                "budgets:Describe*",
                "ce:Describe*",
                "ce:Get*",
                "ce:List*",
                "consolidatedbilling:Get*",
                "consolidatedbilling:List*",
                "cur:Describe*",
                "cur:Get*",
                "freetier:Get*",
                "sustainability:GetCarbonFootprintSummary"
            ],
            "Resource": "*"
        }
    ]
}

Scroll to the bottom of the page and save your changes.

Validate that users assigned this Permission Set on a given account now have read access to Billing and Cost Management pages.

Posts in this series

Using AWS Organizations to standardize security controls across AWS accounts
(Current post) Granting AWS Organization member accounts access to Cost Explorer

References

AWS Organizations User Guide: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html

AWS Cost Explorer Access User Guide: https://docs.aws.amazon.com/cost-management/latest/userguide/ce-access.html

AWS IAM Identity Center Permission Sets User Guide: https://docs.aws.amazon.com/singlesignon/latest/userguide/permissionsetsconcept.html

Using AWS Organizations to standardize security controls across AWS accounts

Wed, 07 May 2025 00:00:00 +0000

Introduction

AWS Organizations provide a helpful way to centralize management of AWS accounts, with support for consolidating billing, role-based permission sets, service and resource control policies, and AWS service configurations.

Service control policies (SCPs) can be used to enforce security controls such as:

Requiring multi-factor authentication (MFA) to complete certain actions
Blocking use of the root user outside of the AWS Organization management account
Blocking certain changes such as leaving the AWS Organization or disabling security tools
Blocking access to specific regions or setting a region allow-list, if your organization has policies restricting where services can be deployed
Blocking granting VPCs direct Internet access

Resource control policies (RCPs) are supported by S3, STS, KMS, SQS, and Secrets Manager, and can be used to enforce security controls such:

Requiring all queries against in-scope resources to be over HTTPS
Restricting access to your S3 buckets, KMS encryption keys, and Secrets Manager secrets to principles within your AWS Organization
Restricting sts:AssumeRoleWithWebIdentity requests to allow-listed OpenID Connect (OIDC) Identity Providers and identities
Requesting KMS encryption to be used for all S3 objects

SCPs and RCPs are provided at no cost, while only up to 5 SCPs and 5 RCPs can be assigned to a given AWS account or AWS Organizational Unit.

Each AWS Organization has a single management account that can’t be changed, and AWS recommends using an account that has no other resources or workloads for this purpose. Access to the management account should be limited to admin users that need to make organization changes. SCPs and RCPs do not apply to the management account.

There are no costs associated with using AWS Organizations, so while you do need to take care when using it (with great power comes great responsibility), it’s an awesome tool to simplify managing multi-account organizations and services.

Creating an AWS Organization

Once you’ve created an AWS account that will be specifically for AWS Organization management, log into that account, navigate to the AWS Organizations service and click “Create an Organization”.

Review the linked recommendations, and click “Create an Organization” to proceed.

You can then view and modify Organization settings and Invite Accounts to join the Organization.

Standardizing user access via IAM Identity Center

From the AWS Organization management account, navigate to the IAM Identity Center service, and enable Identity Center for your Organization if not yet enabled.

Then, under “Settings” > “Identity source”, configure an Identity Source that matches your use case.

You can choose between:

Identity Center directory, where you create users and groups in IAM Identity Center, and users sign in through the AWS access portal with usernames and passwords.
Active Directory, which many companies already have set up to manage network access controls.
Other external identity providers that implement supported identity federation protocols.

For my proof-of-concept, Active Directory and external identity providers aren’t relevant, so I’ll use the native AWS Identity Center directory.

Enforcing multi-factor-authentication (MFA)

Especially if using traditional usernames and passwords, multi-factor authentication is critical to protect your accounts from attackers.

MFA requires that after a user enters a correct username and password, they also provide additional verification that they are who they say they are.

Companies such as Google and Microsoft that have enforced MFA have reported 99%+ decreases in successful account take-overs when requiring on-device prompts, and 100% decreases when requiring physical security keys.

To enforce MFA for your Organization:

Navigate to “IAM Identity Center” > “Settings” > “Authentication”.
Under “Multi-factor authentication”, click “Configure”.
Under “Prompt users for MFA”, select “Every time they sign in”.
Under “Users can authenticate with these MFA types”, select one or more of the following:
- Security keys (eg. YubiKey) and built-in authenticators (eg. Apple TouchID) - recommend always enabling so can users can choose the strongest MFA type available; or
- Authenticator apps (eg. Authy, Google Authenticator) - good back-up if not all users can use the first option.
Under “If a user does not yet have a registered MFA device”, select “Require them to register an MFA device at sign in”.
Click “Save changes”.

MFA will now be automatically enforced for all users.

Adding AWS accounts to an AWS Organization

From the AWS Organization’s “AWS accounts” page, click “Add an AWS account”.

From here, you can either create new AWS accounts under the Organization, or invite existing AWS accounts to join the Organization.

To invite an existing account, click “Invite an existing AWS account”, enter the account’s details, and click “Send invitation”.

You can then log into the invited AWS account, select “Invitations”, and accept or decline the invite.

After accepting the invite, billing for the member account will be managed by the Organization’s management account, and any Organization-level security controls and AWS service configurations will be automatically applied.

Member accounts can leave the Organization at any time from their AWS Organizations dashboard, unless the Organization has enforced a service control policy (SCP) blocking this action. It’s a good practice to implement such an SCP to prevent actors who’ve compromised an account from leaving the Organization to disable its security controls.

Creating cross-account role-based permission groups

A permission set is a collection of IAM policies and permission boundaries that defines the access that will be granted to a logged in user.

Multiple permission sets can be created and assigned to user groups to enable those users to log into a AWS account with one of the permission sets that have been made available to them.

To manage your permission sets, navigate to “IAM Identity Center” > “Multi-account permissions” > “Permission sets”.

If you click “Create permission set”, you can select from a number of AWS-defined template permission sets, or create a custom permission set with any combination of managed and inline IAM policies and an optional permission boundary.

For example, we could create the following predefined permission sets for our Organization:

AdministratorAccess: provides full access to AWS services and resources
PowerUserAccess: provides full access to AWS services and resources, but does not allow management of Users and groups
SupportUser: provides permissions to troubleshoot and resolve issues in an AWS account, and contact AWS support to create and manage cases
ReadOnlyAccess: provides read-only access to AWS services and resources

We can then create a user group and grant that group a subset of these permission sets on specific member accounts in our Organization, depending on the level of access they need on the given accounts for their business functions.

Under “IAM Identity Center” > “Groups”, you can manage permission groups that you can assign role-based permissions, and these groups can be assigned to any selection of Organizational Units or accounts within your AWS Organization.

To create a new group, select “Create group”, enter a group name and optional description, optionally add users, and click “Create group”.

Then open that group and under AWS accounts, select “Assign accounts”.

Select accounts and permission sets that the given group should have on those accounts, the click “Assign” to apply the permissions.

You can then add users to the group, and after they log into their provided AWS access portal, they will be able to select an account and a permission set out of those assigned to their group for that account.

A common use case for setting up multiple permission sets on a group is to allow developers to log into accounts with read-only access by default to validate workflows, and only have them log in with admin permissions when necessary to manually remediate issues.

Setting up Service Control Policies

From your AWS Organization management account, navigate to “AWS Organizations” > “Policies”.

Click “Service control policies” and enable this feature.

Click “Create policy” and create a policy based on your use cases.

For example, to create a policy that blocks member accounts from leaving the AWS Organization, disabling or modifying GuardDuty config, or attaching their VPCs directly to the Internet, we can create a SCP with:

Policy name: DenyBypassOrgSecurityControls
- Note: Will likely want a more specific policy and name, using this for testing purposes.
Description: Prevent member accounts from leaving the AWS Organization, disabling or modifying GuardDuty configurations, or opening direct VPC access to the Internet.
Policy JSON:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeaveOrganization",
      "Effect": "Deny",
      "Action": [
        "organizations:LeaveOrganization"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyModifyGuardDuty",
      "Effect": "Deny",
      "Action": [
        "guardduty:AcceptInvitation",
        "guardduty:ArchiveFindings",
        "guardduty:CreateDetector",
        "guardduty:CreateFilter",
        "guardduty:CreateIPSet",
        "guardduty:CreateMembers",
        "guardduty:CreatePublishingDestination",
        "guardduty:CreateSampleFindings",
        "guardduty:CreateThreatIntelSet",
        "guardduty:DeclineInvitations",
        "guardduty:DeleteDetector",
        "guardduty:DeleteFilter",
        "guardduty:DeleteInvitations",
        "guardduty:DeleteIPSet",
        "guardduty:DeleteMembers",
        "guardduty:DeletePublishingDestination",
        "guardduty:DeleteThreatIntelSet",
        "guardduty:DisassociateFromMasterAccount",
        "guardduty:DisassociateMembers",
        "guardduty:InviteMembers",
        "guardduty:StartMonitoringMembers",
        "guardduty:StopMonitoringMembers",
        "guardduty:TagResource",
        "guardduty:UnarchiveFindings",
        "guardduty:UntagResource",
        "guardduty:UpdateDetector",
        "guardduty:UpdateFilter",
        "guardduty:UpdateFindingsFeedback",
        "guardduty:UpdateIPSet",
        "guardduty:UpdatePublishingDestination",
        "guardduty:UpdateThreatIntelSet"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyOpenVpcInternetAccess",
      "Effect": "Deny",
      "Action": [
        "ec2:AttachInternetGateway",
        "ec2:CreateInternetGateway",
        "ec2:CreateEgressOnlyInternetGateway",
        "ec2:CreateVpcPeeringConnection",
        "ec2:AcceptVpcPeeringConnection",
        "globalaccelerator:Create*",
        "globalaccelerator:Update*"
      ],
      "Resource": "*"
    }
  ]
}

Click “Create policy” to save your changes.

Then, from the “Service control policies” page, select your created policy and select “Actions” > “Attach Policy”.

Select the Organizational Units or specific accounts you want to apply the policy to, and click “Attach policy”.

The security controls will now be automatically enforced for the associated member accounts.

We can test this by logging into a member account using the AdministratorAccess role, navigating to AWS Organizations, and selecting to leave the Organization.

As shown above, the Service Control Policy successfully blocks member accounts from leaving the Organization. Removing member accounts can now only be initiated from the Organization management account, which reduces the risk of a compromised account bypassing Organization security policies.

Posts in this series

(Current post) Using AWS Organizations to standardize security controls across AWS accounts
Granting AWS Organization member accounts access to Cost Explorer

References

AWS Organizations User Guide: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html

AWS Organizations Best Practices: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_best-practices_mgmt-acct.html

AWS Organizations Security Control Policies User Guide: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html

AWS Organizations Security Control Policies Best Practices: https://aws.amazon.com/blogs/industries/best-practices-for-aws-organizations-service-control-policies-in-a-multi-account-environment/

AWS IAM Identity Center MFA User Guide: https://docs.aws.amazon.com/singlesignon/latest/userguide/mfa-configure.html

Reducing Lambda latency by 76% with AWS Lambda Power Tuning

Sun, 14 Jul 2024 00:00:00 +0000

Introduction

Optimizing AWS Lambda memory capacity can decrease customer-facing latencies by up to 2-5 times without significantly increasing hardware costs. However, this takes trial and error, and many teams just pick an amount of memory and stick with it, leaving their services several times slower than necessary.

Other teams spend hours setting up custom code and metrics to measure latencies for each of their service’s use cases, benchmark each use case against various memory capacities, and use the AWS Cost Estimator or AWS Lambda pricing documentation to estimate costs and choose the amount of memory with the best latency-to-cost tradeoff.

This is no longer necessary with the AWS Lambda Power Tuning tool, which can be run against any Lambda function in your AWS account to automatically determine the optimal memory capacity that minimizes execution latency and/or hardware costs.

There is no cost to deploy and run this besides its underlying hardware costs, which is likely free if you only run it a few times before deleting it from your account.

Since it only relies on AWS-infrastructure-level API calls, the tool works regardless of which programming language your Lambda function uses, and doesn’t require any modifications to your service infrastructure or code.

Set up

The AWS Lambda Power Tuning GitHub repo documents multiple ways to deploy the tool, using either the AWS Serverless Application Repository (simplest), AWS SAM CLI, AWS CDK, or Terraform.

I used the AWS Serverless Application Repository since this reduces set-up to a few button clicks, and I planned to tear down the tool after optimizing my Lambda function.

To use this deployment option, you can simply log into your AWS account, visit https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:451282441545:applications~aws-lambda-power-tuning, and click Deploy.

This will create an AWS Lambda Application that encapsulates all the infrastructure for the tuning tool, including the AWS Step Functions State Machine that you’ll invoke to run the benchmark tests.

Click on the powerTuningStateMachine resource to open the state machine, and click Start Execution, then enter the JSON payload to run the benchmark test with, where input parameters are documented on the tool’s GitHub README.

For example, the following payload runs the tool against the given Lambda function, with 15 executions each for 512, 1024, 1536, 2048, and 3008 MB of memory, with a function payload specific to my API service, and the balanced optimization strategy.

{
  "lambdaARN": "arn:aws:lambda:us-west-2:123456789012:function:TestLambdaFunctionName",
  "powerValues": [
    512,
    1024,
    1536,
    2048,
    3008
  ],
  "num": 15,
  "payload": {
    "resource": "/v1/consent-management/services/{serviceId}/users/{userId}/consents",
    "path": "/v1/consent-management/services/TestServiceId/users/TestUserId/consents",
    "httpMethod": "GET",
    "pathParameters": {
      "serviceId": "TestServiceId",
      "userId": "TestUserId"
    },
    "requestContext": {
      "resourceId": "1abc2d",
      "resourcePath": "/v1/consent-management/services/{serviceId}/users/{userId}/consents",
      "operationName": "ListServiceUserConsent",
      "httpMethod": "GET",
      "path": "/v1/consent-management/services/{serviceId}/users/{userId}/consents",
      "accountId": "123456789012",
      "protocol": "HTTP/1.1",
      "stage": "test"
    }
  },
  "parallelInvocation": false,
  "strategy": "balanced"
}

I set parallelInvocation to false after observing Lambda throttling errors with it set to true, since my test Lambda isn’t currently provisioned for high load, and strategy to balanced to equality weight minimizing latency and minimizing costs, while you can configure the tool to only consider one or use a different weighted average.

Analyzing results

Once the execution completes, the Execution input and output tab will display the recommended amount of memory as the power value, the resulting average latency in milliseconds and cost per execution, and the URL to a more detailed visualization.

By navigating to that URL, we can view a graph of average latency and execution costs for each amount of memory measured, along with summarized best and worst memories for latency and cost.

In this case, for my Lambda function, which is written in Java and queries a DynamoDB table, 2048 MB of memory resulted in the lowest average latency, while 1024 MB of memory had the lowest runtime costs.

We can see that 512 MB actually costs more than 1024 MB, and this is due to the duration being several times higher which results in higher GB-second charges.

This was only run for 15 iterations per memory allocation, so I increased the sample size and reran against 1024, 1536, and 2048 MB by setting powerValues and num to "powerValues": [1024, 1536, 2048], "num": 50.

I executed the Lambda function a couple times first with a test payload to eliminate cold starts as a compounding factor, and then ran the state machine with the new config, which resulted in the following output and visualization:

{
  "power": 1536,
  "cost": 3.2760000000000005e-7,
  "duration": 12.266666666666667,
  "stateMachine": {
    "executionCost": 0.00023,
    "lambdaCost": 0.00012891480000000002,
    "visualization": "https://lambda-power-tuning.show/#AAQABgAI;3t2NQUREREFERExB;ilmiNADhrzRWgeo0"
  }
}

The more detailed visualization indicates that for our particular use case, we’re unlikely to see significant performance improvements from increasing memory above 1536 MB, and the marginal cost increase from 1024 MB to 1536 MB is acceptable for us.

You can see a more detailed table view of the underlying data by going to the step function execution’s Detail tab, selecting the Table view, selecting the Analyzer task, and selecting the Analyzer panel’s Output tab.

Tear-down

When you no longer need the tool, you can open the AWS CloudFormation console and delete the serverlessrepo-aws-lambda-power-tuning CloudFormation stack.

Outcome

The tool took under 10 minutes to deploy, execute, and fine-tune, and resulted in me changing my test Lambda’s memory allocation from 512 MB to 1536 MB.

This lowered my API’s average latency from 50ms to 12ms, a 4.17x improvement, AKA 76% latency reduction. Duration costs increased by 8% to $0.3276/million executions, which is minimal for my service’s scale.

Given the latency improvements of choosing the right amount of memory, and how easy this tool is to use, I’d recommend it to anyone building services on AWS Lambda.

References

AWS Lambda docs introducing AWS Lambda Power Tuning: https://docs.aws.amazon.com/lambda/latest/operatorguide/profile-functions.html

AWS Lambda Power Tuning GitHub repository with usage details: https://github.com/alexcasalboni/aws-lambda-power-tuning

AWS Lambda pricing: https://aws.amazon.com/lambda/pricing/

Serializing and deserializing DynamoDB pagination tokens to support paginated APIs

Sat, 18 May 2024 17:00:00 +0000

When using AWS’s Java 2.x SDK, DynamoDB scan and query responses provide pagination tokens in a Map<String, AttributeValue> lastEvaluatedKey object, which represents the primary key of the last processed DynamoDB item. You can then pass this value as the “exclusive start key” for the next query to get the next page of results.

When your service retrieves all pages of results locally, this isn’t a problem. However, when you want to provide a paginated API backed by DynamoDB, you’ll need to convert this attribute value map into a format that can be passed over HTTP, AKA “serialize” the object into a string.

When your client requests the next page of results with that string pagination token, you’ll also need to convert that string back into the Map<String, AttributeValue> format that the AWS SDK expects, AKA “deserialize” the string to the original data structure.

Prior method for serializing/deserializing pagination tokens

Before May 2023, building paginated APIs backed by DynamoDB was not very convenient, as you’d have to build your own custom serialization and deserialization code.

Example implementation using Immutables and Jackson, with a sample DynamoDB table primary key that has both a partition key and a sort key:

import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import org.immutables.value.Value.Immutable;
import org.immutables.value.Value.Parameter;
import org.immutables.value.Value.Style;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;

import java.io.IOException;
import java.util.Base64;
import java.util.Map;

/**
 * Serializable representation of a Product DynamoDB pagination token.
 * Using Immutables to generate safe, immutable value objects.
 * @see https://immutables.github.io/
 */
@JsonDeserialize(builder = ProductNextTokenBuilder.class)
@JsonSerialize
@Style(visibility = Style.ImplementationVisibility.PRIVATE)
@Immutable
interface ProductNextToken {
    @Parameter
    String getPartitionKey();
    @Parameter
    String getSortKey();
}

/**
 * Class encapsulating logic to convert DynamoDB pagination tokens between attribute value
 * maps used by the AWS SDK, and string values that can be passed over HTTP.
 */
class ProductSerializer {
    private static final PRODUCT_TABLE_PARTITION_KEY = "YourDynamoDBTablePartitionKeyName";
    private static final PRODUCT_TABLE_SORT_KEY = "YourDynamoDBTableSortKeyName";

    ProductSerializer(final ObjectMapper objectMapper) {
      this.objectMapper = objectMapper;
    }

    /**
     * Serialize a lastEvaluatedKey from an attribute value map to a string.
     *
     * @param lastEvaluatedKey attribute map returned by paginated DynamoDB queries.
     * @return serialized String token that can be passed over HTTP.
     * @throws JsonProcessingException exception thrown if unable to parse the key.
     */
    public String serializeLastEvaluatedKey(final Map<String, AttributeValue> lastEvaluatedKey) throws JsonProcessingException {
        if (lastEvaluatedKey == null) {
            return null;
        }

        final ProductNextToken tokenObject = new ProductNextTokenBuilder()
            .partitionKey(lastEvaluatedKey.get(PRODUCT_TABLE_PARTITION_KEY))
            .sortKey(lastEvaluatedKey.get(PRODUCT_TABLE_SORT_KEY))
            .build();

        return Base64.getUrlEncoder().encodeToString(objectMapper.writeValueAsBytes(tokenObject));
    }

    /**
     * Deserialize a lastEvaluatedKey from a string to an attribute value map.
     *
     * @param lastEvaluatedKey attribute map returned by paginated DynamoDB queries.
     * @return serialized String token that can be passed over HTTP.
     * @throws IOException exception thrown if unable to decode encodedLastEvaluatedKey.
     * @throws JsonParseException exception thrown if unable to deserialize the decoded key into a ProductNextToken.
     */
    public Map<String, AttributeValue> deserializeLastEvaluatedKey(final String encodedLastEvaluatedKey) throws IOException, JsonParseException {
      if (encodedLastEvaluatedKey == null) {
          return null;
      }

      final ProductNextToken deserializedToken = objectMapper.readValue(
          Base64.getUrlDecoder().decode(encodedLastEvaluatedKey),
          ProductNextToken.class
      );

      final AttributeValue partitionKeyValue = AttributeValue.builder()
          .s(deserializedToken.getPartitionKey())
          .build();

      final AttributeValue sortKeyValue = AttributeValue.builder()
          .s(deserializedToken.getSortKey())
          .build();

      return Map.of(
          PRODUCT_TABLE_PARTITION_KEY, partitionKeyValue,
          PRODUCT_TABLE_SORT_KEY, sortKeyValue
      );
    }
}

This is a lot of code to maintain and test, with multiple exception cases. We can remove the dependency on specific key structure by generalizing the code to iterate over the map and JSON key-value pairs, as shown in https://github.com/aws/aws-sdk-java-v2/issues/3224, but this is still more complex than should be necessary for what we’d prefer to be simple “stringify” and “unstringify” methods.

Serialization/deserialization with the DynamoDB Enhanced Document library

Since May 2023, AWS’s Java 2.x SDK includes an Enhanced Document library that simplifies converting pagination tokens between the AWS SDK’s objects and JSON strings that can be passed over HTTP.

The software.amazon.awssdk.enhanced.dynamodb.document.EnhancedDocument class includes utility methods that make serialization and deserialization one-liners.

AWS blog post demonstrating use cases: https://aws.amazon.com/blogs/devops/introducing-the-enhanced-document-api-for-dynamodb-in-the-aws-sdk-for-java-2-x/

Sample code for converting between Map<String, AttributeValue> pagination tokens and JSON strings:

import software.amazon.awssdk.enhanced.dynamodb.document.EnhancedDocument;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;

import java.io.UncheckedIOException;

/**
 * Class encapsulating logic to convert DynamoDB pagination tokens between attribute value
 * maps used by the AWS SDK, and string values that can be passed over HTTP.
 */
class ProductSerializer {
    /**
      * Convert a DynamoDB attribute value map to a JSON string.
      * @param attributeValueMap DynamoDB item key represented as a map from attribute names to attribute values
      * @return String JSON string representation of the DynamoDB item key
      */
    public String serializeLastEvaluatedKey(final Map<String, AttributeValue> attributeValueMap) {
        return EnhancedDocument.fromAttributeValueMap(attributeValueMap).toJson();
    }

    /**
      * Convert a JSON string representation of a DynamoDB pagination token to the format required by DynamoDB API calls.
      * @param paginationTokenJson JSON string representing the last paginated API call's last evaluated record key
      * @return Map<String, AttributeValue> exclusive start key for the next paginated DynamoDB scan/query API call
      * @throws UncheckedIOException exception thrown if fail to parse pagination token
      */
    public Map<String, AttributeValue> deserializeLastEvaluatedKey(final String paginationTokenJson) throws UncheckedIOException {
        return EnhancedDocument.fromJson(paginationTokenJson).toMap();
    }
}

This is much more manageable, with serialization and deserialization functionality now provided out-of-the-box as part of the standard AWS SDK.

We can pass this serialized JSON string to our API clients as the client-facing pagination token. Optionally, if it’s important to us to obfuscate our internal DynamoDB key structure from clients, we can add back a Base64 encode/decode layer on top of the JSON strings using the same code snippets from the earlier example.

Concurrency from single host applications up to massively distributed services

Fri, 01 Dec 2023 03:00:00 +0000

Concurrency is when multiple software threads or programs are run at the same time, and is a key aspect of many modern applications.

Web browsers run dozens of concurrent processes based on your activity, querying servers, downloading files, and executing scripts all at once.

Online services with millions of active users run a scaled up number of concurrent processes across thousands of servers, with various distributed system design patterns to support this.

This post will describe several levels of concurrency, how they’re commonly applied, and pros and cons of each approach.

Levels of concurrency

Multi-threaded applications

Within application code, we can run multiple threads concurrently.

This approach can be locally applied regardless of whether an application is run on a single host or in a distributed service. However, multi-threaded code increases code complexity and introduces thread safety issues, and an error in one thread may take down the entire application.

A common use case for multi-threading is when we need to make multiple requests to other services that may each take multiple seconds to complete. We can trigger each request in a separate thread to run them concurrently, and collect the results at the end of the longest running call, rather than synchronously making one request at a time after the prior response has returned.

Latency trade-offs

Example 1: Suppose we will run 4 requests that each take 2 seconds, 5 seconds, 5 seconds, and 1 second to complete, and each thread adds 20 milliseconds of overhead to start and close. We’ll exclude the time to combine results as equivalent between multi-threaded and synchronous approaches. Our runtime with multi-threading will be max(2, 5, 5, 1) + 0.02*4 = 5.08 seconds, compared to the synchronous approach taking 2 + 5 + 5 + 1 = 13 seconds to make all requests. In this scenario, multi-threading reduces our latency by 7.92 seconds.

Example 2: Splitting tasks into threads does not come for free and may not worthwhile for very short-lived requests. For example, if we have 1000 requests that each take 0.01 seconds to complete, running each request in a separate thread would take max(0.01) + 0.02*1000 = 20.01 seconds, compared to the synchronous approach taking 1000*0.01 = 10 seconds. In this case, the synchronous approach is twice as efficient as multi-threading.

Since the cost of such a high branching factor is high, in reality, we’ll typically break this workflow up into batches of requests per thread, such as 200 requests per thread.

Example 2b: Given 1000 requests that each take 0.01 seconds to complete, if we split the work into 5 batches of 200 requests per thread, computing all the results would take max(200*0.01, 200*0.01, 200*0.01, 200*0.01, 200*0.01) + 0.02*5 = 2.1 seconds, compared to the synchronous approach taking 1000*0.01 = 10 seconds. By batching the work before applying multi-threading, we can reduce latency compared to synchronous calls by 7.9 seconds.

Multi-threading provides the most latency reduction when we’re able to run multiple long-running tasks in parallel, especially multi-second tasks, whether each task is a single long-running request or a series of requests adding up to seconds.

Multi-container hosts

Within a host, we can run multiple containers which each receive allocated memory and run an isolated instance of application code.

This allows us to fully utilize a host’s CPU and memory, while we will eventually get to a point where the host no longer has sufficient CPU or memory capacity to add more containers, or where performance begins to drop due to increased context switching and IO bottlenecks.

Isolating concurrent applications in separate containers also improves system reliability, since regardless of individual application failures, the other containers can continue running. However, we are still vulnerable to host-level failures.

A single host can be sufficient for some small-scale services that only have a few hundred concurrent requests and are acceptable to periodically take offline for maintenance. For services that need to provide 24/7 availability or handle more traffic, we will graduate to distributed services where this host will be a single unit of a larger architecture, leading us to the multi-host cluster.

Multi-host clusters behind a load balancer

When we require high availability or more concurrency than a single host can support, we can set up a load balancer that distributes traffic across multiple hosts, forming a cluster of hosts.

This allows us to horizontally scale, that is, add or remove servers to our resource pool as needed. Horizontal scaling makes our service more robust to individual host failures and enables more flexibility in our infrastructure, allowing us to swap out different types of hosts at will, patch or update individual hosts without affecting service availability, and pay for just as many hosts as are needed to meet current demand.

This is often the go-to design pattern for services that need to process thousands of concurrent requests, which a single host may no longer be able to handle.

Multi-cluster services

When we have more traffic than a single load balancer can handle, we can set up a DNS load balancer to distribute traffic across multiple clusters.

This is rarely the starting point for a new service. We only want to add this level of complexity when absolutely necessary, such as when scaling up to millions of concurrent requests, or after hitting infrastructure restrictions on load balancer concurrent connections or maximum attached endpoints.

Many cloud providers provide distributed DNS load balancers that remove the single point of failure of a traditional load balancer, scale to millions of concurrent users, and automatically route traffic to the closest regional cluster.

DNS load balancer trade-offs

DNS load balancers are more limited in functionality than many specialized load balancers. For example, AWS network load balancers can support more granular access controls and security configurations, and integrate with compute services to automatically replace unhealthy hosts that fail to respond to the load balancer.

DNS also requires its connected endpoints to be accessible to the Internet, which is not always ideal. Following the security principle of defence-in-depth, when protecting critical data or infrastructure, anything that doesn’t need to be connected to the Internet, shouldn’t be. Network load balancers can be set up in protected virtual private networks to only allow access from allow-listed hosts or other trusted networks.

For these reasons, in some scenarios it will make sense to have the added complexity of both a frontend DNS load balancer to distribute traffic to the closest cluster, and backend application load balancers that provide more functionality and integration with your local infrastructure.

If you don’t need any functionality that isn’t supported by a DNS load balancer, can live with your servers being accessible from the Internet, and already manage your own health monitoring and host replacement strategy, then you can simplify your architecture by having a DNS load balancer directly route traffic to your backend servers.

Summary

We’ve discussed how concurrency can be applied at multiple levels:

Multi-threaded applications that run multiple tasks in parallel, such as querying several websites simultaneously
Multi-container or multi-process hosts that run multiple applications in isolation from one another, so that a given application can continue running if others fail
Multi-host clusters that enable horizontally scaling a service to process hundreds of thousands of concurrent requests
Multi-cluster services that enable routing traffic to local load-balanced clusters that can be independently scaled, to process millions of concurrent requests

Many distributed services now start with multi-host clusters for reliability and scalability reasons, so that any given host can be replaced without impacting customer service, and additional hosts can be added as needed.

A single load balancer and backend compute cluster can often handle hundreds of thousands of concurrent requests or more, while the load balancer may become a single point of failure for your service. Distributed DNS load balancers can help to mitigate this concern when it’s acceptable for your servers to be accessible from the Internet.

For applications where you need to handle millions of concurrent requests and have business requirements not met by a single DNS load balancer, such as needing granular access control for your backend servers or integrations with other infrastructure, a DNS load balancer in front of multiple load-balanced clusters can meet these demands with the trade-off of an additional layer of complexity.

Addendum

Before scaling your service to process millions of concurrent requests and paying hundreds of thousands of dollars to do so, make sure this is really necessary.

Would it be more efficient to extract some of your use cases to a separate microservice?

Are your hosts really doing unique work on every call? Could some of that work be deduplicated, or could the right application of a caching layer reduce your traffic and/or average latency by orders of magnitude?

Also, note that millions of concurrent users do not always translate into millions of transactions per second. If each user only needs to make a server request every few seconds, with multiple seconds between where they locally interact with rendered results, you may only have tens to hundreds of thousands of transactions per second, which while still high, lowers the required complexity of the system.

Software architecture design is an iterative process, and the optimal design will change along with the business, so it’s often worth starting with the simplest approach that meets current needs and can be scaled up or down as needed based on customer traffic. There’s no prize for building the most expensive service that no one uses.