Posts
-
Process for designing distributed systems
In this post I'll step through my process for designing distributed systems, with example questions and artifacts associated with each step.
We will take the following steps to design a new service:
1. Validate whether this service needs to exist
2. Clarify business requirements
3. Estimate scale
4. Define system interfaces and data models
5. Define data flow and storage
6. Define high-level system components
7. Design individual components
The artifacts of each step can be validated with stakeholders to ensure we're on the right track before continuing. They collectively add to a design document that can be referred to both while building the service, and afterwards to understand its inner workings.
-
Choosing between AWS compute services
When building a new service in AWS, it can be difficult to decide between all the available compute services. In this post I’ll give a brief overview of the main options and describe how I compare and choose between them for a given project.
-
AWS caching options
AWS provides several ways to cache data depending on your use case and infrastructure requirements. In many cases, you don’t need to invent the wheel and can use a fully-managed solution that does not require significant code changes.
By caching at the appropriate layer, you can optimize latencies while minimizing unnecessary load to your backend services, allowing you to scale at a reasonable cost.
-
AWS S3 bucket creation dates and S3 master regions
While working on functionality that depended on AWS S3 bucket ages, I noticed that published bucket CreationDate values didn’t always reflect when the buckets were created.
For example, when I called the S3 ListBuckets API a few minutes after updating a bucket access policy, the CreationDate value returned for that bucket was the time that I had modified the policy rather than the time that I had created the bucket. This was also reproduced when using the AWS CLI via the
aws s3api list-bucketscommand. -
Choosing a logging library for Kotlin or Java AWS Lambda functions
There are a lot of logging libraries to choose from when writing AWS Lambda functions in Kotlin or Java. Since Kotlin is fully interoperable with Java, Kotlin projects have access to both Kotlin-based and Java-based logging libraries.
This post compares some of the major options and evaluates which are most suitable for Lambda functions.
-
Lessons learned from using AWS Data Pipeline
One of the projects I worked on last year had a requirement to sync daily snapshots of data from Amazon RDS to Amazon S3 in order to support other internal services that ingested from S3 data providers.
I decided to use AWS Data Pipeline since it seemed to be a good fit for our use cases and worked well in a proof-of-concept.
This turned out to be a less than ideal solution although it did the job, and I learned several lessons from the experience.
-
Lessons learned from playing Go
Managing too many choices
Go forces us to ask ourselves those all-encompassing questions:
What should I do now?
What things are most important to me?
What should I be focusing on?
Or combined into a more immediate, actionable form: What is the most important thing for me to do right now?
-
Investigating 'You don't have enough free space in /var/cache/apt/archives/' errors
I was recently having issues upgrading Debian Linux due to
You don't have enough free space in /var/cache/apt/archives/errors, and it took an hour or so to resolve the root cause of the issue (one program consuming far too much space). Hopefully this post will help cut down the debugging time for other people who run into this problem.
subscribe via RSS