AWS Posts

I spend most of my time building production systems on AWS. I also spend a growing fraction of my time working with LLMs to design and implement those systems. That combination raises a question I kept coming back to: how much does the model actually know about AWS? Not "can it write a CloudFormation template" or "can it debug a Lambda timeout." Those are execution tests. I wanted something more fundamental. If I ask a model about VPC peering limits, ElastiCache shard maximums, or the four-step Secrets Manager rotation lifecycle, does it know the answer? Does it know the current answer, or is it stuck on a value from two years ago?

Database costs are the second largest line item on most AWS bills I review, right behind compute. The problem is that RDS and Aurora pricing has enough moving parts to keep teams overspending for years without realizing it. Instance hours, storage, I/O operations, backup retention, snapshots, data transfer, Extended Support surcharges. Each component has its own optimization lever, and most teams only pull one or two of them. After years of running production databases on AWS and auditing bills across dozens of accounts, I have a reliable playbook for cutting RDS and Aurora spend by 40-65% without sacrificing availability or performance. This article lays out that playbook: every strategy, the math behind it, and the operational tradeoffs you need to understand before applying each one.

Every distributed system I have built forced a conversation about consistency before it forced a conversation about performance. Sometimes that conversation happened during design. More often it happened at 3 AM when a customer reported stale data, a checkout double-charged, or a dashboard showed a record that had been deleted twenty seconds earlier. The CAP theorem gets referenced in every system design interview and every architecture document, yet most teams still get caught off guard by what it actually means in production. The theorem itself is simple. Living with its consequences is where the engineering happens.

I have built real-time features into more systems than I can count: chat, live dashboards, IoT telemetry pipelines, collaborative editors, trading feeds, notification systems. Every one of them started with the same question: which protocol? The answer has never been the same twice, and the wrong choice has cost me weeks of rework more than once. WebSockets get all the attention. SSE gets overlooked. gRPC streaming gets misunderstood. Long polling gets dismissed too quickly. MQTT gets ignored entirely outside IoT circles. Each of these protocols solves a different problem, and the differences become painfully obvious only after you have built, deployed, and tried to scale the wrong one.

NAT Gateway costs are the silent budget killer in multi-account AWS environments. I've audited organizations spending $15,000/month on NAT Gateway data processing alone, spread across dozens of VPCs, each with its own pair of NAT Gateways. When I showed them they could cut that by 40-70% with a centralized egress VPC and Transit Gateway, the conversation shifted fast. The architecture is straightforward. The cost math requires attention. The routing setup has genuine gotchas that will break production if you get them wrong.

Every e-commerce team starts with a synchronous checkout. The API receives a cart, charges the card, decrements inventory, and returns a confirmation. It works until it doesn't. Payment processors time out. Warehouses operate on batch cycles. Inventory reservations race against each other across regions. I have rebuilt checkout and fulfillment pipelines three times across different organizations, and every rebuild ended at the same place: an asynchronous state machine with compensating transactions. AWS Step Functions is the right tool for this job, and this article covers the specific patterns, cost math, and operational lessons from running cart-to-delivery workflows in production.

I have built video analysis pipelines that process thousands of uploads per day, routing each file through multiple ML models for content moderation, face recognition, transcription, and object detection. The architecture I keep returning to uses SageMaker Pipelines as the orchestration backbone, with open-source models deployed across Processing Jobs and Batch Transform steps. This approach gives you full control over model versions, GPU instance selection, and inference logic without per-API-call pricing from managed AI services. The tradeoff is real: you own every container, every model artifact, and every failure mode. This article is the architecture reference for building that pipeline. I cover model selection for each analysis domain, the SageMaker Pipeline DAG design, GPU instance sizing, and the operational patterns that keep it running at scale. If you need a deeper understanding of how SageMaker Pipelines work under the hood, start with SageMaker Pipelines: An Architecture Deep-Dive.

I have built video content moderation pipelines both ways: one using AWS managed AI services orchestrated by Step Functions, another using open-source models running on SageMaker endpoints orchestrated by SageMaker Pipelines. Both architectures process uploaded video, detect unsafe visual content, transcribe audio for toxic language analysis, and route flagged material to human reviewers. They solve the same problem with fundamentally different trade-offs in cost, accuracy, operational overhead, customization depth, and data control. This article is the comparative analysis. I break down every dimension that matters when making this architectural decision, with real pricing data, accuracy benchmarks, and operational experience from running both approaches in production. For the full implementation details, see the companion articles: Video Content Moderation with Step Functions and AWS AI Services for the managed services approach and Video Content Moderation with SageMaker Pipelines and Open-Source Models for the open-source approach.

S3 is the most used service on AWS and, for many organizations, the single largest line item on the bill after compute. The insidious thing about S3 costs is that they creep. Nobody notices when a bucket grows from 10 TB to 50 TB over six months because the data is "just sitting there." Then the bill arrives and the storage line has tripled. I have audited AWS accounts where S3 spending dropped 60-70% after a week of lifecycle policies, storage class changes, and cleaning up forgotten multipart uploads. The savings were always there. Nobody had looked.

Every production architecture conversation I've had in the last five years eventually lands on the same question: can we go active/active across regions? The answer with Aurora has historically been "sort of, with significant caveats." Aurora Global Database gives you cross-region reads and fast failover. Write forwarding lets secondary regions send writes to the primary. Aurora DSQL promises genuine multi-region active/active with strong consistency. Each of these represents a different point on the spectrum between "one region writes, everyone else reads" and "any region writes, strong consistency everywhere." I've deployed all of them. The operational reality of each is more nuanced than the marketing suggests.

Every platform that accepts user-uploaded video faces the same operational reality: a single piece of unmoderated content can produce legal liability, advertiser flight, and reputational damage that takes months to repair. I have built content moderation systems for platforms processing thousands of hours of video per day, and the architectural pattern I keep returning to is a Step Functions orchestration layer coordinating AWS managed AI services. Rekognition scans frames for nudity, violence, hate symbols, and other policy violations; it also identifies celebrities and labels objects and scenes. Transcribe pulls the audio track into a timestamped transcript. Step Functions ties these asynchronous, variable-duration jobs into a single deterministic pipeline that writes a structured metadata package back to S3 alongside the original video. This article is the architecture reference for that pipeline: the service integrations, the ASL definitions, the failure modes, the cost model, and the operational lessons that only surface under production load.

Every relational database argument eventually becomes a religion debate. I have no interest in that. What I care about is how these engines behave under load, where they break, and what happens when you deploy them on Aurora's distributed storage layer. After running both MySQL and PostgreSQL in production on Aurora across dozens of services, the differences that actually matter have little to do with SQL syntax preferences. They live in storage engine internals, MVCC implementation, connection handling, and the operational failure modes that surface at 3 AM when your on-call phone goes off.

EC2 is the single largest line item on most AWS bills. It is also the line item where the gap between what teams pay and what they should pay is the widest. I have audited AWS accounts where compute spending dropped 45% in the first month after applying the strategies in this article. No performance loss. No architectural changes. Just purchasing mechanics, instance selection, and scheduling discipline. The savings were always available. The team had never looked.

DynamoDB sits at the center of more AWS architectures than any other database service. I've used it for everything from mobile backends handling millions of daily active users to event-sourced systems processing tens of thousands of writes per second. Most teams treat it as a simple key-value store, plug it in, and move on. That works until they hit a hot partition at 3 AM, discover their GSI is throttling independently of the base table, or realize their on-demand table costs three times what provisioned capacity would have. After years of running DynamoDB at scale, I've accumulated enough operational scars to fill this reference. Patterns, trade-offs, cost traps, and the internal mechanics that explain why DynamoDB behaves the way it does.

User authentication looks simple from the outside. A sign-up form, a login page, maybe a "Forgot Password" link. Behind that surface sits a sprawling system of token management, federation protocols, MFA enrollment, session lifecycle, Lambda triggers, and security hardening decisions that are expensive to reverse once users are in the system. I have built authentication layers on AWS Cognito for applications ranging from internal tools with fifty users to consumer platforms with hundreds of thousands, and the lessons from those projects inform every recommendation in this article.

Deployment automation is the single most impactful investment a team can make in operational reliability. Manual deployments (SSH into a box, pull the latest code, restart the service, pray) are slow, and they are the root cause of a disproportionate number of production incidents. Every manual step is an opportunity for human error: the wrong branch, a missed configuration file, a forgotten service restart, a deployment to the wrong environment. Having spent years building and operating deployment pipelines across hundreds of EC2 instances, Lambda functions, and ECS services, I have watched CodeDeploy evolve from a simple EC2 deployment tool into the foundational deployment engine that underpins most serious AWS CI/CD architectures. It lacks glamour and thorough documentation of its deeper behaviors, yet it is the service that actually puts your code onto your compute.

Nobody wants to own build infrastructure. Everybody depends on it. I have spent years managing Jenkins clusters, debugging flaky build agents, patching security holes on build servers, and scaling CI/CD capacity for growing engineering teams. The operational overhead? Wildly disproportionate to the business value. AWS CodeBuild kills that burden. It is a fully managed, container-based build service. Fresh, isolated compute for every build. Automatic scaling to any workload. You pay only for the minutes you actually use. The architectural decisions baked into CodeBuild (ephemeral containers, pay-per-minute pricing, deep AWS service integration) reflect hard-won lessons about what matters in build infrastructure. And what does not.

Most teams bolt messaging onto their architecture after the first production outage caused by synchronous service-to-service calls. A payment service calls an inventory service directly, the inventory service is slow, the payment service times out, the customer gets charged twice. Suddenly everyone agrees the system needs a queue. I have spent years designing event-driven systems on AWS: order processing pipelines handling millions of transactions per day, IoT telemetry ingestion at hundreds of thousands of events per second, multi-region fan-out architectures coordinating dozens of microservices. AWS offers at least six distinct messaging and eventing services. Each solves a different problem. Choosing wrong means either overengineering a simple notification flow or discovering at 3 AM that your architecture cannot handle the throughput your business requires. This article is not a getting-started guide. It is an architecture reference for engineers who need to pick the right service, configure it correctly, and avoid the failure modes that surface only under production load.

I keep running into the same mistake across teams. They treat their build tool and their pipeline orchestrator as one thing. They'll jam deployment logic into CodeBuild buildspec files or Jenkins jobs, and six months later nobody can explain why a release failed or who approved what. The release process turns brittle, opaque, impossible to audit. CodePipeline exists to fix this. It coordinates builds, tests, and deployments into a workflow you can actually observe and reason about, with clearly defined gates, approvals, and rollback boundaries.

Having spent years packaging Lambda functions as zip archives, I hit the wall that every team eventually hits: the 250 MB deployment package limit. The first time it happened was an ML inference function with a PyTorch model and its dependency tree. We burned weeks trying to strip binaries, use Lambda Layers creatively, and shave megabytes from scipy. When AWS launched container image support for Lambda in December 2020, it raised the size ceiling to 10 GB and fundamentally changed how I think about Lambda packaging, base image standardization, CI/CD pipelines, and the boundary between serverless and container workloads. Container images let you use the same Dockerfile, the same build toolchain, and the same base image across Lambda, ECS, and Fargate, which eliminates an entire category of "works in my container but not in Lambda" problems.

Manually shipping containerized Lambda functions works for experiments. Build the image locally, push it to ECR, update the function, verify it works. Fine for one function updated once a week. The moment you have multiple functions, multiple environments, or more than one engineer deploying? It falls apart. Someone forgets to tag the image. Someone pushes to the wrong ECR repository. Someone updates production instead of staging. I have personally done all three of those in a single bad afternoon. The worst one is deploying a broken image with no way to roll back except pushing the previous image and hoping you remember which tag it was. I have watched this exact progression on enough teams to know the pipeline question is never "if" but "when," and the answer is almost always "after something breaks in production at 2 AM."

Any iOS app with real users generates telemetry. Session starts, feature usage, error events, performance metrics, purchase funnels. Most teams start by shipping all of it to Amplitude or Mixpanel and calling it done. That works for a while. Then the monthly invoice triples, you discover the vendor's data model cannot answer a question your PM asked three days ago, and you realize you are paying somebody else to store your data in a format optimized for their business.

Infrastructure as Code is one of those concepts that every cloud team claims to practice, yet the architectural differences between the tools they use (and the downstream implications for team velocity, operational safety, and organizational scaling) are rarely examined with the rigor they deserve. I have provisioned and managed infrastructure across hundreds of AWS accounts using all four major IaC tools over the past decade, from wrestling with early CloudFormation YAML to adopting CDK for its high-level abstractions to running Terraform at scale across multi-account landing zones. That experience has given me strong opinions about when each tool shines and where each one will hurt you in production.

Five ways to expose a Lambda function over HTTP. At least. AWS keeps adding more. Most teams pick API Gateway on day one and never revisit that decision. Fine. API Gateway handles a lot.

I have deployed SageMaker Pipelines across production ML platforms ranging from simple training-to-deployment workflows to multi-model ensembles with conditional quality gates. It is a fundamentally different orchestration paradigm than what most teams expect. The SDK trades orchestration flexibility for zero-cost execution, native SageMaker integration, and first-class support for the ML lifecycle patterns that actually matter in production: parameterization, caching, experiment tracking, and model registration. This article goes deep on the internal workings. How the execution engine resolves dependencies. How caching decisions happen. How data moves between steps. How to design pipelines that hold up under real operational pressure. If you are still deciding between Pipelines and Step Functions, I cover that comparison in Building Large-Scale SageMaker Training Pipelines with Step Functions. I assume here that you have already committed to Pipelines and want to know what is actually going on beneath the Python API.

I have spent the last several months orchestrating ML training pipelines that coordinate dozens of SageMaker jobs: preprocessing, feature engineering, distributed training, hyperparameter tuning, evaluation, conditional deployment. The pattern I keep seeing is that teams pour effort into model architecture and training code while treating the orchestration layer as an afterthought. Then the orchestration layer is exactly where the ugliest production failures happen. This article is my architecture reference for building training pipelines on AWS Step Functions at scale. If you have already read my AWS Step Functions: An Architecture Deep-Dive, the execution model and state types will be familiar. Here we get into the problems specific to ML pipelines: training jobs that run for hours, spot instances that vanish mid-epoch, models that need human sign-off before they touch production traffic, and the retraining loops that keep everything from going stale.

AWS OpenSearch Service runs behind more production workloads than most engineers realize: log analytics, full-text search, security event monitoring, vector similarity search. Lots of teams deploy it. Few really understand what's happening underneath. I've designed and operated OpenSearch clusters for years now, everything from small dev setups to multi-petabyte production deployments ingesting billions of documents a day. My first cluster went red within 48 hours. I've learned a lot since then.

I recently published a deep-dive into CloudFront's architecture covering its internals, origin architecture, cache behavior, security, and edge compute capabilities. The most common follow-up question: should we use CloudFront or Cloudflare?

Amazon CloudFront is one of the most underestimated services in the AWS portfolio. Most teams think of it as a caching layer you put in front of your S3 bucket or Application Load Balancer to speed up static asset delivery. That understanding was roughly correct in 2015. It is incomplete today. CloudFront has evolved into a globally distributed edge compute and security platform that handles request routing, WAF enforcement, DDoS mitigation, authentication, A/B testing, header manipulation, and serverless compute, all before a request ever reaches your origin. This article covers the architectural patterns and operational lessons I have accumulated from architecting systems that serve traffic through CloudFront across dozens of AWS accounts.

ElastiCache looks easy. Deploy a managed cache, point your app at the endpoint, enjoy sub-millisecond reads. Then production happens. Engine selection, cluster topology, eviction policy, replication strategy, connection management, failover behavior: every one of these choices determines whether your caching layer holds up or collapses at 3 AM on a Saturday. I've spent years building and running ElastiCache clusters serving millions of requests per second. Some fronted relational databases with multi-terabyte datasets. Others were dead-simple session stores. All of them taught me something. Usually through failure first.

I've yet to ship a production architecture on AWS that doesn't involve Elastic Load Balancing somewhere. Most teams slap a load balancer in front of their service and move on. Fair enough. That works until it doesn't. After debugging enough 502 cascades at 2 AM, I can tell you: the differences between the four ELB types, and what happens when you pick wrong, deserve way more attention than they typically get. So here it is. Patterns, trade-offs, and operational scars from years of running load-balanced architectures at scale.

Most teams ignore Step Functions until they find themselves writing ad-hoc state management code inside Lambda functions, chaining queues together with brittle retry logic, or building homegrown saga coordinators that nobody wants to maintain. The service is a fully managed state machine engine that coordinates distributed components (Lambda functions, ECS tasks, DynamoDB operations, SQS messages, human approvals, and over two hundred other AWS service actions) through a declarative JSON-based workflow definition. I have spent years building production orchestration on Step Functions: ETL pipelines processing billions of records, saga-based transaction systems spanning dozens of microservices, real-time data enrichment at tens of thousands of events per second. This article captures what I have learned about the internals, the trade-offs, the failure modes, and the patterns that survive contact with production traffic.

Amazon API Gateway sits in front of most serverless and microservice architectures on AWS. Three distinct API types, a control plane versus data plane split, a layered throttling hierarchy, a caching layer, a rich integration model. Most teams deploying API Gateway never dig into these mechanics. I have spent years building and operating API Gateway-backed systems handling everything from low-traffic internal tools to production APIs processing tens of thousands of requests per second, and I learned most of the hard lessons the hard way.