Beyond CRUD: Engineering a Reliable Backend

Introduction

Most backend tutorials end the moment an endpoint returns a 200 OK. In reality, that's where the interesting engineering begins.

Recently, I received a take-home assignment to build a URL auditing service called PagePulse. At first glance, it looked like another backend project. But the actual objective wasn't just to build an API—it was to build a service that could handle real-world scenarios such as concurrent traffic, slow external services, repeated requests, and failures gracefully.

This article isn't a tutorial on building a backend API. Instead, it's a walkthrough of the engineering concepts I implemented to take a simple CRUD service a step closer to something that resembles a production-ready backend.

For this project, I used FastAPI as the backend framework, but the concepts discussed apply to backend services regardless of the technology stack.

Why Basic CRUD APIs Aren't Enough

Many developers think backend development is simply writing CRUD endpoints that read and write data. Most beginner tutorials stop there, making it seem like that's all there is to backend engineering.

While CRUD operations are the foundation of many applications, they're only a small part of a reliable backend. A service that works perfectly during development may fail under heavy traffic, slow third-party APIs, invalid user input, or unexpected failures.

Building reliable software means thinking beyond business logic. You also need to consider performance, resilience, scalability, observability, and fault tolerance.

Input Validation

One of the first things every backend should do is validate incoming requests before performing any expensive operations.

Rejecting invalid requests early prevents unnecessary database queries, external API calls, and CPU usage. It also ensures that the server only processes data in the format it expects.

FastAPI makes this extremely convenient through Pydantic, which validates request bodies, query parameters, and path parameters before the request even reaches the business logic.

Timeouts and Retries

External services aren't always reliable. Sometimes an API is temporarily slow, sometimes it doesn't respond at all.

Without proper timeouts, a request may hang indefinitely, consuming server resources and degrading the experience for every other user.

Adding sensible timeout values ensures that requests fail quickly instead of waiting forever. Combined with retries for transient failures, the application becomes much more resilient while avoiding unnecessary resource consumption.

Redis Caching

One of the biggest performance improvements came from introducing caching.

The idea is simple: if the application has already fetched and processed the same information recently, there's no reason to perform the exact same work again.

Instead of repeatedly calling external services for identical requests, the application first checks Redis. If the data exists, it's returned immediately (a cache hit). Otherwise, the application fetches fresh data, stores it in Redis, and serves the response (a cache miss).

This significantly reduces response times, decreases external API calls, and improves higher latency percentiles like p95.

For this project, I used a Redis Docker container during local development and migrated to Railway's free Redis instance for deployment.

Rate Limiting

A public API should never allow unlimited requests from a single client.

Without any restrictions, one client—whether intentionally or accidentally—can consume a large portion of your server resources and negatively impact every other user.

Rate limiting solves this by defining how many requests a client can make within a specific time window. If the limit is exceeded, the server responds with a 429 Too Many Requests response.

Besides protecting infrastructure, rate limiting also encourages fair usage and helps prevent abuse, accidental request storms, and simple denial-of-service attempts.

Concurrency Limiting with asyncio.Semaphore

Concurrency refers to how many requests your application can process simultaneously.

Imagine your API suddenly receives hundreds of requests at the same time. Allowing every request to execute immediately may overwhelm your server, exhaust available resources, or even cause failures.

Instead of allowing unlimited concurrency, I used asyncio.Semaphore to limit the number of requests being processed simultaneously. Additional requests simply wait until resources become available.

Latency is commonly measured using percentiles.

• p50 represents the median response time experienced by half of the users.
• p95 represents the slower requests experienced by only about 5% of users.

Optimizing p95 is particularly important because it reflects the worst experience that a noticeable portion of users receive.

For this assignment, I performed load tests with approximately 100, 250, 500, and 1000 concurrent requests. After multiple rounds of testing, I found that allowing around 20 concurrent requests produced the best balance between throughput and reliability for this particular application.

Every application is different, so concurrency limits should always be determined through testing rather than guesswork.

Structured Logging

Logging is often overlooked during development, yet it becomes one of the most valuable tools once an application is deployed.

Simple log statements such as:

INFO Request received
INFO Response sent
ERROR Timeout occurred

don't provide enough context when debugging production issues.

Structured logging organizes log data into consistent fields such as request ID, endpoint, latency, status code, and timestamps.

{
  "request_id": "...",
  "endpoint": "/api/v1/audit",
  "status": 200,
  "latency_ms": 142
}

These logs are significantly easier to search, filter, and analyze using centralized logging systems.

For this project, I used structlog to produce structured JSON logs throughout the application.

Continuous Integration with GitHub Actions

Writing code is only one part of software development. Ensuring that every change is safe to deploy is equally important.

To automate this process, I configured GitHub Actions as the project's Continuous Integration (CI) pipeline.

Whenever code is pushed to the repository, GitHub automatically runs linting and the test suite. If any test fails, the deployment process stops immediately.

This prevents broken code from reaching production, catches regressions early, and provides confidence that every deployment has passed a basic quality check.

Automating these checks also removes the need to manually verify every change before deployment.

Final Thoughts

Building PagePulse changed the way I think about backend engineering.

Writing the endpoint was actually the easiest part. The real challenge was making the service reliable under failures, slow dependencies, repeated requests, and concurrent traffic.

Some of the biggest takeaways from this project were:

• Good logging makes debugging dramatically easier.
• Caching introduces its own design trade-offs but significantly improves performance.
• Rate limiting protects your infrastructure from abuse.
• Concurrency control helps maintain stability under load.
• Infrastructure and operational concerns are just as important as application code.

A backend isn't "production-ready" because it has more features. It's production-ready because it continues to behave predictably when things don't go as planned.

Production software isn't defined by the number of features it offers—it's defined by how gracefully it handles failure.