Image credit: Retro Tech Dreams

I was scrolling through old tech nostalgia pictures online and ran into this screenshot of a classic Windows 98 desktop, mountain lion wallpaper, AIM running man, Encarta 99, the works. It stopped me mid-scroll, because it looked almost exactly like the desktop I grew up with.

In January 2000, my father brought home our first computer: an assembled Pentium III, somewhere between 2 and 5 GB of hard drive space, and 128MB of RAM. It had a Creative sound card installed, which felt like a big deal at the time, everything suddenly had real sound instead of beeps. I was 9 years old, and that machine became the center of my world for the next few years.

I spent hours on Need for Speed, Sonic, a Star Wars game I only remember as “Sith,” and whatever “ultra fighter” game I could get running. In between the games, I quietly taught myself PowerPoint and Word, no one showed me, I just clicked around until things made sense. It’s funny looking back now, since that’s still basically how I learn most software today.

That PC broke down more than once over the years. Each time, my father just took it in and got it repaired rather than giving up on it. I didn’t think much of it then, but it’s clear now that those repairs were him making sure I kept having a machine to learn on.

Finding that screenshot today was a small thing, but it pulled up a lot. That beige tower with the Creative sound card was the start of a path that eventually turned into a career in software.

Why Vanilla Gradient Descent Struggles

Loss surfaces aren’t bowls — they’re full of narrow ravines, where the gradient in one direction is much steeper than in another. With a single fixed learning rate, you either oscillate back and forth across the steep direction, or move too slowly along the shallow one. Picking a learning rate that’s safe for the steep direction makes progress along the shallow direction painfully slow.

Momentum

Momentum keeps a running average of past gradients and updates the weight using that average instead of the raw gradient:

v = beta * v + (1 - beta) * grad
w = w - lr * v

Intuitively, it’s a ball rolling downhill: gradients that consistently point the same way accumulate velocity, while oscillating gradients (the back-and-forth across a ravine) cancel out in the average. This damps oscillation and speeds up movement in directions where the gradient is consistent.

RMSProp

Momentum doesn’t address that different parameters can need very different step sizes. RMSProp tracks a moving average of the squared gradient per parameter, and divides the update by its square root:

s = beta * s + (1 - beta) * grad^2
w = w - lr * grad / (sqrt(s) + eps)

Parameters with consistently large gradients get their effective learning rate shrunk; parameters with small, infrequent gradients get it boosted. Each weight effectively gets its own adaptive learning rate.

Adam

Adam combines both ideas: momentum for the direction, RMSProp-style scaling for the step size, plus a bias correction term because the moving averages start at zero and are biased low early in training.

m = beta1 * m + (1 - beta1) * grad
s = beta2 * s + (1 - beta2) * grad^2

m_hat = m / (1 - beta1^t)
s_hat = s / (1 - beta2^t)

w = w - lr * m_hat / (sqrt(s_hat) + eps)

t is the current training step, and the bias correction terms matter most early on — without them, m and s are artificially small for the first several updates.

Why It Matters

This is why optimizer=Adam is the default in nearly every training script: it inherits momentum’s resistance to oscillation and RMSProp’s per-parameter adaptivity, with defaults (beta1=0.9, beta2=0.999) that work reasonably well across a huge range of problems without manual tuning. Understanding what each piece corrects for makes it much easier to diagnose training instability — a loss that oscillates wildly often means the effective step size is too large for the steepest direction in the loss surface, regardless of which optimizer is doing the stepping.

Event sourcing solves this by changing what you store: not the current state, but every event that led to it.

The Core Idea

Instead of a balance column you update in place, you store an append-only log of events:

AccountOpened    { account: "acct_1" }
FundsDeposited   { account: "acct_1", amount: 1000 }
FundsWithdrawn   { account: "acct_1", amount: 200 }
FundsDeposited   { account: "acct_1", amount: 50 }

The current balance isn’t stored anywhere — it’s derived by replaying the events: 0 + 1000 - 200 + 50 = 850. The events are the source of truth; every view of “current state” is a projection computed from them.

Why This Fits Financial Systems

• Audit trail for free. Regulators and auditors don’t want “the balance is 850” — they want to know how it got there. With event sourcing, the full history is the data model, not a separate audit log bolted on afterward.
• Point-in-time reconstruction. Replaying events up to a given timestamp reproduces the exact state at that moment, which is exactly what reconciliation and historical reporting need.
• Corrections without overwrites. If a deposit was recorded in error, you don’t edit history — you append a FundsDepositReversed event. The mistake and its correction both remain visible.

Snapshotting

Replaying years of events on every read doesn’t scale. The common fix is snapshotting: periodically persist the derived state (e.g., the balance after every 1,000 events), and on read, load the nearest snapshot and replay only the events after it.

CQRS

Event sourcing pairs naturally with Command Query Responsibility Segregation (CQRS): writes append events, while reads are served from one or more projections (materialized views) built by consuming the event stream. This lets you build a fast “current balance” table for queries while keeping the event log as the durable, authoritative history.

The Trade-offs

It isn’t free:

• Complexity. You’re building an event store, projections, and replay logic instead of an UPDATE statement.
• Eventual consistency. If projections are updated asynchronously, a read immediately after a write can be stale.
• Schema evolution. Old events were written under old assumptions; your event handlers need to keep understanding them as the system evolves.

When It’s Worth It

Event sourcing earns its complexity when the history itself is a business requirement — ledgers, payment systems, trading platforms, anything with compliance or audit obligations — rather than systems where “what is the value right now” is the only question anyone ever asks.

Idempotency keys are the standard fix, and the pattern shows up everywhere from Stripe’s API to internal settlement systems.

The Problem

Imagine a POST /transfers endpoint. A client sends a request, the server processes the transfer, but the response is lost to a network blip before the client receives it. The client, seeing a timeout, retries. From the server’s point of view, two valid requests arrived — so it processes the transfer twice.

Retrying is the correct client behavior. The server’s job is to make sure retries are safe.

The Pattern

The client generates a unique key — typically a UUID — and sends it on every attempt of the same logical operation:

POST /transfers
Idempotency-Key: 7e3f9c1a-7b2e-4e29-9c34-3a1d9f9b9b40

{ "from": "acct_1", "to": "acct_2", "amount": 500 }

On the server:

1. Look up the key in a dedicated store (a table or cache with a TTL).
2. If it doesn’t exist, mark it as “in progress,” process the request, store the result, and return it.
3. If it exists and is “in progress,” reject or hold the duplicate request — a concurrent retry shouldn’t process the transfer a second time while the first is still running.
4. If it exists and is “complete,” return the stored response without reprocessing anything.

key_record = store.get(idempotency_key)

if key_record is None:
    store.put(idempotency_key, status="in_progress")
    result = process_transfer(payload)
    store.update(idempotency_key, status="complete", response=result)
    return result

if key_record.status == "in_progress":
    raise ConflictError("request already in flight")

return key_record.response

Edge Cases That Matter

• Key reuse with a different payload. A client might accidentally send the same key with a different amount. Store a hash of the request body alongside the key, and reject mismatches — otherwise you silently return the wrong result for the second request.
• Race conditions. Two retries can arrive within milliseconds of each other. The “in progress” state needs an atomic write (a unique constraint in a relational table, or a conditional SETNX in a key-value store) so only one request wins the race.
• Expiry. Keys can’t live forever. A TTL of 24 hours is common — long enough to cover realistic retry windows, short enough to bound storage growth.
• Scope. Keys should be scoped per client or per API credential, not global, to avoid collisions between unrelated callers.

Why It Matters

This is exactly how Stripe, Adyen, and most payment processors prevent double charges, and it’s a standard expectation for any API that mutates money, inventory, or anything else that can’t simply be re-run. If you’re building APIs for a finance system and retries aren’t idempotent, you have a double-processing bug waiting to happen in production — it’s just a matter of when a slow network finds it.

The Core Idea

A neural network is a function composed of many simpler functions. Training it means finding weights that minimize a loss function — typically the difference between predicted and actual outputs. Gradient descent does this iteratively: compute how much each weight contributes to the loss, then nudge it in the opposite direction.

Backpropagation is how we compute those gradients efficiently.

A Simple Example

Consider a two-layer network with a single training example. The forward pass computes:

z1 = W1 * x + b1
a1 = relu(z1)
z2 = W2 * a1 + b2
loss = (a2 - y)^2

The backward pass works in reverse, computing dLoss/dW for every weight by repeatedly applying the chain rule:

dLoss/dW2 = dLoss/da2 * da2/dz2 * dz2/dW2

Each term is easy to compute locally. The key insight is that each layer only needs the gradient flowing in from the layer ahead of it — this is what makes backprop efficient.

Why It Matters

Before backpropagation, training deep networks was largely intractable. The 1986 paper by Rumelhart, Hinton, and Williams popularized the algorithm and kicked off the first wave of neural network research. Forty years later, the same fundamental algorithm powers systems with hundreds of billions of parameters.

Understanding it at this level — not just using it via loss.backward() — changes how you think about network architecture, initialization, and why certain training problems (like vanishing gradients) happen in the first place.

Next Steps

If you want to go deeper, implement a small autograd engine from scratch. Andrej Karpathy’s micrograd is an excellent reference — around 150 lines of Python that implement the core of what PyTorch does under the hood.