A Lambda That Dies at Import Cannot Page Itself

On the night of July 11th, UFC 329 ran fourteen fights, and my site reported the results of exactly none of them. Live event results are the single most time-sensitive feature on UltimateFightingStats.com. The event page and the homepage hero showed 0 of 14 fights complete for the entire card, including hours after fights had visibly finished on television. The component responsible, a scheduled Lambda that polls for results every five minutes, was crash-looping on every single invocation. It had been dying since 19:33 UTC, roughly ninety minutes before the first fight, and it kept dying until 05:22 UTC the next morning. Nearly ten hours of continuous failure.

Not one alert fired.

That is the part of this postmortem worth writing about publicly. The bug itself was small and its fix took minutes. The reason it ran unobserved for ten hours is a structural mistake I suspect a lot of engineers are making right now: every piece of monitoring I owned lived inside the process it was supposed to watch.

What I had built to survive failure

I want to be precise about the state of the system going in, because the lesson only lands if you see how much resilience was already there.

The ingestion path for live results had real defensive engineering inside it. Results flowed through an SQS queue rather than being written inline. The writer used an atomic claim, so the same result applied twice could never double-write or corrupt a fight record. The serving layer was deliberately uncached and force-dynamic while an event was live, so the moment data existed in the database it would display. And the poller carried its own watchdog: a check that would page me on Slack if a live event was underway and zero results were coming in.

On paper, that watchdog is exactly the alert you want for the failure that happened. In practice it was worthless, because it lived inside the handler. The handler never started.

The deploy that armed the bomb

Ninety minutes before the event, I deployed a change that gave the poller a Lambda layer containing headless Chromium, so it could render one of its sources in a real browser when a plain HTTP fetch was not enough. Attaching the layer meant setting an environment variable in the SAM template:

Layers:
  - !Ref ChromiumLayer
Environment:
  Variables:
    NODE_PATH: '/opt/nodejs/node_modules'

That one variable is the whole incident. Here is the mechanism.

AWS Lambda's Node.js runtime ships the AWS SDK v3 preinstalled at /var/runtime/node_modules, on the default module resolution path. To keep bundles small, my esbuild config marked @aws-sdk/* as external for most handlers, with a comment that says exactly what the assumption was: provided by Lambda runtime. The poller imports @aws-sdk/client-sqs to queue results, and for weeks that worked, because the runtime supplied it.

Setting NODE_PATH replaces Node's module search path. After the deploy, the poller's resolution path pointed only at the Chromium layer's node_modules, which contains no AWS SDK. The runtime's copy at /var/runtime/node_modules was no longer reachable. And because the bundler had marked the SDK external, it was not in the deploy artifact either.

Neither bundled nor resolvable. Every cold start died on the import statement, before a single line of my code executed:

Runtime.ImportModuleError: Error: Cannot find module '@aws-sdk/client-sqs'
Require stack: /var/task/event-poller.js

The sharpest detail is that my codebase already knew about this trap. Every other handler that mounts a layer and sets NODE_PATH bundles the AWS SDK for precisely this reason, with comments saying so. The invariant existed. It just existed as convention and copy-paste, not as anything a machine enforced. When I added the layer to a new handler under time pressure, I violated an invariant that nothing was checking.

Ten hours of noise, four deploys, zero signal

The last clean poll cycle ran at 19:28:36 UTC. The first import error hit at 19:33:37. From there, EventBridge did its job perfectly: it fired the schedule every five minutes, all night, and every invocation died at import.

What I did during the event made things worse in an instructive way. Watching results fail to appear, I went firefighting. Between 21:15 and 03:27 UTC I authored and merged four separate pull requests: making the queue path survive failures, adding a second data source as a fallback, probing a schema mismatch, fixing a query bug. Every one of those changes fixed a real latent issue. None of them touched the actual failure, and each merge redeployed the same broken bundle, which kept crash-looping exactly as before.

There is a name for what I was doing: hardening the inside of a request path that was never being entered. I was adding fallbacks and redundancy downstream of an import statement that threw before any of that code could load. The churn also buried the one true signal, a repeating ImportModuleError sitting in CloudWatch logs, under the noise of my own deploys.

At 03:28 UTC I ran a manual emergency reconcile from a CI workflow, bypassing the dead Lambda entirely. It wrote thirteen results directly to the database in a sixteen-second burst. The fourteenth, the main event, was not yet marked final at the source in that moment, so even the manual path missed it, and nothing was alive to retry.

I should be honest about the blast radius. UltimateFightingStats was pre-launch at the time. There were no paying subscribers refreshing that page; the person watching 0 of 14 all night was mostly me. What failed was not a revenue stream but the product's core promise, on the exact kind of night the product exists for, and the fact that I only learned about it by looking at my own website is the finding.

The next morning, following data instead of hunches

I stopped guessing and started reading evidence, in order.

First, database timestamps. All thirteen backfilled results shared a sixteen-second write window six and a half hours after the event started, and the main event row had never updated at all. That ruled out real-time ingestion having partly worked. Whatever wrote those rows, it was a bulk backfill.

Second, the source. I queried the upstream live API directly and the main event result was sitting right there: Holloway def. McGregor, TKO, round one, 1:09. The source was fine. The failure was mine.

Third, CloudWatch logs for the poller, which showed the same Runtime.ImportModuleError on every invocation from 19:33 onward. The smoking gun.

Fourth, the why. Reading the SAM template next to the bundler config made the mechanism obvious: NODE_PATH shadowing the runtime SDK while the bundle externalized it.

The fix was surgical. I rebuilt the poller bundle with the AWS SDK included and pushed it directly with aws lambda update-function-code, to the poller and to the reconciler, which carried the same latent bug. The first clean run happened at 05:22:41. One poll cycle later, the pipeline found all fourteen fights, ingested the main event at 05:23:04, and marked the event complete. Thirteen results had landed in a sixteen-second manual burst overnight; the system self-healed the rest in a single cycle the moment the import succeeded.

That recovery is worth noticing. The atomic-claim writer meant that a chaotic night of manual backfills and redeploys never double-applied or corrupted a single row. The uncached serving layer displayed data the instant it existed. The inside of the pipeline was genuinely sound. That was never the problem.

The fixes are boring on purpose

The permanent changes fall into two categories, and the difference between them is the entire lesson.

The first category fixes the bug. The poller and reconciler now bundle the AWS SDK like every other layer-mounted handler. More importantly, the build script now fails the build if any handler whose Lambda config sets NODE_PATH also externalizes @aws-sdk. The invariant that used to live in comments and convention is now enforced by tooling. This exact class of outage can no longer ship, regardless of who is tired or rushed when they add the next layer.

The second category fixes the blindness, and it was the number one action item in the postmortem: CloudWatch alarms on the Lambda Errors metric for both functions, delivering to an SNS topic and out to email. The alarm fires when two of three five-minute windows record an error, so a crash-loop gets detected in roughly ten to fifteen minutes instead of being discovered by a human looking at a web page. The recovery state notifies too, so I know when it clears.

Notice what that alarm is not. It is not clever. It knows nothing about fights, results, or live events. It cannot tell me what went wrong, only that invocations are failing. And it is the only thing on this list that would have caught the actual outage, because it is the only monitor that does not depend on my code successfully starting.

Two proposals came out of the same postmortem and follow the same logic from outside the process: a post-deploy smoke invoke of scheduled Lambdas, asserting the invocation returns without a function error, which would have caught this ninety minutes early; and a freeze window on non-trivial ingestion deploys in the hours before a scheduled event, which would have prevented it from being introduced that day at all.

Resilience has a direction

I have written before about the engineering discipline that lets a small team ship fast without breaking things. This incident is that discipline's shadow, the failure mode it did not cover.

Here is the pattern I now watch for. When engineers harden a system, we instinctively harden the inside of the request path: retries, fallbacks, queues, idempotent writes, second data sources, watchdogs in the handler. All of that work is real, and on the night in question all of mine held up. But every one of those mechanisms shares an unstated precondition: the process has to start. The failures that actually get you are the ones before and around the code, at the boundaries where your process is being loaded, resolved, scheduled, or configured, because that is exactly where your in-process instrumentation does not exist yet.

A watchdog inside the handler and a crash at import are not just a mismatch. They are the same design error viewed from two sides: I had assumed the thing being monitored and the thing doing the monitoring could safely be the same process.

They cannot. Whatever else you build, some dumb, external check has to sit outside the blast radius and count failures it does not understand. Mine costs almost nothing, knows nothing about my domain, and is the only reason the next ten-hour outage will last fifteen minutes.