A tenant registers a delivery callback URL. The Hub sends a POST whenever Resend reports an email bounced or was opened. The body is JSON. The header X-Hub-Signature carries an HMAC-SHA256 digest computed over that body using a secret that only the Hub and the tenant know. The tenant recomputes the digest on receipt and compares. If they match, the request is authentic. If they do not, the request is dropped.
This is the standard webhook-signing recipe. GitHub uses it. Slack uses it. Resend uses it on the way in. Now the Hub uses it on the way out.
The obvious implementation looks like this:
const body = JSON.stringify(event);
const digest = crypto.createHmac('sha256', secret).update(body).digest('hex');
It worked in tests. It worked in the integration suite. It worked against a mock callback server. Then I tried to write a verification snippet for the tenant docs in Python, and the digests did not match.
Why JSON.stringify Sinks Webhook Signing
JSON.stringify is not deterministic in any language. The order of keys in the output depends on the order they were inserted into the object. Two semantically identical objects can serialize to different bytes:
JSON.stringify({ a: 1, b: 2 }) // '{"a":1,"b":2}'
JSON.stringify({ b: 2, a: 1 }) // '{"b":2,"a":1}'
Both encode the same data. Both hash to a different digest under HMAC. A tenant who receives the callback, parses the JSON, then re-serializes it to compute their own signature, will produce different bytes than the Hub did, even when nothing about the payload changed.
This is the silent failure mode of webhook signing. Both sides are doing exactly what they were told. Both digests are correct for the bytes they were computed over. The bytes are different. The signatures mismatch. The tenant rejects the callback as a forgery, and there is no error message that explains why.
The Hub side controls the bytes that go on the wire. Whatever I sign, I send, and as long as the tenant hashes the raw bytes of what arrived, the digests agree. But "hash the raw bytes" is fragile. Express's body parser, FastAPI's request handler, any middleware in the chain that touches the body, will replace the original bytes with a re-stringified version. The tenant integration breaks not because of a bug in their code, but because of a default in the framework they used.
Three Things The Scheme Has To Do
I needed three things. The signing scheme had to be deterministic, so two encodings of the same data produced the same digest. It had to be specifiable, so a tenant could implement verification in any language without reading the Hub's source. And it had to be reusable, because Phase 7 added the delivery callback first but planned to extend the same signing pattern to suppression callbacks, generic webhook fan-out, and alert callbacks downstream.
The scheme I picked is canonical JSON: sort object keys recursively, build the JSON string by hand, hash that. Any tenant in any language that performs the same recursive sort produces the same bytes and the same digest. The contract becomes the canonicalization rule, not the byte stream.
Canonical JSON in Seventy Lines
The signing module is small. Three exported functions, one private helper, around 70 lines total in src/lib/outbound-signing.ts.
export function canonicalJson(value: unknown): string {
if (value === null || typeof value !== 'object') {
return JSON.stringify(value);
}
if (Array.isArray(value)) {
return '[' + value.map((item) => canonicalJson(item)).join(',') + ']';
}
const keys = Object.keys(value as Record<string, unknown>).sort();
const parts = keys.map((k) => {
const v = (value as Record<string, unknown>)[k];
return JSON.stringify(k) + ':' + canonicalJson(v);
});
return '{' + parts.join(',') + '}';
}
Three things to notice. Primitives delegate to JSON.stringify, which handles string escaping and number formatting correctly per the spec. Object keys get sorted lexicographically before serialization. The function is recursive and uses no whitespace, so the output is byte-identical to what a JCS-style canonicalizer in Python or Go would produce given the same input.
What the function deliberately does not handle: Date, Map, Set, BigInt. None of those round-trip through JSON cleanly. Outbound callback payloads are intentionally pure JSON values, type-checked at the call site. Refusing to support exotic types keeps the canonicalization rule auditable in one paragraph rather than a footnote.
The signing function builds on top:
export function buildSignedOutboundRequest(
event: unknown,
secret: string,
): { body: string; signatureHeader: string } {
const body = canonicalJson(event);
const digest = crypto.createHmac('sha256', secret).update(body).digest('hex');
return { body, signatureHeader: `sha256=${digest}` };
}
The body and the signature are returned together. This is the design decision that took the longest to reach. The first version returned them separately and let the caller choose what to send over the wire. That created room for a class of bugs where the caller signed one set of bytes and sent a different set: log the body, then send JSON.stringify(event), sign with canonicalJson(event), and ship the wrong thing. Returning them as a pair means the bytes that get hashed are the bytes that get transmitted. The caller cannot accidentally split them.
The sha256= prefix on the header value follows the convention GitHub and Slack use. It exists so that the scheme can later add SHA-512 or another algorithm without breaking parsers, and so tenants writing verification code can split on = and validate the algorithm name first.
The Test Story That Inverted
I assumed the testing story would be the hard part. Cryptographic functions resist mocking. You cannot mock HMAC into returning a fixed digest, because the whole point is determinism over real input. So I expected a lot of test infrastructure: helper builders, fixture payloads, golden digests checked into the repo.
The opposite happened. Because the function is pure and deterministic, the tests are five-line affairs. Pass an input. Assert on the output digest. No mocks. No setup. The HMAC determinism tests run in microseconds:
test('signs canonically: key order does not affect digest', () => {
const secret = 'test-secret';
const a = signOutboundPayload({ a: 1, b: 2 }, secret);
const b = signOutboundPayload({ b: 2, a: 1 }, secret);
expect(a).toBe(b);
});
That single test catches the entire class of bug that motivated the canonicalization in the first place. If JSON.stringify ever sneaks back into the signing path, this test fails immediately. It catches regression in any future refactor.
The harder testing problem turned out to be on the dispatch side, not the signing side. The route that triggers a callback uses void dispatchDeliveryCallback(...).catch(...). That is fire-and-forget: the webhook handler can return 200 to Resend without waiting for the tenant's callback URL to respond. Production behavior is correct: the Hub never blocks Resend on a slow or down tenant. But naive integration tests race against the dispatch. The test calls app.inject(), gets a 200 response, then asserts on the database row that should have been updated by the callback handler. The callback might still be in flight.
Three options for testing this. Option one: add a test-only hook that returns the in-flight promise so the test can await it. Option two: use setImmediate() and poll. Option three: keep the production fire-and-forget contract untouched and use a polling helper in the test that waits up to two seconds for the assertion to become true.
I picked option three. The production behavior is the contract. Adding a test-only side channel means production code carries a hook that exists only for tests, which inverts the relationship: the test is supposed to verify the production code, not constrain it. The polling helper is uglier but bounded: typical cases resolve in 10 to 50 milliseconds because both the mock callback server and the database are local, and the upper bound of two seconds catches a real hang without hanging the test suite forever.
What Ships in Phase 7
The signing module ships in batch 011 of Phase 7. The webhook integration in batch 012 wires the dispatcher to call it on every Resend delivery event. The end-to-end test in batch 013 fires a mock Resend webhook into the Hub, watches the database row get inserted, then watches a mock callback server receive a signed POST with the right digest. All three batches add 14 new tests on top of the existing 309. None of them mock HMAC.
The tenant verification snippet in USER_SETUP.md Section 10 is intentionally explicit about reading raw request bytes. In Node, express.raw({ type: 'application/json' }) instead of express.json(). In Python, request.get_data() instead of request.json(). The snippet works because the Hub signs canonical JSON and sends those exact bytes. A tenant who hashes the raw body bytes verifies correctly. A tenant who reparses and re-serializes does not, and the snippet says so up front.
Phase 7 7b extracted canonicalJson and signOutboundPayload from the delivery-callback module into src/lib/outbound-signing.ts so the next callback family (suppression notifications, generic webhook fan-out, alert callbacks) uses the same scheme without copy-paste. The canonicalization rule is now load-bearing for every outbound HTTP call the Hub will ever make.
The signing module is 73 lines. The bug it prevents took an afternoon to find and would have taken a tenant a week to debug on their side.