The Lovable app took a 4242 card. Stripe Checkout opened. The success page rendered. The webhook returned 200.
A week later the same customer was charged twice for one booking. Stripe had retried payment_intent.succeeded. The handler ran again. Nobody had written a line of the code they were reviewing.
That pattern showed up talking to Base44 and Lovable builders while building FetchSandbox. Junior, mid-level, senior. Same two beliefs. Skill level did not change them.
Short answer
Stripe test mode and a green CI job do not prove the generated integration survives a second delivery. Force the retry on purpose. Deduplicate on the stable event.id. Confirm one payment creates one side effect. Then run that same failure on the still-broken code and the patched code so an exit code, not another model, decides whether the fix held.
Lovable, Base44, Bolt, v0, and Replit are good at the first charge. Production is still where most teams meet the retry.
What two beliefs keep shipping unverified payments?
The first belief is that the model will handle edge cases because the handler looks smart.
It often does look smart. The generated webhook verifies the signature, switches on payment_intent.succeeded, writes a booking row, and returns 200. That is the path Stripe test cards exercise. It is also the path a reviewer can read in five minutes.
The path the model did not see is the one Stripe documents as normal: at-least-once delivery. The same event arrives again with the same id. A card declines after the first attempt. The success response never reaches your app, so the client retries.
The second belief is that mocks and CI already cover this.
They cover what you stubbed. A mock that returns one payment_intent.succeeded will not replay it. Lovable's own Stripe docs tell you the integration does not work in preview, so you deploy, use 4242 4242 4242 4242, and watch Checkout complete. RapidDev's 2026 guide stops at the same checkpoint: deployed URL, test card, webhook delivery log showing 200.
That is a checkout test. It is not a retry test.
Why does Stripe test mode miss the retry?
Stripe retries when your endpoint is slow, returns a non-2xx, or times out. The retried body keeps the same event id. A handler like this treats every HTTP POST as a new business event:
app.post("/webhook", async (req, res) => {
const event = stripe.webhooks.constructEvent(req.body, sig, secret);
if (event.type === "payment_intent.succeeded") {
await createBooking(event.data.object.metadata.booking_id);
}
res.json({ received: true });
});
Signature verification can succeed on both deliveries. HTTP 200 can succeed on both deliveries. createBooking still runs twice.
The durable check is not "did Stripe accept the test card." It is:
- insert
event.idunder a unique constraint before the side effect - one
payment_intentcreates one booking - a second POST with that same
idreturns200and does nothing else
A patch that stops the duplicate by creating no booking at all will also look green if your only assertion is "the count stopped increasing." The invariant has to name the desired side effect, not the disappearance of a symptom.
Is this a junior-only mistake?
No. A senior said the same sentence a bootcamp graduate said: it looked fine until it didn't.
The gap is not whether someone can read Stripe's retry docs. The gap is whether the second delivery ran against this repository before a customer did it. Generated code plus a reviewer who did not write it makes that gap wider, not narrower. Diff review cannot see a retry that never executed.
Lightrun's 2026 survey of 200 enterprise SRE and DevOps leaders measured the same loop at a different scale. 43% of AI-generated changes still needed manual debugging in production after passing QA and staging. 88% needed two or three redeploy cycles to confirm a fix that already looked right. None of the respondents could verify an AI-suggested fix in a single deploy.
That survey is enterprise SRE work, not a Base44 poll. The operational shape matches what builders described: production is the environment that finally exercises the failure.
How do you test the second delivery before production?
Do not replace Stripe test mode. Keep it. Add a run that Stripe test mode will not volunteer.
- Create and confirm a payment against a stateful Stripe twin.
- Deliver
payment_intent.succeededonce. - Redeliver the same event
id. - Read application state, not only the HTTP status.
- Require the old handler to create two side effects and the patched handler to create one.
FetchSandbox's webhook_retries scenario replays each webhook with the same upstream event id. That is a stand-in for Stripe's backoff, not a claim that every production retry interval was reproduced. The useful part is the duplicate-delivery invariant, which you can keep in CI after it holds locally.
Connect FetchSandbox MCP in Cursor or Claude Code and ask the agent to run stripe accept_payment under webhook_retries, then call prove_fix before writing the diff to disk. prove_fix needs the still-broken tree. A model saying the new test looks good is not the gate.
The eval contract is the same one in deterministic evals for coding agents: buggy tree exits 1, fixed tree exits 0, anything else stays unproven. For the broader AI-builder setup, see integration testing for Lovable and Bolt apps.
Paste the receipt into the pull request by hand. FetchSandbox does not post GitHub comments automatically.
- name: Run API integration workflows
run: |
npx fetchsandbox run "$FETCHSANDBOX_ID" --all --json \
> fetchsandbox-workflows.json
Receipts
Here is a real Stripe accept_payment run under webhook_retries, captured 2026-09-15:
POST /v1/customers→ 200POST /v1/payment_intents→ 200POST /v1/payment_intents/pi_GAXMB2ZHA61M6LWRJ42ATUTS/confirm→ 200POST /v1/payment_intents/pi_GAXMB2ZHA61M6LWRJ42ATUTS/capture→ 200GET /v1/payment_intents/pi_GAXMB2ZHA61M6LWRJ42ATUTS→ 200
The attached duplicate-delivery probe used event evt_dup. The buggy reference handler charged twice (charge_count: 2, then GET /charge-check → 409). The fixed reference handler charged once and labeled the second POST a duplicate.
Full timeline: fetchsandbox.com/runs/2d64d21288?flow=run_da21c6c9-679c-4fe4-92aa-2718f592b106
This receipt proves the failure class on FetchSandbox reference handlers. It does not prove a fix in your Lovable or Base44 repo. That still requires prove_fix on the still-broken tree.
Questions about testing AI-built payments before production
Is Stripe test mode enough for a Lovable or Base44 app?
No. Test mode plus a 4242 card proves Checkout can complete on a deployed URL. It does not replay payment_intent.succeeded with the same event id, which is how Stripe retries.
Do mocks catch webhook retries?
Only if you wrote a mock that redelivers the same event. Most generated test doubles return one successful payload and stop.
Why isn't a green CI job enough?
CI is green for the assertions you have. If those assertions never execute a duplicate delivery, production is the first runner that does.
Can the same model that wrote the handler verify the retry?
It can propose the check. The verdict has to come from measured behavior on old code and patched code. If the probe cannot reproduce the double charge on the old tree, the fix is not proven.
Does this apply only to juniors using Lovable?
No. The same two beliefs showed up at every skill level. The missing piece is a deterministic eval loop before you ship, not more confidence in the generated diff.