These two terms get conflated all the time, and the confusion costs teams real engineering hours. Contract testing and sandbox testing answer different questions. If you only do one of them, you have a gap.
What contract testing actually does
Contract testing tools (Pact, Spring Cloud Contract, OpenAPI-based contract validators) verify that two services agree on the shape of the data they exchange. The consumer says “I expect responses with a customer_id field as a string.” The provider runs a verification step to confirm its real responses match that expectation.
This is great for catching breaking changes. If the provider renames customer_id to customerId, the contract test fails before you ship.
What contract testing does not do
Contract testing does not tell you whether your code actually works. It only tells you whether your assumed shape matches the real shape. You can have a perfectly valid contract and still write code that:
- Reads the customer ID into the wrong variable
- Skips a required idempotency check
- Mishandles the second event in a multi-step flow
- Persists state in the wrong database row
- Returns 500 when the response has 0 items
None of those bugs show up in contract testing, because contract testing is not running your real handler against real responses across a real flow. It is just verifying shapes.
What a stateful API sandbox does
A sandbox runs the actual provider API (or a stateful simulation of it), with persistent state and lifecycle behavior. Your code runs against it the same way it runs against production. You can:
- POST a customer, then PATCH it, then GET it back, and confirm the chain works
- Trigger a webhook event and watch your handler process it
- Replay an event 10x and confirm idempotency
- Test error paths by switching scenarios (rate limited, declined, expired)
A sandbox proves the integration works. Contract testing proves the integration fits the contract. They are different proofs.
The right setup is usually both
On a real production team, you should have both:
- Contract tests in CI to catch breaking schema changes from the provider. Run on every PR.
- Sandbox-based integration tests in CI to catch logic bugs in your handler, your retry policy, your idempotency, your state management. Run on every PR.
- Real provider tests in staging for the final 5% of edge cases that only the real API exhibits.
Where teams usually go wrong
The two common failure modes:
- Only contract tests. Your shapes match. Your code is buggy. You ship and find out in production.
- Only end-to-end tests against the real provider. Your tests are slow, flaky, and impossible to run in parallel. You stop running them. Eventually you ship things that broke a year ago because nobody noticed.
The healthy middle is both, with most of the test volume living in the sandbox layer.