English source · 19 min
Composed release rehearsal
On 10 November 2026 a fresh PostgreSQL 17.11 cluster started Core, the
independent Checkout and Connector, and an app worker on Darwin arm64. Each
journey provisioned a fresh Core database, a Checkout session database and a
standalone Connector database. The latter two used dedicated runtime roles.
Cargo built the three binaries from the sources below with --locked.
The worker used the same Core binary in its app-worker role.
Resolve the release inputs
| Component | Source used to build | Runtime and contract |
|---|---|---|
| Core | Veyndo at a3d9b8d3 |
0.5.0. Store API, merchant orders, receipts, app gateway and Flow Builder. The migration ledger runs through 112-hosted-checkout-recovery. |
| App worker | Same source as Core | Same native order outbox and versioned Flow definitions. The worker opens the migrated database in serve mode. |
| Checkout | Checkout at eaa34f27 |
0.1.0. Session API under /v1/sessions and OpenAPI at /openapi.json. Checkout database schema marker veyndo-checkout-sessions, version 1. |
| Connector | Connectors at fc36cad5 |
0.1.0. Standalone namespace profile, pinned contract/actions.json and schema emitted by the binary. |
The Core and worker source used locally has the same tracked runtime inputs
as the hosted Core 99b0aec6 and worker 687bbae8. The hosted deployments
report those two distinct build revisions. Northflank also reported the
Checkout and Connector source revisions listed above. These observations
identify source and build provenance. The rehearsal used local Cargo
binaries. Rebuilding and booting the full set of production OCI images with
resolved digests, hosted runtime configuration and every supporting service
remains a separate release gate.
The input fingerprints below are SHA-256. For the two independent schemas,
hash the stdout of the corresponding binary with --print-schema.
Core Cargo.lock 8b5ea57ffffaf8c19e6671af92cda0a0cd470c06bb0d769789de04f0afff9f75
Core components.lock 27888ec784bd2d1a4541f398e6cf65db00c2634737793edb4a98cd7d21d26233
Core migration 112 7282172a7c6edceedc6e639d9e8592d389c32af45ef1f8362dab522a67f806de
Checkout Cargo.lock caf1b66985a1f63e49757358ecbf09ca48899cb4df5a03017a4ad08dae352cdf
Checkout schema d9ba0fbc7902fff8e77fcce01fb19c443fb61069fc13c767c2de3e1af496c541
Connector Cargo.lock c26f89bf44307374d61a98ee7dd156cda21facbe5abe88c3a119b646d42da4c0
Connector actions 18238ab8cb67a1c5d56bacc5206b645ade124eaec61656ed1016f8fd54294a5f
Connector schema a16139728207f4222676702663ef3466dee1396363ecefcecd14c917b118b83d
Run the recovery checks
Build Core using cargo build --locked --bin veyndo, Checkout using
cargo build --locked --manifest-path service/Cargo.toml, and Connector using
cargo build --locked. The run used Cargo and Rust 1.99.0. In a Veyndo checkout
that includes the new regression, make the pinned Core executable available
at target/debug/veyndo. Use a disposable PostgreSQL cluster and the
Connector test configuration.
For the direct native runner, set TEST_PSQL to the local executable.
Set MERCHANT_TOKEN and PLATFORM_SECRET_KEY to disposable test values.
Then run:
DEMO_CATALOG=legacy-furniture \
CHECKOUT_TEST_BINARY=/absolute/path/to/veyndo-checkout-service \
CONNECTORS_BINARY=/absolute/path/to/connectors \
python3 scripts/email_tests.py \
Contracts.test_independent_checkout_rollback_and_worker_restart \
Contracts.test_independent_checkout_capture_drives_invoice_mail_and_refund
| Interruption | Observed recovery |
|---|---|
| Client discards a successful confirmation body | A restart of independent Checkout followed by the original confirmation returns the committed order. The shop has one order and one stock decrement. |
| Independent confirmations are paused and the app worker is stopped | The paused route returns HTTP 503. The original customer still reads the same order and invoice. A distinct native cart can buy, and repeating that purchase returns the same second order. There are two orders and two stock decrements. |
| The worker restarts and independent confirmations resume | The pending native order gains one invoice and one loopback SMTP delivery. Resuming the separate route and replaying the first confirmation preserves both orders, the first invoice and the total of two messages. |
| A separate payment fixture delays verified capture | Before capture, no invoice or mail is produced. After capture, the configured Flow generates one invoice and one matching Connector/SMTP receipt. Replaying the refund keeps the original payment, order, invoice and stock effect. |
This is a rollback of purchase routing. Database schema downgrade and a failure inside the uncommitted Core transaction are separate tests. The fixtures cover neither hosted mail delivery nor provider uptime or capacity. The release checklist tracks those separately.
Compare the two purchase routes under load
The same pinned binaries ran the two existing mixed traffic tests in separate disposable shops. Each run used 200,000 synthetic products, 240 catalog reads, 48 simulated purchases and 16 assistant requests. Each test verified one stock decrement, one invoice and one SMTP receipt per purchase. The assistant used a local model fixture.
| Purchase route | p50 | p95 | p99 | Purchases retried | Total retries |
|---|---|---|---|---|---|
| Native | 33.33 ms | 55.63 ms | 58.86 ms | 0 / 48 | 0 |
| Independent | 35.65 ms | 2051.50 ms | 2070.54 ms | 8 / 48 | 12 |
The median differs by 2.32 ms in this run. The slower independent confirmations include the explicit retry waits when admission is refused. A low median therefore does not describe what every buyer experienced. See the paired test commands. The measurements are for local debug binaries. They do not establish the same percentiles or the capacity of the deployed services.
For the hosted comparison, use a dedicated fixture deployment or isolated test tenant with enough inventory and independent Checkout enabled for the test tenant only. Keep image digests, database placement, replica counts, concurrency, fixture data and request schedule fixed between the two runs. Record pre-forward refusal counts and retry waits separately from accepted request latency. Compare the purchase attempts with committed orders, stock allocations and the outbox after each run. Report provider callback capacity as a separate workload.
Northflank rehearsal with deployed executables
A follow-up on 10 November used a separate Northflank project in Europe
West with one manual job, no public ports, no linked production secrets,
and no production database. The Linux x86-64 job used the
nf-compute-200-4 plan and a fresh Debian PostgreSQL 17.11 cluster.
fsync, synchronous_commit and full_page_writes were enabled.
Core, Checkout, the app worker and Connector ran as separate processes
inside that job and used loopback addresses. Checkout and Connector had
separate databases and dedicated runtime roles.
The executables were copied from the four running services. Their SHA-256
checksums matched both the production container files and the files started
in the test job. Core and its worker had different build revisions and
byte-identical executables. The rehearsal code was pinned to Core
76ca4eb2c13ed257a408c42462018ab4205ece2b.
| Service | Deployed source revision observed | Executable SHA-256 |
|---|---|---|
| Core | 76ca4eb2c13ed257a408c42462018ab4205ece2b |
485b851124edddf174d094441e32beb65fab256910bb7f280487a3f0da0916d8 |
| App worker | 687bbae83dd3c5c07d3bf97009fb2f07e2d2510c |
485b851124edddf174d094441e32beb65fab256910bb7f280487a3f0da0916d8 |
| Checkout | eaa34f27386e0de2b34de6085fef78ac237943e9 |
be10b8fe602248796ba0bd224819ccf0a180d809d442f7e5b8d226c8a20b9120 |
| Connector | fc36cad517533a18922966c7f3aa2322860ce19c |
a4de4879de28fae10b4b31171b9536449c67c0913619371f15e8d49e1603c8d7 |
Both recovery tests above passed. The run recovered the original order after response loss, preserved its invoice during the routing pause and processed a distinct native order when the app worker restarted. The payment fixture completed capture, invoice, mail and refund. The job exited with code 0 at 12:19 UTC.
Hosted paired load
Each arm used a fresh test shop with 200,000 synthetic products, 48 purchase confirmations, 240 catalog reads and 16 assistant calls. The same driver used up to 12 purchase clients, four catalog clients and one assistant client concurrently. Both arms matched each of the 48 orders to one stock decrement, one invoice and one loopback SMTP message.
| Route | Confirm p50 | Confirm p95 | Confirm p99 | Orders with retries | Total retries | Worker restart to last mail |
|---|---|---|---|---|---|---|
| Native | 260.48 ms | 430.84 ms | 540.24 ms | 0 / 48 | 0 | 10.54 s |
| Independent | 222.44 ms | 3195.01 ms | 4171.13 ms | 11 / 48 | 27 | 13.45 s |
The independent confirmations include the wait after a retryable HTTP 429. The 3.20 s p95 is buyer-visible time in this driver. The sample does not separate that wait from accepted-request processing or isolate provider callback throughput. Since the services share one job, these numbers exclude the service-to-service network and separate resource limits of the production deployment. The purchase and SMTP tests use fixtures and do not establish provider availability or mailbox arrival.
To repeat the job, copy only the approved executable files from the
active containers and verify their checksums again in the test container.
In a disposable postgres:17-bookworm job install Python, cryptography and
OpenSSL, start PostgreSQL on loopback, and run the recovery commands
above followed by the two mixed traffic tests.
On Debian Python 3.11 invoke the suite with the script directory after the
standard library so scripts/platform.py cannot shadow Python's module:
python3 -P -c 'import runpy,sys; sys.path.append("scripts"); runpy.run_path("scripts/email_tests.py", run_name="__main__")' \
Contracts.test_independent_checkout_rollback_and_worker_restart \
Contracts.test_independent_checkout_capture_drives_invoice_mail_and_refund
The complete hosted gate still needs pulls by OCI digest, the full set of services with their own deployments and a separate provider callback capacity test. An executable checksum covers the program file used here. It does not cover the libraries, startup configuration or filesystem of its production image.
Hosted relay admission comparison
The follow-up used the same isolated Northflank job plan and PostgreSQL
settings as above. The Core executable came from running build
workable-river-5946 at 9ed1c0bb.
Its SHA-256 was
8bcfb2db2ec2f9aa6c61950b863db5a871a0cdc15ad1e887e272164a1004c992.
The job checked this fingerprint before execution. It used the original
Core executable from the preceding table as the baseline and ran the
same Worker, Checkout and Connector executables for all three arms. The
Worker stayed on 687bbae8, Checkout on eaa34f27, and Connector on
fc36cad5. Set APP_WORKER_TEST_BINARY to an absolute Worker binary
path to reproduce that distinction with the existing test.
Each arm used a fresh test shop, 200,000 synthetic products, 48 purchase confirmations, 240 catalog reads, 16 assistant requests and a worker restart. The new Core runs in the public shop. The test workloads ran in separate databases inside the isolated job with simulated payments and loopback SMTP.
| Route and Core | Confirmation p50 | Confirmation p95 | Confirmation p99 | Orders with retries | Retries |
|---|---|---|---|---|---|
| Independent, before | 269.03 ms | 4,078.00 ms | 4,202.43 ms | 11 / 48 | 30 |
| Independent, with bounded waiting | 476.62 ms | 1,717.01 ms | 1,924.65 ms | 9 / 48 | 9 |
| Native, updated Core | 226.34 ms | 298.18 ms | 380.77 ms | 0 / 48 | 0 |
Each purchase committed one order and one stock decrement and produced one invoice and one SMTP fixture receipt. The independent p95 improved by 58% while its median rose. The test includes the buyer's retry wait. The updated Core also passed route pause, worker restart, lost confirmation recovery and provider-fixture capture and refund. Local Chromium passed the desktop pre-dispatch retry and mobile lost response scenarios, preserving the original receipt and empty cart after reload.
The isolated run fb6f5c9a-b2c7-4629-b1fe-80e0710347ae completed at
13:02 UTC. It used the production executables with fixture credentials.
The later separate service placement test
measured both demo purchase routes. Provider callback load remains open.
Match the deployed builds to image manifests
At 13:50 UTC, Northflank's deployment API reported these source revisions and build IDs for the active services. For each build, the build log reported exactly one distinct manifest digest pushed to the managed registry. The image library reported each matching build tag ready. Core and the app worker use the same build service with different build IDs. The recorded digests identify the images to use in the next rehearsal.
| Service | Deployed source | Northflank build | Pushed image manifest SHA-256 |
|---|---|---|---|
| Core | 333f604592a7a2ba3d3a2de808ca6947e1a3ea57 |
joyful-turn-9296 |
3364bd60d7049343e5784a5c93ffbfa3dea8e0eb387c1c7b15d9878331538bf3 |
| App worker | 687bbae83dd3c5c07d3bf97009fb2f07e2d2510c |
last-mine-6458 |
a2c15af0b71c8bf2b0e6d85370659cf1368834cae956b1ec149020e2b07e2a21 |
| Checkout | eaa34f27386e0de2b34de6085fef78ac237943e9 |
puffy-soup-2325 |
8524590889e27bc2163058b97b5966265f8c9525028d519515101a09d365de1e |
| Knowledge | 874534dd80425f741d7b1a749b64281efda62340 |
sleek-belief-4263 |
df734b6fcd415f81d48f0aa3eae83afa27676ddcb7c7214b8c283aaf55bab16c |
| Connector | fc36cad517533a18922966c7f3aa2322860ce19c |
mint-meeting-7629 |
4ffb567056f7a6187d0e2a579532a6a9f840ac8e1fa33218ca5398dcb621cdef |
| Payments | 4946228a4033c99b4399f0d1d1fb6ca696f11187 |
bold-flower-2375 |
05bd0e1cb994e90d8594648b1d1cbaf2da897a92d0ab0375046a4093b80a8b37 |
| Automation | c4e50f6151d2beb1f4db19565235347040d287c7 |
Imported source revision tag | 31c04335a7db0fade07ce0d090a1e75281aedd3688397319064853f303f9e9ca |
Automation uses the imported image version
sha-c4e50f6151d2beb1f4db19565235347040d287c7. The deployment points to
the exact import record, which the image API reported ready for Linux amd64
with the digest above.
Northflank image metadata API.
The first digest-pinned lookup returned 403 from the private registry with the current Northflank session. The table establishes build provenance from Northflank metadata. The clean boot requires registry read access, verified image pulls, empty databases migrated at startup, healthy instances, and payment and mail traffic routed to fixtures. Then repeat the purchase, response-loss recovery, worker restart and routing rollback with the same assertions for order, stock, invoice and mail. Northflank image retrieval.
Boot the original Core and Worker images
The private Northflank release lab started the original Core build
joyful-turn-9296 and App Worker build last-mine-6458 by their exact
Northflank build IDs. Only the lab could reference the build service during
setup. The two runtime containers had no public ports and used synthetic
credentials. Their executable fingerprints matched the previously recorded
Core 8bcfb2db… and Worker 485b8511….
The separate PostgreSQL 17 fixture database had zero tables in the public
schema before Core started. After bootstrap it had 104 migration ledger
entries, including 112-hosted-checkout-recovery. Core returned HTTP 200 on
/health. The fixture used Core's automatic bootstrap with the test database
owner. The Worker ran in serve mode with PROCESS_ROLE=app-worker.
| Test | Observed result |
|---|---|
| Discard the first successful native order confirmation and retry it | The retry returned the original order. The test shop had one order and stock moved from 80 to 79. |
| Start the separate Worker | Its order Flow created exactly one invoice for the original order. |
| Stop the Worker and place a second simulated order in a new fixture shop | Core returned the original second order on retry, decremented its stock once and left its invoice pending. |
| Resume the pinned Worker and recheck both orders | Each shop still had one order, one stock decrement and one invoice for its original order. |
The test used simulated native payment and the order Flow for invoice generation. Northflank denied the independent Checkout build reference from the lab with HTTP 403. The complete image-level purchase through independent Checkout, payment and Connector remains open. The next run replaced the database owner at runtime with two restricted roles. The lab's runtime services were paused after the run, and the temporary build access was removed. Northflank cross-project build access.
Run the original images with restricted database roles
The same original Core and Worker images ran as separate private Northflank
services against another empty PostgreSQL 17 database. A one-shot Core
migration job used the database owner and applied 103 migration ledger
entries with demo seeding disabled. The database then reported signed
scope enforcement active with distinct system and tenant logins. Core and
the Worker ran with BOOTSTRAP_MODE=serve, DB_RLS_REQUIRED=true and
DB_SCOPE_REQUIRED=true. Neither received the migration connection or a
staged exception. See the database role setup.
| Test | Observed result |
|---|---|
Give the fixture system login TRUNCATE and start the original Core image |
Startup exited with code 101 and reported an unsafe runtime role. Core passed health after the grant was revoked. |
| Buy from two separate fixture shops and discard each first successful confirmation | Retrying returned the original order in each shop. Each had one order and stock moved from 80 to 79. |
| Query the other shop's orders and invoice records in both directions | A foreign shop header was denied with HTTP 403. Searching for the other shop's receipt within the caller's shop returned an empty result. |
| Run the signed-scope probes with the actual tenant login | Forged scope, actor and signature attempts failed. The role could not read signing keys or private identity data, change the schema or write to managed projections. |
| Resume the Worker after Core has dispatched both orders | Each order gained exactly one invoice. Reading the receipt and retrying again after the Core restart preserved the order, stock and invoice. |
Core must run an outbox consumer for this setup. The initial
PROCESS_ROLE=http configuration accepted the purchases and left the
invoices pending. Changing Core to PROCESS_ROLE=all with
APP_WORKER_ENABLED=false dispatched both committed orders while the
separate app-worker was still paused. Resuming that Worker processed the
two Flows. The test used native simulated payments and checked invoice
generation. Provider payment, mail delivery and independent Checkout and
Connector images remain separate checks.
Connect the Payments adapter and replay a callback
Core at 3195dd4c
ran on Darwin arm64 with a separate app worker. Checkout at eaa34f27,
Connector at fc36cad5 and Payments at 4946228a were built from their
pinned sources with Cargo's lockfiles. Checkout and Connector used their
own databases and logins. A migration process initialized a separate
Payments database before the restricted Payments login opened it. The
only external interfaces in this run were a loopback payment provider
fixture and a loopback SMTP receiver.
The registered independent checkout and mail journey used the actual Payments adapter with the private repository's provider fixture. The client discarded the first checkout response and recovered the same order and attempt after restarting Checkout. Before verified capture there was no invoice or mail. Payments rejected a forged callback, accepted repeated deliveries of the same event, and also handled a separate event for the same payment. The completed run had one capture request at the provider fixture, one order, one stock decrement, one invoice and one SMTP message matched to the Connector receipt by Message-ID. Replaying the full refund preserved the original receipt and made one refund request at the provider fixture.
The private Payments repository's existing native checkout regression also checks repeated approval callbacks. The fixture does not call an actual payment provider or deliver to a public mailbox.
Start the original Checkout, Payments and Connector images
Northflank rejected the cross-project image reference for Checkout. Three manual jobs in the images' source project could start the exact build IDs instead. Each had an empty runtime environment, no public listener and a 90-second deadline. The jobs checked the program file fingerprint inside the running image. Checkout and Connector also executed their schema command successfully.
| Service | Build | Northflank job run | Executable SHA-256 |
|---|---|---|---|
| Checkout | puffy-soup-2325 |
fa9c97fd-49f5-4487-bef0-d5bdf2db4f49 |
be10b8fe602248796ba0bd224819ccf0a180d809d442f7e5b8d226c8a20b9120 |
| Payments | bold-flower-2375 |
711cd0b9-fcb7-433f-aa08-a586bf148734 |
79f68bb3b7801cebb923f5b196616152d6c154003590f19158a8f39b7cbb1653 |
| Connector | mint-meeting-7629 |
81662e5d-3df2-4212-b90f-6e989d306266 |
a4de4879de28fae10b4b31171b9536449c67c0913619371f15e8d49e1603c8d7 |
These manual jobs checked startup only. The later five-workload rehearsal connected the services with private fixture databases.
Release the simulated invoice correction
The Payments simulation deliberately omits the provider risk session identifier. Invoice approval now accepts that simulated session and keeps the identifier mandatory for provider invoice confirmation. The Payments CI uses the current Core contract for safe diagnostics on saved payment failures. The complete Payments workflow passed with disposable PostgreSQL. It includes repeated approval callbacks, saved method revocation, all 17 simulated methods, and receipt replay after a service restart.
Northflank deployed build luminous-ink-2350 from 1b52f500 without a
schema migration. Its image manifest digest is
sha256:95a11b66220fa52cdc6c1823e02b4a73e2ef05f0c99ac8ce703679cb2c937eea.
The old replica retired and the active provider endpoint reported
status: ok, mode: sandbox and implementation version 1.6.6.
Core health and the Lumen storefront also responded. The hosted check
verified service readiness. The payment and invoice behavior was
verified against provider and simulation fixtures in CI.
Run five separate workloads on Northflank
On 10 November the original hosted builds ran as private Northflank services in their source project. The workloads referenced the existing builds directly. PostgreSQL 17 held four fresh databases. The Core and app worker shared the Commerce database through distinct restricted system and tenant logins. Checkout, Payments and Connector each had their own database and login. The test verified that each login could connect only to its assigned database and lacked superuser, role creation and row-security bypass rights. Core's signed database scope was enforced. The two one-shot migration jobs finished before the application workflows started.
| Workload | Northflank build | Source revision |
|---|---|---|
| Core | zippy-foot-5455 |
c783c6158f2078692cb15e16b98cd097f100d522 |
| App worker | last-mine-6458 |
687bbae83dd3c5c07d3bf97009fb2f07e2d2510c |
| Checkout | puffy-soup-2325 |
eaa34f27386e0de2b34de6085fef78ac237943e9 |
| Payments | luminous-ink-2350 |
1b52f5005a954a2780153af8ee63fde7c7cc3e33 |
| Connector | mint-meeting-7629 |
fc36cad517533a18922966c7f3aa2322860ce19c |
All listeners were private. The test resources had synthetic credentials and no links to production secret groups or databases. A temporary test helper forwarded fixed loopback fixture addresses to the private services and received synthetic mail on the Connector's loopback interface. The services used the existing commerce, payment and mail contracts. This run shared the provider project network with production.
| Exercise | Observed result |
|---|---|
| Independent Checkout with simulated demo payment | Discarded the successful confirmation body, restarted Checkout on a new instance and replayed the original confirmation. One order and one stock decrement remained. The next shopping cart was empty. The completed cart remained readable for receipt recovery. |
| Native purchase with the separate Payments service | The simulated capture settled after the adapter's signed notification, without browser reconciliation. An invalid approval nonce and a forged notification were denied. Two valid replays left one notification in Core's inbox and one simulated provider order. |
| Paid order through the app worker and standalone Connector | Before capture the order had no invoice. The completed Flow ran its invoice and confirmation actions once. The Connector receipt and the SMTP listener recorded the same Message-ID once. An unauthenticated action returned 401 and a foreign shop request returned 403. |
| Refund and replay | The original attempt reached refunded for the full amount. Repeating the same refund preserved the order count, stock, invoice and one delivery. |
The independent Checkout run used the demo payment. The capture, refund and mail run used the native purchase route with the separate Payments simulator. The strict Core profile reserves the separate Checkout provider route for the actual PayPal Sandbox. The public Lumen pilot records the native and Storyfront journeys through that provider. This private rehearsal used no external payment account or inbox. All six temporary services, including PostgreSQL, were paused after the test.
Compare the checkout routes across separate services
The Northflank load test used the Core, App Worker, Checkout and Connector builds from the five-workload rehearsal above. Each ran as one private service. The measurement client ran in a separate container in the same Northflank project. PostgreSQL 17 held three fresh databases, with signed database scope active in Core. The two synthetic shops each had 200,000 products. All services used test credentials and the SMTP receiver was local to the Connector container.
Each run issued 48 purchase confirmations from 12 clients alongside 240
catalogue page requests from four clients. The native route used guest
carts. The independent route used one signed-in synthetic customer and the
same set of product positions and stock levels. Each run used the demo
payment and prepared its carts before the timed portion. Confirmation
timings include wait and retry time. Retries kept the original purchase
request and honoured the server's retryable 429 and Retry-After.
| Route | Confirmation p50 | Confirmation p95 | Confirmation p99 | Catalogue p95 | Orders retried | Retries |
|---|---|---|---|---|---|---|
| Native | 495.29 ms | 589.72 ms | 605.53 ms | 364.27 ms | 0 / 48 | 0 |
| Independent | 656.90 ms | 3,109.45 ms | 4,013.84 ms | 186.23 ms | 12 / 48 | 17 |
Each shop recorded 48 distinct orders. Replaying each confirmation returned its original order. The 48 purchased products each went from 20 units to 19. The 199,952 other synthetic products retained 20 units. Each route generated 48 invoices and 48 completed Connector delivery jobs. The SMTP receiver recorded 96 distinct Message-IDs across both routes.
The independent path needs further work on latency under contention. Twelve confirmations retried after admission refused the original request. This run used one replica per service, omitted model calls and measured the simulated purchase path. The earlier co-located comparison used a different profile with assistant traffic and a worker restart. Provider callback capacity and production throughput remain open. All six test services including the client and PostgreSQL were paused after the run.
Compare confirmation waits across separate services
The next isolated Northflank run used Core build succinct-snow-553 from
5cd5642f7a239b64d3bdbffd07fe83f99d838ad7 in both arms. The app worker,
Checkout and Connector used the same builds as the preceding comparison.
Each ran as one private service with a fresh PostgreSQL 17 cluster and
signed Core database scope. The two shops each contained 200,000 synthetic
products. Each arm ran 48 independent purchase confirmations at concurrency
12 and 240 catalogue reads at concurrency four. Core was restarted between
arms to change only the confirmation admission setting. The driver used
demo payments and measured elapsed time including retries.
| Confirmation wait | p50 | p95 | p99 | Catalogue p95 | Local wait timeouts | Cluster quota refusals | Purchases retried | Total retries |
|---|---|---|---|---|---|---|---|---|
| 300 ms | 594.89 ms | 2987.44 ms | 3685.80 ms | 173.11 ms | 16 | 0 | 11 / 48 | 16 |
| 1000 ms | 976.98 ms | 1071.11 ms | 2490.97 ms | 181.01 ms | 0 | 2 | 2 / 48 | 2 |
Neither arm filled the local waiter queue. Each confirmation replay recovered the original order. PostgreSQL recorded exactly 48 orders and 48 stock decrements per shop. The remaining 199,952 products per shop retained their initial stock. There were 96 invoices, 96 completed Connector jobs and 96 distinct SMTP Message-IDs. A separate local test discarded Commerce's successful order response on its way to Checkout after commit. Restarting Checkout and replaying the original confirmation recovered one order with one stock movement, invoice and SMTP acceptance.
The longer wait traded a higher median for fewer buyer retries and a lower p95. Cluster refusals remain measurable. These runs describe one replica per service and demo payment. The external provider callback workload is still separate.
Verify scoped Automation in the private composition
The private run also used the released Automation image from
c4e50f6151d2beb1f4db19565235347040d287c7, with a private relay to the
fixture Core. Automation had no database. It checked the current shop key
with Core before evaluating the supplied rule facts.
The driver made 48 alternating eligible and ineligible calls per
transport. HTTP and MCP returned the same decisions and registry
fingerprint, with no effects. Authenticated p95 was 14.48 ms over HTTP
and 12.67 ms over MCP in that private fixture. A key with the additional
catalog.read permission returned 403 on both. Revoking the
automation.read key made the next call return 401 on both. The test
used synthetic credentials. All seven temporary services were paused
after the tests.
Compare signed payment callback delivery
Both arms used Core build succinct-snow-553, one private Payments
replica and separate Core and Payments databases on the same disposable
PostgreSQL 17 cluster. Each arm started with a fresh shop and 24 native
orders awaiting payment. Eight clients approved orders through the real
Payments API in simulation mode. No client asked Core to reconcile.
Core settled after the Payments service's signed notification. The driver
timed approval and polled for the confirmed capture in Core.
| Payments build | Approval p95 | Capture visible p50 | Capture visible p95 | Capture visible p99 | 24 approvals settled |
|---|---|---|---|---|---|
luminous-ink-2350, 1b52f500 |
34.78 ms | 16.22 s | 16.46 s | 16.55 s | 47.68 s |
swanky-secretary-700, 3a5035e7 |
45.53 ms | 4.42 s | 4.77 s | 5.11 s | 13.56 s |
In the original run, Core's inbox received one notification about every two seconds. The worker slept after processing each row. The changed worker sleeps when a scan finds no due row or fails. It continues to claim one row at a time and honours each row's retry time.
Both runs rejected an invalid approval nonce and replayed all 24 valid approvals. Each preserved exactly 24 orders, distinct capture IDs, Core inbox rows and capture events. Stock went from 100 to 76 per shop, including after replaying the order requests. All four temporary services were paused. The workload used simulated funds and did not contact the provider Sandbox.