AI Wrote the Service. How Do We Review It—and What Does That Cost at Runtime?

Lately, I write very little implementation code by hand. AI agents produce a substantial part of it. The work did not disappear; it moved from typing code to defining constraints, reviewing changes, and evaluating the result.

The faster an agent can produce a service, the more uncomfortable the next question becomes: how can a human verify what was built without reconstructing the system from thousands of generated lines?

In an earlier article, I proposed a typed architectural graph as a shared system representation for developers, AI agents, validators, generators, and observability. A follow-up tested whether that graph could generate real Go, Python, C++, and Rust services without forcing the four runtimes into one artificial programming model.

Once the approach produced working systems, diagrams were no longer enough. There was a practical question to answer:

What is the runtime cost of making a generated system easier to understand, modify, validate, and observe?

Review the Architecture Before Reading the Implementation

A regular code diff tells a reviewer which lines changed. It does not directly say that a new parallel branch appeared, an external service was introduced, a deadline changed, or an error path became explicit. The reviewer has to recover that meaning from transport code, handlers, configuration, task pools, and tracing setup.

When an agent works inside a formal architectural model, review can happen at two levels. First, the human reviews the architectural change:

+ Inventory Service connected over gRPC
+ explicit error path
~ FunctionCall replaced with PriorityTaskPool
~ soft deadline changed from 1000 to 500 ms

The graph validator checks topology and type compatibility. The generator then creates the familiar infrastructure deterministically. Human attention can move to the smaller handwritten area: does the business function preserve its contract and behave correctly on failure or cancellation?

This does not eliminate code review, tests, or security scanning. It changes their scale. The reviewer sees architecture as architecture first, then reads the code where non-standard decisions were actually made.

The Scenario: HTTP → Graph → gRPC → Graph → HTTP

The benchmark uses the same order-processing system as the multi-runtime case study. An Order Service accepts an HTTP request, splits an order into items, calls an Inventory Service over gRPC, maps the result, merges the branches, and returns an HTTP response. A soft-deadline branch and an explicit error path remain part of the generated topology.

You can inspect the actual graph in Open Service Architect. It is not a static illustration: it contains service boundaries, typed connections, call semantics, deadlines, and error routes.

For every language I compared two implementations:

• generated — the path produced from the graph and executed through its ServiceLib runtime;
• native — a direct implementation of the same HTTP-to-gRPC scenario using the same ecosystem.

The comparison is always generated Go against native Go, generated Rust against native Rust, and so on. This is not a language ranking. Each pair isolates a more useful question: what additional capacity does the framework-backed path consume relative to a direct implementation in the same runtime?

Why the Benchmark Contains Almost No Business Logic

The measured request contains one item with a deliberately missing stock identifier, so every request follows the same OUT_OF_STOCK path. There is no database, external API, artificial delay, or substantial domain computation.

That makes this an intentionally hostile benchmark for the framework. The request still pays for HTTP parsing, serialization, gRPC, scheduling, graph traversal, result propagation, and tracing integration, while useful business work is close to zero.

This is both a limitation and the point. The result does not tell us the universal cost of a node. It estimates the additional cost of this concrete end-to-end path when almost nothing can hide framework overhead.

What the Framework Is Paying For

The generated path does more than call one handler. It constructs typed stream contexts, dispatches messages through operators, propagates results, manages fan-out and merge semantics, carries cancellation, and creates consistent tracing boundaries.

Part of the cost comes specifically from the streaming execution model. The native baseline can implement this predetermined scenario almost linearly: parse HTTP, call gRPC, transform the response, return JSON. The generated version preserves a more general structure in which execution semantics can change without rewriting the handler.

In direct code, changing from a synchronous call to a pool—or from sequential processing to parallel fan-out—can affect control flow, cancellation, error propagation, synchronization, tests, and instrumentation. In the graph, the same architectural change can be a configuration-level decision. Tracing, metrics, and topology-aware spans remain wired in.

Test Conditions

• one ARM64 machine running macOS and Docker Desktop;
• two service containers: Order Service and Inventory Service;
• k6 load generator limited to six CPU cores;
• five-second warm-up and one 20-second measurement run;
• one-CPU run at 256 virtual users;
• two-CPU run at 512 virtual users to keep the additional service capacity loaded;
• zero request errors in every measured variant.

Docker CPU limits are quotas, not physical core pinning, and Docker Desktop adds a shared virtual machine. Absolute numbers from different machines should not be compared. The useful comparison is generated versus native within the same language and environment.

Results: One CPU and Two CPU

Latency cells show the one-CPU value first and the two-CPU value after the slash.

* Python caveat: each Python service ran as a single process, without multiple worker instances. Because of the GIL and the single main asyncio event loop, the two-CPU configuration does not use both cores in the same way as Go, C++, and Rust. The values are included for completeness, but they are not a valid measurement of two-core Python scaling. That would require multiple processes and load distribution between them.

Turn Throughput into Capacity Cost

A throughput percentage makes the framework look especially expensive when the useful work is almost zero. A clearer engineering view is to convert achieved throughput into capacity time:

capacity time (µs/request) = 1,000,000 / requests per second

framework cost = generated capacity time - native capacity time

This is not p50 latency and not CPU time measured by a profiler. Two service containers have independent CPU quotas, so throughput alone cannot recover the precise CPU microseconds spent inside each process. It is the additional system capacity time per completed request under equal conditions.

C++ came close to doubling generated throughput in the second load regime. Go grew by about 1.7× and Rust by about 1.4×. These are observed changes between two different load regimes—not pure CPU-scaling coefficients—because the VU count also changed from 256 to 512.

The additional capacity cost decreased in the two-CPU regime for Go, C++, and Rust. Python remained close to its one-CPU result, but the single-process limitation means it should not be used to judge two-core framework scaling.

Put 5–44 Microseconds Next to Real Work

Now imagine that the request performs only 2 ms of useful work: a local database query, a cache-miss path, a computation, or an external call. Compare that with the measured framework cost:

This is not a prediction of production throughput. I/O, contention, queues, and concurrency interact non-linearly. It does show the scale: overhead dominates the story only while the benchmark request does almost no useful work. At 10 ms of domain or I/O work, the same measured cost is roughly 0.05–0.44% of that timescale.

What the Numbers Say About the Four Runtimes

Go shows a visible cost for stream dispatch, contexts, and result propagation while retaining substantial throughput. It remains the straightforward choice for ordinary orchestration-heavy backend services.

C++ has the lowest generated throughput in the one-CPU run and a distinctive latency distribution, yet scales strongly in the second regime. Calling it the automatic choice whenever microseconds matter is debatable when Rust is sitting in the next row. 🙂

Python has the highest absolute framework cost, but its intended use is rarely an empty request path. When a service wraps an ML model or performs substantial I/O, tens of framework microseconds are unlikely to be the dominant term. A proper multi-core comparison still needs multiple worker processes.

Rust produced the strongest generated result in this test. Rust is often avoided because of its learning curve and infrastructure complexity. Generation changes that trade-off: developers do not need to assemble the HTTP server, gRPC wiring, tracing, metrics, configuration, and runtime structure before writing the first business rule. They still need to understand Rust to maintain custom logic, but the amount they must design from scratch becomes much smaller.

What This Benchmark Does Not Prove

• It does not rank programming languages or their runtimes.
• It does not isolate the cost of an individual graph node.
• It does not measure a production network, database, or real domain workload.
• It does not establish multi-core Python scaling.
• One measurement run is a snapshot; small differences require repeated runs and profiling.
• It does not prove that every service should use a framework.

It answers a narrower question: for this concrete HTTP → graph → gRPC → graph → HTTP path with almost no business logic, what additional system capacity separated a generated implementation from a direct implementation in the same language?

When the Framework Is Worth It

The framework becomes attractive when the architecture is expected to change, when several services need consistent contracts and observability, or when humans and AI agents both need a compact representation of the system.

• execution semantics can change from a direct call to a pool or parallel branch;
• timeouts, cancellation, retries, and error paths must remain explicit;
• tracing and metrics should work consistently out of the box;
• multiple services—or multiple languages—share one validated topology;
• AI agents need small tasks with fixed boundaries instead of a repository dump;
• reviewers need to understand architectural changes before reading generated artifacts.

A direct service is still the better choice when the path is tiny, stable, extremely latency-sensitive, and the team is willing to own all infrastructure code manually. If measured overhead exceeds the system budget, the benchmark has already answered the decision.

A lightweight direct service does not invalidate the architectural approach. It can remain one user-owned component connected to the larger graph through an explicit contract. The model does not need to generate every line in order to describe the system.

Reproduce It—or Make It Faster

The generated examples are available as regular repositories:

• Go example
• Python example
• C++ example
• Rust example

The four runtime libraries are open as well: servicelib, pyservicelib, cppservicelib, and rustservicelib. If you see an avoidable allocation, queue transition, context copy, or scheduling decision, optimization ideas and pull requests are welcome.

The measured 5–44 µs is not the price of another way to call a function. It is the price, in this benchmark, of making execution structure explicit, changeable, reviewable, and observable.