To determine whether API throughput and integration load are the real scalability bottleneck of your web application, you need to record traffic volume, concurrency, HTTP 429 status codes, latency percentiles (p50, p95, p99), and moments of buffered or deferred traffic for each integration.
Essential Baseline Metrics for API Throughput
Capturing the right baseline metrics is crucial to understanding whether API throughput and integration load form a web application's scalability bottleneck. These metrics help identify external constraints that affect application performance.
- Identify the impact of external APIs on web server worker pools to prevent worker exhaustion.
- Assess whether infrastructure expansion worsens API bottlenecks by triggering stricter rate limits and retry storms.
- Use downstream latency percentiles and queue age to obtain reliable diagnostics of integration issues.
- Compare local CPU and memory values with external wait times to reveal hidden capacity issues.
Why integration load is a hidden scalability constraint
Integration load does not arise solely from the number of connections. The limiting factor often lies in where a connection sits within processing, how work is distributed across available capacity, and the traffic pattern. As a result, a web application can appear calm locally while in practice being slowed down by an external CRM, ERP, or other API. A scaling decision that looks only at its own servers then misses the dependency that determines how much work can actually be processed.
The sharpest boundary occurs when an integration is synchronous in the HTTP request-response path. While waiting for an external API, a web worker remains occupied. That worker therefore cannot handle the next user request, even when its own processing requires little additional capacity. In that case, the application's responsiveness is directly tied to the response time and availability of the external service. This makes integration load difficult to see: the cause lies outside the application, but the queue of user requests and visible delay arise on the application side.
Asynchronous processing changes that relationship. A queue can absorb peaks without the user interface freezing while it waits. However, this does not remove the external constraint; it shifts the question to how quickly the backlog can be processed again. Moreover, when bulk processing and time-sensitive tasks do not have separate queues or worker pools, one slow external API can hold up all other background tasks and web processes. The technical dependency then becomes a shared capacity limit rather than an isolated incident.
The shape of traffic also determines whether the integration reaches its limit. Sudden peaks and synchronous cron jobs consume fixed API quotas per second much faster than a smooth, predictable flow. An average load measurement can therefore appear reassuring, while short concentrations of traffic hit the quota, concurrency, or wait time limits of a connected service. Without usage, performance, and capacity data for each integration, there is a risk that additional local compute power is viewed as the answer to a limit that in fact lies outside the web application.
Sources for this section: aws.com, microsoft.com, shopify.engineering
Risks of missed checks and incorrect assumptions
A scaling issue is quickly labelled as “insufficient application capacity” when users see slow pages or error messages. Without a recorded baseline for relevant integration data, however, that explanation cannot be tested. The visible failure may just as easily result from wait time at an external ERP, quotas at a downstream API, or a pattern of consecutive calls to a CRM. Those who scale based on the symptom alone may invest in capacity that cannot speed up the limiting dependency.
A concrete risk arises when the latency of an external ERP increases during a peak. Synchronous web workers then remain in I/O wait until the external response returns. Once the worker pool reaches its maximum capacity, new requests can no longer be processed normally and the reverse proxy may return HTTP 504 Gateway Timeouts. The error message appears at the edge of the web application, but without a baseline it does not reveal whether the pressure comes from local processing or from the external wait time holding workers. That distinction determines whether additional instances have an effect.
A second incorrect assumption concerns throughput to external services. When a downstream API quota is exceeded, HTTP 429 status codes may be returned. If the application retries without control, the load grows at the very moment the external service is imposing limits. This can result in IP blocks and a complete synchronization outage. Without previously recorded error codes, traffic patterns, and response behaviour, there is a risk that this chain is recognized only after the integration can no longer recover under normal load.
The structure of an integration can also distort performance. In the Distributed N+1 Query pattern, an application makes synchronous REST calls to an external CRM for individual records within an iteration loop. Network latency then accumulates to seconds per page request. A measurement that shows only total page time may incorrectly point to the application as a whole. The baseline therefore does not only have a signalling function: it prevents a visible delay from being confused with where that delay originates, thereby limiting inefficient investments in the wrong layer.
Sources for this section: microsoft.com, shopify.engineering
What needs to be verified and why
Verifying integration load begins by distinguishing between local activity and work held up by an external dependency. Low CPU and memory values are not sufficient evidence that a web application still has capacity available. A process may perform little computation while still holding capacity because it is waiting for an external API or ERP. When the integration is also experiencing rate limiting or timeouts, a false health signal emerges: the server appears idle, but the chain cannot process additional user requests or synchronizations.
Therefore, first verify web worker pool occupancy and saturation in relation to the duration of external calls. During synchronous I/O waits, runtime workers, such as PHP-FPM workers, remain occupied until a slow external response arrives. If the pool becomes saturated, new incoming requests are rejected with HTTP 502 or 504. The relevant question is not solely how many requests the web application receives, but how many workers are waiting at a given moment, how long they wait, and whether that wait coincides with one external dependency. This reveals whether throughput is constrained by work distribution or external response time.
For asynchronous processes, verification shifts to the queue. When bulk synchronizations and time-critical tasks share the same background queue, task wait times can increase to hours. Priority tasks are not necessarily executed slowly; they may simply not get their turn because bulk work keeps the available workers occupied. The age of tasks in the queue, broken down by work type and connected external service, therefore shows whether a backlog is a temporary volume effect or a structural capacity conflict between processes.
These three checks should be assessed as one picture: local CPU and memory, synchronous worker occupancy, and asynchronous queue age, each alongside integration errors and external wait time. Only then can a team determine whether the application itself is saturating or is primarily functioning as a waiting room for a constrained connection. This verification prevents an apparently healthy server from being used as an argument for an incorrect scaling assessment.
Sources for this section: sre.google, aws.com, microsoft.com
Checklist for capturing baseline metrics
Record this data for each external API or connected platform and compare it with the timing of peak load. The combination distinguishes between normal delay, quota pressure, and an integration that limits available processing capacity.
- Traffic volume, concurrency, and HTTP 429 status codes per integration. Record how many outbound calls take place, how many are running concurrently, and when an external SaaS or CRM service returns HTTP 429. These three measurements belong together: quotas can be hit by excessive throughput or excessive concurrency, even when average traffic appears limited. Record the values separately for peak periods, because that is when it becomes clear whether the external service is blocking traffic. The baseline thus provides a factual basis for determining whether API throughput is the constraint.
- p50, p95, and p99 latency of external calls, alongside average response time. An average alone can mask outliers. Percentiles show how the typical request, the slower group of requests, and the extreme tail of delay behave. In particular, p95 and p99 can demonstrate that a downstream integration causes periodic delay that would otherwise be incorrectly attributed to local server or database capacity. Record these values for each external call, not only for the complete page or API response.
- Moments when traffic is buffered or deferred. If outbound traffic is regulated to prevent quota blocks, record when bulk operations slow down or are deferred. This makes the trade-off visible between immediate throughput and avoiding HTTP 429 errors from external partners. An increasing amount of deferred work does not automatically indicate a defect, but it does show that available API capacity is below current demand. Without this data, a lower error rate may incorrectly be read as sufficient integration capacity.
Sources for this section: sre.google, shopify.engineering
What can go wrong if checks are skipped
Skipped checks can easily turn a capacity problem into an amplified chain failure. The following risks show why a measurement without realistic dependencies or without cause attribution provides a misleading basis for scaling.
- Local scaling can overload the external service. When high response times are observed without a baseline for the involved integration, engineers may scale containers and virtual machines under the assumption that the web layer is too small. Additional instances, however, increase the parallel load on the same slow CRM. If that downstream system is already operating at its concurrency limit, it may collapse completely under this additional pressure. The investment then increases demand on the limiting service rather than expanding processing capacity for users. The error is not in scalability as a goal, but in the lack of evidence about where the delay begins.
- Uncontrolled retries can block recovery. A faltering integration needs room to process its existing workload. Automated synchronous retries without exponential backoff and jitter do the opposite: they repeatedly send requests to a service that is already under pressure. As a result, the load increases exponentially and the external service can no longer recover. If HTTP errors and retry behaviour are not checked together, a retry storm may remain invisible until the delay spreads through the entire chain. The observed outage is then larger than the original disruption at the connection.
- Isolated load tests produce overly favourable conclusions. A substantiated test report includes downstream dependencies simulated with realistic delays and network errors. A test with mocked responses without latency primarily assesses the web application without the integration load that may be decisive under production load. As a result, an application may appear able to process a high volume in the report, while the actual connection holds up workers and queues during delay or errors. The test outcome is then not confirmation of chain capacity, but only of part of it. This difference affects both investment decisions and operational planning around peak load.
Sources for this section: microsoft.com, shopify.engineering
Frequently asked questions about integration load and scalability
These questions concern architectural choices that change the meaning of the baseline. They do not answer whether a connection is already the bottleneck, but rather what operational consequence follows when processing is organized differently.
- “Isn't synchronous processing simpler when an immediate status is needed?”
Yes. A synchronous integration simplifies program code and provides immediate confirmation of the status of the external operation. In return, scalability and availability are directly tied to the external service. If that service slows down or is unavailable, the user request waits as well. An asynchronous queue decouples both systems and prevents user interaction from having to remain pending during that wait. The trade-off lies in the application: the user interface must handle status while processing is not yet final, and data may temporarily not have the same state everywhere. The relevant choice therefore depends on the need for immediate confirmation versus the acceptability of deferred processing. - “Does local caching solve the throughput constraint of a CRM or ERP?”
Local caching or duplication of CRM or ERP data can remove external network delays; in the described situation, this concerns delays under 10 ms. However, this does not remove the responsibility to determine how current the local data is. Cache invalidation makes processing more complex and introduces the risk that inventory or pricing information is outdated. Caching therefore changes the dependency from a direct response dependency into a question of data freshness. This boundary should remain explicit when interpreting baseline data: low local read times do not prove that source data is current at that moment.
Sources for this section: microsoft.com
Important considerations for scalability and integration load
The usefulness of a baseline ultimately depends on the traceability of every loss of time and on the period over which the pattern is captured. These two criteria connect the technical measurement with the risk of an incorrect capacity decision.
- Make the source of latency verifiable. Use standardized distributed tracing according to OpenTelemetry and W3C TraceContext specifications, with waterfall views that assign time to application code, database transactions, or external HTTP calls. This turns a total response time from a symptom into a breakdown of individual wait times. A peak can then be assessed based on where it arises: within the application, in a database transaction, or during an external call. For integration load, this distinction is directly operational. Without this attribution, the same high page latency can lead to a local scaling expansion while the actual delay lies outside internal capacity. The financial consequence is that additional infrastructure costs may arise without users experiencing less waiting time.
- Base capacity on percentiles across representative multi-week periods. Record p50, p95, and p99 over representative periods of several weeks instead of relying on averages or superficial uptime statistics. Percentiles keep normal processing and the slow tail visible separately. A system can respond quickly on average while the p95 or p99 value shows that some requests take much longer during recurring pressure. Over a multi-week period, differences between steady load and recurring peaks become more visible than in a snapshot. This prevents a quiet measurement day from being used to dismiss a capacity limit, or a single incident from being treated as a structural pattern. The operational boundary remains the wait time that can demonstrably be attributed in the trace to an external HTTP call, database transaction, or application code.
Sources for this section: sre.google
