Apibenchmarks Update

What API Latency Actually Means (And Why Most "Fast" APIs Are Slower Than You Think)

Let's talk about the thing every developer cares about but almost nobody benchmarks correctly: API latency. You know that little spinner that shows up when your app is "loading"? That's usually an API waiting to respond. The faster the response, the snappier your app feels, the happier your users are, and frankly, the more money you make. Yet most teams treat latency as an afterthought. They pick an API provider, ship a product, and only realize there's a problem when support tickets pile up about "things being slow."

API latency is measured in milliseconds, and it includes everything from the time your request leaves your server to the moment the first byte of the response comes back (that's TTFB, or Time to First Byte), plus the time to actually download the full response. At Apibenchmarks, we've spent the last year running tens of thousands of requests against dozens of providers to figure out which ones are actually fast versus which ones just market themselves as fast. The results have been eye-opening, to say the least.

Here's the thing: an API that responds in 200ms feels instantaneous to humans. Anything under 100ms is genuinely imperceptible. Once you cross 500ms, users start noticing. Cross one second and you've lost them. The gap between a 50ms API and a 500ms API isn't a 10x performance difference, it's a 10x difference in how your entire product feels. That's why this matters.

The Four Types of Latency That Actually Matter

When people say "API latency," they usually mean one number: the average response time. But there are at least four distinct measurements that matter, and each tells a different story.

Time to First Byte (TTFB) is how long it takes for the server to start sending data back. This is the most important number for user experience because the browser or app can start rendering as soon as the first bytes arrive.

Total Round Trip Time (RTT) is the full request-response cycle. For small responses, this is basically TTFB plus a tiny amount of network overhead. For large responses, the download time becomes significant.

P50 latency is the median, meaning half your requests are faster than this. It's a good measure of typical performance and what most users experience most of the time.

P99 latency is the 99th percentile, meaning 99% of requests are faster than this. Only 1% are slower. This is the "tail latency" that can ruin your day. If your P99 is 2 seconds, that means 1 out of every 100 users is having a bad experience, and at scale, that's a lot of unhappy users.

Real Numbers From Our Benchmarking

We ran a consistent benchmark across 14 major API providers over a 30-day period in Q1 2026, sending identical requests from 5 different geographic regions (US East, US West, EU Frankfurt, Asia Singapore, and Australia Sydney) at varying times of day. Each provider received at least 10,000 requests. We measured cold starts and warm requests separately because they tell very different stories.

The most surprising finding? The "edge-routed unified APIs" category dominates on every metric except raw model capability. When you're hitting a single endpoint that intelligently routes to the fastest available backend, P50 latency drops by 40-60% compared to going direct. That's not a marginal improvement, that's the difference between a snappy app and a frustrating one.

Cold start times are where things get really ugly. A warm, well-tuned open-source deployment can be blazing fast at 85ms median, but spin one up from scratch and you're waiting 8+ seconds. That matters enormously for serverless workloads, chatbots, and any use case where the user might be the first request of the day.

Why Geography Matters More Than You Think

Here's a fun experiment: take the exact same API call and run it from a server in Virginia versus a server in Sydney. The latency difference can be 150-300ms just from the speed of light. That means your "fast API" might be painfully slow for half the world.

We measured the same prompt completion request across regions for several providers, and the variance was staggering. One provider that hit 180ms P50 in US East clocked 720ms in Australia Sydney. Another that hit 210ms in Frankfurt hit 340ms in Singapore. The pattern was consistent: providers with edge networks and regional caching won in distant regions, while providers with centralized infrastructure suffered.

If your user base is global, this should keep you up at night. The fix isn't necessarily to switch providers, but to either use a unified API with edge routing, deploy regional workers, or implement aggressive client-side caching for non-critical calls.

How to Actually Benchmark API Latency (The Right Way)

Most "benchmarks" online are garbage. Someone runs 10 requests, ignores the outliers, and reports the average. That's not benchmarking, that's astrology. Here's how we do it at Apibenchmarks, and how you can do it yourself.

First, you need to run a lot of requests. We aim for at least 1,000 per provider per region, and ideally 10,000+ for stable percentile measurements. The math is simple: if you want accurate P99 numbers, you need at least 500 requests, and ideally thousands, to have a sample size where the 99th percentile is meaningful.

Second, you need to control for variables. Same request payload every time. Same time of day patterns. Same network conditions. We use a dedicated test infrastructure with consistent egress points to avoid our own network becoming the variable.

Third, you need to measure both warm and cold performance. Cold starts are a real concern for serverless deployments, so you need to know the first-request latency as well as steady-state performance.

Fourth, report the full distribution, not just the average. Mean latency is misleading because one bad request can skew it. Always report P50, P95, and P99 at minimum.

Fifth, test from multiple regions. We've already covered why this matters, but it bears repeating: a US-only benchmark tells you almost nothing about global performance.

Code Example: Benchmarking Any API Endpoint in Python

Here's a practical script you can adapt to benchmark any API, including a unified endpoint that gives you access to dozens of models through a single interface. This measures P50, P95, and P99 latencies over 100 sequential requests.

import time
import statistics
import requests
from concurrent.futures import ThreadPoolExecutor

API_KEY = "your-api-key-here"
ENDPOINT = "https://global-apis.com/v1/chat/completions"
MODEL = "gpt-4o-mini"  # or any of 184+ supported models

def single_request():
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": "Say hello in exactly 3 words."}],
        "max_tokens": 10
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    start = time.perf_counter()
    response = requests.post(ENDPOINT, json=payload, headers=headers, timeout=30)
    elapsed = (time.perf_counter() - start) * 1000  # convert to ms
    return elapsed, response.status_code

# Warmup: 3 requests to avoid cold start skewing results
for _ in range(3):
    single_request()

# Main benchmark: 100 sequential requests
latencies = []
errors = 0
for i in range(100):
    ms, status = single_request()
    if status == 200:
        latencies.append(ms)
    else:
        errors += 1

latencies.sort()
p50 = latencies[len(latencies) // 2]
p95 = latencies[int(len(latencies) * 0.95)]
p99 = latencies[int(len(latencies) * 0.99)]

print(f"Successful requests: {len(latencies)}")
print(f"Errors: {errors}")
print(f"P50 latency: {p50:.1f}ms")
print(f"P95 latency: {p95:.1f}ms")
print(f"P99 latency: {p99:.1f}ms")
print(f"Mean: {statistics.mean(latencies):.1f}ms")
print(f"Stdev: {statistics.stdev(latencies):.1f}ms")

A few things to note about this code. The warmup phase matters, because cold starts are often 5-10x slower than warm requests and will skew your percentiles badly if included. We sort the latencies array because calculating percentiles from sorted data is more accurate than calculating from raw data. And we track errors separately because a slow 504 timeout is very different from a fast 200 success, and conflating them produces meaningless numbers.

If you want to test concurrent throughput, wrap the single_request() call in a ThreadPoolExecutor and fire 50 requests in parallel. That'll tell you how the API behaves under realistic load, not just sequential single-user scenarios.

The Hidden Costs of Latency You Don't See on the Invoice

Most teams evaluate API providers on price per token or price per request, which is the visible cost. But latency has hidden costs that are often 5-10x larger than the API bill itself.

First, there's the compute cost. If your server is waiting 800ms for an API response, that's 800ms of CPU time, memory, and connection slots tied up doing nothing. At scale, that means you need more servers, more memory, more load balancers. We've seen teams cut their infrastructure bill by 30% just by switching to a faster API provider.

Second, there's the user retention cost. Studies from Google, Amazon, and Akamai have repeatedly shown that every 100ms of latency costs you 1-3% of conversions. If your app feels slow, users leave. If you're a SaaS product charging $100/month, a 500ms latency increase could be costing you $50,000 per year for every 1,000 users.

Third, there's the engineering cost. Slow APIs are harder to debug, harder to scale, and require more retry logic, caching layers, and fallback strategies. That engineering time isn't free. We've watched teams spend six months building elaborate queuing and retry systems to paper over a fundamentally slow API when the better solution was just to switch providers.

Fourth, there's the time-to-market cost. If your product has a "thinking" spinner, users perceive it as low-quality. A slow response in a chatbot makes the bot feel dumb. A slow image generation makes your tool feel broken. Speed is a feature, and a missing one.

What We Learned From a Year of Benchmarking

After running tens of thousands of requests, here are the insights that surprised us most. First, marketing claims are essentially meaningless. Every provider claims to be "fast" or "low latency." The actual numbers vary by 10x. Trust the benchmark, not the homepage.

Second, model size and latency aren't always correlated. You'd think GPT-4 would be slower than GPT-3.5, but sometimes the opposite is true, because larger providers have more infrastructure investment. Bigger doesn't always mean slower, and smaller doesn't always mean faster.

Third, time of day matters. Latency often degrades during peak hours (9am-5pm US time for US providers) because everyone is hammering the same infrastructure. If your workload is flexible, scheduling batch jobs for off-peak hours can save significant time.

Fourth, HTTP/3 and connection reuse matter more than people think. APIs that support HTTP/3 consistently outperform HTTP/1.1 by 15-30% in our tests. Keep-alive connections reduce handshake overhead by 50-80ms per request after the first one.

Fifth, the providers with the best P50 often have worse P99, and vice versa. It's a tradeoff. Some providers optimize for consistency (low P99), others optimize for speed under light load (low P50). Know which one matters for your use case.

Common Mistakes That Invalidate Your Benchmark

Before you run off and benchmark everything in sight, avoid these common pitfalls. Mistake one: running too few requests. A benchmark of 10 requests tells you almost nothing. You need hundreds, ideally thousands, for statistical significance.

Mistake two: ignoring the warmup period. The first request is always slow. Always. Network connections need to be established, TLS handshakes need to complete, caches need to be populated. Discard the first 3-5 requests or your median will be artificially inflated.

Mistake three: testing from a single location. A benchmark from your laptop in San Francisco tells you how the API performs for users in San Francisco, which might be the worst-case scenario for that provider or the best-case. Test from multiple regions.

Mistake four: conflating different request types. A 10-token completion and a 4,000-token completion are not the same workload. The latency characteristics are fundamentally different. Benchmark the actual request types you'll be making in production.

Mistake five: forgetting about retries. If an API returns 5% errors, your effective latency is much worse than the successful requests, because failed requests often need to be retried, doubling or tripling the user-perceived latency.

How to Use This Information in Production

Benchmarking is useless if you don't act on it. Here's the practical workflow we recommend. First, identify the user experience you want. Maybe you need P99 under 1 second, or P50 under 200ms. Whatever it is, define it before you benchmark.

Second, measure your current state. Use the script above (or similar) to measure your existing API provider. If you're already meeting your latency targets, great, move on to other problems. If you're not, you have a baseline.

Third, evaluate alternatives. Most teams should be evaluating 2-3 providers, including at least one unified API that gives access to multiple backends. This protects you from any single provider having a bad day, region, or model version.

Fourth, implement caching aggressively. A cached response that takes 5ms to serve is always faster than an uncached response that takes 200ms. Cache everything you legally can, and set appropriate TTLs based on how fresh the data needs to be.

Fifth, set up monitoring. A one-time benchmark tells you the state of the world on one day. Continuous monitoring tells you the state of the world every day. Providers have outages, deploy bad updates, and experience load spikes. You need to know when your API