Cloudflare, a provider of CDN, security and edge compute services, recently released new performance metrics of its Workers platform, claiming it is now 196% faster than Fastly’s [email protected]
Cloudflare based these results using a time to first byte (the length of time between a client making an HTTP request to when the client receives the first byte of the request’s response) function run on 50 nodes using Catchpoint’s data from across the world.
According to Cloudflare’s blog post, tests were performed on November 8, 2021, using a free tier account for both Cloudflare Workers and Fastly’s [email protected]
The results of the Cloudflare test. Credit: Cloudflare.
However, according to Laura Thomson, VP of Engineering at Fastly, Cloudflare’s recent benchmarks would be seriously flawed.
“The biggest problem is that they conflate network round trip time with ‘execution at edge’ time,” she explained in a Twitter post.
“[Cloudflare] is measuring cherry-picked [round-trip time] instead of edge compute performance.”
“We will publish comparisons once we have enough data to be statistically significant, because that is kind of important.”
In addition, the Fasty executive explained that benchmark details can be constructed to suit specific systems.
“To build a meaningful benchmark a great deal of care has to go into setting up the test correctly. You have to make sure the test doesn’t hit artificial limits (for example, free trial resource limits) as primary evidence suggests happened in this case.”
According to Thomson, Cloudflare’s methodology was also flawed, as the company did not publish which CatchPoint nodes were used, thus preventing anyone else from doing direct comparisons.
“The benchmarks claim to compare @fastly ’s JS implementation which is in beta vs CF’s JS, which is not,” she explained. “I wonder why they didn’t measure Rust, which is much more mature on our platform than theirs.”
Finally, Thomson called for users to run their own benchmarks on Fastly, while also bearing in mind there are CPU limits on free trial accounts.
“You won’t be able to benchmark on Cloudflare (and we can’t either) because their terms of service forbid it,” she concluded.
Cloudflare did not reply to the Twitter post directly. This story is still under development, and we’ll make sure to update it should Fastly or Cloudflare provide any more clarifications.
A spat over benchmarking results might seem to be small potatoes. However, with speed being a key element of edge computing from a developer perspective, it’s not surprising that Cloudflare and Fastly are competing hard on that front. Winning the speed war is important, but other factors such as which languages are supported and developer experience (including platform ease of use) are also important factors that edge platform providers will need to offer developers.
Jim Davis, Principal Analyst, Edge Research Group
Catchpoint | CDN | Cloudflare | edge PaaS | Fastly | performance