Go Channels slow down with more CPUs - Grant Stephens, Fastly

Fastly uses Go extensively in our logging systems. The hosts that handle these logs typically have 128 Cores and 1TB of memory and handle around 1M logs per second. Channels are used extensively in this code, however on these high CPU count hosts there is some unexpected behavior which can cause significant performance bottlenecks. Hitting this bottleneck on a production workload is not fun- customers want their logs now, not later. Building on the channel speed analysis done by Phil Pearl we found that a high core count host’s channel performance could be 10 times slower than low core count machines. This is especially relevant if bottlenecks exist where large parts of the data eventually flow through a single channel.The combination of the high CPU count and high throughput per channel pose some interesting possibilities for solutions as the two factors are interlinked. A number of solutions to this problem exist, however there is no panacea. Batching channels, parallelizing workflows and buffering were all investigated.

[@namandave1359]

what was the last question?

[@peppybocan]

Yeah, I don't think Go's scheduler is necessarily optimised and/or built for 128 core scheduling and contention🤔Usually microservices have ~1 core or units of cores. This use case seems a bit of an odd edge case. I would, instead of having 128 cores, downsize the nodes to 32 core ones, which, yes, increases the number of connections but I don't think that's necessarily a problem. 🤔

[@sortof3337]

It is kind of still slow on 32 cores even. Also go build times are atrocious on giant project. Recently started work on a huge project with lots of libs and close to 2mil lines of code. Build times are just as bad as rust. :( Also I don't like that it doesn't give you any information when its building stuff. :|

[@jcbritobr]

Thats bizarre...