Investigate user reported performance issue

[plorenz]

Initial Report

So in my first set of tests, I had two routers talking directly to each other over the private network. This was to get an idea of the best possible connection I could get. But in a real deployment, the routers at the endpoint would be private and behind firewalls, with a public router hanging out on the internet, which seems to be the typical setup in any case.

When I ran iperf through this setup, one connection seemed fine. Maybe slightly faster. Great! But then I moved on to telling iperf to add a second connection and later, 4 or 8 connections.

In all the cases where the number of connections were greater than one, I saw something like, iperf showing some initial data moving on all the connections, and then...nothing. 0mbit/s...then maybe a little bit more data moving...then 0mbits/s for a while, etc. And the more connections, the less data made it through to the point where 8 connections essentially moved 0mbits/s of data except in the first second.

Now, what's interesting here is that if I tell iperf to, say, throttle this down to 100mbits/s on each connection, then two connections or 4 connections work OK. Data moves continuously. But, if I add more connections it reverts back to the situation where I get 0mbits/s again.

This suggests that some kind of congestion collapse is happening if multiple streams really want to use a lot of throughput...but what's strange is that this only happens in this 3 router setup. With the 2 router setup, I could have as many connections as I wanted (though, after the combined aggregate bandwidth hit the max (between 1 and 2 gigabits/s), adding more connections just meant each connection got a smaller share of the overall max bandwdith).

The last update to my public thread seemed to indicate some issue with throttling or fairness with out of order packets? Maybe this is just tickling that bug or there's some basic issue around handling so much data on so many connections? But I thought you should know. Seems like the basis for a regression test if it's a real bug.

The logs didn't really seem to be saying much even on debug mode. The only thing that seemed off was that the private edge routers were complaining periodically that they couldn't store cached info because the index wasn't initialized? But it was unclear why and I'm not even sure it was important as the routers could clearly establish a connection, they just couldn't deliver data after a certain amount of saturation.

Second Note

So when I did my testing I took some baseline measurements with iperf. One that was just the "bare" network throughput which was ~25Gbit/s Obviously, that was going to be very high on a local network and wasn't going to be representative, but I wanted to know it to bracket what was possible.

So to get some more realistic idea of what a point to point TLS connection over this link would be I setup stunnel and piped iperf through that. That got ~4Gbits/s. Now, stunnel is a very simple proxy and only really seems to consume one CPU per connection...but it shows a basic idea of what is possible on that link if you do TLS and burn a CPU doing it just going point to point.

So when I did the 2 router Ziti setup, I assumed OpenZiti would be somewhere around 4Gbit/s since, ultimately, it would be tunneling my data over an SSL link to the other side. Now, granted, OpenZiti does more with the connection than stunnel does. So, going in, I figured that I might see something like ~3.2Gbit/s...or maybe even 4Gbits/s or more since maybe OpenZiti might be using more CPUs or doing something else that stunnel could not.

So it was pretty deflating to see 700-800Mbit/s through OpenZiti with two nodes.

Third Note

If you route iperf through a OpenZiti network and you profile the transmitting router using strace, and you look at the reads and writes, and look the intervals between reads and the intervals between writes, you can see some interesting things. The biggest thing is that reads and writes are fairly balanced but there are periods where data is read into the router but the corresponding writes just stall.

The standard deviation for the writes is higher than the reads and if you plot the write times on a histogram, you'll see the problem lies in the tail of the distribution. Basically, a small number of packets have very high latency and this seems to spoil the overall throughput.

I wonder if this gets worse as more high bandwidth connections are added and suddenly the whole network collapses... thus yielding those 0Mbit/s connections I mentioned above. Profiling like this and looking at how you can deal with it when handling the packets and sizing the buffers will probably help things a lot.