1/ Just got the FIBRE stats page updated for the network architecture I switched to a year+ ago. For those who missed how awesome FIBRE is and how low-latency block relay is (mostly) a solved-problem, keep reading bitcoinfibre.org/stats.html
2/ The top graph is the most important one, and shows (by default in log-scale) the time it takes to send a block around the globe in excess of the Speed of Light. Right now it shows that 92% of blocks got to every node in under 10ms more than the SoL between servers!
3/ This graph also shows that 85% of blocks were delivered in under 7.5ms, which is *faster than the time to send 1MB through a 1Gbps link*! Note that the time here does *not* add up as you take more hops, even when we go through 4 servers on the way, its *still* just 10ms + SoL!
4/ Next we have some scatter plots showing the time to reconstruct a block in relation to the amount of data used. Obviously we can't send the full 1MB block or we wouldn't be beating 7.5ms, and you can actually see the link speed of the servers in these graphs!
5/ Time past SoL is a critical performance metric, but another bit of magic behind (the public) FIBRE (net) is the careful layout that keeps said SoL as low as possible. Relay times and Tn time show 98% of blocks making it to every node in 165ms! AWS can't even get close to that!
6/ Of course any relay net is only as good as it's block sources, so we track the number of round-trips (ie compact block reconstruction failures or blocks provided via non-compact-blocks) as that implies it took longer than necessary to get blocks!
7/ Even worse than an extra 10ms to get a block is getting a block from somewhere other than the original pool who found the block. While we don't know pool IPs so can't track this directly, we *can* indirectly measure this by looking at how many nodes beat our internal relay.
8/ Bc our internal relay is ~as good as you can get, if we get a block from more than one source, it implies that the block was already propagating globally by the time our first more got it. Ofc many pools have their own ~optimal relay nets so this requires some interpretation.
9/ We can also measure direct peering via source stability. In the header, you can filter all the stats by the pool which (probably) found the block. Then, looking at the number of sources for the 1st recvs, you can see whether blocks always come from the same sources or not.
10/ That, in combination with looking at the filtered number of secondary sources, gives you a good indication of whether a pool is directly connected or not. Of course it requires some interpretation as some pools have many servers or their own relay nets.
11/ Finally, there are pie charts breaking down where each individual node received blocks from, allowing me to ensure the network layout is working as expected (bonus: check the JS debug console for comparison of UDP and ICMP ping times between nodes, it's easy to spot the GFW!)
12/ TL;DR: Each stat on the pub FIBRE net stats page is there for a reason and has gotten lots of manual optimization, and there's a lot there! Not just that but everything is public to make it easy to copy my work (and some have!)
13/ When someone tells you block relay needs to be expensive or can be done on AWS, point them to that stats page and shake your head. Plus, pools run their own nets based on this work, preventing any censorship that a single relay net could do already! bluematt.bitcoin.ninja/2016/07/07/rel…
14/ Finally, thanks to the folks who threw a few-k into the yearly server costs for 2018: @BITMAINtech @BitfuryGroup @slushpool and, as always, @ChaincodeLabs for providing a home for this (and other) research!
Also, if any solo-miners/pools have a PoP in Canada, I do have a Montreal node, I just don't currently expose it (@SGBarbour, or @sysmannet maybe?)
