...
Infiniband: assuming you want to do RPCs, you’ll have to use either its reliable connections, which are stream-based and thus have many of the same problems as TCP streams, or its unreliable datagrams, which have all the awkwardness of UDP. Infiniband congestion control is based on priority flow control (PFC), which I believe has serious problems that have been widely documented. Infiniband’s one-sided RDMA operations can be very fast, but they only work well for tasks with relatively static data layouts (in general I’d argue that distributed shared memory is not a good abstraction for distributed systems, but that’s another discussion). I will say that the NICs developed in the Infiniband world (especially those from Mellanox) are spectacularly fast; fortunately this NIC technology can also be used for protocols like TCP and Homa.
RocE: it has all of Infiniband’s problems and contaminates our Ethernet-based networks with PFC, which seems like a bad idea.
SCTP: I don’t know enough about this protocol to make meaningful comments, but it looks like it’s designed for WANs. Any particular reason to believe it will solve TCP’s problems in the datacenter?
QUIC: my knowledge is limited, but QUIC appears to be designed primarily for long-haul communication between browsers and Web servers. Thus, it addresses a variety of problems that made HTTP-over-TCP slow. I haven’t seen any indication that its designers considered issues related to low-latency communication in datacenters, so it would be fortuitous if it turned out to solve that problem. I have heard QUIC characterized as a protocol for RPCs, which would seem to make it similar to Homa, but its abstractions are reliable streams and unreliable datagrams, which sound more like TCP and UDP. Its congestion control mechanism is sender-based and has a lot of similarity with TCPs. Thus there is reason to believe that QUIC would suffer from many of TCP’s problems if used for datacenter applications.
...