`silvic-net`

The collectives distributed training runs on, over a TCP ring: TcpCollective implements the Collective of silvic-core. silvic.ray.train creates one per Ray actor; any other launcher can do the same.

let listener = TcpCollective::bind_advertised("0.0.0.0", &node_ip)?;
let address = listener.address(); // gather every rank's, in rank order
let mut ring = TcpCollective::connect(listener, rank, &addresses, job_token, Duration::from_secs(1800))?;

Public API

TcpCollective::bind(host), TcpCollective::bind_advertised(host, node_ip): A listener on an ephemeral port. bind_advertised listens on host, such as 0.0.0.0, and advertises node_ip; bind("0.0.0.0") is an error, since other nodes cannot reach that address.
listener.address(): The ip:port the other ranks connect to.
TcpCollective::connect(listener, rank, addresses, job_token, timeout): Joins the ring. timeout bounds the rendezvous, then how long a collective waits on a silent peer; set_timeout changes it later. A world of one opens no sockets.
allreduce_sum_f64, allreduce_sum_i64, allreduce_max_f64, allgather_bytes: The Collective methods: every rank calls the same ones, in the same order, with buffers of the same length.

The ring

Each rank connects to the next and accepts from the previous. The handshake carries a version, the job token, the rank and the world size, and rejects stray or mismatched peers. An allreduce is a reduce-scatter and then an allgather around the ring; below 4 KiB it is a single allgather. Every chunk folds in a fixed rank order, so a result is the same on every run for a given world size, and exact in any order for the integer-valued histograms quantized training reduces. Sockets are non-blocking and polled, so a ring step sends and receives at once and large buffers cannot deadlock.

Failures

Every frame carries the operation, a sequence number and the length, so ranks that call different collectives fail with an out-of-step error instead of exchanging garbage. A socket error, a closed peer or a timeout fails the collective with a Collective error naming the peer rank; the rank then shuts its sockets, so the failure travels around the ring, and every later call fails the same way.