Download
🎧 Podcast Summary
Listen to the narrated summary of the paper:
Abstract
Between them, the members of a research group or a circle of friends own several consumer computers, none large enough alone to run a capable large language model. Existing systems pool such capacity across open swarms anyone may join, which a group admitting only trusted machines cannot use. Bounding membership removes what those systems depend on: a swarm holds each part of the model on several peers and routes around a slow one. A bounded session must use every device it admits. Its pipeline advances at the pace of whichever device received a share it cannot serve quickly, so the division has to be right before serving begins. We propose Kafila. Its protocol assembles a ring from behind independent NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device’s memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head, which holds the embedding and output projection, together with that division rather than beforehand. On unequal machines in three countries Kafila shortens the slowest stage by $4.2\times$ against uniform division and $2.6$ to $3.5\times$ against a memory-proportional one, and serves models for which uniform division admits no assignment at all. What that is worth to a user depends on how much of a token is computation rather than network. Across continents most of it is network; on members sharing a site the same division returns $1.6\times$ the throughput to one user and $2.7\times$ the aggregate to four.
Figure 1: Session Architecture
