Building KRXY — Part 7: From Two AI Nodes to Distributed Learning

Part 6 ended with two Raspberry Pis answering independent requests at roughly twice the batch throughput of one. That was a useful result, but it was still inference: each Pi ran published model weights, and neither was learning anything from those requests. The more ambitious part of KRXY had always been to investigate whether small, unreliable edge computers could participate in training a model instead of only serving one.

The next two days, 30 and 31 October 2026, changed the project in two different ways. First, the two-node platform had to become reliable enough to survive restarts, context changes and ordinary chat workloads. Then the research returned to a problem that had resisted the first experiments: training a small model from random weights, generalising beyond its training examples and spreading that learning across two physical Pis. The development logs and experiment records are useful here because some gates passed, others failed, and the distinction matters more than a headline speed-up.

30 October: fixing what two nodes exposed

The previous comparison had revealed a scheduling race. Two concurrent requests could choose the same supposedly free node, leaving the second request with node_busy even though another compatible Pi was available. On 30 October, the admission path gained a bounded retry rather than a permanent queue. In a live two-request check, both short CPU chats completed on different nodes, with their executions overlapping. When the entire compatible fleet was genuinely busy, the platform still returned node_busy instead of waiting indefinitely.

CPU generation also began streaming useful text through the agent and gateway before the job finished, while Hailo remained buffered. An initial stream check delivered its first content chunk after 3.3 seconds and finished its short reply after 5.6 seconds. That was a responsiveness improvement, not a claim of faster decoding. The earlier benchmarks had measured complete buffered responses; from this point, time to the first visible token could become a separate metric.

Longer conversations exposed another weakness. A character-count estimate had been deciding whether CPU chat fitted the configured context, even when the real tokenised conversation would fit. The gateway was changed to count the Qwen2.5 chat template with its tokenizer, include the output reservation and compare that total with the runtime's reported context. Both CPU services were moved to 4,096 tokens of configured context. That is not the upstream model's maximum context, and it did not change the compiled 2,048-token Hailo limit.

A multi-turn contact-page test then showed why the extra room mattered. Longer follow-ups could be admitted without pretending a character estimate was a token count. The tests also uncovered a timeout: a 3B reply was still being generated when the agent's older 240-second client limit ended the request. The wait was increased to 600 seconds, and ordinary CPU chat without an explicit limit was given a 2,048-token output allowance. The smaller 256-token evaluation budgets remained explicit and unchanged. A longer budget can make a complete answer possible, but a Pi can still spend minutes generating it.

Trying more CPU threads was not a shortcut

The same day, one CPU-only Pi was used for a controlled thread comparison on the Qwen2.5 3B GGUF. The experiment compared two, three and four llama.cpp CPU threads. On a fixed 128-token decode, two threads took 19.77 seconds, three took 19.62 seconds and four took 20.97 seconds. The difference between two and three was small; four was slower.

For the full contact page, two threads produced a closed page in 146.1 seconds of compute using 882 output tokens. Three threads wrote a longer answer, finishing in 160.0 seconds at 976 tokens. Four threads ran for 176.7 seconds, hit the 1,024-token cap and still left the HTML incomplete. More threads did not make this workload more useful or faster. The service stayed at two threads. The tests were on one Pi, with three measured repeats after a warm-up, rather than a general rule about every model or workload.

Remote power: a failed restart before a working one

The operational milestone was less glamorous but just as necessary. A provider dashboard with Restart and Shut down controls is not useful if pressing Restart stops inference services and then fails to reboot the board. That happened in the first authorised Node 02 test. The agent had been told it could use a narrowly scoped sudo helper, but the agent's systemd service ran with NoNewPrivileges=yes. The helper could not gain the permission it needed. The agent stopped the CPU servers before discovering that failure, leaving the node unavailable for new jobs until the services were restored and scheduling resumed.

The attempted restart was recorded as failed, not as a success. The helper was removed to prevent the same failure from being repeated, and the design was corrected. The replacement used systemd-logind with a restricted policy and checked that the action was actually permitted before stopping the managed inference processes. The agent only advertised restart and shutdown when the running process could use the mechanism.

The next authorised test was successful. Node 02 was closed to new scheduling, acknowledged the command and requested a reboot. Around two minutes later it returned with a different boot identity, the same enrolled node identity, one agent process and both CPU inference services running at the expected 4,096-token context. The platform recorded a verified restart, after which the provider explicitly resumed scheduling. Node 01 kept serving while its neighbour restarted.

Node 01 then needed its own service configuration brought into line with the processes actually running. Earlier, hand-started CPU servers did not match the older systemd units: a cold restart could have brought back a smaller context and no 3B service. Those processes were adopted under systemd, the newer power-capable agent was installed, and the Hailo service was left in place. Later that day, an authorised shutdown of both Pis was followed by switching both back on. Each returned with its intended services active, and the previous boot's logs were available. The dashboard needed one more correction because it still described returned nodes as offline after a shutdown request; the status was changed to show that they were back on but deliberately held from scheduling until Resume.

The significant result was not merely that two Raspberry Pis could power cycle. It was that the platform had a checkable distinction between a power command being requested, an operating system being asked to act, and a node actually returning from a different boot. The design was still a development setup, not a production availability guarantee.

The first training baseline that generalised

While the serving platform was being stabilised, the research track revisited the failed early training experiment. An earlier tiny model had memorised arithmetic examples without solving held-out ones, so comparing distributed versions of that model would not have answered anything useful. On 30 October, a revised byte-level transformer with approximately 0.86 million parameters was trained from random initialisation on three-digit addition with reversed answer digits. The training, development and test sets were separated by sum values, meaning the held-out targets were not simply copies of sums seen during training.

The centralised baseline, RQ001-E0004, passed its pre-registered gate. All three seeds achieved 100% exact match on the recorded train, development and held-out test evaluations. The test gate required at least 90% exact match, and the seed-0 test set contained 2,000 held-out pairs. This was not a general language model, nor evidence that KRXY had created an alternative to Qwen. It was a controlled small task with a measurable learning target, which the previous experiment had lacked.

The first baseline ran on the development Mac's Linux virtual machine, taking 263–284 seconds of optimiser time per seed for the fixed 8,000-step budget. After this, the research rule was tightened: the Mac would remain the control plane and interface, while model-training experiments would run on the Pis.

The same model and training code were then run on the CPU-only Pi as E0004b. It also reached exact-match 1.0 on all three seeds, but took about 3,931–3,970 seconds of optimiser time per seed, around 14–15 times the time measured on the Mac VM under their respective thread settings. That was a useful baseline rather than an embarrassment. If decentralised edge training was to help, it had to overcome genuinely slower workers, not assume a Raspberry Pi was a workstation. The Pi peaked at 69.7°C during this run, without a recorded throttle event or chat interruption.

31 October: simulated workers, then physical workers

The next experiment, E0005, tested eight heterogeneous simulated training workers, not eight physical Pis. Actual model updates were computed, but worker speeds, links and failure events were controlled by a simulator. Asynchronous updating met its registered simulated time-to-target scaling rule on all three seeds, while synchronous averaging did not. However, the scheduled worker interruptions happened after the learning target had already been reached. That was a weakness in the experiment design, not proof of robust learning through early failures. The timing result was also sensitive to the measured cost of small worker batches.

E0006 addressed that gap by moving the simulated leave and rejoin events earlier, before convergence, and by testing smaller float16 payloads. Training still reached the required held-out target after the interruptions on every seed. The asynchronous run was 1.64×, 2.70× and 1.64× faster than its simulated centralised comparison across seeds 0, 1 and 2, but it failed the predeclared requirement of at least 2× on every seed. Float16 payloads roughly halved the transfer bytes and improved time to target without a loss of final test accuracy in this experiment. Those remained simulation results, not claims about physical network performance.

The more important step came next. E0007 ran asynchronous float16 training as real processes on two Raspberry Pis, with one Pi also coordinating updates. That coordinator mixed parameters but did not train on examples. Both Pis contributed to the learning process, and a training worker was deliberately killed and restarted while the rest of the platform was left running.

On this small arithmetic task, two-Pi training reached the development threshold 2.19×, 3.88× and 2.09× faster than the registered one-Pi baseline across the three seeds. But these were times to a learning threshold, not pure hardware throughput. The two-worker method used smaller batches and sometimes needed fewer examples. The sustained throughput improvement was closer to 1.46× in this initial comparison. More importantly, the recovery gate failed: the surviving worker reached the learning target just before the killed worker came back. The rejoined worker was accepted and contributed later, but the experiment had not proved recovery to the target after a rejoin.

That led to E0008, a separately registered test later on 31 October. The one-Pi comparison now used the same batch size as the two-Pi method, and the worker was killed and restarted earlier so that the learning threshold would still be ahead of it. This time, the distribution, recovery and learning gates all passed.

E0008 seed	One Pi to target	Two Pis to target	Speed-up
0	285.3 s	118.0 s	2.42×
1	192.2 s	98.8 s	1.95×
2	185.8 s	83.8 s	2.22×

The matched full-budget throughput improvement was approximately 1.88–1.96×. In the recovery arm, one worker's training process was killed at 6,400 examples and restarted at 12,800. At the moment of restart, the development score was below the registered 0.99 target on every seed. The target was then reached later, after the rejoined process had already contributed between seven and eleven accepted updates. Final held-out test exact-match was 1.0 across the tested recovery runs. This was the recovery condition E0007 had failed to establish.

The real training test still had boundaries. It was two trusted Pis on one local network, learning a tiny arithmetic rule under a trusted coordinator. It did not involve Hailo-based training, internet-connected untrusted providers, or a general-purpose language model. The coordinator was still needed for parameter mixing. But for the first time, the project had measured physical distributed learning and recovery, not just a simulator of both.

What the model library looked like then

The model library remained an important interface, but the five-model view shown in later screenshots is not the 30–31 October setup. At this point the active hosted deployments were Qwen2.5 1.5B on Hailo, Qwen2.5 1.5B Q4_K_M on CPU, and Qwen2.5 3B Q4_K_M on CPU. Node 01 could advertise all three. Node 02 had the two CPU deployments and no Hailo accelerator. The Hailo entry was the default, while CPU 3B remained a selectable candidate.

The later Qwen3.5 2B and 4B experimental services, with their larger configured contexts, were added in November and should not be backdated into this article. Likewise, the tiny from-scratch arithmetic model was a research experiment, not a new hosted chat option in Open WebUI. Showing the distinction between a consumer model library and an experimental training artifact is part of reporting the project accurately.

What these two days established

Part 6 showed that a second Pi could almost double throughput for independent inference jobs without shortening an individual response. Part 7 marks a different milestone: two physical Pis also participated in learning a model together. The progression was not a straight series of successes. The first remote restart failed before the mechanism was corrected. The eight-worker simulation left an important failure-timing gap, so a second simulated experiment moved the interruptions earlier. The first physical recovery test missed its own criterion and stayed failed. Only the later registered physical experiment demonstrated learning after the worker rejoined.

The two paths—serving and learning—were still separate. Hosted Qwen models answered chats; the from-scratch transformer learned a small, controlled arithmetic task. Neither result proved a decentralised foundation model or removed the trusted coordinator. What changed was that the original research question was no longer supported only by diagrams and simulated worker claims. There was now a real two-node training result to investigate, replicate, stress and attempt to generalise.

The next question was how robust that result would remain when the network stopped behaving like a quiet local lab: slower links, changing bandwidth, lost connectivity and workers that could not always stay available. Those conditions would become the next research tests.