The next two days, 28 and 29 October 2026, were the first attempt to answer that. A second Raspberry Pi 5 with 16 GB RAM and SSD storage joined the work, this time without the Hailo AI HAT+ 2. That difference was deliberate. If the scheduler really understood model capabilities rather than hardware names, a CPU-only Pi should be able to advertise the same pinned GGUF as the first node and accept an appropriate job. The two days also included an unexpected outage, postponed measurements, cooling work, concurrent inference tests and the beginnings of remote power control. Some results were encouraging; others made the limits of the first implementation clearer.

A second physical node, without Hailo

The original node combined a Raspberry Pi 5 with a Hailo-10H accelerator and CPU-based llama.cpp inference. The second node was another Raspberry Pi 5 with 16 GB system RAM and a roughly 256 GB USB-attached SSD, but no Hailo accelerator. For the platform, that distinction was important: it was no longer safe to assume every provider had identical hardware or the same inference runtimes.

At this stage the available hosted deployments were still the three Qwen2.5 variants from Part 5. The two later Qwen3.5 deployments had not been commissioned or advertised as usable hosted models on these dates.

This was not five models, and the entries should not be confused with five separately trained KRXY systems. These were hosted deployments of published pretrained weights. The Hailo compiled model and the CPU GGUF were not identical files merely because the model family had a similar name. KRXY continued to match deployment ID, runtime and reported artifact digest before placement. Neither Pi was training those published weights.

28 October: an outage before the comparison

The first day did not produce the clean two-node speed comparison I had planned. While the 3B model file was being copied onto the new Pi, Node 02 stopped answering on the local network. The cause was not established. A later boot recorded an ext4 orphan-cleanup message, consistent with an unclean filesystem unmount, but that did not identify why the earlier interruption happened. There was no persistent previous-boot journal to settle the question. I did not treat the incident as a solved thermal or network fault.

The platform could still serve a request from the original node while the second one was offline. A short CPU 1.5B routing check returned two output tokens with a reported compute time of 562 milliseconds and a client wait of 4.435 seconds. That showed that an offline provider was not being selected for this request. It was not a useful speed benchmark.

I also traced a real Open WebUI contact-form reply back to the original node. It used the CPU 3B deployment, reported 813 output tokens, finished naturally, and took about 135.5 seconds of node execution time. Only Node 01 was claiming that exact 3B deployment at the time. This was evidence of the path actually taken, not evidence that two nodes had shared the work.

An initial six-request matrix on Node 01 followed, using the same 3B model with temperature 0 and a 512-token limit. The mean reported compute time was about 83.8 seconds. Every response ended with finish_reason=length, so these results measured a fixed output budget, not completion of a working HTML page. The remaining comparisons were postponed because the second Pi did not yet have its CPU cooling fan fitted. A faster-looking test would not have been worth much if its hardware conditions were inconsistent.

29 October: cooling, health and a fairer baseline

Once both Pis had PWM cooling fans, the comparison was repeated under a more consistent setup. The tests used the same pinned Qwen2.5 3B Q4_K_M model, temperature 0, two CPU threads and one generation slot on each Pi. Holding one node at a time produced single-node measurements; releasing both allowed the scheduler to place overlapping requests.

The thermal readings were not identical. At the start of the recorded session, Node 01's fan was around 3,003 RPM and Node 02's around 1,742 RPM. During generation they rose to roughly 9,500 and 5,300 RPM respectively. One Node 01 sample reached 81.8°C and latched the historical soft-temperature-limit flag, although active throttling bits were clear when checked. Node 02 reached 70.8°C in a mid-job sample without a throttling flag. There was no observed undervoltage, out-of-memory error or storage reset during this run, but the earlier outage remained unexplained.

Adding cooling was necessary to run a more credible comparison, but it did not magically make the first Pi faster. Its mean compute time for the 512-token stylish prompt was 83.779 seconds on 28 October and 83.755 seconds on 29 October. That small difference was not evidence of a meaningful performance improvement.

Two Pis could finish two jobs in the time of one

The main result came from overlapping independent requests. In the fan-fitted 512-token matrix, the original node took about 83.8 seconds of compute for the stylish prompt and the second took about 82.6 seconds. With both available, paired requests overlapped for roughly 81–84 seconds. A documented pair produced 1,024 output tokens across two jobs in an 83.9-second span, approximately 12.2 tokens per second of batch wall throughput.

For comparison, a single 512-token request had about 86–88 seconds of client wall time, roughly 5.9 output tokens per second from the caller's perspective. The overlap result therefore showed about twice the batch throughput when two independent jobs could run together. It did not show one answer being generated twice as fast.

The both-nodes row describes the average individual request within the paired workload, not the time required to finish both jobs together. That distinction is the point of the experiment. Each model was still generating on one Pi. What changed was how many jobs the network could progress at the same time.

All 36 requests in this fixed 512-token matrix ended at the token limit. They were not 36 valid completed contact forms. In a separate 1,024-token comparison, the stylish page naturally finished after 882 output tokens. Node 01 took about 148.9 seconds of compute and Node 02 about 145.7 seconds. Two pages could overlap in roughly the wall time of one, but neither individual page became a short task just because another node existed.

Trying to split one answer across nodes

The next question was whether two machines could cooperate on one deliverable instead of just taking separate chats. I ran an opt-in contact-form experiment that assigned markup and styling as separate child jobs, then combined them locally and checked the result. It was not the normal chat path and was not a model-sharding system.

The two child jobs overlapped by about 27.9 seconds, so the scheduler had achieved some parallelism. But the assembled result did not satisfy the validation contract: the expected .contact-submit CSS selector was missing. A repair job added roughly 111 seconds and still did not produce a valid result. End-to-end wall time was about 153 seconds, slower than the single-node complete-page comparison. Parallel work alone had not improved the final answer.

That was a useful failed experiment. Breaking a task into pieces introduces interface contracts, assembly rules and repair overhead. If the pieces do not join cleanly, the apparent concurrency is irrelevant to the person waiting for a usable result. KRXY had demonstrated parallel independent inference. It had not demonstrated that splitting a single answer was better.

A scheduler still needed a better admission path

The first fully simultaneous attempt also exposed a routing edge case. One request was admitted while the other returned node_busy, even though another compatible Pi could have taken it. The scheduler chose a free node and then locked it; two competing admissions could choose the same node before the state changed. This was not proof that the whole fleet was full.

The issue informed the next serving change: retry placement briefly after node_busy so the other free node could be selected. That improvement belongs to the work immediately following this benchmark, not to the measured 29 October results. I am not counting a later fix as something already proven here.

Node health and remote power controls

The second day also moved the provider dashboard beyond an online/offline label. Node-health work added compact displays for measured resources and temperature alongside the existing capability reporting. The intention was to show the operational cost of running a model, not only its name. CPU usage, memory, storage, temperature and cooling state matter when nodes can have different physical conditions.

Remote restart and shutdown commands were also added at the platform layer. The design distinguished a command being accepted from the operating system actually acting on it, and a power request was meant to stop new scheduling while existing work drained. That safety boundary mattered, particularly with two providers.

The API rollout happened on 29 October without restarting the Pis. Both existing agents continued heartbeating, and a short inference job completed afterwards. However, the running agents were not upgraded to execute the new power commands: local installation required a privileged prompt. Real remote restart and shutdown were not verified on this date. Those controls were therefore a platform milestone, not a working remote-power milestone. The later power-agent installation and recovery tests belong in the next article.

What two days established

By the end of 29 October, KRXY had crossed an important line. It was no longer just one Raspberry Pi offering several local runtimes. Two physical nodes with different hardware could advertise compatible CPU deployments, and the scheduler could run independent 3B requests on both. The measured benefit was about twice the throughput for overlapping jobs, not a faster individual answer.

The same work gave me a list of things not to claim. The 28 October outage was not explained. The Node 01 temperature warning was real even though active throttling was not observed during the check. The 512-token responses were truncated. Splitting a contact-form task failed its validation contract and added time. The first simultaneous admission needed a retry improvement. Remote power existed in the API but had not yet been exercised by upgraded agents.

None of this demonstrated distributed model training. The Pis were serving published pretrained weights; they were not yet learning a common model across unreliable hardware. But this was the first physical evidence that the platform could gain useful inference capacity by adding a heterogeneous node without pretending the machines shared memory or jointly decoded one answer.

The next work would make that two-node setup more robust: repairing the admission race, improving streaming and context handling, aligning live services with startup units, and testing power recovery rather than assuming a node would come back correctly. Those operational details would become essential before relying on the network for longer experiments.

This part follows the evaluation work begun late on 25 October 2026 and the changes recorded on 26 and 27 October. There was still one physical Raspberry Pi. A second CPU model, a larger hold-out test, a model library and a very visible incomplete HTML response all changed how I thought about model choice. The records are the evaluation reports, the frozen hold-out specification, the node runbook and the development commits. Where the tests were small or automatic, I have kept those limitations attached to the results.

From a working route to a measured answer

After the routing changes on 25 October, KRXY gained a Model Evaluations area. The first practical template had 12 tasks covering things such as drafting a reply, extracting structured information, correcting statements and answering technical questions. A run could select two hosted deployments, send the same cases to each, store the returned text and record pass, fail or review decisions. An output limit and generation settings were saved with the run so that a later comparison did not depend on remembering what had been selected in a browser.

The first 24-request comparison used the Hailo Qwen2.5 1.5B deployment and the CPU Qwen2.5 1.5B GGUF, one repetition of the 12 tasks per model. It completed, but the early scoring rules exposed a problem of their own. Simple text matching can mark a reply wrong because it uses a different but valid expression, or overlook the fact that a letter addresses the wrong person. The evaluation logic was revised without overwriting the original stored answers or assessments. The corrected report showed seven whole-answer passes, three fails and two reviews for the CPU deployment, against two passes, nine fails and one review for Hailo.

Those were 12 synthetic tasks per deployment, not a reliable ranking of two models in general. There was no independent human judging of every reply, no proof of model honesty, and no claim that one set of scores represented real-world accuracy. The important advance was having the replies and their scoring revisions saved together. The PDF exports carried the settings, category results, timing summaries and the actual answers, instead of reducing a comparison to one unexplained percentage.

26 October: trying a larger CPU model

The next step was to add a pinned Qwen2.5 3B Instruct model running through llama.cpp on the same Pi. The existing Hailo deployment and CPU 1.5B model remained available. The new 3B deployment was a candidate, not a replacement for the default. Its model file, runtime configuration and artifact digest were fixed, and the agent was extended to advertise it as a distinct deployment.

This was also a useful reminder of the limits of a small board. The extra model ran as another local process; it did not create a second provider. The runbook recorded that the 3B process was hand-started and that its prepared systemd unit was not enabled. The agent also had to be treated carefully: the live agent was running manually while its systemd service was inactive. Starting a second copy would have meant two agents polling as the same node. A model appearing in a library should not hide that kind of operational detail.

For the comparison, I did not want to reuse only the questions that had already influenced the earlier development work. A new 24-case hold-out was frozen before the candidate's answers were examined. It contained six email tasks, six extraction tasks, six correction tasks and six technical tasks. All three deployments were run with temperature 0, a 256-token output budget and one repetition. The recorded run issued 72 requests; 71 completed. One Hailo case returned no_eligible_node, so the formal paired comparison used the other 23 cases rather than pretending every model had produced an answer.

More passes did not mean promotion

On those 23 paired cases, the 3B candidate produced 14 whole-answer passes, compared with 12 for CPU 1.5B and seven for Hailo. Looked at alone, that would have made the larger model attractive. But the same comparison recorded six failures for 3B, four for CPU 1.5B and 13 for Hailo. The 3B candidate also took a median of 8.496 seconds of node execution time, compared with 3.623 seconds for CPU 1.5B and 6.756 seconds for Hailo on the paired set.

The promotion rule had been written before those answers were read. The candidate needed more passes than both controls, no more failures than the better control, no more length stops, and a median execution time no greater than twice that of CPU 1.5B. It cleared the pass-count condition and the length-stop condition. It failed the failure-count and execution-time conditions. Twice the CPU 1.5B median was 7.246 seconds; the 3B median was 8.496 seconds. The default therefore stayed on Hailo, and the 3B deployment remained selectable as a candidate.

That result was more valuable than automatically promoting the largest model. It showed why I wanted explicit acceptance rules in the first place. A larger model can improve some answers and still be the wrong default under the criteria selected for this particular project. These were automated checks on a small synthetic hold-out, not a population-level accuracy result. Time to first token, true decoding throughput and power consumption were not measured in that run, so I did not infer them from gateway timing.

Building a model library around real availability

With three hosted deployments in play, the node page needed a clearer way to show what was actually installed, ready and selectable. On 26 October, the platform gained the first model-library controls and recorded model load and unload operations. The 3B CPU candidate could be managed through a pinned command, rather than accepting an arbitrary model path or shell command from the page. Hailo remained tied to its existing service and was not presented as something the page could freely load or unload.

On 27 October, that library was made more compact. It put the model family, runtime, size, quantisation, context allowance and current state in a readable card, with the operational controls and evaluation link close to the relevant deployment. Technical details were available without making every row a page of diagnostics. The layout was checked on both wide and narrow screens, rather than treating the Raspberry Pi dashboard as a desktop-only tool.

The visual details had to be grounded too. A Qwen mark was used with its source and licence recorded. Gemma and Liquid entries used text labels rather than assuming their trademarks could be copied. The memory and storage rail showed host measurements, while accelerator memory was explicitly marked not measured. An execution-time range was shown only where completed jobs supported it. A token-per-second figure was not invented from total gateway duration because that includes work other than decoding.

The library was becoming a record of the difference between catalogue entries, installed artifacts and live services. That distinction came directly from the routing work in Part 4. A model can exist in the catalogue without being available on a node. It can be on disk without a ready runtime. It can be ready without being the recommended default.

The incomplete contact form

A different problem appeared during an ordinary Open WebUI chat on 27 October. I asked the 3B model to generate a styled HTML contact form. The saved response stopped in the middle of the code, ending with a lone <. Open WebUI showed the message as finished, but the form was missing its closing structure. That looked at first like an interface problem or a model that simply could not finish the task.

The stored KRXY job gave a more precise answer. It had completed as a job, but its finish_reason was length. The client had not requested a token limit, so the gateway applied its 256-token default. The runtime had returned exactly 256 output tokens. The incomplete text in Open WebUI matched the text stored with the job. The interface had not lost the rest of the answer; the rest had never been generated.

A direct test against the same local runtime helped separate the causes. At 256 tokens, the output remained incomplete. With a larger allowance, a sample finished the HTML structure and eventually stopped after 877 output tokens. That was one observed sample, not a guarantee that every similar prompt would need the same number. It was enough to demonstrate that the earlier 256-token boundary was preventing a useful answer from finishing.

The gateway was then changed so an ordinary interactive chat that omitted its limit received an allowance of 1,024 tokens. Evaluation runs that explicitly requested 256 retained that budget, as did the separate provider hardware-job default. The hosted-job wait was increased to allow for a longer CPU generation. The stored original result was not rewritten. When a future reply actually hits its limit, the client-facing text can show a visible [KRXY output limit] notice while the raw job retains its own finish reason.

The verification mattered. After the change, the original prompt produced a response that ended normally after 662 output tokens and included the closing HTML tags. A separate Open WebUI contact-form request used 945 output tokens and also returned complete closing tags and the explanation that sending email requires a backend or form service. That confirmed the improvement in those tested cases. It did not mean the model would always produce valid or production-ready HTML.

What the two days changed

By the end of 27 October, KRXY had more than a list of models that could be called. It had stored evaluations, a frozen hold-out for comparing a 3B candidate with existing deployments, an explicit decision not to promote that candidate, and a model library that separated what existed from what was ready. The contact-form investigation also clarified that a successful job status and a completed answer were different things. An answer can be returned successfully and still stop because of an output limit.

There were still important boundaries. The tests ran on one physical Pi, using published pretrained model artifacts. The evaluation verdicts were automated and limited to the tasks used. A model result was not an independently verified provider contribution. The 3B model remained a candidate, the Hailo deployment remained the default, and there was not yet a second real provider to compare node capacity across machines.

The next architectural step was no longer just to ask one Pi to host more things. It was to see what changed when a second physical Raspberry Pi joined the network, especially one without the Hailo accelerator. That would test whether the compatibility and scheduling rules from Part 4 still held when the platform had genuinely different machines to choose from.

This part covers that change and the work that followed during the morning and afternoon of 25 October. There was still only one physical Raspberry Pi. The second node, the larger evaluation runs, and the distributed-training results belong to later entries. The evidence here comes from the development log, the node runbook, the capability-scheduling decision, and the recorded live checks.

From an online node to a compatible deployment

Until this change, the gateway knew about a Hailo-hosted Qwen2.5 1.5B deployment. With only one real provider online, it could send a request there without making much of a placement decision. That was useful for commissioning, but it would not scale to a network where different nodes offered different runtimes and model files. Merely knowing that a Raspberry Pi was online did not tell the gateway which model it could execute.

The new placement rule matched the requested deployment against capabilities reported by a live node. The deployment catalogue recorded what the platform expected, while the agent reported what its running environment supported. A consumer's model choice was now a constraint, not a suggestion. The CPU diagnostic model was being prepared alongside the existing Hailo model, but preparation was not the same as availability.

On the first rebuilt gateway check that morning, the running agent was still 0.3.2-hailo. The model endpoint listed only hailo:qwen2.5-instruct:1.5b. A short request completed and returned Ready.. The newer agent files, CPU environment configuration, and a local llama-server process were already present on the Pi, but restarting the agent needed a local sudo password. The running process remained unchanged, so it did not advertise the CPU deployment. This was a repeat of the lesson from Part 3: the code on disk is not necessarily the code answering requests.

Why a model name was not enough

A name such as Qwen2.5 1.5B identifies a model family and size, but not a particular executable artifact. The Hailo path used a compiled HEF through hailo-ollama. The CPU path used a pinned Q4_K_M GGUF through llama.cpp. They were different artifacts, running through different software, even though both were based on published Qwen2.5 models.

The next routing change required a hosted-model claim to include its model ID, the catalogue runtime, and a 64-character artifact digest. That gave the gateway a stricter basis for matching a job to a provider. An existing enrolled Hailo node received a temporary legacy exception during the transition; it was not a general permission for nodes to claim models without evidence of the expected artifact. The agent's signed identity established which node made a claim. It did not independently prove that the claim was honest or that the model output was correct.

The difference became visible when the gateway received a request for cpu:qwen2.5-1.5b-instruct:q4_k_m while the live agent still reported only Hailo. The gateway returned no_compatible_node. It did not replace the CPU model with Hailo or fall back to the development simulator. Unsupported sampling options were rejected rather than quietly ignored. That refusal was a useful result: a plausible answer from the wrong model would have been a routing failure, even if the text looked good.

Making the CPU deployment available

Later that afternoon, the local sudo prompt on the Pi was accepted. The CPU service, krxy-cpu.service, started, and the running agent became 0.3.4-hailo. The node then advertised both hailo:qwen2.5-instruct:1.5b and cpu:qwen2.5-1.5b-instruct:q4_k_m. The gateway could see two hosted deployments on one provider, without treating them as two physical nodes.

The first live checks were deliberately modest. A Hailo request returned Ready. on the Hailo deployment ID, and a CPU request returned Ready. on the CPU deployment ID. The consumer page kept Hailo selected by default while making the CPU option available. These short responses checked routing and execution, not whether one model wrote better answers than the other. The CPU llama-server listened on the Pi's loopback interface. No second Raspberry Pi or remote CPU worker had been added.

The hosted jobs still had verification marked unsupported and reserved credits of 0. They produced no provider contribution or usage-ledger entry. Both the HEF and the GGUF were published model artifacts, not models trained by KRXY. A completed inference request was evidence that the route worked, not proof of model integrity, useful contribution, or a claim to payment.

Removing a capability without losing the node

The next test asked whether the catalogue would keep showing a model after its runtime had stopped. Only krxy-cpu.service was stopped. On the next catalogue read, the CPU deployment disappeared, and a request for it returned no_compatible_node. Hailo remained available and answered a separate request. Starting the CPU service again restored its model entry, and another CPU job completed.

This mattered more than simply proving that both runtimes could start. The model list now reflected a live capability instead of a record that software had once been installed. If the CPU runtime went away, its deployment went away with it; the Hailo route did not have to disappear as well. For a future network with mixed hardware and intermittent availability, the scheduler needs to distinguish a node that is connected from a specific deployment that is ready to take work.

Two runtimes, one busy Raspberry Pi

With both deployments present, the next check sent overlapping Hailo and CPU requests. Hailo succeeded, while the CPU request returned node_busy. The agent still allowed one in-flight generation on the provider node. Having two local inference processes did not automatically make them two independent workers, and the scheduler did not count them that way.

This was a conservative limit, but an intentional one. Running several models on the same machine is not equivalent to adding another machine to the network. The Pi's system memory, CPU and other resources are shared. Until concurrent capacity is measured and represented properly, a clear busy response is preferable to accepting work that the agent cannot safely run.

A final check repeated a request using the same idempotency key. The gateway returned the same job rather than creating a second one. Retries are normal in distributed systems: a client may lose its connection after a request has been accepted, or may send the request again because no reply arrived. Treating a retry as new work could eventually produce duplicate executions and duplicate accounting. That did not happen in this check.

What the day established

By the end of 25 October's routing work, KRXY could select an exact hosted deployment, refuse an incompatible request, show two runtimes on one node, withdraw the CPU capability when its service stopped, report a busy provider, and recognise a repeated job request. The important change was not the addition of a CPU model by itself. It was that a reported capability had become something the gateway used to make and enforce a placement decision.

There were limits to the result. Only one physical Raspberry Pi was operating as a provider. The model claims were not independently attested, the hosted responses were not verified contributions, and no hosted contribution credits were paid. Open WebUI was not signed in for the final activation checks. A training baseline also ran on the Mac that day, but its predeclared comparison gate was not met; that research has a separate story and is not evidence of successful decentralised training.

The next question followed naturally. Routing a request to the right model did not tell me whether that model was any good at the task. The evaluation work later on 25 October began measuring answer quality separately from the scheduler's ability to deliver a request to the intended runtime. That is where Part 5 will begin.

A local model does not join a network by existing. Something has to say which computer it is, whether it is still there, and what it is allowed to claim it can do. On KRXY that something is the node agent. This part is that program, as it stood on 24 October 2026, including the hour when the wrong copy of it was the one talking.

Why a node needs an agent

The control plane keeps the registry and the jobs. It does not sit on the Pi, and it cannot see /proc on the Pi. The Pi has to open the connection. That matters on a network where the worker may be behind an address that changes. The Mac does not dial the board. The agent calls out.

By the evening of 24 October the agent did four things in a loop. It signed a description of the machine and posted it as a heartbeat. It posted a telemetry sample. It asked whether a job was waiting. If one was, and it was for the Hailo model this agent knew, it called hailo-ollama on 127.0.0.1 and posted the text back. The model runtime produced the tokens. The agent did not. The agent is the worker's mouth, not its brain.

The default interval was five seconds. After the 15:20 reboot the commissioning notes say heartbeats resumed at about that interval. The platform's default is to call a node offline when the newest heartbeat is older than 30 seconds. Five minutes of silence is not a healthy node that happens to be quiet. It is a node the scheduler should stop choosing.

Giving Node 01 an identity

Six days before this hardware was commissioned, the platform could already enrol a worker. On 18 October the repository gained a mock gateway, enrolment, and a local stack. A simulated node could register. That is why the Pi did not invent the protocol on the day it booted. It used one.

Joining is a one-time token. The agent generates an Ed25519 key if it does not already have one, stores the private half in a file with mode 600, and posts the public half with the token. The platform returns a node id. The agent writes that id into a small identity file, also mode 600, and deletes the token. There is no second use of the token. A reboot reads the identity file and the same key, and the heartbeats continue under the same id. That is what happened at 15:20: the model was not downloaded again, and the id did not change.

A new key, without that identity file, is a new enrolment. Replacing the key on a machine that still has the old id makes later signatures fail, because the platform is holding the public key from the first enrolment. I did not rotate the key on this day. The id that came back after the reboot was the one from the afternoon.

The platform can also mark a node revoked. A revoked node is refused if it tries to enrol again. I did not revoke Node 01. The state exists so a lost or untrusted worker can be refused. It is not something this day's log exercised.

What a heartbeat is allowed to mean

Each heartbeat and each result is signed. The signature covers a timestamp and a hash of the body, using the node's key. The transport that evening was ordinary HTTP on the development LAN. There was no TLS on that API. A signature shows that the holder of the key sent the bytes. It does not show that the bytes are true, and it is not end-to-end encryption. Anyone on that LAN could read the JSON. They could not produce a fresh signature without the key.

The body is a capability document, schema krxy.capability.v1. On this agent it included the architecture, the operating system, the kernel, the CPU model string, the logical core count, system memory, a filesystem size for the root disk, the CPU temperature when the sensor returned a plausible reading, and one accelerator entry. That entry said Hailo-10H, copied the firmware string from hailortcli, and set inference to true only when that identify text contained HAILO10H. Training, validation and preprocessing were false. The note on the accelerator memory said it was catalogue capacity, not a live measurement. The 8 GB figure on the product page stayed a product-page figure.

A separate telemetry post carried the sample: CPU percent over a short interval, memory used and available, disk used and free, load, uptime, the agent version, a boot id, byte counters for the default interface, and the throttle flags. A fan speed was included only when the agent could read a hwmon tachometer. A missing fan was omitted. It was not stored as zero RPM. Accelerator temperature and utilisation were not in the runtime's report, so they were not invented.

Cooling could also be declared in the agent's environment, as present or absent. The code labels that declaration as the operator's word, and it says so in the payload: it is not a tachometer reading. Copying the agent onto another machine does not copy a fan claim, because the declaration comes from that machine's environment.

None of this is a measurement the control plane checked. The architecture note for this runtime says heartbeats and telemetry are reported claims, not verified hardware. A completed job is the node returning text. Verification was unsupported, reserved credits stayed zero, and no contribution row was written. Part 2 already recorded that for the two jobs that came back that afternoon. I am repeating it here because it is the limit on what the agent may say.

When the wrong agent was reporting

At 21:27 the systemd unit for the agent was enabled and inactive. It had been stopped at 18:08. Two processes, started at 18:16 from a detached shell, were still heartbeating. The running image identified itself as agent 0.2.0-hailo. The file on disk was already 0.3.0-hailo. The control plane stored the old version, and it stored the fan as absent, with no speed.

I had expected the unit to be the process. The dashboard had no way to distinguish "a process with this node's key is answering" from "the file I just edited is the process answering". Both conditions look like an online node. Inference was unaffected, because hailo-ollama was already under systemd and was not the process that had been left behind.

At 21:49 a restore script checked the on-disk version, signalled the detached processes, and let systemd start the unit. A further restart left a single agent in that unit, reporting 0.3.0-hailo and a fan speed near 2,970 RPM. The node id did not change. The earlier "fan absent" line was a stale process, not a missing cooler.

The first copy of the physical agent that this repository actually commits is that 0.3.0-hailo file, later the same evening. 0.2.0-hailo is the version the detached process reported. I do not have that older file as the committed source. The lesson does not need it. A healthy heartbeat proves that something is answering. It does not prove that the intended program is the thing answering.

A working node that looked offline

The same loop that posted heartbeats also ran the job. hailo-ollama was called, and the agent waited. A short prompt returns in well under a second, which is why the afternoon soak never tripped this. A longer reply does not.

The platform's offline window is 30 seconds. A generation longer than that window stopped the heartbeats, the node looked offline, and the next request came back as no_eligible_node. The Pi was busy computing. The scheduler treated it as gone. That is a bad failure mode: a working node is hidden, a retry looks justified, and the status page is describing the loop, not the accelerator.

The 23:04 change, agent 0.3.2-hailo, moved the heartbeat onto a side thread. The job loop can block inside one generation. The heartbeat thread keeps posting, then sleeping for the same five seconds. The comment in that version says a 256-token reply can take longer than the offline window, which is why the split exists.

The runbook records one check. During a 28.8-second generation the newest sample stayed within 15 seconds, and the following requests returned 200. That is one reply, on this agent, against a 30-second window. It is not a result for every prompt length, and it is not a result for a stuck runtime. If the heartbeat thread itself blocks, the node will look offline again. I had not tested that.

The same change removed a silent 64-token cap that had been applied to every Hailo call. Part 2 described the reply that ended mid-line. From 23:04 the platform stores the runtime's own stop reason, and it infers "length" only when the measured output actually meets the allowance. A missing reason is not called a clean stop. That is a reporting fix. It is not a claim that the text was correct.

What I had, and what I had not

By the end of 24 October the worker had a name that survived a reboot and a process cleanup. The agent reported a version, a Hailo identity when hailortcli said so, system memory, a root filesystem size, CPU temperature, throttle flags, and a fan speed when it could read one. It refused, in the payload, to claim training. It labelled catalogue memory as catalogue memory. Jobs for the one Hailo model id were fetched, run on loopback, and posted back. Anything that was not that model was skipped.

I did not have a check that the reported hardware was the hardware. I did not have TLS. I did not have a second physical node, and I am not going to treat the next machine as part of this story. If the Mac slept, heartbeats failed until it woke, and the agent kept retrying. Local inference did not need the Mac. The registry did. Accelerator utilisation was still absent, and an absent number stayed absent. No contribution was accounted. No training step was run.

The interesting constraint is the one the stale process made obvious. The control plane believes the key. It does not believe the file on disk, because it cannot see the file on disk. Version, fan speed and "I can run this model" are whatever the answering process puts in the JSON. The signature makes that process consistent. It does not make it current.

Where the next part starts

The following morning the gateway began to place a named model onto a node that reported it, instead of assuming the Hailo worker. Later the same day a job was bound to a digest, so a model name was no longer enough. That is the next thing worth writing down: what a node is allowed to claim it can run, once a claim can actually move a job.

Building KRXY series

Part 1 — The Beginning. What I wanted to build.

Part 2 — The first node. What the hardware actually did on 24 October.

Part 3 — The agent. This entry.

Later parts will follow the work in order. I will link them when they exist.

This part is the first day that question had an answer. The day is 24 October 2026. The record is a commissioning sheet, a node runbook, and the decisions that led up to the order. Where those records disagree with Part 1, I follow the records.

From the plan to an order

On 16 September 2026 the reference design was a Raspberry Pi 5, preferably with 16 GB of system RAM, a Raspberry Pi AI HAT+ 2, active cooling, a USB-attached SSD, and Gigabit Ethernet. The reason for USB was already fixed. The Pi 5 has one external PCIe connection. The HAT uses it, so the SSD cannot take the same connection.

The next day the purchase was staged. One physical worker, extra workers simulated, and my existing development Mac as a temporary control plane. The RAM target for that first board was cut to 8 GB, on the argument that the HAT has its own memory and that a larger board should wait for a measurement. A management Pi was left for later. Combining the coordinator and the worker on one board would have made the control plane vanish every time I rebooted the worker.

On 18 October the order went the other way on RAM. The board ordered was a Raspberry Pi 5 with 16 GB, plus the AI HAT+ 2. Storage, the enclosure, cooling and the power supply were still only planned. There was no commissioned machine that day. The repository, as recovered that morning, held the September documents and no application code. Choosing 16 GB was a procurement decision. It was not evidence that 8 GB would have failed.

What was actually on the bench

The machine that was measured on 24 October was a Raspberry Pi 5 Model B, revision 1.1. Linux reported four aarch64 processors and a MemTotal of 16,607,536 kB, about 15.8 GiB. That is how a 16 GB board usually appears. It is the system RAM. It is not the accelerator's memory.

The HAT enumerated on PCIe as a Hailo-10H. The link on that boot was 8.0 GT/s, one lane, so the single external connection was running at a PCI Express 3.0 rate. The runtime named the architecture HAILO10H and the firmware 5.1.1. Raspberry Pi's product page for the AI HAT+ 2, product code SC2166, specifies a Hailo-10H, 8 GB of memory on the HAT, and 40 TOPS of INT4 inference. I am quoting that page. The commissioning tools did not print a measurement of those 8 GB, and they did not print a TOPS figure. The 8 GB is device memory. It does not appear in the Pi's 16 GB.

The boot disk identified itself as a Samsung PM871b M.2 2280, 256 GB. That model is a SATA drive, not an NVMe drive. The kernel saw it through a Realtek RTL9210 USB bridge, at 5 Gbit/s, with the UAS driver. The root filesystem was ext4 on that disk, and the boot firmware partition was on it too. At 15:26 the filesystem reported 9.3 GB used and 216 GB free, on a volume of about 235 GB. The planning documents had preferred a 1 TB NVMe SSD. Part 1 named the enclosure as a fanxiang MD87. The commissioning sheet records the bridge chip and the Samsung model. It does not record the retail name of the enclosure, and it does not record a price. The vendor's advertised speed for that enclosure was not what the USB link negotiated.

The cooler, the supply and the board case are official Raspberry Pi parts. The Active Cooler, product code SC1148, is an aluminium heatsink and a blower that plugs into the Pi 5 fan header. Raspberry Pi rates that fan at up to 8,000 RPM. The supply is SC1149, the white UK 27 W USB-C unit, specified at 5.1 V and 5 A. The case is SC1159, red and white. A distributor order dated 28 October 2026 bought three coolers, two supplies and two cases. The unit prices, before VAT, were £3.79, £9.09 and £7.58. Shipping was not charged, and the VAT on the order was £8.94. That order is four days after these measurements, and the quantities are more than one node, so it identifies the parts bought for the machines. It is not a receipt for the single board that booted on 24 October, and the Pi and the HAT are not on it.

A fan was already present on the 24th. Once the agent that could read it was the one actually running, telemetry stored a speed near 2,970 RPM, well under the Active Cooler's stated maximum. The commissioning sheet does not name the part. Across the temperature samples that day the Pi's throttle flag stayed 0x0, including the undervoltage bit. That is a firmware report from those samples. It is not a measurement of watts drawn from SC1149. The sheet also does not say the red and white case was fitted. The HAT occupies the top of the board, and the Active Cooler is the part sold for a Pi that is not using the case's own fan.

The network fact is smaller than the plan. The plan preferred Ethernet. On this day the address was on Wi-Fi. The Ethernet interface was present and had no IPv4 address. I am leaving the addresses themselves out of a public post.

There was no recovery microSD. A failed boot disk would have meant imaging a card and moving the Pi onto it. Copies of boot files are not a disk image.

The software that made it a node

Raspberry Pi OS on the board was Debian 13 (trixie), kernel 6.18.50+rpt-rpi-2712.

The Hailo software had a fork I had to get right before anything ran. hailo-all is the older Hailo-8 stack and conflicts with the HAT+ 2 packages. The install was dkms and hailo-h10-all 5.1.1 from the Raspberry Pi archive, plus the Hailo GenAI package at the same 5.1.1 version. A newer GenAI package existed. It was left alone, because it was not the release paired with that driver. Nothing in the record says I installed the wrong stack and then recovered. The constraint was known, and the paired versions are what landed on the machine.

hailo-ollama is the local server that loads a compiled Hailo model and exposes a small HTTP API. It was bound to 127.0.0.1:8000 on the Pi, so the model service was not on the LAN. The KRXY agent, running as the user krxy, called that address. The Mac kept the control plane. The Pi did not get a copy of the Mac's SSH key, and it did not get the platform environment file. The enrolment token was removed after the first successful enrolment. The node key stayed on the Pi. Replacing that key would have been a different node.

The model was not trained for KRXY. The catalogue id was qwen2.5-instruct:1.5b: Qwen2.5-Instruct at 1.5 billion parameters, quantised Q4_0, compiled as a Hailo HEF. The pull log recorded 2,359,716,464 bytes. The installed HEF's compiled context is 2,048 tokens. The manifest's licence field was empty. I can say what was loaded. I cannot treat that file as a model this project trained.

The day, in the order it happened

The sheet does not give a delivery date between the 18 October order and this day. It does record an idle CPU temperature of 46.1°C taken before the HAT was fitted. There is no matching idle reading from later, with the HAT on and the services stopped.

By early afternoon the provisioning log and the model-pull log had been written. At 15:14 the provisioner recorded that it was finished, and the model was marked pulled. A ten-minute soak then sent 717 sequential chats to the local server. The prompt was Reply with the digit 1., with a limit of eight predicted tokens. There were no errors. Wall-clock latency around those HTTP calls was 0.688 s minimum, 0.703 s median, and 1.44 s maximum. Sampled CPU temperature was 58.7–65.3°C, the throttle flag stayed clear, and about 15.5 GiB of system memory stayed available. That median is the latency of this tiny request. It is not a tokens-per-second result, and it is not a number I would quote for a normal prompt.

At 15:20 the Pi was rebooted. hailo-ollama was back at 15:20:09 and the agent at 15:20:18. Provisioning did not run again, and the model was not downloaded again. /dev/hailo0 came up owned by root:render. The account krxy is in the render group, which is what allows it to open the device. Heartbeats resumed, and the node identity was the same one as before the reboot.

A local chat after that boot, the same short prompt, returned 1. The runtime reported two evaluated tokens, a stop reason of stop, and a total duration of about 0.66 s. The wall clock around the call was 0.705 s. During the request the server held the Hailo device open.

The control plane then assigned work. One job completed with the marker KRXY-HAIL-OK. A second, after the reboot, completed with KRXY-BOOT-OK. The second reported about 14.3 s of wall time and about 1.3 s in the accelerator field. The wall time includes the model load after boot. Both results were stored as completed. Verification was unsupported, reserved credits were zero, and no contribution row was written. Completed means the node returned a result. It does not mean a validator accepted it.

Later that afternoon a browser session on the Mac asked the node, through KRXY, what three plus three is. The reply that came back was "The answer to your question is 6." The prompt had also asked the model to mention a marker. The stored reply did not contain it.

What went wrong on the same day

The first was a permission inherited from the GenAI package. Its install makes the model directory world-writable so a pull can store files. After the reboot, hailo-ollama logged that it could not create hailort.log, and it kept serving requests anyway. That evening the directory was set to mode 0750, owned by krxy. The log warning did not return. A world-writable model store on a machine that accepts other people's prompts is a bad default, even on a lab bench.

The second was which process was the agent. At 21:27 the systemd unit was enabled and inactive. It had been stopped at 18:08. Two processes started at 18:16 by a detached shell were still heartbeating. The running image was agent 0.2.0-hailo. The file on disk was already 0.3.0-hailo. The control plane stored the old version, and it stored the fan as absent, with no speed. Inference itself was fine: hailo-ollama was the process systemd was already supervising. At 21:49 a restore script checked the on-disk version, signalled the detached processes, and let systemd start the unit. A further restart left a single agent in that unit, reporting 0.3.0-hailo and a fan speed near 2,970 RPM. The node identity did not change. The earlier "fan absent" line was a stale process, not a missing cooler.

The third showed up when the reply was longer than the soak prompt. The agent in use that afternoon limited output to 64 tokens. A longer generation the same night stopped on that allowance and ended in the middle of a line. At 23:04 that silent cap was removed. From then on, the platform recorded the server's own stop reason. A missing reason is treated as "length" only when the measured output actually meets the allowance. The same evening, a generation longer than about 30 seconds had also stopped the heartbeats, so the node looked offline while it was working. The following agent build heartbeats on a side thread. During a 28.8 s reply the newest sample stayed inside 15 seconds.

None of these were theoretical. Each one left the machine looking healthier in a status page than it was.

What that day proved

By the end of 24 October I had one physical worker with a recorded identity.

• It booted from the USB disk, with the Hailo driver and both services coming back after a reboot.
• The accelerator was visible as /dev/hailo0, firmware 5.1.1, and a local server answered the selected HEF.
• The Mac could assign a job, the Pi could return it, and the same identity survived the reboot.
• A short arithmetic question asked in the browser came back answered.
• CPU temperature under the tiny-prompt soak stayed between 58.7°C and 65.3°C, with no throttle flag. About 15.5 GiB of system RAM stayed free, which fits a model that lives on the accelerator.
• The fan, once the current agent was the one reporting it, was spinning.

That is an initial node. It is one machine, on one LAN, with the coordinator on a computer that has to stay awake. If the Mac sleeps, heartbeats fail until it returns, and the agent keeps retrying. Local inference on the Pi does not depend on that Mac. The registry does.

Accelerator temperature and utilisation were not available from this runtime, so they were omitted rather than stored as zero. SMART data for the boot disk was blank. TRIM had not been verified. There is still no recovery card. Output was not verified by anything other than "the node sent text". No contribution was accounted. No training step was run. The long-term question in Part 1, whether learning can scale across unreliable edge devices, was not touched.

A 1.5 billion parameter instruct model, compiled by someone else and loaded as a HEF, is a useful first workload. It is not a KRXY model.

What I would not conclude

The 16 GB board did not prove that 8 GB was too small. This model did not sit in system RAM. The next workload might.

The SATA disk behind a 5 Gbit/s bridge did not prove anything about NVMe. The plan still describes NVMe, because that was the storage I thought I was buying. The disk that booted is the one in the table above.

The 0.7 s median is not a benchmark against any other runtime. A later comparison would need the same prompt, the same token allowance, and a record of what was measured. I did not have that on this day.

Ethernet remains the intended baseline. It was not the path this node used.

Where Part 3 starts

The interesting residue of the day is the agent. A file on disk, a systemd unit, and a process started from a shell had been three different things, and only one of them was talking to the control plane. The node identity survived that. The next entry, when I write it, stays with that agent: what it reports, what it is allowed to claim, and how a worker keeps one name across a restart.

Building KRXY series

Part 1 — The Beginning. What I wanted to build.

Part 2 — The first node. What I actually started building. This entry.

Later parts will follow the work in order. I will link them when they exist.

KRXY started with a fairly ordinary question.

What if ordinary people could contribute small amounts of AI computing from inexpensive hardware, in much the same way that other decentralised networks are built from independently operated nodes?

Cloud AI is powerful, and it is highly centralised. The capacity sits in large facilities, and the cost of using it follows that concentration. My first thought was smaller than inventing a new kind of model. I wanted to know whether small devices, particularly Raspberry Pis together with the AI accelerators then appearing as add-on boards, could be useful building blocks. One Pi would never resemble a datacentre GPU. A network of independently operated machines might still accept a job, run it locally, and return a result.

A harder question grew out of that one, and its status has to stay clear. It was a long-term research objective. It was not something KRXY had done.

Could a future AI architecture be designed so that learning itself could scale across unreliable, heterogeneous edge devices, rather than requiring a permanent centralised high-bandwidth training cluster?

At the beginning, the prior question was whether a managed edge node could work at all. I had an answer to neither. What follows is the plan.

What KRXY was meant to be

KRXY is an experimental decentralised AI platform and a research project. The platform was the part that might eventually be operated. The research had to remain free to fail.

The picture at the planning stage was a set of independently operated edge nodes connecting to a coordination platform. Depending on the machine, a node might offer inference, CPU compute, accelerator compute, space for models, a statement of its availability, and hardware telemetry. Later, the same node might take part in distributed learning experiments. The platform was supposed to ask what a machine could do, and then trust the answer only as far as it could check it.

The topology is familiar from other community-operated infrastructure:

Provider hardware → secure outbound connection → coordination layer → scheduler → workload → result

Each node would keep its own memory, its own storage and its own failure domain. I was proposing a way to place independent jobs. I was not proposing that many small computers would merge into one larger computer, or share a single pool of memory. Links between ordinary houses are too slow, too uneven and too fragile for that, and the software problem would have been a different project.

That became one of the original architectural assumptions. For consumer internet connections, distributing independent jobs is much more realistic than splitting every layer of one large model across the public internet. Latency, bandwidth, reliability and hardware differences make fine-grained distributed inference difficult. A system that accepts a request, chooses a node which already holds a suitable model, runs it there, and returns the result is a coherent first system. Shuttling the activations of a single forward pass across home broadband is a deeper problem. I intended to earn the first before speaking as though I had the second.

Why a Raspberry Pi

The Pi interested me because someone could actually buy one and leave it running.

Next to a server or a GPU it is inexpensive, widely available, and modest in power. It has a mature Linux ecosystem, Ethernet and Wi-Fi, USB for expansion, and, on the Pi 5, a PCIe link for a faster peripheral. NVMe storage is a practical option. There is a large community that already knows how to install an operating system on a small board and keep it alive. The CPU has become useful in its own right: the Pi 5 is a 2.4 GHz quad-core 64-bit Arm Cortex-A76, still a small processor, and a long way from the Pis of a decade earlier. Dedicated accelerators were arriving as HATs, so "edge AI" was beginning to mean a particular board rather than a slogan.

The limits belonged in the same plan. A Pi has far less memory than a GPU server, and less memory bandwidth. Left to work in a warm corner, it will throttle. A great deal of AI software assumes x86 or CUDA, and the Arm path is often missing or awkward. Accelerators want their own model formats and their own compilers. The models that fit are small beside datacentre models. A home connection varies from hour to hour. And a node can vanish without warning, because it was unplugged, rebooted, powered off, or because the network in that house changed.

I treated that last limit as part of the research problem. A design which works only when every participant stays online, on a quiet link, with identical hardware, has assumed away the reason for investigating KRXY.

The first proposed node

The machines are Raspberry Pi 5 boards with 16 GB of RAM. That is the host for every node, not a special case for the first one. Storage is the same idea on each of them: a 256 GB NVMe SSD in a USB enclosure, so the drive is not sitting on the Pi's PCIe connector. The AI accelerator is the part that is not shared. One Pi carries a Raspberry Pi SC2166 Raspberry Pi AI HAT+ 2. The others are the same 16 GB board and the same sort of USB NVMe storage, without that HAT.

Heterogeneous hardware was one of the objectives, so this was a feature of the setup rather than a missing part. A node with no accelerator still had to be a normal participant. Work was supposed to be scheduled from a capability report. The protocol had to cope with one machine that could offer the Hailo device and with others that could not.

The enclosure is a fanxiang MD87. The listing describes it as a USB 3.2 Gen 2 case, stated at up to 10 Gbps, for an M.2 NVMe or SATA SSD. I am quoting that as the product description. It is not a speed I had measured on a KRXY node, and a 256 GB drive does not become faster because the case claims support for larger ones.

The HAT figures are Raspberry Pi's own specification: a Hailo-10H, 40 TOPS of INT4 inference, and 8 GB of memory on the HAT. Raspberry Pi's documentation also describes that memory as support for local language and vision-language models, in their words up to around six billion parameters. I kept that as a vendor description of the one board fitted with the HAT. It was not a KRXY measurement, and the early experiments were meant to stay well below that ceiling.

On the Pi that has the HAT, the reason for USB storage is physical. The Pi 5 has a single PCIe 2.0 x1 link, and the AI HAT+ 2 uses it, so that machine cannot also boot from an M.2 HAT on the same connector. The other Pis use the same enclosure and the same 256 GB NVMe size, which keeps the storage path comparable. A microSD card was only a recovery tool. An earlier pass of the plan had considered an 8 GB Pi, then the machines actually used were the 16 GB models. Choosing 16 GB was not evidence that 8 GB would have failed. I had not yet measured what a KRXY workload would draw, or how hot it would get. The supply Raspberry Pi recommends for the Pi 5 is the 27 W USB-C unit, specified at 5.1 V and 5 A.

The yellow area is what every Pi has. The node agent is the only part that faces the network, and it opens the session outbound. The coordination layer sits above that area because it is not part of the Pi. The green box is the accelerator, a Raspberry Pi SC2166 Raspberry Pi AI HAT+ 2, and it is fitted to one Pi only.

The network I imagined

Around that node I sketched a coordination platform. Putting a name on a box meant I expected to need the software. It did not mean the software existed.

As a plan, the platform included a node agent, a node registry, authentication and identity, capability reporting, heartbeats and health, a model registry, a scheduler, a job queue, routing, verification of results, some notion of reputation, and contribution accounting. If use ever had to be recognised fairly, credits or billing could follow. Applications would call a gateway. I also wanted enough observability to see a failure, rather than guess at one from silence.

Security was a requirement in the first sketch, not a layer to add once a demonstration worked. Nodes on ordinary connections should make outbound encrypted connections. The design did not ask home users to expose a service to the internet. Consumer routers sit behind NAT and firewalls. A node that has to be a public server is awkward to run, and it is a larger attack surface than I wanted to hand to participants.

Users, node operators and the coordination layer were eventually to be treated as mutually untrusted, wherever that could be made practical. A prompt is user data. A node is someone else's computer. Joining the network is weak evidence that the machine is safe. Deciding where a job runs is a placement decision, and it is not, by itself, a privacy guarantee. Reputation might one day change how much work a node is offered. It would still have to sit beside a check on the work itself, and at this stage I did not know how to design that check.

The scheduler was supposed to choose. A request should go to a node that holds the model, has hardware that fits, has a healthy recent heartbeat, has spare capacity, and has declared a matching capability. Machines that fail those checks should be left alone. That was a design intention. The awkward cases were still ahead: a node that reports a model it cannot run, a heartbeat that is dishonest, two eligible nodes whose speeds are nothing like each other.

The first realistic objective

Before any attempt at decentralised learning, the project needed a much smaller milestone. Could several independent edge nodes behave like a managed compute network?

1. A node can register itself.
2. It can report its capabilities.
3. It can receive a workload chosen for it.
4. It can run an AI job locally.
5. It can return a result.
6. It can disconnect or restart without taking the rest of the system with it.
7. The coordination side can see its health.
8. Nodes with different hardware can coexist, and can be told apart.

That much would already be useful. Several people could operate nodes, and a scheduler could place independent requests on the machines fit to run them. The value would be ordinary and specific: identity, placement, failure, and a result that comes back. It would not yet be a claim about a new learning algorithm.

This was a target sequence for distributed inference. I had not run it.

I was also unwilling to buy a cluster merely to resemble the drawing. The first physical lab would be a single worker. My existing development computer would be the temporary control plane: registry, scheduler, a place to watch the node, and simulated extra workers. A separate always-on control computer could wait until coordination needed to stay up while the worker rebooted. Further physical workers could wait until an experiment needed real timing, real failure, or a hardware difference that simulation could no longer stand in for. The control plane coordinates. It is not the source of the intelligence, and a worker was supposed to keep some useful local operation if that coordination disappeared.

The larger research question

A network that only ran inference would have been worth building. It was not the whole ambition.

The long-term question became:

Can an AI system be designed whose learning process naturally tolerates unreliable, heterogeneous edge nodes?

Traditional distributed training usually assumes a world KRXY could not safely assume: powerful GPUs, fast interconnects, predictable machines, low latency, workers that stay online, and enough shared bandwidth to keep them in step. The machines I was contemplating had a different character. Raspberry Pis. CPUs, and sometimes different accelerators. Unequal amounts of RAM. Domestic broadband. Latency that moves. Nodes that join and leave. Jobs that stop halfway through. Performance that simply differs from one house to the next.

I wanted to investigate whether the learning mechanism itself could be designed around those conditions, rather than treating them as defects to clear away before serious work could start. I did not know how to do it. Writing the question down opened a research programme. It did not supply a method.

These were the questions I expected to meet. I also expected some of the honest answers to be "this split is too expensive" or "this update should have been refused":

• How should training work be divided across machines this small?
• How much communication can be tolerated before the learning is worth less than the traffic?
• What happens when a worker disappears mid-task?
• Can partial work be recovered, or does it have to be discarded?
• How can a contribution be verified?
• How can a bad update, or a malicious one, be limited?
• How can uneven hardware contribute fairly, without the whole system being held to the weakest node, and without a weak node being credited for the work of a strong one?
• Can learning continue asynchronously, when nodes are out of step?
• How much bandwidth does a useful update actually require?
• Can a model improve without every node being synchronised all the time?
• When is central coordination still necessary?
• Which workloads belong at the edge, and which do not?

A rule about the models sat beside those questions, because it changes what I was willing to call a result. Primary KRXY research models were to start from random initialisation. They were not to be fine-tunes or distillations of an existing foundation model. Tools, operating systems, libraries and published methods could be used. The weights the project would treat as its own needed a lineage that could be written down: architecture, data, code and checkpoints. Early models would be modest on purpose. There would be no large pretrained model waiting offstage if the small experiments disappointed.

The accelerator raised a narrower question, and it applied to one machine. I planned to treat the Hailo-10H first as an inference device on the single Pi fitted with the HAT. Whether it could contribute to learning was untested. The other Pis had no such device, so anything they ran had to run on the CPU. My working hypothesis for the accelerated node was that the HAT might still help with forward evaluation, validation, scoring or specialist jobs, while a training step that needed a different kind of computation stayed on the CPU. That was a hypothesis about a division of labour. It was not a finding.

The first bar I set for the learning work was small, and it was simulated. I wanted a small model, started from random weights, trained across simulated nodes that differed in speed, bandwidth, reliability and availability, with a centralised run kept beside it as the baseline. The demonstration I was aiming at was specific: about eight simulated nodes, at least three speed profiles, artificial limits on latency and bandwidth, at least one node leaving and one later rejoining, and training that continued without a global restart. Loss against the baseline, network traffic, and the effect of stale updates would all be recorded. None of that had been run. It was the definition of an early result I would be willing to take seriously.

The roadmap I thought the work required

I did not believe that question could be approached in one jump. The order below is the initial roadmap, the sequence I thought the risk demanded. It is not a diary of work already done.

Phase 1 was one physical node: it boots, it has an identity, it reports its hardware, and an agent can ask it to run a local job. Any further participants would mostly be simulated.

Phase 2 was the smallest network behaviour worth having. A gateway accepts a request, and a node returns a result. One machine is enough to expose most of the mistakes in a protocol.

Phase 3 was more than one participant. Simulated nodes could come first. A second physical node would be justified when an experiment needed real time, real failure, or a real difference in hardware.

Phase 4 was scheduling and recovery. The system should choose a node for a reason. A node that vanishes should cost that job, not the platform.

Phase 5 was measurement. Before any claim about quality, or about what an accelerator was worth, I needed a comparison of what a node actually produced on the hardware it actually had.

Phase 6 was the learning experiments: asynchronous updates, stale work, nodes that come and go, and a centralised baseline. This phase was allowed to fail. A negative result would still be a result.

Phase 7 was the long horizon. Could coordination become helpful rather than essential? Could more than one place aggregate updates? Could nodes exchange anything useful with less reliance on a permanent central training cluster? I filed that as a direction to earn, one experiment at a time.

What I would have counted as success

Success, at the start, was not "build an AI better than the large commercial labs". That would have been a different project, and it would have invited claims this hardware could not support.

Early success was plain. One Pi can take part reliably as a managed node. Models can be found and executed. A workload can be assigned from outside the machine. Health can be observed. When the node restarts, the platform notices.

Intermediate success would mean that different nodes can coexist, and that the scheduler picks a suitable one. A failed node leaves unrelated work standing. Several jobs can be in flight. Contribution can be counted in specific units: work done, failures, bytes moved, and whether the result passed a check.

Research success was still a hypothesis about evidence I did not have. Learning work survives a worker being interrupted. Contributions can be verified rather than trusted. Adding an edge node produces a benefit I can measure. Useful learning still happens on a real home network, with limited upload and machines that are sometimes absent.

The long-term criterion remained a question. I wanted to investigate whether an AI architecture can scale its learning across a large population of heterogeneous edge devices, with less dependence on a training cluster that has to stay permanently in the middle. That is the reason for the project. It is not a route I had already found.

What could go wrong

I expected much of this to go wrong. A series that remembers only the path which happened to succeed would be a brochure.

Models and software were the first hazard. A model that runs on a workstation may have no usable Arm build. An accelerator model is often a compiled artefact, tied to one toolchain and one version, rather than a file that can be copied unchanged between frameworks. I expected the Hailo tools to accept a narrower set of models than a general CPU runtime, and I expected to meet that limit as a compiler error. Licences constrain which weights may be placed on other people's machines. Training research models from random initialisation also means the project cannot borrow a huge pretrained model to cover a disappointing experiment.

The machine itself has three different ceilings: system RAM, accelerator memory and SSD space. A model can miss any one of them. Heat will throttle a board that looked fine for five minutes. A weak power supply will produce faults that look like software. If the work is ever compared per watt, power becomes a research measurement, and I did not yet have the readings.

The network is the next constraint. Home connections are asymmetric, buffered and sometimes metered. NAT and firewalls remove any design that needs strangers to connect inbound. A bandwidth budget that is comfortable for a short reply can be absurd for a checkpoint. Nodes will disappear, and the protocol has to treat that as ordinary.

Trust is a separate problem from packet loss. An operator can be careless or hostile. A result can be wrong, empty, or written to look right. A prompt sent to someone else's Pi has left the user's machine; a small computer is not automatically a private one. Signed identity, isolation, retention, and a refusal to treat "joined" as "trusted" were all open. I could name them. I could not yet specify them tightly enough to build.

Learning has its own ways to fail. The machines can spend more time communicating than computing, especially when they are small and the uplink is slow. An update from a slow node arrives late, and a late update can be useless or harmful. Some workloads do not belong on a Pi. Scheduling becomes a different problem as soon as the nodes differ. Underneath those technical questions is a plain one about cost: even a mechanism that works still has to be worth the electricity, the hardware and the effort of coordinating it. I did not assume that it would be.

Why I am writing it down in public

I want a record that is awkward to tidy up after the fact. This series should show what was proposed, what was built, what worked, what failed, what was measured, which assumptions were wrong, and where the architecture changed. Where an observation can be repeated, I want enough detail for someone else to try.

I am not expecting every experiment to support the original hypothesis. A negative result is still an engineering result. If jobs can be placed across a few small machines and the learning question then turns out to have the wrong shape, that is worth knowing. If an accelerator is excellent for one family of models and useless for another, that is worth knowing too. The failure I most want to avoid is a write-up that keeps the diagram and loses the measurement.

Where Part 2 starts

The planning diagrams were the easy part. They did not get hot, they did not undervolt, and they did not drop the network halfway through a job.

The next question was whether a Raspberry Pi 5 and an AI accelerator could become a reliable KRXY node, rather than another drawing that never survived contact with the hardware.

Part 2 will begin with building the first node.

Building KRXY series

Part 1 — The Beginning: Designing a Decentralised Edge AI Network

Later entries will follow the project chronologically. I will link them when they exist. As the work reaches them, I expect the series to cover the first node, the Raspberry Pi 5 and the Hailo-10H, the node agent, a control centre, connecting workloads, CPU inference beside the accelerator, model libraries, evaluating quality, a second node, scheduling and resilience, distributed-learning experiments, the lessons inside the failures, and what it takes to go beyond two machines.

Those are intentions for the series. They are not results.