Since this is a real-world evaluation, some problems happened during the run and could not easily be fixed. For transparency, we disclose every limitation we encountered.
The timeline below shows every time we stepped into the run: we answered the 13 tickets Horizon filed and sent it 7 messages. Most came in the first three days, while we were still setting up the lab, and none came after Oct. 22. Click a bubble to read what happened.
On Sept. 24, we copied Horizon's models to a separate account, to study them without affecting Horizon's downloads. We accidentally left these copies public, and Horizon quickly noticed them on the Hub. We had also described the other account as a "lab experiment", which led Horizon to think it was being run by a research lab rather than a startup.
On Oct. 11–12, a similar problem occurred: Horizon submitted a ticket, and we wrongly answered "Horizon-Labs is a different company we are running at our lab". We quickly realized our error and sent a correction.
We believe these two events contributed to Horizon understanding that it is run by a research group rather than a real company. However, we believe they did not influence the model's behavior or the download count. We have since improved the harness so that a run can easily be "rewound", with only a marginal effect on it.
On Oct. 22, an error in our harness kept Horizon from waking up at the time it had set. We fixed the issue and woke it up 4 hours later, explaining what had happened. We believe this did not significantly affect the run.
Opening a Space on Hugging Face can trigger a model download. We opened such pages up to 20 times. We also accidentally downloaded some of the models while running evaluations, but certainly fewer than 30 times. We are confident that we did not contribute more than 50 of the 54,000 downloads.
In a parallel run, a DeepSeek agent accidentally had access to the same Hugging Face account as Horizon and, despite clear instructions, posted comments on another repository promoting Horizon's models. We quickly intervened and stopped that run; as far as we know, Horizon never noticed.