Back to Blog

AI

My AI Got Stuck in Line Behind Itself

By Khai Tran · · 1 min read

On Wednesday, several copies of Red were working at once and all of them needed the same spot in line. It took three fixes, and part of the problem wasn't the line at all.

Featured image for My AI Got Stuck in Line Behind Itself

Week ending September 25.

On Wednesday, my AI got stuck in line behind itself.

I usually have several Red sessions working at once now. When a session finishes something, it sends the work up to a shared copy. This week, only one session could do that at a time.

On Wednesday there were five to eight sessions, and each one held that spot for seven to twelve minutes while it ran the full test suite. The scheduled push that keeps the shared copy current would wait ten minutes, give up, and try again later. For a couple of hours, it gave up every time.

The first fix was a real line: first come, first served, with a ticket for each session. That evening we found a gap. If the session at the front got paused, everyone behind it would freeze, so every place in line now comes with a deadline. Later that night, the main line got priority over side work, and side branches stopped running the full suite before sending, since they get checked again later anyway.

Some of the failures that evening came from a separate problem. The computer was overloaded, with as many as seven test runs going at once.

So now, when a test run times out, the system has to say why: a budget that's too small, a machine that's too busy, or neither.