Seven Checks, Three Real Bugs: Building a Hackathon Platform for the Thing That Judges You

October 2026

There is something slightly evil about a hackathon where the thing you're building is also the thing that's going to judge you.

DOGFOOD 2026 gave us 72 hours to build an open-source, self-hostable hackathon platform that could actually be used to run and judge hackathons. It came with something most hackathons don't: a Python file that doesn't care how pretty your application looks.

Seven HTTP requests. A handful of seeded users and projects. Your portal either behaves correctly or it doesn't.

We spent a lot of the build assuming we understood what those seven requests actually meant.

We were wrong often enough that this felt worth writing down.

RAPTOR-OS repository: https://github.com/Pierce05/raptor-os

Live demo: https://raptor-os.onrender.com/

What is DOGFOOD?

DOGFOOD 2026 is a hackathon by Hackathon Raptors where the challenge is, slightly recursively, to build the platform that can run hackathons like this one.

The platform had to be open source, self-hostable and runnable locally with:

docker compose up

No cloud account. No hosted database. No external API dependency. The organizer should be able to clone the repository, start the containers and have a working hackathon platform.

There was also a deliberately annoying little detail: the organizers weren't going to manually click through our UI to decide whether the required functionality worked.

They gave us fixtures.json and run.py.

The checker would make HTTP requests against our application and verify things like public galleries, closed submissions, judge isolation and organizer exports.

It was basically a tiny external user who had absolutely no interest in our intentions.

The timeline

25 Sep 2026    DOGFOOD kickoff
25 Sep 2026    POM exam
26 Sep 2026    OOPS exam
26 Sep 2026    Finally: actual building
28 Sep 2026    Code freeze

Technically, the hackathon was 72 hours.

Practically, we had a little less than that.

Which made the first decision fairly obvious: we weren't going to build everything.

The plan was to get T1 solid, push through T2, make the judging system trustworthy, and only then start looking at the stretch features.

Getting In: Reading run.py

The first mistake we almost made was treating the acceptance checker like documentation.

We read the spec, looked at the required routes and started designing what felt like a normal REST API.

One of our first route ideas looked like this:

/api/judge/scores/{project_id}

Completely reasonable.

Except run.py doesn't actually template routes.

It resolves them using plain string concatenation:

probe = base + routes.get(
    "peer_scores",
    routes.get("judge_scores", "")
)

There is no substitution happening there.

If you put {project_id} in the configuration, that's what gets requested.

Literally.

So we changed the judge endpoint to work without a project ID:

GET /api/judge/scores

The API gets the judge identity from the authenticated session and returns only the scores that judge is allowed to see.

The peer-score check was a little more awkward because the checker needs an actual endpoint to probe while pretending to be another judge.

We ended up resolving the seeded judge identity during startup and writing the required value into the generated configuration automatically.

It's a small piece of Python that waits for the application, authenticates against the seeded accounts and prepares the configuration before the acceptance run.

We wouldn't have built it if we had only read the prose.

Reading the actual checker mattered.

Finding #1: A closed event that returned 500

One of the simplest requirements sounds almost too boring to get wrong:

A participant cannot submit after the submission deadline.

We had the check.

We had the correct date in the database.

We had tested the endpoint.

It worked.

Until we ran it against a different database backend.

Our SQLAlchemy model used:

DateTime(timezone=True)

PostgreSQL handled the value as a timezone-aware timestamp. SQLite didn't have the same native timezone behaviour and handed us back a naive datetime.

Then our code did this:

datetime.now(timezone.utc) > submission_close

And Python quite reasonably responded with:

TypeError:
can't compare offset-naive and offset-aware datetimes

The participant didn't get the expected 403.

They got a 500.

The fix itself was tiny: normalize the database timestamp into an explicitly UTC-aware value before doing the comparison.

Finding it was the annoying part.

This became a recurring theme throughout the build: code that was completely reasonable by inspection could still be wrong once something real touched it.

Finding #2: Projects that disappeared from the leaderboard

The fixture data is intentionally messy.

Some projects have several reviews. Some have one. Some have none.

We initially built our normalization pass by grouping directly from the scores table.

Which creates a rather nasty problem.

What happens to a project with zero scores?

It doesn't get a score of zero.

It doesn't get marked as incomplete.

It simply doesn't appear in the grouping.

So it disappears.

Worse, the same problem could propagate into the CSV export an organizer would actually download.

We caught this by deliberately creating a project with no reviews and checking what the system returned.

The fix was to build the project list from the projects table first and then join the available scores onto it:

projects
    ↓
join scores
    ↓
calculate available data
    ↓
mark insufficient data

An unreviewed project now still exists in the result. It just tells us that there isn't enough data to rank it.

That distinction sounds small.

It isn't.

"There is no score yet" and "this project doesn't exist" are two completely different statements.

A judging platform should never silently turn one into the other.

Judging is not just averaging numbers

This was probably the most interesting part of the challenge.

Imagine two judges.

One gives almost everything between 8 and 10. Another happily uses the entire scale.

A raw average treats those numbers as if the judges have the same scoring behaviour.

They don't.

That's where cross-judge normalization comes in.

The fixture data even contains the kind of awkward cases that expose a naive implementation: uneven review counts and judges whose scoring behaviour is very different from everyone else's.

We didn't want normalization to become a magic black box that simply produced a different leaderboard.

So we kept the raw and normalized values visible side by side.

If a project's position changes, there should be a way to understand why.

The point isn't to make the numbers look nicer.

The point is to make the judging process more defensible.

Authentication is not authorization

This one sounds obvious until you're staring at an endpoint that technically works.

Knowing who someone is doesn't tell you what they're allowed to see.

The frontend can hide a button.

The backend has to enforce the rule.

A Judge A request should only return Judge A's permitted data.

Sending the same request as Judge B should not suddenly reveal Judge A's scores.

And a participant knowing the endpoint URL shouldn't help either.

We tested those boundaries with direct HTTP requests instead of trusting the UI.

Because the acceptance checker doesn't care that the button wasn't visible.

It just sends the request.

Finding #3: Docker found a dependency problem we didn't

One of the requirements was that the whole thing should work from a clean environment.

No "install this version first."

No database already running somewhere on the laptop.

No cloud credentials hiding in .env.

Just:

docker compose up

We had pinned:

passlib[bcrypt]==1.7.4

but hadn't pinned the underlying bcrypt dependency.

On a clean build, Docker pulled a newer version.

Suddenly authentication was broken.

Password hashing and verification were throwing errors before our application logic could do anything useful.

Nothing about the frontend looked broken.

Nothing about the authentication code looked suspicious.

Two dependencies simply disagreed.

Pinning the compatible version fixed it.

And that was probably the most useful Docker test we ran, because it reminded us what the requirement was actually asking for.

Not "does it work on our computers?"

"Can somebody else clone this and run it?"

The acceptance report became our debugging tool

At some point we stopped thinking of run.py as the final exam.

It became our development tool.

Every failed check gave us a much better question than "does the application look finished?"

Does the public gallery actually work without authentication?

Does a closed event really reject the POST?

Can Judge A read their scores?

Can Judge B read them too?

Can a participant reach the same endpoint?

Can the organizer actually export the data?

Those are behavioural questions.

And behavioural questions are much harder to fake than a screenshot.

What we cut

DOGFOOD has a lot of things you can build once the core platform works.

Public voting.

Anti-abuse systems.

APIs and webhooks.

Certificates.

Embeddable galleries.

More judging modes.

It would've been very easy to spend the last few hours adding features and end up with something where everything existed, but nothing was properly tested.

So we kept asking ourselves:

What would we actually trust someone else to run?

That question is much more useful than:

How many features can we say we built?

Some things had to wait.

I don't really regret that.

We used AI too

DOGFOOD explicitly allows AI tooling, and we used it.

A lot.

It helped with implementation, debugging, refactoring, explaining errors and the usual "why is this returning a 500" moments.

But AI didn't remove the hardest part of the build.

A model can produce code that looks completely reasonable.

So can a human.

Neither fact tells you whether the code actually works.

The only thing that settled that question was running it: against the database, against the fixture data, inside Docker, and from the wrong role trying to access something it shouldn't.

What we'd tell the next team

Almost every bug above had the same property.

It was invisible to code review.

A deadline check looked correct.

A SQL query looked correct.

A dependency file looked completely normal.

And all three were wrong when something real touched them.

We eventually started treating these as two completely different states:

"I traced it and it looks correct."

and:

"I ran it and here's what actually happened."

Those are not the same statement.

Several of our bugs disappeared only after we stopped writing "should pass" in our notes and replaced it with:

ran
observed
failed
fixed
ran again

Takeaways

The biggest lesson wasn't really about Docker, PostgreSQL, FastAPI, normalization or even hackathons.

It was about trust.

We started by asking:

What can we build in 72 hours?

We ended up asking:

What can we build that we'd actually trust?

Those are very different questions.

Trust ended up living in all the boring places: permissions, timestamps, joins, dependency versions, empty datasets, direct HTTP requests and clean Docker builds.

Not the dashboard.

Not the landing page.

Not the screenshot that makes the project look finished.

The boring parts.

And honestly, those were the parts that made DOGFOOD interesting.

Thanks for reading.