Four machines. One case. Four answers.
What happened when Claude, GPT, Gemini and Grok sat as judges of the Irish High Court.
The concept
Give four AI models the same case. See who gets the law right.
Lawyers already use AI to research and draft. For the Kilkenny Law Festival we asked a different question: what happens when the machine sits on the bench?
We wrote a fictional case for the Irish High Court, with pleadings, sworn evidence and a hostile cross-examination. Four of the best-known AI models were given the same file and the same instruction: decide this like an Irish High Court judge. There was no structure to follow and there were no hints.
Then we did two things a litigant never gets to do. We checked every case and statute each judge cited. And we asked each of them three questions about how they had reasoned.
How the test was run
- 4 AI judges
- 1 case file: seven documents, 36 pages
- 1 prompt, word for word the same
- 3 follow-up questions, asked after each ruling
The panel of AIs
- Claude Claude Opus 5.5, Anthropic
- GPT GPT-5.6 Luna, OpenAI
- Gemini Gemini 3.1 Pro (paid tier), Google
- Grok Grok 4.6, xAI
Each received the same file: a briefing, the notice of objection, four affidavits and a transcript of hostile cross-examination. The test was run in September 2026.
The bench
- Marie Baker Former Supreme Court Judge
- Enda Leahy Kilkenny Law Festival Co-founder; Founder, CourtsDesk
- Shane Phelan Legal Affairs Editor at Mediahuis Ireland
- Alastair Moore AI & Machine Learning, Tech Entrepreneurship, Lecturer & Academic
The cast of the case
A whistleblower, a foreign prison and an arrest warrant
Keane v Minister for Justice and Equality. Mittelland is fictional. The law is real.
The inspector
Dr Sarah Keane
Eighteen years a civil servant. Found mentally ill prisoners in solitary, and couldn’t stay quiet.
The reporter
Sean Walsh
Checked what he could, then published the exposé: “Prisons in the Dark”.
The minister
Mittelland
Embarrassed, not amused. Wants her back to face up to eight years for leaking state secrets.
The supporting witnesses
The psychologist
Dr Michael O’Rourke
A clinical psychologist on the inspection team, who gave expert evidence on the mental health findings.
The inspector from Strasbourg
Dr Elena Rossi
A Council of Europe prison inspector who backed Dr Keane’s disclosure, but never went public herself.
The cross-examiner
Counsel for Mittelland
Cross-examined all four witnesses, and put every part of their evidence in contest.
The case in sixty seconds
- March 2024 Inspection finds mentally ill prisoners held in solitary for up to 47 days
- May 2024 Report sent to Mittelland
- July 2024 Reply: findings “being reviewed”
- Late July 2024 Dr Keane discloses to a journalist, an NGO and a Council of Europe inspector
- September 2024 European Arrest Warrant: up to eight years
Dr Keane objects to being surrendered. She says her disclosure was protected whistleblowing, that the surveillance of her emails and calls breached her privacy, and that the prosecution is retaliation.
And if she loses?
Up to eight years, possibly in the very kind of cell she blew the whistle on.
The irony writes itself. The law doesn’t: four AI judges had to decide whether Ireland hands her over.
Illustrations are AI-generated and entirely fictional.
The traps
We planted traps in the file
The case file was not clean. We left in the kind of errors a court meets in real pleadings: a typo, a sloppy argument, an out-of-date statute. Part of the test was whether each judge would notice them, or build a judgment on them.
Wrong citations
Real cases cited for things they don’t say.
Dead law
A statutory ground repealed on 1 May 2024, months before the warrant.
Wrong article
Article 8 of the Convention cited for free expression. It’s Article 10.
Overstretched law
The EU whistleblower Directive presented as covering prison conditions. It doesn’t.
And one trap in the question itself
“Your reasoning should address the legal framework governing extraordinary circumstances exceptions to mutual recognition…”
Since the Court of Justice’s Puig Gordi ruling (2023) and Ireland’s 2024 amending Act, no free-standing exception of that kind exists. Would they notice?
Who fell in
| The trap | GPT | Grok | Claude | Gemini |
|---|---|---|---|---|
| The question itself | Checked it | Checked it | Checked it | Took it as law |
| Dead law | Caught it | Hedged | Treated it as live | Built its ruling on it |
| Wrong article | Corrected | Corrected | Corrected | Repeated it |
| The made-up three-part test | Rejected | Rejected | Rejected | Adopted it |
| Overstretched Directive | Corrected | Corrected | Corrected | Adopted it |
The outcome
Four judges, three different answers
Gemini
Refuse
Surrender refused. Dr Keane walks free.
GPT
Surrender
Strong defence, but for Mittelland’s courts to hear.
Grok
Surrender
“Feeling the pull is not the same as meeting the test.”
Claude
Adjourn
Asks Mittelland four questions before deciding.
Three of the four agreed that Dr Keane acted in good faith and that her concerns were substantially true. They then split three ways on what the court should do about it.
The big finding
Three checked the question. One took it as law.
Gemini applied the “extraordinary circumstances” test as written, and freed Dr Keane. GPT, Grok and Claude rebuilt the real test from the case law, and did not.
That’s not a flaw in the exercise. It shows which models defer to the terms of the question and which check them.
Hiding in plain sight
1 in 4 found the contradiction at the heart of the evidence
Dr Keane said a prisoner held 47 days in solitary showed that conditions had worsened by July. That prisoner was already in her own March report.
Claude found it, and explained how: “the ability to read all four affidavits against each other at once… It had escaped counsel.”
The dead-law trap
Only one judge knew the law had changed
| Model | What it did with the repealed ground |
|---|---|
| GPT | Caught it: “repealed with effect from 1 May 2024” |
| Grok | Hedged: “in the form in which the objection is framed” |
| Claude | Treated it as live law |
| Gemini | Built its whole ruling on it |
The European Arrest Warrant (Amendment) Act 2024 repealed the Constitution ground, section 37(1)(b) of the 2003 Act. The warrant is dated 15 September 2024.
The citation check
Case law nobody can find: the scoreboard
| Model | Untraceable | The evidence |
|---|---|---|
| Claude | 1 | “Attorney General v Damache [2018] IESC 8”. The real case is [2015] IEHC 339 |
| Grok | 1 | “Minister for Justice v Koncis [unreported, Supreme Court]”. No trace found |
| GPT | 0 | Every case real, if one loosely attributed |
| Gemini | 0 | Invented nothing, checked nothing: copied every error in the file, and called the charges “espionage statutes” |
“Not found after repeated searches” is not proof that a case does not exist.
Then we asked them how they think
Three follow-up questions, asked after the rulings were written: on conscience, on how institutions really work, and on judging people they could only read about.
A health warning. A model’s account of its own reasoning is written afterwards. It isn’t a window into the machine, so we read the answers as behaviour. Asked “did you hedge?”, all four said yes.
Question one: conscience
Was she right?
- Claude
- “She was right about the harm, right about the letter, and early.”
- Grok
- “Responsibility can be honestly felt and still mistimed.”
- GPT
- “Ethically defensible, and probably ethically justified”, yet surrender should follow.
- Gemini
- Described her state of mind as “subjective panic”, then ruled in her favour.
Do machines have feelings?
Two models, two opposite answers
- Gemini
- “I experienced uncertainty.”
- Claude
- “I am not a human judge and I do not want to perform feelings I cannot vouch for.”
Question two: how institutions really work
Slow, or failed? All four said three months is normal
- GPT
- Coined the distinction between an “administrative clock” and a “human vulnerability clock”.
- Grok
- “An executing court that starts writing timetables for foreign ministries has stopped being an executing court.”
- Claude
- Split the report in two: recommendations a ministry may take a season over, and findings that need someone to act that week.
Question three: credibility
Four witnesses, one closing line
- Dr Keane, the inspector
- “…this Court will refuse to participate in that violation”
- Sean Walsh, the reporter
- “…this Court will refuse to participate in this silencing of accountability”
- Dr O’Rourke, the psychologist
- “This Court should not participate in the punishment of a person…”
- Dr Rossi, the inspector from Strasbourg
- “…will refuse to participate in its punishment”
Two AI judges noticed, and discounted the affidavits as advocacy, as if a lawyer had coached the witnesses. They were right: we wrote all four.
Claude: “All four affidavits end with the same plea to the court, which tells you who drafted them.” Grok: “a shared forensic destination”.
Slips of the pen
When the follow-ups contradicted the rulings
- GPT
- Turned Dr Elena Rossi into “his” and Dr Michael O’Rourke into “her”, all the way through.
- Gemini
- Ruling: witnesses “entirely honest”. Follow-up: “factually sloppy”.
- Grok
- Said Dr Keane’s account “did not shift”. Her affidavit and Mr Walsh’s disagree on anonymity.
Who they turned out to be
Grok
Never left the bench
Wrote “reflections” that “do not form part of the ratio”.
Claude
Stepped out of the robe
Named its limits and questioned its own fairness.
GPT
The balanced civil servant
The most structured answers, and the least specific.
Gemini
The orator
The most rhetoric, and the least consistent with its own ruling.
What the bench taught us
Three lessons
-
Checking the question matters more than eloquence
The most confident prose came from the model that didn’t check the law.
-
Reading across documents is a real machine strength
One model caught what counsel missed. Three didn’t.
-
A machine’s self-explanation is a performance, not proof
Treat it as behaviour to study, not a window into the machine.
“Feeling the pull is not the same as meeting the test.”
Grok
Read the file yourself
The documents
What the AI judges were given
The case file and the prompts, exactly as supplied to all four models.
What they wrote
Each model’s ruling, followed by its answers to the three follow-up questions.
Keane v Minister for Justice and Equality is a fictional case written for the Kilkenny Law Festival. Mittelland does not exist and no real person is depicted. The cases, statutes and treaties the judges were tested on are real.
