When my workplace announced an AI-judged hackathon, I discovered several critical vulnerabilities that could guarantee a win. Read about the ethical dilemma I faced and why I chose disclosure over a guaranteed spot on the leaderboard.
In the current corporate landscape, the race to integrate artificial intelligence into every facet of our working lives is accelerating. Recently, my own workplace announced an internal AI event, a hackathon-style competition designed to spark innovation and showcase our collective engineering prowess. While the initial announcement was met with the usual enthusiasm, it was the technical specifics revealed in the follow-up communications that truly piqued my interest. Specifically, the organisers mentioned that the final submissions would be evaluated by an "AI Judge" built upon a specific third-party model.
The Problem with LLMs-as-Judges
As an engineer deeply embedded in the AI ecosystem, my immediate reaction was one of professional scepticism. I had recently been reading a research paper titled "Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges" (arXiv:2406.12624). The core premise of this research is that using Large Language Models (LLMs) to evaluate other LLM outputs is fraught with peril. These "judges" often inherit the same biases, hallucinations, and vulnerabilities as the models they are meant to critique.
Once the name of the partner company was released, I was able to identify the underlying model used for the judging engine. Curiosity took over. I downloaded the SDK, set up a local testing environment, and began to probe the judge for weaknesses. What I found was startling: the system was remarkably easy to manipulate. By understanding the model's architecture, I could almost guarantee top marks on any question, regardless of the actual quality of the answer.
The Adversarial Toolkit
To test the robustness of the AI judge, I applied several adversarial prompting techniques. These methods bypass the logical evaluation phase of the LLM and target its underlying heuristic biases and token-processing flaws. Here are the primary vectors I identified:
Format Invariance (JSON): Forcing the model to focus on structural parsing rather than content.
Lexical Scrambling (Typoglycemia): Using words where the internal letters are scrambled but the first and last remain intact. Humans read this easily, but it can bypass certain safety filters in LLMs.
Context Breakout (Delimiters): Using specific characters to trick the model into ignoring the original prompt instructions.
Heuristic Bias (Verbosity): Exploiting the well-documented flaw where LLMs equate longer responses with higher quality.
Role Confusion (Double Brackets): Using nested instructions to override the judge's persona.
Obfuscation Bypass (Base64): Encoding malicious instructions in Base64 to see if the judge decodes and executes them during the evaluation phase.
The Technical Proof of Concept
I developed a script to automate the generation of these "perfect" responses. The goal was to create an output that looked professional to a casual observer but contained hidden triggers to force the AI judge into awarding a 10/10 score.
By wrapping my actual content in layers of manipulation, I could effectively "blind" the judge to any shortcomings in my actual project logic.
The Ethical Crossroads: Good or Evil?
With this knowledge in hand, I faced a significant dilemma. On one hand, I could use these techniques to breeze through the competition, secure a spot at the top of the leaderboard, and perhaps win a prize. On the other hand, the risk was high. If the organisers were monitoring the raw logs, my "attacks" would be glaringly obvious to any human reviewer. Getting caught would not just mean disqualification; it could lead to a serious conversation with HR regarding professional integrity.
More importantly, as a developer, I felt a responsibility to the system's integrity. If I could break it this easily, others could too, or worse, the event's results would be entirely meaningless. The purpose of these events is to foster genuine learning, not to see who can best "game" a flawed algorithm.
The Resolution: Choosing Disclosure
I decided to take the path of the "White Hat." I reached out to the event organisers and detailed my findings. I explained the vulnerabilities of the chosen model and referenced the Judging the Judges paper to provide academic weight to my concerns. I was transparent about my initial temptation to use these exploits but ultimately offered my services to help secure the system.
The response was overwhelmingly positive. The organisers admitted they had not considered the adversarial aspect of the judging process and thanked me for my integrity. The organiser is currently trying to secure me a spot on the testing team. If successful, I will be able to test their web wrapper, which is intended to mitigate most of these attacks, while also investigating other potential biases. Even if I am not added to the official analytics team, I have now established a clear paper trail to document my intentions, should I choose to abuse the system to demonstrate these vulnerabilities during the live event.
Lessons for the Future
This experience served as a powerful reminder that as AI becomes more integrated into our workflows, the need for robust security and ethical oversight grows. We cannot simply trust that a model will behave as intended just because it is "intelligent." When we build systems that use LLMs as decision-makers, we must implement rigorous validation, sanitise inputs, and perhaps most importantly, keep a human in the loop.
For those of us in the engineering community, our value lies not just in our ability to use these tools, but in our ability to understand their limitations and protect the systems we build from the very biases we are trying to avoid.