I tried to retrain a model to write my Vitest unit tests automatically. The initial results were amazing, but then it hit a wall of real world complexity. A look into the practical limits of model retraining, the challenges of mocking, and the prohibitive costs that made the project unviable.
As developers, we're always looking for ways to automate the tedious parts of our job so we can focus on creative problem-solving. For me, the holy grail has always been automating unit testing. So, with the power of modern language models, I embarked on an exciting project: retraining my own model to write basic Vitest unit tests for my components.
The goal was simple: feed the model the Vitest documentation and a component's source code, and get a functional test file back. The initial results were nothing short of magical.
The Promising Start
For simple, presentational components, the kind that take some props and render some HTML, the model was fantastic. It correctly imported the component, wrote describe and it blocks, and made basic assertions about the rendered output. It understood props, slots, and simple event emissions. For a moment, I felt like I had cracked it. The outcomes looked incredibly good, and I started to imagine a future with a significantly reduced testing workload.
Hitting the Complexity Wall
My optimism, however, was short-lived. As soon as I moved from simple components to more complex, real-world examples, the model's performance collapsed. The two biggest hurdles were:
Complex Component Logic: When a component had internal state, multiple methods, or conditional rendering based on intricate logic, the model struggled to create meaningful tests. It could write the boilerplate, but it couldn't grasp the "why" behind the component's behaviour.
Mocking Dependencies: This was the breaking point. Modern components rarely exist in a vacuum. They import services, call APIs, and interact with global stores. My model had no concept of how to mock these dependencies. It would try to write tests as if these complex external parts just existed, leading to code that was syntactically plausible but functionally useless and would never pass.
The Realisation: Data and Cost
The diagnosis was clear. My model's "education" was far too shallow. The basic Vitest documentation wasn't enough. To handle the complexity and mocking inherent in real-world applications, the model would need to be trained on a massive dataset of actual, complex test suites something far beyond the scope of a personal project.
This led to the second, more formidable wall: cost.
To give the model the level of learning it truly needed, I would have to process a huge amount of data. The costs associated with using a platform like AWS SageMaker for training and fine-tuning, coupled with the token costs of using the OpenAI API as a base, quickly spiralled. For a personal project, the potential expense made it completely unviable. The cost of building a truly effective test writing AI would far outweigh the time it saved me.
Conclusion: A Valuable Lesson
This experiment was a fascinating and humbling lesson in the practical limitations of applied AI. While the dream of a fully automated, context aware test writer is compelling, the gap between a simple proof of concept and a production ready tool is immense. It's a stark reminder that an AI model is only as good as its training data, and acquiring and processing that data at scale remains one of the biggest barriers for individual developers and small teams.
For now, it seems my unit tests will continue to be written the old fashioned way: by me (and a little from co-pilot).