Flaky CI pipelines are often a symptom of growing pains rather than poor engineering. Discover how migrating to Playwright and integrating agentic testing can restore developer confidence and velocity.
Nothing destroys engineering velocity quite like a flaky Continuous Integration pipeline. When developers stop trusting the test suite, every failure becomes easier to dismiss as 'probably CI,' and the safety net gradually loses its value. But in a fast-moving startup environment, an unreliable pipeline is not usually the result of bad engineering; it is usually a symptom of a company outgrowing its initial tooling.
Recently, I spent time looking at how we validate our application, recognising that the sensible trade-offs we made yesterday were becoming constraints today.
The CI Ghost and the Playwright Pivot
We had a persistent ghost in our End-to-End (E2E) suite: a flaky WebSocket test that passed locally every single time but would randomly fail in the cloud. Real-time asynchronous communication can be particularly awkward to test because CI environments introduce timing and scheduling differences that may never surface consistently on a developer machine.
The immediate temptation in these scenarios is always to slap a sleep(5000) in the test block and hope for the best. Instead, I tore the test down to identify the actual race condition happening between the WebSocket connection handshake and the UI assertion hooks. By replacing arbitrary timeouts with deterministic synchronisation hooks, forcing the test to wait for explicit socket state changes before proceeding, we systematically killed the flake.
But fixing individual tests only gets you so far. As the application grew, the characteristics we needed from our E2E tooling changed. Cypress served us well during our early build, but as we scaled, Playwright's approach to concurrency, parallel execution, and overall execution model was simply a better fit. When developers know that a red X in CI means they actually broke something, they merge with confidence.
Agentic Testing: Exploration vs. Assertion
Even with Playwright running smoothly, traditional E2E testing serves a deterministic purpose. It relies on strict DOM locators and hardcoded assertions. To bridge the gap between deterministic checks and actual human behaviour, I have been exploring the use of AI agents to actively control the browser. I don't see agentic testing as a replacement for deterministic tests. If a checkout flow must produce a particular state, I want a deterministic assertion proving it.
Where agents become interesting is in the space around those assertions: exploring an application semantically and identifying functional behaviour we did not think to encode into the suite. This relates to the broader shift I discussed in my article Why Agentic Coding Won't Make Software Developers Extinct, where we move beyond simple scripted automation into contextual intelligence.
We are currently working out the infrastructure to integrate this agentic experimentation into our workflow. By combining strict Playwright assertions with the fuzzy, exploratory nature of AI, we are building a dual safety net. Similar to how I approached system reliability in my AI Engineering Playbook Part 6: Making It Fast & Cheap, the goal here is to maintain high quality without sacrificing the speed that defines a successful startup.
Conclusion: Testing as a Living Capability
Testing infrastructure is never a static decision you make on day one and forget about. It is a living capability that must mature alongside the product. Whether you are managing Context Aware AI Guidance System logic or standard web forms, recognising when to evolve the pipeline ensures that as your product's surface area expands, your team's confidence in shipping expands right alongside it.