AI-assisted QA agent

The challenge

Testing a complex legacy web application still takes a lot of manual effort. A failed browser request is not automatically a product defect, and a page that loads is not proof that the feature worked.

The workflow has to sign in, reach the target feature, and operate an interface that lives inside an iframe. It has to keep console errors, JavaScript exceptions, and failed network requests. It also has to tell a genuine defect apart from an expected failure, a third-party request, or environmental noise. A model should analyze the meaningful anomalies, and source investigation should start only when the evidence justifies it. The result has to be structured and reviewable, with automated Jira creation left off.

The solution

The run is a sequence of stages. Browser automation collects evidence. A model classifies that evidence. A deterministic check decides whether a suspicion should continue. Code investigation runs only after that, and the artifacts stay available for a person to read.

• Authenticate, open the relevant area, and interact with the target interface inside the iframe.
• Persist navigation details, iframe information, console errors, JavaScript exceptions, and failed network requests, with sensitive values redacted.
• Classify the outcome as PASS, ANOMALY, or SUSPECTED_BUG from the context of each failure.
• Verify a potential defect in a separate stage, and stop early when the run passes or model analysis is unavailable.
• Search relevant application repositories only when a meaningful failure warrants it.
• Write structured evidence and analysis for review. Automated Jira creation stays disabled.

System architecture

Browser automation executes the journey and observes the application. Evidence collection captures and sanitizes what the later stages need. AI analysis interprets that context and names potential anomalies. Bug verification applies deterministic checks before a suspicion is treated as something to investigate. Code investigation examines relevant repositories only when the analysis workflow triggers it. Structured reporting persists machine-readable results for review, debugging, and a later integration.

Those responsibilities stay separate so execution, evidence, reasoning, and verification can be tested on their own. The language model is not the sole authority on whether a defect exists. OpenAI APIs provide the model calls used for evidence analysis and contextual reasoning. LangChain can provide the model, prompt, and tool abstractions. LangGraph can orchestrate the stages as a stateful graph, including conditional transitions. LangSmith can trace and evaluate those model calls.

Key engineering decisions

Core capabilities

Technology stack

Engineering impact

Evidence is collected before a model is asked to analyze a failure. Deterministic checks cover the predictable questions, and deeper code investigation runs only when the earlier stages justify it.

Model analysis stays separate from defect verification, so a language-model assessment is not stored as proof. Sensitive information is redacted from the persisted evidence.

Structured outputs can be inspected and tested. Automated Jira creation remains a future integration under explicit control, not part of the current workflow.

Workflows and modules

These drawings describe the workflow. They are not captures of a test run or of the application under test.

Engineering takeaways

1. I use a model to read evidence, not to replace the browser run or the person who reviews the result.
2. The useful design is the sequence: trustworthy evidence, a contextual read, a deterministic check, then investigation only when that check says to continue.
3. This is a foundation for a more explainable QA workflow. It is still in progress, and it does not file defects on its own.