Building AI Agents with LangChain: Why Calling an LLM Is Only the Beginning

One area that caught my attention is software testing.

Automated tests are great at checking whether something behaves as expected. But working with an existing application is rarely that straightforward. A test can fail because of a genuine bug, an unavailable service, a timing issue, or an unexpected response from a third-party dependency.

Someone still needs to investigate what happened, understand the context, and decide what to do next.

That got me thinking: Can we build an AI agent that does more than execute tests? Can it help us understand failures and investigate potential bugs?

This is where frameworks such as LangChain become interesting.

What makes an AI agent different from a chatbot?

Most of us have used an LLM to explain code, generate a function, or answer a technical question. These are useful applications, but an AI agent can take things a step further.

Instead of responding to a single prompt, an agent can work through a series of steps, use tools, inspect results, and decide what action makes sense next.

Consider a simple example.

You run an automated test against a web application, and the test fails. A conventional automation script might report the failure and stop.

An AI-assisted workflow could take a different approach:

1. Capture the browser console errors and failed network requests.
2. Examine the evidence to understand what happened.
3. Determine whether the failure looks relevant to the feature being tested.
4. Investigate the application code if the evidence suggests a possible defect.
5. Produce a structured report explaining the findings.

The important difference isn't that AI is involved. It's that the system uses information gathered during execution to guide what happens next.

Of course, this doesn't mean an agent will always make the right decision. Its conclusions still need appropriate validation.

Where does LangChain fit in?

LangChain provides abstractions for building applications powered by language models. It helps developers connect models with prompts, tools, and application logic without having to implement every integration from scratch.

For an AI-assisted QA system, tools might include browser automation, evidence collection, repository search, and report generation.

Imagine that a browser automation tool captures this information:

• The application loaded successfully.
• A primary API request returned HTTP 200.
• A third-party analytics request failed.
• No JavaScript exception was recorded.

A poorly designed system might treat every failed request as a defect.

An AI-assisted system could examine the evidence and recognize that the analytics failure may be unrelated to the feature under test. It could then classify the execution as healthy, provided the remaining evidence supports that conclusion.

LangChain can help connect the language model to the tools and instructions needed for this kind of analysis.

But there is an important distinction: using LangChain doesn't automatically make an application a reliable agent. The reliability comes from how the workflow is designed, what evidence is available, which decisions are delegated to the model, and how those decisions are verified.

My use case: An AI-assisted QA agent

I'm exploring this idea through an AI-assisted QA agent for an existing web application.

The application has a more complicated setup than a standalone page. The user navigates through a parent application, opens a target feature inside an iframe, and interacts with a separate frontend.

That makes testing more interesting. A failure can originate from the parent application, the embedded application, a network dependency, or the test environment itself.

Simply collecting errors isn't enough. The system needs to understand which errors matter.

My approach is to break the process into distinct stages.

1. Execute the test and collect evidence

The first step is conventional browser automation.

The system follows the expected navigation path and captures relevant information, including browser console errors, JavaScript exceptions, failed requests, and details about the target iframe.

The purpose is to establish what actually happened before asking an AI model to interpret it.

I also consider privacy part of the design. Persisted evidence should not expose authentication tokens, sensitive query parameters, or other unnecessary personal information.

2. Analyze the evidence

Once the evidence is available, the AI analysis stage examines it in context.

Rather than asking a vague question such as "Did the test find a bug?", I want the system to reason about the observed behavior and classify the result.

The workflow distinguishes between three outcomes:

• PASS: The available evidence supports a healthy execution.
• ANOMALY: Something unexpected occurred, but the evidence does not establish a product defect.
• SUSPECTED_BUG: The evidence suggests a potential defect that warrants further verification.

These categories help avoid a common mistake in AI-powered automation: treating an unusual event as proof that something is broken.

An anomaly deserves attention, but it isn't necessarily a bug.

3. Verify before investigating further

This is one of the most important design decisions.

I don't want the language model to have the final say on whether a defect exists.

LLMs can produce convincing explanations that are not supported by the available evidence. For that reason, I'm separating AI-assisted analysis from deterministic verification.

The verification stage applies predictable checks to decide whether the workflow should continue.

If the execution passes, there is no reason to spend more time investigating it. If the AI analysis is unavailable, the system should handle that situation explicitly rather than pretending that the test succeeded.

When a suspected bug requires further investigation, the workflow can move to the next stage.

4. Investigate the relevant code

A useful QA agent should help answer more than "What failed?"

Ideally, it should also help answer "Where should I look to understand why it failed?"

The agent can search the relevant application repositories when the collected evidence justifies code investigation.

This step is conditional. A healthy execution shouldn't trigger unnecessary repository searches or additional model calls.

That decision matters because AI workflows can become expensive and slow when every stage runs regardless of whether it is needed.

The goal isn't to use AI everywhere. It's to use it where it adds value.

Do we need LangGraph as well?

LangChain and LangGraph are related, but they solve different problems.

LangChain provides building blocks for working with language models and tools. LangGraph is designed for orchestrating stateful workflows involving multiple steps, conditional transitions, and potentially human intervention.

For a small application that takes one input and produces one response, a straightforward function call may be enough.

A QA agent is different.

It has execution stages, intermediate evidence, classification results, verification decisions, and conditional investigation. Some stages should run only when previous stages produce specific outcomes.

This kind of workflow can benefit from an explicit state machine or graph.

For example:

Run test → Collect evidence → Analyze → Verify → Investigate if necessary → Generate report

The actual path depends on the result of each stage.

LangGraph is worth considering when the workflow becomes sufficiently complex to benefit from explicit state management and conditional routing. However, it isn't mandatory simply because the application is called an AI agent. A simpler implementation can be the better engineering choice until the complexity justifies another abstraction.

What about LangSmith?

Building an agent is one thing. Understanding why it behaved a certain way is another.

When an LLM produces an unexpected result, several things could be responsible: the prompt, the available evidence, the model response, tool execution, or the application's decision logic.

Without useful observability, debugging these workflows can become frustrating.

LangSmith provides tooling for tracing and evaluating LLM-powered applications. It can help developers inspect execution traces and understand how a workflow behaved.

For a QA agent, that could be useful when investigating questions such as:

• Why was an execution classified as an anomaly?
• Which evidence was provided to the model?
• Did the model return the expected structured output?
• Which workflow stage failed?
• Did a prompt change improve or reduce classification quality?

These are important questions if an AI-powered tool is expected to become part of a team's engineering workflow.

As with any observability tool, sensitive data must be handled carefully. Traces should not inadvertently expose credentials, tokens, or confidential application information.

LangSmith is an option to evaluate as the agent develops, rather than a requirement for every LangChain project.

The part I wouldn't leave entirely to AI

It's tempting to let an agent decide everything: whether a bug exists, whether the evidence is sufficient, and whether a ticket should be created.

I would approach that cautiously.

For this use case, I prefer a combination of AI reasoning and deterministic rules.

The language model is useful for interpreting contextual evidence and identifying patterns that are difficult to capture with a fixed rule for every possible scenario.

Deterministic code is better suited to predictable checks, explicit workflow transitions, data validation, and enforcing safety boundaries.

The two approaches complement each other.

I also want the output to remain reviewable. A useful report should explain the observed behavior, identify the evidence supporting the conclusion, and distinguish confirmed facts from hypotheses.

Automatically creating Jira tickets can be a later step, once the detection and verification process has demonstrated sufficient reliability. For now, keeping that action under human control is a sensible boundary.

Keeping AI costs under control

Cost is another practical consideration.

An agent can make repeated model calls, inspect large amounts of source code, and invoke tools unnecessarily. Those decisions can increase latency and cost without improving the result.

For a system like this, I'd focus on a few principles:

• Collect relevant evidence before invoking the model.
• Keep prompts focused on the decision being made.
• Avoid sending entire repositories when a small amount of relevant code is sufficient.
• Use deterministic logic for checks that don't need a language model.
• Skip code investigation when the execution is healthy.
• Measure how often the agent's conclusions are useful and correct.

The best architecture isn't necessarily the one with the most sophisticated AI framework. It's the one that solves the problem reliably without introducing unnecessary complexity.

Final thoughts

I'm interested in AI agents because they offer a way to rethink how we approach engineering tasks that involve both structured processes and contextual reasoning.

In software testing, browser automation can execute a journey, deterministic logic can validate expected behavior, and an LLM can help interpret evidence that requires more context.

Combining these capabilities is promising, but the integration needs careful design.

LangChain can help connect models and tools. LangGraph can help orchestrate more complex workflows. LangSmith can help with tracing and evaluation. OpenAI APIs can provide the underlying language-model capabilities.

None of these tools replaces the need for good engineering decisions.

For me, the real challenge is building a system that knows when to use AI, when to rely on deterministic logic, when to investigate further, and when there simply isn't enough evidence to reach a conclusion.

That's the difference I'm interested in exploring: not just adding AI to an existing workflow, but designing a workflow where AI has a clear purpose.