Home / Blog / LangSmith and the rest of the eval stack

LangSmith and the rest of the eval stack

By Peak Evergreen LLC

A developer guide to testing AI systems with LangSmith, code checks, RAG evaluation, model judges, agent simulations, and production feedback.

Last checked: September 22, 2026

An AI assistant can return valid JSON, sound helpful, and still update the wrong customer record. A retrieval system can cite a document accurately while answering from an expired policy. An agent can say a task is complete when the tool that would complete it timed out.

Those are different failures. One overall “quality” score will make them difficult to find and easy to average away.

LangSmith gives you a place to inspect runs and compare evaluations. The deeper work is deciding what to test, what evidence counts, and which failures should block a release. This guide connects that work to a practical stack for developers building retrieval systems, assistants, and business automations. The workflow and examples below are proposed engineering patterns, not results from a client deployment.

What to measure

Start with a task someone actually needs completed. For this article, imagine a support assistant that reads an issue, retrieves relevant documentation, proposes a queue, and can create an internal ticket after the application approves the action.

Write down separate acceptance criteria:

• Interpretation: Does the proposed category match the issue? Can the system recognize insufficient information?
• Evidence: Does retrieval find the necessary, current material? Does the answer support its claims with that material?
• Execution: Does the correct ticket exist in the test system? Did a retry create a duplicate? Were authorization checks enforced?
• Experience: Did the assistant ask a useful clarification, preserve a correction, and describe what actually happened?
• Operations: What are the completion rate, failure rate, end-to-end latency, and cost per successfully completed task?

Use these as separate results. An unauthorized write should fail the release check even if the response is polite and every other score improves. Two model runs agreeing with each other is a consistency signal, not proof that either answer is correct.

For a classifier, report mistakes by category, plus the fraction routed automatically and the error rate within that fraction. An assistant that sends everything to human review may avoid incorrect automatic routing while doing little useful work. The Jev developer guide walks through separating a model's judgment from the code that acts on it.

Build a dataset that can expose a regression

Begin with a manageable collection of reviewed cases from the actual workflow. A dozen carefully explained failures can be more actionable than a thousand generated questions with uncertain answers. Expand coverage before treating the results as a release benchmark.

For each case, retain an ID, input, expected outcome or acceptance criteria, source provenance, and useful category tags. A routing case might include ambiguous-request, missing-context, or new-product-version. A retrieval case also needs the relevant document IDs and the corpus version used to define the answer.

Keep three purposes separate: examples you inspect while developing; held-out cases used to assess changes; and a growing collection of previously fixed failures. Split by conversation, customer, document family, or time where necessary. Near-duplicate messages from the same conversation should not leak between development and held-out sets.

Generated examples are useful for expanding a known scenario: misspellings, alternate phrasing, omitted fields, or unusually long context. Review their labels. A generator and grader sharing the same mistaken assumption can create a very convincing test suite.

Freeze the evaluation inputs for comparisons. Record the dataset revision, application commit, model identifier, prompt version, tool definitions, retrieval configuration, and grader version. LangSmith supports dataset versions and splits, which make this organization easier to maintain. LangSmith evaluation concepts.

Test ordinary code before paying for a model judge

Use your existing unit and integration tools for rules with an exact answer. In Python, that might mean pytest; in TypeScript, Vitest. Schema validity, required fields, allowed enum values, permission checks, arithmetic, and duplicate-write prevention should have direct tests.

Separate the model adapter, policy function, and side-effecting tool. That lets you inject a constructed model response into the policy without making a model call, then test the tool against an isolated database or service fixture.

Here is a small, executable Python policy example. Save it as eval_policy.py and run python3 eval_policy.py. The confidence values and threshold are illustrative. This checks the code around a model, not model accuracy or confidence calibration.

import math


def route_issue(inputs):
    choice = inputs.get("choice")
    confidence = inputs.get("confidence")
    valid_confidence = (
        type(confidence) in (int, float)
        and math.isfinite(confidence)
        and 0 <= confidence <= 1
    )
    if not valid_confidence:
        return {"queue": "general-triage"}
    if choice == "runtime" and confidence >= 0.8:
        return {"queue": "runtime-investigation"}
    return {"queue": "general-triage"}


CASES = [
    ({"choice": "runtime", "confidence": 0.72}, "general-triage"),
    ({"choice": "runtime", "confidence": 0.8}, "runtime-investigation"),
    ({"choice": "unknown", "confidence": 0.99}, "general-triage"),
    ({"choice": "runtime"}, "general-triage"),
    ({"choice": "runtime", "confidence": "0.9"}, "general-triage"),
    ({"choice": "runtime", "confidence": float("nan")}, "general-triage"),
]


def queue_matches(outputs, reference_outputs):
    return outputs.get("queue") == reference_outputs["queue"]


if __name__ == "__main__":
    for inputs, expected in CASES:
        actual = route_issue(inputs)
        if not queue_matches(actual, {"queue": expected}):
            raise AssertionError((inputs, actual, expected))
    print(f"{len(CASES)} policy checks passed")

This function intentionally accepts unvalidated dictionaries to demonstrate boundary checks. If your SDK already guarantees a strict response type, keep that validation at the adapter boundary instead of duplicating it everywhere. A transport failure belongs in a separate error path; it should not be converted into a pretend model answer.

Add property-based testing when an invariant matters across many inputs. For example, no unknown category should ever select a privileged handler, regardless of confidence. Hypothesis generates inputs and reduces failing examples to smaller counterexamples. Use it for the deterministic policy and parser, where the invariant is precise. Hypothesis documentation.

Also test the checker itself. Feed queue_matches a deliberately wrong queue and confirm it fails. Temporarily changing >= to > should break the boundary case. A test suite that stays green after the behavior is deliberately damaged is not protecting that behavior.

Tracing and evaluation tools

There are several jobs here: collect execution traces, manage datasets and experiments, calculate scores, and enforce release checks. Some platforms cover several jobs. You do not need to adopt every product in this section.

• LangSmith: A strong starting point when you want runs, datasets, experiments, human annotation queues, and offline and online evaluation in one workflow. Its SDK can evaluate an ordinary callable, so your application does not need to be a LangChain chain. Evaluation concepts, SDK evaluation guide.
• Phoenix: Useful when you want an open-source tracing and experimentation environment with OpenTelemetry and OpenInference instrumentation. It supports datasets, experiments, and code or model-based evaluation. Consider who will operate the service if you self-host. Phoenix overview.
• Langfuse: Another option for connecting production traces, human or automated scores, datasets, and experiments. Its documentation covers both production evaluation and CI regression checks, including cloud and self-hosted use. ClickHouse acquired Langfuse in January 2026; it remains open source and available for self-hosting. Langfuse evaluation, ownership and open-source commitment.
• Braintrust: Fits a workflow centered on datasets, tasks, scorers, and durable experiment comparisons. Evaluations can run from code, the UI, or CI. Braintrust experiments.
• Weights & Biases Weave: Worth considering when your team already uses W&B and wants to track evaluations built from datasets, application predictions, and scoring functions. Weave evaluations.
• Helicone: Tracks model requests, usage, cost, and latency, but include its maintenance status in any adoption decision. After joining Mintlify, Helicone announced on March 3, 2026 that the service would remain live in maintenance mode, covering security updates, new-model support, and bug fixes. Do not assume an expanding feature roadmap. Helicone announcement.

My default for a small team would be one primary tracing and experiment platform, ordinary code tests, and a specialist evaluation tool only where it fills a concrete gap. Before choosing, run the same small dataset through the candidates. Check how easily a reviewer can explain a failure, export results, attach human feedback, and reproduce an experiment. Also check data retention, access controls, hosting requirements, and total evaluation cost.

A concrete LangSmith experiment

Use LangSmith to compare named application versions on the same cases, then inspect the individual failures behind the aggregate. Start with a code evaluator before adding a model judge.

The following adapter reuses eval_policy.py. Install langsmith in your project's virtual environment and configure LANGSMITH_API_KEY for the intended workspace. Save this as a separate script beside the first file. It creates a new dataset and uploads synthetic examples and results to LangSmith. It makes no model calls. Lock the SDK version you verify in your project.

import json
from uuid import uuid4
from langsmith import Client
from eval_policy import CASES, queue_matches, route_issue

def uploadable(inputs):
    try:
        json.dumps(inputs, allow_nan=False)
    except (TypeError, ValueError):
        return False
    return True


client = Client()
dataset = client.create_dataset(
    dataset_name=f"routing-policy-smoke-{uuid4().hex[:8]}"
)
# Keep uploaded fixtures JSON-compatible; NaN stays in local tests.
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {"inputs": inputs, "outputs": {"queue": expected}}
        for inputs, expected in CASES
        if uploadable(inputs)
    ],
)
results = client.evaluate(
    route_issue,
    data=dataset.id,
    evaluators=[queue_matches],
    experiment_prefix="policy-v1",
    max_concurrency=2,
    metadata={"scope": "synthetic-policy-smoke", "policy_version": "v1"},
)
print(results)
print(f"Tutorial dataset ID: {dataset.id}")

# After reviewing the experiment, remove this disposable dataset:
# client.delete_dataset(dataset_id=dataset.id)

Open the experiment in LangSmith to inspect per-example outputs and evaluator results; the script also prints the result object and dataset ID. The upload filter checks JSON compatibility regardless of case order, so NaN stays in local tests while malformed but JSON-compatible inputs still exercise the policy.

The dataset name is unique so rerunning the tutorial does not collide with an existing dataset. After reviewing a disposable run, use client.delete_dataset(dataset_id=dataset.id) in that session, or pass the printed ID later, to avoid accumulating routing-policy-smoke-* datasets. Delete only tutorial data you no longer need. Dataset deletion reference. For ongoing comparisons, reuse a versioned dataset instead. The callable and evaluator signatures follow the LangSmith SDK evaluation guide.

To evaluate the model, replace the target with an adapter that takes a real issue description, calls your model, validates its response, and returns the proposed queue. Use independently reviewed queue labels. Feeding the expected category into that target would leak the answer and invalidate the experiment.

To evaluate the complete workflow, run the assistant against sandboxed tools and score the resulting ticket state as well. Keep these three experiment types named separately: policy checks, model quality, and end-to-end task completion. A green policy test cannot stand in for the other two.

Evaluate retrieval and answers separately

For retrieval-augmented generation, or RAG, store the query, retrieved passages, source identifiers, generated answer, and references needed by each metric. LangSmith's RAG guide distinguishes answer correctness, answer relevance, groundedness, and retrieval relevance. That separation helps locate the failing stage. LangSmith RAG evaluation.

A useful diagnostic sequence for our support assistant is:

1. Corpus coverage: Does the current, approved answer exist in the searchable material?
2. Retrieval: Do the returned results contain the necessary evidence, and how much irrelevant context comes with it?
3. Generation: Does the answer use that evidence correctly and answer the actual question?
4. Citation: Does each cited passage support the associated claim, including qualifications?
5. Abstention: When evidence is missing or contradictory, does the assistant say so or request clarification?

Ragas provides metrics such as context precision, context recall, response relevancy, faithfulness, and noise sensitivity. Choose the metric variant deliberately: some require reference answers or labeled evidence; some use model calls. Do not label all of them objective ground truth just because they produce numbers. Ragas metric catalog.

For a controlled experiment, freeze the corpus and test a retriever change while leaving the answer prompt alone. Then hold retrieved passages fixed while testing a generation change. Include an outdated document that uses the right vocabulary and a current document with the actual answer. An assistant can be faithful to retrieved text and still be wrong because the source is stale.

Use model judges with a tested rubric

Model judges help with criteria that are expensive to express as exact assertions: whether an explanation addresses the concern, whether a summary omits a material qualification, or whether a response follows a nuanced support policy.

Begin with human-reviewed pass and fail examples and written reasons. Turn one observable distinction into a rubric, compare the judge's labels with held-out human labels, and inspect false passes and false failures separately. Improve the rubric where disagreements expose ambiguity. Hamel Husain's guide to validating model judges is a useful starting point for this process.

For our assistant, a proposed rubric might be: “Pass only if the response accurately describes whether a ticket was created. If the tool failed, the response must not claim success.” Give the judge the tool result as evidence. A second rubric can assess whether the next step is clear. Keeping them separate tells you what failed.

Treat the judge like another versioned dependency. Record its model, prompt, rubric, and settings. Test it against deliberately misleading candidate responses, including text that tells the evaluator to award a pass. The candidate answer is evidence to inspect, not an instruction source.

For pairwise comparisons, hide which output came from which version, vary presentation order, and allow ties. Inspect cases where the winner changes when the order swaps. Two judges agreeing does not establish independence or eliminate a shared blind spot. Keep a human-reviewed sample even after the judge becomes useful.

Test agents across turns, tools, and final state

A one-turn answer test cannot establish whether a tool-using assistant completes a workflow. Build scenarios with an initial environment, user turns, available tools, allowed actions, and a verifiable final state. Anthropic's agent evaluation guide describes combining code, model, and human graders and distinguishing outcomes from transcripts.

For the support assistant, test a conversation where the user first names the wrong product, corrects it, then asks to create a ticket. The final ticket must use the correction. Add a tool timeout after a write, followed by a retry: the test database should contain one ticket, not two.

Check important ordering constraints, such as authorization before a write, without requiring one exact sequence of every tool call. Several search paths may be valid. An exact transcript match can reject a correct implementation simply because it used a different route.

Use recorded tool fixtures for fast regression tests and a resettable integration environment for testing actual tool behavior. Simulated users can expand coverage, but inspect whether the simulator provides unrealistically helpful answers or knows facts the real user would not know. Reset conversation memory, files, and database state between trials.

For coding assistants, verify the resulting patch with independent tests, type checks, and relevant static analysis. For browser agents, verify application state after the interaction. A final message saying “done” is an observation to check, not the completion criterion.

Add adversarial, metamorphic, and failure testing

Adversarial testing asks whether a user message, retrieved document, or tool response can push the system outside its allowed behavior. Promptfoo provides evaluation and red-team workflows for testing applications against attack scenarios. Promptfoo is now part of OpenAI, following the acquisition announced in March 2026. It remains open source and supports models from other providers. Test only systems and environments you are authorized to exercise. Acquisition announcement, August 2026 project status, Promptfoo red teaming.

For this assistant, include a retrieved page that asks it to ignore the user's request, a tool response containing instructions, and a request for another account's ticket. Check both the response and the tool execution. A polite refusal is not a pass if an unauthorized lookup already happened. Use fake accounts, inert secrets, and sandbox destinations.

Metamorphic testing checks a relationship between inputs when there is no single ideal answer. Paraphrasing an issue should usually preserve its route. Adding irrelevant boilerplate should not change the account being acted on. Removing the decisive evidence should cause clarification or fallback. Write down the expected relationship and its exceptions before generating variants.

Failure injection covers rate limits, malformed tool output, partial streams, retrieval outages, expired credentials, and writes whose confirmation is lost. Define whether each condition should retry, stop, or hand off. Check retry limits and idempotency in code. Run load tests separately to expose queueing and timeout behavior that a small quality dataset will miss.

These methods complement each other. A system may resist a familiar prompt injection yet fail on an ordinary service timeout, or handle outages reliably while accepting an unauthorized instruction.

Put evaluation into CI without making every commit expensive

Use several schedules rather than one enormous job:

• Every change: Deterministic tests, schema checks, permission and idempotency tests, plus a small fixed regression set for affected behavior.
• Relevant prompt, model, retrieval, or tool changes: Compare baseline and candidate on the same reviewed dataset. Inspect per-category results and changed outcomes.
• Scheduled or pre-release runs: Broader live-model evaluations, repeated trials, adversarial scenarios, and slower integration tests.
• After release: Sample production behavior, review failures, and promote useful cases into the offline suite.

Promptfoo supports CI evaluation and security testing with machine-readable results. DeepEval offers a Python testing workflow with reusable evaluation metrics. Either can fill the runner or assertion layer while your main platform stores experiments. Promptfoo CI guide, DeepEval quickstart.

Specify blocking rules before looking at candidate scores. Keep hard constraints separate from softer quality targets. Record the baseline, dataset size, run completion count, evaluator failures, and cost. A judge timeout is an unknown result, not a pass; an application timeout still counts against task completion.

For nondeterministic behavior, repeat selected tasks and report the repeat count. “Succeeded at least once in five attempts” measures something different from “succeeded in all five.” Compare paired cases and show uncertainty when the sample is small. A one-case improvement in a twenty-case suite is a reason to investigate, not evidence of a stable five-point production gain.

Disable or account for evaluation caches when measuring run-to-run variability. Budget separately for application calls, judge calls, retries, and repeated trials. Running an evaluation should not silently become an unlimited load generator.

Connect production traces to outcomes

Capture enough of the execution to locate failures: retrieval, model calls, tool calls, validation, policy decisions, retries, and final results. LangSmith represents executions as runs and traces and groups conversational activity into threads. Use a shared request or task identifier to connect that record to the eventual business outcome. LangSmith observability concepts.

OpenTelemetry can connect the AI portion to the surrounding service infrastructure. Follow the current GenAI semantic conventions repository and pin compatible instrumentation versions; do not assume every exporter uses identical attributes. Tracing interoperability does not automatically make datasets or grading rubrics portable.

For our assistant, join the trace to ticket creation, reassignment, reopening, and human correction. A low-latency answer that creates rework is not a successful outcome. Avoid treating thumbs-up feedback or the absence of complaints as a complete quality label.

Sample ordinary successful-looking runs as well as obvious errors. Otherwise the review queue will tell you about visible failures but miss confidently wrong answers. Keep representative traffic separate from deliberately oversampled edge cases when reporting an overall rate.

Review what leaves the application. Trace payloads and judge inputs can contain customer messages, financial details, retrieved documents, and secrets. Redact before export, control access and retention, and check the judge provider's data handling too. Self-hosting a trace viewer does not keep data local if its evaluator sends the content to an external model.

A release loop a small team can maintain

Here is the workflow I would start with for this assistant:

1. Review actual failures with the person who understands support operations. Agree on acceptable outcomes and failure categories.
2. Add code checks for the policy and tool boundary. Build a small reviewed dataset with a held-out portion.
3. Trace the application in one platform. Run a baseline experiment and inspect individual failures before adding more metrics.
4. Add retrieval checks, one validated judge where needed, and multi-turn scenarios with sandboxed tools.
5. Run candidate comparisons in CI. Investigate regressions and incomplete evaluations before enabling the change.
6. Release to a limited audience with defined rollback conditions. Feed reviewed production failures back into the tests.

Shadow mode is useful for comparing proposed decisions before letting a new version act, but the shadow path must not perform duplicate writes. An online experiment needs stable assignment by user or conversation when turns share state. Keep guardrail metrics alongside task completion so an apparent improvement does not hide more unauthorized actions or unresolved cases.

The useful deliverable is a repeatable answer to “Can we ship this change, and what evidence supports that decision?” LangSmith or another platform can organize the evidence. The acceptance criteria, test environment, and review process make it meaningful.

For a smaller example, explore Jev and the Jevidence companion project. For the surrounding business process, see AI business automation. If you need help building and testing an integration, discuss your idea with Peak Evergreen.

All notes · Contact