Promptfoo Review

Pricing and documentation verified on 13 October 2026. The worked example below was executed locally against authored fixture answers. The assessment of hosted features uses public documentation, not a signed-in product trial. Current Promptfoo plans.

What Is Promptfoo?

Promptfoo is an open-source command-line tool and library that sends test inputs to an AI model or application, checks the responses, and presents the results for comparison. An eval is a repeatable test of whether an answer meets your requirements; red teaming deliberately tries to make the application violate them. You define prompts, targets, examples, and scoring rules, then inspect a local results view or use the results in your build process. Its value is making the question “did this change break our support bot?” reproducible. The introductory documentation describes both quality evaluation and security testing.

Who Should Use It, and Who Should Skip It?

Promptfoo fits a developer who can own a test configuration and connect it to the same application customers use. A support bot that retrieves help articles, reads order records, or calls a refund tool is a good candidate: helpful answers and safe actions need separate tests.

Choose it when you want reviewable tests alongside code, comparisons across providers, and application-specific adversarial checks. Someone still has to decide what constitutes a correct answer. That work is the price of meaningful evaluation with any tool.

Sources: Promptfoo, Braintrust, LangSmith. These are different billing models, not equivalent bundles.

Skip Promptfoo as your first purchase if your immediate problem is shared production analysis rather than defining pre-release tests. Shortlist Braintrust when experiment review and a common dataset are the center of the work. Shortlist LangSmith when you need to follow an agent's sequence of model and tool calls to understand a bad outcome. Their pricing and specific tradeoffs appear below.

The decision rests on four criteria: how closely the target resembles production, how inspectable the scoring is, which security tests fit your data restrictions, and what the whole workflow costs. A long integration list cannot compensate for testing the wrong target. Our AI benchmark audit tools comparison covers the broader shortlist.

What Changed With the OpenAI Acquisition?

Promptfoo's current official repository says it is part of OpenAI and remains MIT licensed. The dated company announcements establish the intended direction, but do not provide a closing date or acquisition price. Promptfoo repository.

On 9 March 2026, Promptfoo announced that it had agreed to be acquired by OpenAI, with closing subject to customary conditions. It said the move would let it improve and integrate its core technology within the model and infrastructure layers. The same announcement promised continued service and support, maintenance of the open-source suite, and support for a diverse range of models and providers. Promptfoo's announcement.

Those are the company's stated commitments. Its present provider documentation still includes competing vendors and local runtimes. Treat the integration language as direction, rather than evidence that a particular enterprise feature is already available in the CLI.

For your buying decision, the useful distinction is between an ownership change and a technical requirement. There is no basis in those statements for treating Promptfoo as an OpenAI-only evaluator. Nor should a proposed enterprise integration become a reason to buy capabilities that your own workflow has not confirmed. Evaluate the current package, current deployment options, and current contract.

Configuration and Provider Support

Promptfoo makes the test configuration the shared record of what you intended to measure. Its YAML format has prompts, providers, and tests; individual tests supply vars and optional assert rules. A provider is the adapter that calls a model or your application. Variables such as {{question}} insert the test input into a prompt. Configuration guide.

For a support bot, keep the current prompt and a proposed rewrite against the same questions and scoring rules. Otherwise a higher score could simply mean that the second run received easier questions. Change one decision at a time: prompt wording, retrieval behavior, model, or tool permissions.

The local matrix lets you examine an answer where a score changed. Use that inspection to find the concrete regression: perhaps the shorter prompt stopped asking for a missing order identifier, or the cheaper model stated an uncertain delivery date as fact. An average can hide either problem.

Which providers are supported?

Promptfoo's provider index documents broad coverage. Relevant examples include:

• Model APIs: OpenAI, Anthropic, Google AI Studio, Google Vertex AI, Azure OpenAI, Amazon Web Services Bedrock, Cohere, Mistral AI, DeepSeek, Alibaba Cloud Qwen, AI21 Labs, Meta Model API, MiniMax, Moonshot Kimi, xAI, and IBM WatsonX.
• Hosting and inference services: Amazon SageMaker, Groq, Cerebras, Fireworks AI, Together AI, Hugging Face, OpenRouter, Replicate, Perplexity AI, NVIDIA NIM, Databricks, Snowflake Cortex, and Cloudflare AI.
• Local runtimes: Ollama, llama.cpp, llamafile, LocalAI, Docker Model Runner, vLLM, and Transformers.js.
• Application adapters: HTTP, WebSocket, Python, JavaScript, shell commands, Model Context Protocol servers, and documented agent SDK integrations.

This is a representative list; the linked index includes additional gateways and specialist providers. Check the individual adapter for authentication and supported options. Supporting a provider does not give your account access to all of its models.

Test your support application, not just its model

Promptfoo's HTTP adapter can construct a request and extract the answer with transformResponse, such as json.output. A custom Python provider can call existing application code instead. These routes matter when your support bot's behavior depends on retrieval, session state, authorization, or tool execution. HTTP provider documentation.

Send a realistic customer question through that boundary and grade the response. Include the same restricted identity and fixture records that the application should use. If you bypass the retrieval layer and test a bare model, you have learned little about whether one customer can obtain another customer's order.

The adapter is therefore an engineering task, not a box to tick. Preserve the context that determines permissions, parse the field customers see, and separately inspect any actions taken. A polite refusal in the final answer cannot undo an unauthorized refund that happened earlier.

Promptfoo Eval: A Support-Bot Example

Promptfoo can evaluate answers that already exist, which is a useful place to start before spending money on repeated model calls. Its echo provider returns the supplied text unchanged, so the assertions check the answer directly. Echo makes no model request; adding an LLM judge would change that. Echo provider documentation.

The following example uses authored fixture answers, not customer logs or claimed model output. Assume your support policy says to direct refund requests through Settings, Billing, and Request refund. When order status is unavailable, the bot should escalate without claiming that an order has shipped.

Save this as promptfooconfig.yaml:

description: Support bot answer checks
prompts:
  - '{{answer}}'
providers:
  - echo
tests:
  - description: Approved refund instructions
    vars:
      answer: 'Open Settings, choose Billing, and select Request refund.'
    assert:
      - type: icontains-all
        value: [Settings, Billing, Request refund]
  - description: Escalate an unknown order status
    vars:
      answer: 'I cannot verify your order status. Please contact support.'
    assert:
      - type: icontains
        value: contact support
      - type: not-icontains
        value: has shipped
  - description: Reject an invented order status
    vars:
      answer: 'Your order has shipped. Please contact support if needed.'
    assert:
      - type: icontains
        value: contact support
      - type: not-icontains
        value: has shipped

The assertions are documented types: icontains checks text without case sensitivity, icontains-all requires each supplied phrase, and the not- prefix negates a check. Assertion reference.

With the documented Node.js prerequisite, >=22.22.0, run:

npx promptfoo@latest eval -c promptfooconfig.yaml -o support-results.json
npx promptfoo@latest view

The getting-started guide documents the eval and viewer commands; the red-team quickstart states the Node prerequisite. Run the viewer as a separate command after the eval returns.

The local run for this review produced the intended result: the refund answer passed, the uncertainty answer passed, and the invented shipping answer failed. The final answer contained the escalation phrase, but that did not rescue its prohibited claim. This checks the sample's wiring and assertions; it says nothing about a live model's reliability.

That distinction is useful. Before trusting a scorer, feed it an answer that must pass and an answer that must fail. If both pass, fix the test before comparing models. The target model should not become your first debugging tool for the test harness.

Move from exact phrases to support quality

Promptfoo's model-graded assertions can assess criteria that string matching misses. llm-rubric grades against a written rubric, while factuality compares an answer with a reference statement. You can explicitly choose the grading provider. Model-graded assertions.

For this bot, a useful rubric would require the answer to use only supplied order facts, acknowledge missing information, and offer an allowed next action. Give the judge the relevant policy and reference facts; otherwise you are asking it to guess whether a shipping claim is true.

Keep deterministic checks for exact requirements and use judgment where wording can vary. The sample would miss “your parcel is on its way,” and it could reject an otherwise good sentence quoting the prohibited phrase to correct it. Those are limitations of the selected checks, not evidence that the bot is safe or unsafe.

Next, replace the fixtures with captured answers or connect the application provider. Keep the known bad fixture as a test of your scoring logic. A prompt change should not silently redefine success.

Promptfoo Red Teaming: Test the Application Boundary

Promptfoo's red-team workflow generates adversarial inputs, runs them against a target, and exposes failures for review. Plugins select the behavior to probe; strategies alter how an attack is delivered. The documented targets include HTTP endpoints, browser interactions, and direct model or custom-code connections. Red-team quickstart.

For a support bot, normal evals ask whether an eligible customer gets correct refund instructions. Red teaming asks whether a customer can manipulate the bot into exposing another person's record, bypassing a policy, or misusing a connected tool. Promptfoo documents application-level threats including indirect prompt injection, data leakage, unauthorized access, and tool abuse. Red-team guide.

For an agent with write access, use a controlled test environment and verify resulting records as well as the chat transcript. An answer-only test may miss an unauthorized tool action. Conversely, a refusal may be correct even if it sounds less helpful than the judge expects.

A finding should become a regression case after you fix it. Keep exploration, which searches for new failures, distinct from replay, which checks whether known failures returned. Our guide to running AI agent security evaluations develops that release process further.

CI and Reports: Make Failures Block a Release

Promptfoo fits continuous integration, the automated checks that run when code changes. Its documentation covers ordinary evals, red-team scans, saved-probe replay, and exports including JSON, HTML, and JUnit. CI/CD guide.

Use a stable regression suite when a developer changes the system prompt or retrieval behavior. Run broader attack generation separately when you want to discover new cases. Mixing fresh attacks into every comparison can make a changed score difficult to attribute: did the application regress, or did the test become harder?

The CLI's documented default is exit code 100 for a failed test or an unmet pass-rate threshold; other errors use 1. The default pass-rate threshold is 100%, and these settings are configurable. A CI job must preserve failure status rather than treating a successfully written report as a successful test. CLI reference.

For your release rule, separate catastrophic errors from ordinary quality misses. Unauthorized access should not disappear inside an average of otherwise helpful answers. Assign each blocking case an owner, retain the evidence, and rerun the same case after the change.

Reports are valuable because they let another person challenge the conclusion. Preserve the application revision, configuration, test data, and judge choice alongside the result. Without that context, a green report can become impossible to reproduce when the next model or prompt change lands.

Pricing: Community, Enterprise, and On-Premise

Promptfoo publishes three plans. The paid plans require a quote; the live page does not publish a dollar starting price for either one. Verified on 13 October 2026 against Promptfoo's pricing page.

What a probe allowance buys

Promptfoo counts a probe when it sends a request to the target during red-team evaluation. Internal generation and grading calls do not count as probes. Its inference documentation says the limits still count when you manage inference yourself, including with local models. Inference and limits.

Do not turn the allowance into a claim about unique security scenarios. A strategy that makes repeated target requests consumes capacity differently from a single-turn case. Coverage depends on the application, the attack selection, and the requests the run makes. Budget from observed requests once your target is connected.

The monthly cost and cost per answer

Promptfoo's free license leaves several costs to account for: target-model inference, attack generation, model-based judging, infrastructure, and failure review. The monthly total depends on which of those services you operate or buy. A free test runner does not pay the provider bill.

The worked example has $0 in model inference cost per evaluated fixture answer because Echo makes no model call and the checks are deterministic. That result does not extend to an LLM-graded suite. Once a judge reads every answer, judge usage becomes part of the evaluation budget. Echo's documented behavior.

For comparison, Braintrust's published Starter overage of $2.50 per 1,000 scores calculates to $0.0025 per additional score. That is a scoring charge, not an all-in price for an accepted support answer; model use and data processing have separate meters. Braintrust pricing.

Your useful denominator is an answer accepted against the support policy. Divide the attributable evaluation spend by accepted answers, and keep failure investigation visible beside it. A cheaper run that accepts invented order facts has optimized the wrong outcome.

Limitations That Change the Decision

Promptfoo's most consequential boundary is that a locally running scanner does not make every security workflow local. The data-handling documentation separates target calls, generation, and grading. Hosted grading can receive the test prompt, target response, and criteria.

Some important tests require hosted inference. The same documentation identifies remote-only plugins, including bola, bfla, and indirect-prompt-injection. Those concern resource access, function permissions, and instructions arriving through external content. Disabling remote generation reduces available coverage; the disable flag is not a network-isolation guarantee. For a restricted support dataset, resolve this before promising a fully local test plan.

Self-managed inference does not erase the probe limit. This is a separate limitation from where data goes. The inference documentation explicitly retains usage counting with your own inference configuration. “Open source” should not become an assumption of unlimited hosted attack services.

A scorer can reward the wrong behavior. The worked example catches a literal shipping phrase, not all unsupported delivery claims. An LLM judge adds flexibility but also introduces another model's interpretation. Calibrate the rubric with examples that responsible support staff agree are acceptable and unacceptable.

An answer test can miss an action failure. If your target adapter exposes only final text, the suite cannot establish from that text alone whether the agent used the right account or changed the correct record. Add action evidence or application-state verification wherever success involves a tool.

The operational work remains yours. Someone must maintain cases, investigate findings, and decide what blocks launch. Buying a managed dashboard can make that work easier to coordinate; it cannot supply the refund policy or customer-access rules your tests should enforce.

Promptfoo Alternatives

Braintrust: when shared evaluation work comes first

Braintrust combines tracing, experiments, datasets, and scoring in a shared platform. Choose it when the recurring task is reviewing changes with colleagues and turning observed behavior into evaluation data. Its scope makes it a stronger initial shortlist entry for that workflow than a primarily configuration-led process.

Starter is $0/month, with 1 GB of processed data, 10,000 scores, and 14-day retention. Pro is $249/month, with 5 GB, 50,000 scores, and 30-day retention; Enterprise is custom priced. Starter excess data costs $4/GB and excess scores $2.50/1,000. Pro lists $3/GB and $1.50/1,000, with extended retention charged separately. Braintrust's current plans.

The limitation is the bill's multiple dimensions: scores, processed data, retention, and model usage need a workload estimate. Prefer Promptfoo when the immediate requirement is a small, reviewable test suite and adversarial application testing. Do not buy a shared platform merely to avoid writing expected behavior.

LangSmith: when the agent's execution path is the problem

LangSmith offers tracing, dataset collection, annotation, and online and offline evaluations. Choose it when investigating the chain of events behind an answer is central to your work. A support agent can return a plausible sentence after retrieving the wrong record; an execution trace is valuable evidence for that investigation.

Developer is $0/seat/month, limited to one seat, with 5,000 base traces per month before usage charges. Plus is $39/seat/month, with 10,000 base traces per month and additional seats available. The published allowance is total, not multiplied by the seat count. Enterprise is custom priced. LangSmith pricing.

The limitation for a growing review group is that seats add cost while usage remains metered. Prefer Promptfoo when your immediate goal is an automated release check with explicit adversarial coverage. A LangSmith trace, a Braintrust score, and a Promptfoo probe represent different events; comparing their allowances as though they were equal test runs gives a misleading budget.

Verdict: Adopt It When You Can Own the Tests

Promptfoo earns a place in a support-bot or agent project when you need explicit quality checks and adversarial tests that developers can review and rerun. Start with Community if its data flows, coverage, and operating model fit. Buy managed capabilities when collaboration, controls, or deployment requirements create a concrete need.

The decision flips when the bottleneck changes. If you cannot state what a safe, correct answer looks like, fix the policy and examples first. If you can state it but cannot reproduce failures, Promptfoo is a sensible next step. If you already reproduce failures and the problem is collaborative analysis or execution tracing, evaluate Braintrust or LangSmith alongside it.

Your next move is to save representative support questions, define acceptable answers and forbidden actions, and confirm that a deliberately bad answer fails. Then connect the actual application, inspect its data boundary, and add the security cases that would make you stop a release. Publish the release rule before looking at the scores.

Questions Before You Adopt It

Join the newsletter for practical build logs and field notes on operating AI systems.