npm / promptfoo
promptfoo
What changed in promptfoo 0.123 to 0.124, from its official release notes, upgrade guides and advisories. Sources checked 2026-10-19.
23 breaking4 security17 new APIs and options1 performance10 behavior and defaults52 fixes and other
See which of these apply to your code, ranked, in your agent:
npx @unstale/cli initpromptfoo 0.12473 changes
Breaking changes 18
Browser SDKs now require explicit installation.
Browser SDKs now require explicit installation.
The Codex Security SDK is now opt-in rather than included by default.
The Codex Security SDK is now opt-in rather than loaded by default.
The hosted ChatKit provider has been removed.
The hosted ChatKit provider was removed.
The Langfuse SDK is now opt-in rather than included by default.
The Langfuse SDK is now opt-in rather than loaded by default.
Local Transformers is now opt-in rather than included by default.
Local Transformers is now opt-in rather than loaded by default.
OpenAI Agents now requires an explicit install.
OpenAI Agents now requires explicit installation.
The Slack SDK is now opt-in rather than included by default.
The Slack SDK is now opt-in rather than loaded by default.
The CLI now returns a failing exit code when a target error stops an evaluation.
Invalid VLSU concurrency values are now rejected.
WatsonX SDKs are now opt-in rather than included by default.
WatsonX SDKs are now opt-in rather than loaded by default.
Security 1
Red-team stops Meta on invalid JSON requests.
New APIs and options 11
Exposes text-token pricing from the provider catalog.
Adds support for OpenAI Decisions.
Claude Opus 5.5 is now supported as a provider model.
Paused Claude server-tool turns can now be resumed.
Claude Sonnet 5.5 is now supported as a provider model.
Coding agents can now run in isolated workspaces.
GPT-6 Sol and Luna are now supported provider models.
GPT-6.1 Sol and its ultrafast variant are now supported.
Grok 4.7 is now supported as a provider model.
OpenAI safety identifiers are now supported by providers.
A TypeSafe Jev provider is now available.
Performance 1
Indexes extracted tests when filtering evaluation results.
Behavior and default changes 7
Assertions can record cost without requiring a threshold.
Promptfoo now warns when Azure service principal configuration is partial.
Bedrock GPT-6 Sol and Luna are now treated as reasoning models.
Claude providers now report usage per call.
Metadata filtering now supports comma-separated OR values.
Redteam reports which turn the grader flagged.
GPT-6 Chat is no longer capped by OPENAI_MAX_TOKENS.
Fixes and other changes 35
Fixes handling of assertion comments.
Preserves grading for asynchronous expressions.
Preserves Cloud errors.
Retains the prepared custom environment.
Fixes handling of dataset limits.
Samples dataset plugin rows uniformly.
Evaluation now respects the #gid in Google Sheets URLs.
Exposes incomplete Responses status from providers.
Preserves complete JSON objects.
Preserves grading for keyword properties.
Red-team runs report missing target output as errors.
Adds a check for npm aliases.
Code-scan OIDC tokens are minted after CLI installation.
Preserves regex operands after yield.
Array sampling no longer has bias.
Assertions now accept grader result shapes that earlier releases recorded.
Bedrock providers now honor region overrides.
Blank BLEU references are ignored.
Vertex now preserves zero Claude sampling parameters.
CommonJS consumers can now use the package API.
Custom grading results are now validated.
Evaluation configs downloaded from the web UI remain importable.
ElevenLabs TTS now respects the configured cache policy.
The --env-file option and config environment are now honored for prompt separator and assertion concurrency.
Configuration now accepts numeric and boolean environment values.
A literal comma is now allowed in metadata filter values.
Inverse RAG scores are clamped to the documented 0–1 range.
The not- prefix is now honored for model-graded-closedqa and factuality assertions.
Null Ollama tool arguments are now treated as empty.
Ollama tool calls are accepted back in multi-turn conversations.
OpenAI Agents token counts are now validated.
Providers now honor cache overrides supplied per call.
Redteam now preserves zero sampling values.
Iterative, crescendo, and iterative:tree now honor an explicitly configured redteamProvider.
Vercel provider response handling was simplified.
promptfoo 0.12334 changes
Breaking changes 5
credit_balance_exhausted is now treated as a hard quota error.
Ollama embeddings now use /api/embed and honor configuration.
RAG assertion thresholds now default to 0.5.
Inverting a search-rubric assertion now inverts its score along with its pass result.
Non-string search rubrics are now rejected.
Security 3
Agents endpoint credentials are isolated.
Custom header requests have been hardened.
Local configuration is no longer leaked into provider request bodies.
New APIs and options 6
A Gemini 3.8 provider is available for use in evaluations.
Providers now support GPT-Live voice sessions.
A provider for the OpenAI Agents API is now available.
Portable HTTP configuration schemas are available for providers.
Portable MCP configuration schemas are available for providers.
Vertex Live is available as a provider.
Behavior and default changes 3
Gemini provider tool and media support has been updated.
Providers support Ollama 0.34 features.
Ollama provider results now surface thinking output and the finish reason.
Fixes and other changes 17
Image loading now supports AVIF and TIFF, and SDK options are honored.
The normalized Envoy base URL is applied after the user configuration spread.
MCP now accepts fine-tuned model identifiers.
Foundry overrides are now honored.
Prompt Gemini tool policy now takes precedence over passthrough.
The Helicone key now honors --env-file and config env.
The Langfuse base URL now honors --env-file and config env.
Default MCP enablement is preserved.
Grader failures are no longer inverted for not-classifier assertions.
Grader failures are no longer inverted for not-search-rubric assertions.
Nova Sonic providers now honor inference configuration.
Ollama HTTP errors are surfaced instead of being returned as empty output.
Streamed Ollama tool calls are now preserved.
CLI result export now handles large values.
Snowflake provider configuration is now forwarded.
Resolved TogetherAI model parameters are no longer overwritten.
Malformed webhook results are now rejected.