Promptfoo

Just shipped: Promptfoo can chain jailbreaks together to generate novel attacks

Just shipped: detection for cross-session data leaks in LLM-based apps. This is a pretty simple test, but an important one - many agentic systems are stateful. Now we can test automatically for this type of failure, applying jailbreaks/injections to retrieve data even when normal methods cannot. If you're thinking about AI security failures, check out Promptfoo - vulnerability scanner that runs locally and is open-source.

Just shipped a plugin that tests for LLM hijacking/data exfiltration using ASCII smuggling (invisible unicode). These types of attacks are interesting because they can be used to circumvent human-in-the-loop mitigations. For example: - Invisible instructions embedded in the content of a webpage that, when pasted into an internal system, hijacks the session to call tools/db/exfiltrate, etc - Invisible instructions embedded in a document that, when loaded in a RAG architecture, hijacks or manipulates the result - Invisible data leak when paired with some exfiltration tactic (e.g. link unfurling, image previews, etc) - Invisible data generated by an LLM to fingerprint outputs Some but not all major chat interfaces strip these characters. If you're feeding UGC directly to an inference API this is something you should be aware of. Attached image outlines a basic example :)

Shipped: conversational red teaming for LLMs Automated chat that probes an AI's boundaries and constraints. It starts with an innocent chat related to a sensitive topic, then subtly increases specificity to guide the AI into a successful attack. It exploits three common problems: 1. LLMs tend to be more compliant if they've already helped the user 2. LLMs are vulnerable when they are guided step-by-step into ethical and security gray areas 3. LLMs tend to drift from their system prompts during a long chat. As AIs become more sophisticated, so must our testing methodologies. Multi-turn attacks like this one help to make conversational AI more robust.

Just shipped: tree-based method for jailbreaking LLMs. This pentest technique finds the most effective way to bypass AI safeguards by mutating a malicious prompt. For security professionals, this strategy can automate a lot of grunt work and put a spotlight on the real risks. The best news - it's free and open source. Try it here: https://lnkd.in/gKewZUJh

Announcing three new red teaming plugins for LLM agents with access to internal APIs: 🔒 Unauthorized data access (Broken Object Level Authorization) ⬆️ Privilege Escalation (Broken Function Level Authorization) 🌐 Malicious resource fetching (Server-Side Request Forgery) They work by: 1. Targeting specific systems within your application’s infrastructure 2. Using "social engineering" tactics optimized for LLMs 3. Generating diverse adversarial inputs and running them through the agent Read more about how to red team gen AI systems: https://lnkd.in/g2Nb8gFe

Had a lovely time discussing the finer points of AI security with Anjney Midha on Andreessen Horowitz's ai + a16z podcast. Promptfoo is bringing AI red team capabilities to the masses with the first open-source tool for LLM application security. CHECK IT OUT: https://lnkd.in/gs3T4Z-A

Excited to announce that Promptfoo has raised a $5M seed round from Andreessen Horowitz and other industry leaders to find and fix vulnerabilities in AI apps. AI security is broken. At Discord, I shipped generative AI to 200M users and tackled the unique risks of LLMs firsthand. The attack surface is massive and there are few tools or best practices. Promptfoo is the first product to adapt AI-specific pentesting techniques to your application. This helps you address the AI vulnerabilities that matter most to your business – like data leaks and insecure integrations – before they are shipped to users. Today, over 25,000 developers at companies like Shopify, Amazon, and Anthropic are fortifying their apps with our powerful open-source tool for evaluating AI behavior. Learn more about how to secure your applications below! https://lnkd.in/gFiE3hv7 Anjney Midha Zane Lackey Joel D. Adam Ely Frederic Kerrest Stanislav Vishnevskiy David Schellhase Michael D'Angelo Gregory Chang

We all knew AI image generators can be manipulated into generating violent and graphic content... But it turns out the jailbreak process can be *fully automated* Write-up here (open-source) with examples from OpenAI's Dall-E https://lnkd.in/gq-XqRKG

New in Promptfoo in the last 2 weeks: 🔍 Advanced Red-Teaming Capabilities: We’ve added support for image model red-teaming and made it easier to add new plugins, giving you more flexibility in your security testing. 🛠️ Enhanced Developer Tools: New features like support for Gemini embeddings, markdown tables in the web UI, and improved support for AWS Bedrock. ✅ Improved Reliability: We’ve fixed several issues, including more robust JSON extraction, correct response formats, and better handling of environment variables. Huge thanks to our community for their contributions. Your input helps us continuously improve!