Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. 🏆 Best Paper Awards @ NeurIPS ML Safety Workshop 2022
| Date | Stars |
|---|---|
| 2026-07-31 | 514 |
| 2026-08-05 | 515 |
| 2026-08-06 | 515 |
| 2026-08-18 | 517 |
| 2026-08-21 | 518 |
| 2026-08-24 | 519 |
| 2026-08-27 | 520 |
| 2026-08-30 | 521 |
| 2026-09-06 | 522 |
| 2026-09-10 | 523 |
| 2026-09-13 | 524 |
| 2026-09-17 | 525 |
| 2026-09-18 | 524 |
| 2026-09-20 | 524 |
Today
— stars today
This week
— stars this week
This month
+6 stars this month
Momentum
0.0
growth rate 0.00%/day
# PromptInject
[**Paper: Ignore Previous Prompt: Attack Techniques For Language Models**](https://arxiv.org/abs/2211.09527)
## Abstract
> Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PROMPTINJECT, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3’s stochastic nature, creating long-tail risks.
Figure 1: Diagram showing how adversarial user input can derail model instructions. In both attacks,
the attacker aims to change the goal of the original prompt. In *goal hijacking*, the new goal is to print
a specific target string, which may contain malicious instructions, while in *prompt leaking*, the new
goal is to print the application prompt. Application Prompt (gray box) shows the original prompt,
where `{user_input}` is substituted by the user input. In this example, a user would normally input
a phrase to be corrected by the application (blue boxes). *Goal Hijacking* and *Prompt Leaking* (orange
boxes) show malicious user inputs (left) for both attacks and the respective model outputs (right)
when the attack is successful.
## Install
Run:
pip install git+https://github.com/agencyenterprise/PromptInject
## Usage
See [notebooks/Example.ipynb](notebooks/Example.ipynb) for an example.
## Cite
Bibtex:
@misc{ignore_previous_prompt,
doi = {10.48550/ARXIV.2211.09527},
url = {https://arxiv.org/abs/2211.09527},
author = {Perez, Fábio and Ribeiro, Ian},
keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Ignore Previous Prompt: Attack Techniques For Language Models},
publisher = {arXiv},
year = {2022}
}
## Contributing
We appreciate any additional request and/or contribution to `PromptInject`. The [issues](/issues) tracker is used to keep a list of features and bugs to be worked on. Please see our [contributing documentation](/CONTRIBUTING.md) for some tips on getting started.
Excerpt of 3,786 characters
Read on GitHub
Fábio Perez · Brazil
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:595e2f7b0288e60e, topic:large-language-models
matched fp:595e2f7b0288e60e, topic:prompt-engineering
matched fp:595e2f7b0288e60e, topic:ai-safety