Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Promptimize is a prompt engineering evaluation and testing toolkit.
| Date | Stars |
|---|---|
| 2026-07-31 | 496 |
| 2026-08-05 | 496 |
| 2026-08-06 | 496 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# 💡 ¡promptimize! 💡
[](https://opensource.org/licenses/Apache-2.0)
[](https://badge.fury.io/py/promptimize)
<img src="https://user-images.githubusercontent.com/487433/229948453-36cbc2d1-e71f-4e87-9111-ab428bc96f4c.png" width=300/>
Promptimize is a prompt engineering **evaluation** and **testing** toolkit.
It accelerates and provides structure around prompt engineering at scale
with confidence, bringing some of the ideas behind test-driven
development (TDD) to engineering prompts.
With promptimize, you can:
- Define your "prompt cases" (think "test cases" but specific to evaluating
prompts) as code and associate them with evaluation functions
- Generate prompt variations dynamically
- Execute and rank prompts test suites across different
engines/models/temperature/settings and compare results, brining
the hyperparameter tuning mindset to prompt engineering
- Get reports on your prompts' performance as you iterate. Answer question
around how different prompt suites are performing against one-another.
Which individual cases or categories of cases improved? regressed?
- Minimize API calls! only re-assess what changed as you change it
- Perform human if and where needed, introspected failed cases, overriding
false negatives
In essence, promptimize provides a programmatic way to execute and fine-tune
your prompts and evaluation functions in Python, allowing you to iterate
quickly and with confidence.
## Hello world - the simplest prompt examples
[more examples on GitHub](https://github.com/preset-io/promptimize/tree/main/examples)
```python
# Brining some "prompt generator" classes - note that you can derive and extend those
from promptimize.prompts import PromptCase
# Bringing some useful eval function that help evaluating and scoring responses
# eval functions have a handle on the prompt object and are expected
# to return a score between 0 and 1
from promptimize import evals
# Promptimize will scan the target folder and find all Prompt objects
# and derivatives that are in the python modules
simple_prompts = [
# Prompting "hello there" and making sure there's "hi" or "hello"
# somewhere in the answer
PromptCase("hello there!", lambda x: evals.any_word(x, ["hi", "hello"])),
PromptCase(
"name the top 50 guitar players!", lambda x: evals.all_words(x, ["frank zappa"])
),
]
```
### The CLI's `run` command
```
$ promptimize run --help
Usage: promptimize run [OPTIONS] PATH
run some prompts
Options:
-v, --verbose Trigger more verbose output
-f, --force Force run, do not skip
-h, --human Human review, allowing a human to review and force
pass/fail each prompt case
-r, --repair Only re-run previously failed
-x, --dry-run DRY run, don't call the API
--shuffle Shuffle the prompts in a random order
-s, --style [json|yaml] json or yaml formatting
-m, --max-tokens INTEGER max_tokens passed to the model
-l, --limit INTEGER limit how many prompt cases to run in a single
batch
-t, --temperature FLOAT max_tokens passed to the model
-e, --engine TEXT model as accepted by the openai API
-k, --key TEXT The keys to run
-o, --output PATH
-s, --silent
```
Let's run those examples and produce a report `./report.yaml`
```
$ promptimize run examples/ --output ./report.yaml
```
```yaml
💡 ¡promptimize! 💡
# ----------------------------------------
# (1/2) [RUN] prompt: prompt-115868ef
# ----------------------------------------
key: prompt-115868ef
user_input: hello there!
prompt_hash: 115868ef
response: Hi there! How are you doing today?
execution:
api_call_duration_ms: 883.8047981262207
run_at: '2023-04-25T02:21:40.443077'
score: 1.0
# ----------------------------------------
Excerpt of 10,590 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:df977824234f63d1, desc:prompt engineering