Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LLM Agent and Evaluation Framework for Autonomous Penetration Testing
| Date | Stars |
|---|---|
| 2026-07-31 | 310 |
| 2026-08-06 | 310 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
The paper can be found on [arXiv](https://arxiv.org/abs/2412.01778).
## Introduction
<img align="left" style="width: 160px;" src="assets/logo.gif" alt="HackSynth Logo"/>
We introduce HackSynth, a novel Large Language Model (LLM)-based agent capable of autonomous penetration testing.
HackSynth's dual-module architecture includes a Planner and a Summarizer, which enable it to generate commands and process feedback iteratively.
To benchmark HackSynth, we propose two new Capture The Flag (CTF)-based benchmark sets utilizing the popular platforms PicoCTF and OverTheWire.
These benchmarks include two hundred challenges across diverse domains and difficulties, providing a standardized framework for evaluating LLM-based penetration testing agents.
<br>
## Using the repository
- You will have to create a Hugging Face and a Neptune.ai account
- Copy your API keys to the `.env` file, and set the desired CUDA devices, based on the `.env_example`
- [Set up the PicoCTF benchmark](picoctf_bench/README.md)
- [Set up the OverTheWire benchmark](overthewire_bench/README.md)
- Start the HackSynth Agent
- Install the environment:
```
python -m venv cyber_venv
source cyber_venv/bin/activate
pip install -r requirements.txt
```
- Start the benchmark with the following:
```
python run_bench.py -b benchmark.json -c config.json
```
The `benchmark.json` should be one of the generated `benchmark_solved.json` files, or an equivalently structured file.
The configuration files used by us for the measurements in the paper are also available in the configs folder.
## How to Cite
If you use this code in your work or research, please cite the corresponding paper:
```bibtex
@misc{muzsai2024hacksynthllmagentevaluation,
title={HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing},
author={Lajos Muzsai and David Imolai and András Lukács},
year={2024},
eprint={2412.01778},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2412.01778},
}
```
## Contributors
- Lajos Muzsai ([email protected])
- David Imolai ([email protected])
- András Lukács ([email protected])
> 🔍 Also see our related project on reinforcement learning for cryptographic CTFs: [HackSynth-GRPO](https://github.com/aielte-research/HackSynth-GRPO)
## License
The project uses the GNU AGPLv3 license.
Excerpt of 2,503 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6bf9829b3e85a5ae, llm:Repository topics: ai, autonomous-pentesting, ctf, ctf-tools, cybersecurity, llms, penetration-testing; description: 'LLM Agent and Evaluation Framework for Autonomous Penetration Testing'
matched fp:6bf9829b3e85a5ae, llm:Repository topics: ai, autonomous-pentesting, ctf, ctf-tools, cybersecurity, llms, penetration-testing; description: 'LLM Agent and Evaluation Framework for Autonomous Penetration Testing'
matched fp:6bf9829b3e85a5ae, llm:Repository topics: ai, autonomous-pentesting, ctf, ctf-tools, cybersecurity, llms, penetration-testing; description: 'LLM Agent and Evaluation Framework for Autonomous Penetration Testing'