We introduce SDAR, a Self-Distilled Agentic Reinforcement learning method.
SDAR is known as the first open-sourced framework that unifies Agentic RL with OP(S)D, providing a codebase that has enabled the following works:
- SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [Paper] [Code]
- OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning [Paper] [Code]
- UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation [Paper] [Code]
- MAPD: From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search [Paper] [Code]
- ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks [Paper] [Code]
- GRSD: Group-Reflective Self-Distillation for Agentic Reinforcement Learning [Paper] [Code]
- OVCSD: From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents [Paper] [Code]
- PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning [Paper]
- StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [Paper]
- CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [Paper]
2026-7-29: 🔥🔥 We released SkillRise, introducing cross-task skill evolution via agentic RL.2026-7-17: 🔥🔥 We released SEED, introducing self-evolving opd based on SDAR.2026-6-25: 🔥 We released OPID, introducing skill evolving based on SDAR.2026-6-22: 🔥 We fixed a bug (issue #35) about AlfWorld teacher skill retrieval problem. Please clone the repo again and have a try.2026-5-15: 🔥 We released our paper and code for SDAR.2026-4: See our previous works Skill0 and Skill1, about lifecycle of agent skills.
SDAR achieves substantial improvements over the standard RL baseline on ALFWorld, WebShop, and Search-QA.
conda create -n sdar python==3.12 -y
conda activate sdar
pip3 install vllm==0.11.0
pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .Log in to Weights & Biases if you use WandB logging (scripts pass trainer.logger=['console','wandb'] in many cases):
export WANDB_API_KEY=your_key_hereInstall with pip:
pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip3 install alfworldDownload PDDL & Game files and pre-trained MaskRCNN detector (will be stored in ~/.cache/alfworld/):
alfworld-download -fWebShop requires Python <=3.10, so begin by creating a new environment:
conda create -n verl-webshop python==3.10 -y
conda activate verl-webshopInstall WebShop:
cd ./agent_system/environments/env_package/webshop/webshop
./setup.sh -d allAfter WebShop is installed, return to the root directory and install the verl package:
cd repo_root/
pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2
# spacy 3.7.2 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.
# weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.15.2 which is incompatible.The warnings can be safely ignored.
cd ./agent_system/environments/env_package/search/third_party
pip install -e .
pip install gym==0.26.2Prepare dataset (data will be saved at ~/data/searchR1_processed_direct):
cd repo_root/
python examples/data_preprocess/preprocess_search_r1_dataset.pySince faiss-gpu is not available via pip, we setup a separate conda environment for the local retrieval server. Running this server will use around 6GB of GPU memory per GPU, so make sure to account for this in your training run configuration. Build Retriever environments:
# Create and activate the retriever environment with Python 3.10
conda create -n retriever python=3.10 -y
conda activate retriever
# Install PyTorch (with GPU support) and related libraries
conda install numpy==1.26.4 # needed to stop incompatible version of numpy from being installed via pip
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
# Install other Python packages
pip install transformers datasets pyserini huggingface_hub
# Install the GPU version of faiss
conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
# Install the API service framework
pip install uvicorn fastapiDownload the index:
conda activate retriever
local_dir=~/data/searchR1
python examples/search/searchr1_download.py --local_dir $local_dir
cat $local_dir/part_* > $local_dir/e5_Flat.index
gzip -d $local_dir/wiki-18.jsonl.gzStart the local flat e5 retrieval server:
conda activate retriever
# redirect the output to a file to avoid cluttering the terminal
# we have observed outputting to the terminal causing spikes in server response times
bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log All scripts live under examples/ and assume the repo root as working directory. You can run e.g.:
bash examples/sdar_trainer/run_alfworld_3b.sh
bash examples/sdar_trainer/run_search_3b.sh
bash examples/sdar_trainer/run_webshop_3b.shOur reproduced codes for GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, and RLSD are also provided in examples/.
See scripts/model_merger.py for FSDP/Megatron merge examples using paths under ./checkpoints/....
If you find this project useful, welcome to cite us.
@article{lu2026sdar,
title={Self-distilled agentic reinforcement learning},
author={Lu, Zhengxi and Yao, Zhiyuan and Han, Zhuowen and Wang, Zi-Han and Wu, Jinyang and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and others},
journal={arXiv preprint arXiv:2605.15155},
year={2026}
}
@article{lu2026skill0,
title={Skill0: In-context agentic reinforcement learning for skill internalization},
author={Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Han, Chengcheng and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
journal={arXiv preprint arXiv:2604.02268},
year={2026}
}
@article{wu2026seed,
title={SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning},
author={Wu, Jinyang and Yang, Shuo and Lu, Zhengxi and Zhang, Fan and Shen, Yuhao and Feng, Lang and Luo, Haoran and Lian, Zheng and Zhang, Shuai and Wen, Zhengqi and others},
journal={arXiv preprint arXiv:2607.14777},
year={2026}
}
@article{yang2026opid,
title={Opid: On-policy skill distillation for agentic reinforcement learning},
author={Yang, Shuo and Wu, Jinyang and Lu, Zhengxi and Shen, Yuhao and Zhang, Fan and Feng, Lang and Zhang, Shuai and Luo, Haoran and Lian, Zheng and Wen, Zhengqi and others},
journal={arXiv preprint arXiv:2606.26790},
year={2026}
}
@article{shi2026skill1,
title={Skill1: Unified evolution of skill-augmented agents via reinforcement learning},
author={Shi, Yaorui and Chen, Yuxin and Lu, Zhengxi and Miao, Yuchun and Liu, Shugui and Gu, Qi and Cai, Xunliang and Wang, Xiang and Zhang, An},
journal={arXiv preprint arXiv:2605.06130},
year={2026}
}
@article{yao2026skillrise,
title={SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution},
author={Yao, Zhiyuan and Chen, Yuxin and Lu, Zhengxi and Xu, Zishan and Sun, Yueqing and Guo, Yifu and Lu, Yuquan and Cai, Zhengzhou and Zhang, Kangning and Han, Zhuowen and others},
journal={arXiv preprint arXiv:2607.26784},
year={2026}
}This project builds on verl-agent, veRL, ALFWorld, SkillRL, and Search-R1. We thank the authors of those projects.