| Version | Date | Update |
|---|---|---|
 |
2026-07-5 | Supports benchmarking for both the policy loop and target native builtin paths; Incorporates agent verification and failure recovery server |
 |
2026-06-27 | Support for Behavior 1K; SessionVerfier for Agent verification; VerifySessionTool |
 |
2026-06-11 | Cleaned protocol files and docs; game scenario separated to general-game-agent branch; main branch now focused on sim & real |
 |
2026-06-5 | Optimize the user-friendly onboarding process; Communication Protocol Specification; More reasonable coding standards; Game Agent & Benchmarking ready |
 |
2026-05-25 | Strict separation of PolicySkillRuntime / BuiltinSkillRuntime; Game Agent & Benchmarking ready |
 |
2026-05-20 | Perception plugin system: SensorConfig / PerceptionConfig YAML + EnvironmentWriter auditable writeback |
 |
2026-05-18 | Session-Centered Runtime MVP: DummySimTarget + DummyAdapter + DummyClient serial pipeline |
 |
2026-04-29 | Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline |
Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:
| 🔌 | One Codebase, Any Hardware — Adding a new robot means implementing one Target Adapter (~100 lines); zero changes to the scheduling layer. |
| 🛡️ | Three Safety Layers — Critic validation → Strict Preflight → Target-side SafetyGuard; mandatory for real-robot deployment. |
| 📋 | Fully Auditable — State, actions, and perception results are written to Markdown + YAML files; every step is traceable and reproducible. |
| 🔄 | Zero-Friction Migration — The same Session protocol runs identically across sim and real targets. |
▲ Session-Centered Runtime Architecture Overview
| 🔄 | Session-Centered Runtime | WatchdogSupervisor → SessionRunner → SkillRuntime → TargetSessionHandle execution pipeline, replacing the legacy Driver-Center architecture |
| 🎯 | Target-Configured | Three target kinds — debug / simulation / real_robot — registered in TARGETS.md, adapters attached on demand |
| 🧩 | Adapter + Bridge | TargetAdapter + PolicyAdapter + ActionBridge three-way decoupling with explicit observation/action contracts; AdapterPlan auto-composed, eliminating target×skill combinatorial explosion |
| ⚡ | Dual Skill Runtimes | PolicySkillRuntime maintains the policy closed loop + BuiltinSkillRuntime manages built-in loops such as target-native benchmarks and constrained agent interaction |
| 🛡️ | Strict Preflight | Runtime validation checks (target / sensor / perception / adapter contract / action contract / tool); failures are rejected before execution starts |
| 📝 | File Protocol Matrix | TARGETS.md · SKILLRUNTIME.md · SESSIONS.md · ENVIRONMENT.md · LESSONS.md + external YAML configs |
| 🔐 | Multi-Layer Safety | Critic validation → Preflight contract checks → Target-side SafetyGuard → Operator Override |
| 🌐 | Fleet Mode | Multi-robot coordination with shared + per-robot workspaces, priority-based serial scheduling |
| 🖥️ | Built-in TUI | paos tui — full-screen interface with tiling chat, live status/logs panes, provider & settings management, and Morandi themes |
| 1 |
Install git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS
pip install -e . # Python ≥ 3.11
pip install -e ".[dev]" # Dev dependencies
|
| 2 |
Initialize Workspace paos onboard |
| 3 |
Start Agent paos agent # CLI chat
paos tui # full-screen TUI |
| 4 |
Optional: Connect Runtime Services The following example evaluates the official PI0.5 policy on the
# Terminal 1: LIBERO TargetWS
MUJOCO_GL=egl PYTHONWARNINGS=ignore \
conda run --no-capture-output -n libero \
python PhyAgentOS/runtime/targets/remote/libero/server.py \
--host 0.0.0.0 --port 9002 \
--camera-height 256 --camera-width 256 \
--max-steps 300 --num-steps-wait 10 \
--control-mode relative --seed 7
# Terminal 2: official OpenPI PI0.5 policy server
conda run --no-capture-output -n openpi \
python -m PhyAgentOS.runtime.policy.openpi.native_openpi_server \
--policy-config pi05_libero \
--checkpoint-dir gs://openpi-assets/checkpoints/pi05_libero \
--host 0.0.0.0 --port 8000
# Terminal 3: Agent, Watchdog, and Verification Service
paos agent --workspace ~/.PhyAgentOS/workspace -m \
"Evaluate PI0.5 on LIBERO suite libero_spatial. Use target libero_real_remote, the libero_target_benchmark builtin runtime, target_native execution, recovery verification, task ids 0-9, init-state ids 0-49, max_steps 300, replan_every_steps 5, and policy endpoint openpi://127.0.0.1:8000." |
Before starting Terminal 3, enable libero_real_remote, configure Runtime
verification, and set the Target configuration to
retry_instruction_mode: original. The latter keeps the original task
instruction on recovery attempts; use verifier_rewrite when the policy should
receive the verifier's nonempty rewrite instead. See the
Runtime configuration reference
for the complete global, Target, SkillRuntime, Session, benchmark,
verification, and remote-host parameter reference. The LIBERO setup is
included as an example.
| Context Loading | File | Owner | Purpose |
|---|---|---|---|
| Always loaded into the agent system prompt | AGENTS.md |
Agent workspace | Project-level operating rules for the agent |
| Always loaded into the agent system prompt | SOUL.md |
Agent workspace | Identity, high-level behavior, and assistant style |
| Always loaded into the agent system prompt | USER.md |
Agent workspace | User preferences and durable profile notes |
| Always loaded into the agent system prompt | TOOLS.md |
Agent workspace | Tool usage policy and available tool guidance |
| Always loaded into the agent system prompt | SKILLS.md |
Agent workspace | Agent-facing skill discovery and loading rules |
| Loaded when present; filtered by enabled runtime targets where applicable | EMBODIED.md |
Agent workspace | Human-readable target capability descriptions |
| Loaded when present as state, not bootstrap policy | ENVIRONMENT.md |
Agent/runtime workspace | Current target and scene/environment state |
| Loaded when present as memory/state | LESSONS.md |
Agent workspace | Operational lessons and failure notes |
| Loaded when present as task state | TASK.md |
Agent workspace | Multi-step task decomposition and progress |
| Runtime protocol; read before scheduling sessions | RUNTIME.md |
Runtime workspace | Instructions for writing valid runtime sessions |
| Runtime protocol; read before scheduling sessions | TARGETS.md |
Runtime workspace | Enabled targets, endpoint/adapter/config references, supported skill runtimes |
| Runtime protocol; read before scheduling sessions | SKILLRUNTIME.md |
Runtime workspace | Policy/builtin skill runtime registry and execution contracts |
| Runtime queue/state; written by Agent and watchdog | SESSIONS.md |
Runtime workspace | Pending/running/completed execution sessions and results |
SKILLS.md is for agent capabilities and skill discovery. SKILLRUNTIME.md is
for runtime execution contracts; it is paired with TARGETS.md and SESSIONS.md.
PhyAgentOS/
│
├── PhyAgentOS/agent/ # Track A ─ Planner / Critic / Memory
│
├── PhyAgentOS/runtime/ # Track B ─ Execution Plane
│ ├── watchdog/ # WatchdogSupervisor
│ ├── sessions/ # SessionRunner / TargetSessionHandle
│ ├── targets/ # RolloutTarget (debug·sim·real)
│ │ └── remote/libero/ # LIBERO benchmark TargetWS server + proxy
│ ├── skillruntime/ # PolicySkillRuntime / BuiltinSkillRuntime
│ ├── adapters/ # TargetAdapter / PolicyAdapter / Bridge
│ │ ├── libero/ # LIBERO target adapter
│ │ └── openpi/ # OpenPI policy adapters
│ ├── policy/openpi/ # OpenPI client + LeRobot pi0-family server
│ ├── perception/ # Perception Runtime / EnvironmentWriter
│ ├── preflight/ # RuntimeCompatibilityPreflight
│ └── schemas/ # Pydantic Schema
│
├── configs/runtime/ # Sensor / Perception / Contract YAML
├── scripts/ # Utility scripts
├── workspace/ # Agent workspace; runtime files may share it by config
├── docs/ # Documentation
└── tests/ # Tests
| Kind | Location | Examples | |
|---|---|---|---|
| 🐛 | debug |
Local | echo / mock / dry-run — zero-hardware protocol pipeline validation |
| 🧪 | simulation |
Remote | RoboCasa, LIBERO — benchmark evaluation & batch experience mining |
| 🤖 | real_robot |
Remote | Franka, Go2, XLeRobot, AgileX PIPER — real-world deployment |
All targets are registered in
TARGETS.md, identified bytarget_adapter://URI. More examples & demos → Project Website
| Document | Audience | Description |
|---|---|---|
| 🌐 Website | Everyone | Full docs, architecture details, demos |
| 📘 User Manual | Users | Installation, deployment, and operation guide |
| 📙 Dev Guide | Developers | Secondary development, hardware integration, plugin authoring |
PRs and Issues are welcome! Check our development roadmap here → Dev Plan.
Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory & X-Era Lab
MIT License · Copyright © 2025-2026 PhyAgentOS
