Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models
| Date | Stars |
|---|---|
| 2026-07-31 | 406 |
| 2026-08-02 | 406 |
| 2026-08-06 | 406 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models <div align="center"> [[Website]](http://craftjarvis-jarvis1.github.io/) [[Paper]](https://arxiv.org/abs/2311.05997) [[Twitter]](https://twitter.com/jeasinema/status/1723900032653643796)  </div> ## Abstract Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. We introduce **JARVIS-1**, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop **JARVIS-1** on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit **JARVIS-1** with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. **JARVIS-1** is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". **JARVIS-1** performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of ObtainDiamondPickaxe, **JARVIS-1** surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. ## Agent Playing Videos We list a series of videos showing **JARVIS-1** playing Minecraft. You can find the videos on our [Project Page](http://craftjarvis-jarvis1.github.io/). ## Install Dependencies This project is intended for running on Linux only. Support for other platforms is not provided. ### Prepare the Environment We recommend using Anaconda to manage the environment. If you don't have Anaconda installed, you can download it from [here](https://www.anaconda.com/products/distribution). ```bash conda create -n jarvis python=3.10 conda activate jarvis ``` Make sure you have JDK 8 installed. If you don't have it installed, you can install it using the following command: ```bash conda install openjdk=8 ``` To check your JDK version, run the command `java -version`. You should see a message similar to the following (details may vary if you have installed a different JDK distribution): ```bash openjdk version "1.8.0_392" OpenJDK Runtime Environment (build 1.8.0_392-8u392-ga-1~20.04-b08) OpenJDK 64-Bit Server VM (build 25.392-b08, mixed mode) ``` Once you have installed the required dependencies, you can run the `prepare_mcp.py` script to build MCP-Reborn. Make sure you have a stable internet connection before you begin. ```bash python prepare_mcp.py ``` Then you can install JARVIS-1 as a Python package. ```bash pip install -e . ``` <!-- <aside> JARVIS-1 relies on gym==0.23.1, while mineclip and minedojo depend on a different version. If you encounter any errors related to gym versions during installation, you can safely ignore them. </aside> --> ### Download Weights JARVIS-1 relies on the weights of STEVE-I. You can download the weights from the [script](https://github.com/Shalev-Lifshitz/STEVE-1/blob/main/download_weights.sh). Then you need to set the weights path in the file `jarvis/steveI/path.py`. <!-- Some controller weights from GROOT have not been released yet. We will release them in the future. --> <!-- You also need to download our multimodal memory from the [huggingface link](https://huggingface.co/zhwang4ai/jarvis_memory). --> ## Usage You need to set the environment variable `TMPDIR` and `OPENAI_API_KEY` first. ```bash export TMPDIR=/tmp export OPENAI_API_KEY="sk-******" ``` ### Learning with dynamic memory
Excerpt of 6,782 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:77c84284ade5ad9d, topic:language-model