liten-vla

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #golang #android #deep-learning #macos #nodejs #awesome #cli #kubernetes #vue #pytorch

(★ 27)

LITEN: Learning from Inference Time Execution for VLAs

liten-vla Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LITEN Companion Code

This repository provides the code used in [Learning from Inference-Time Execution (LITEN)](https://arxiv.org/abs/2510.19752). We provide instructions on how to adapt the code to your own hardware setup, details on the prompts used in our experiments on a DROID-like robot setup, and some pre-saved examples for you to try out LITEN yourself.

## Installation

Our implementation used a miniconda Python environment with Python 3.10.16.

Install the packages in `setup.py` with the command:
```
uv run python3 -m pip install -e . 
```
`uv` is STRONGLY recommended for setup. You can also use a .venv instead of miniconda.
 
Once everything is installed, you should make sure your API keys are added in the `utils` folder. For OpenAI, create a .txt file called `utils/openaikey.txt` and directly copy and paste your key in there. With your keys added, you can then start running our methods.

## Using LITEN on your own hardware setup

In order to run LITEN on your hardware, you'll need to provide an interface for the method to provide instructions to the VLA-controlled robot, and then reset the hardware in between executions. In `world_stubs.py`, we provide a base template for this interface (that we subclassed and used in our experiments with the Pi05-DROID VLA). In this file, you should subclass `BaseWorldStub` and implement the `.act(...)` and `.reset(...)` methods as instructed in that file.

Then, in `run.py`, you should implement the function `setup_world(...)` so that at the beginning of any hardware-based method, the interface to the robot will be established and usable. You may want to add arguments to the top-level methods in `run.py` and pass those arguments to `setup_world(...)` to more easily configure your interface.

Once you've set your hardware interface up, you can run LITEN with the following command:
```
python3 run.py liten <experiment_name> 
```

You'll then be prompted to provide a task instruction for LITEN to solve. From there, you'll reset your environment to the proper initial configuration and then let LITEN generate a candidate plan to execute (in the form of python code.) Once the plan has been executed, LITEN will go through each subtask that was executed in your plan and assess it. Those assessments will be saved in a directory and added in-context to future plan generations. We provide a default directory `liten_context` that starts out empty. You can provide your own path to a directory instead, but that directory must contain subdirectories in the proper format for LITEN to be able to process.

To run baselines, the procedure is the same as above, but you'll need to provide the directory to past experience yourself (which can of course be an empty directory).

```
python3 run.py <ablation_name> <experiment_name> <path_to_experience_directory> 
```

## Additional details on prompting for in-context adaptation for your own system

Here, we go into detail on each prompt we use and the level of specificity to your hardware setup that the prompt should contain.

### Generating Plans
We use the following prompts, all of which are in `vlm_hl/prompts/plan_reasoning`:

* `generate_program_with_context.txt`: This top-level prompt contains system-level instructions for LITEN to generate plans. It importantly provides the programmatic API we use in `world_stubs` for the program-plans to invoke in the VLM-generated code. The API contains a few more methods than we use in our evaluations. These features are purely experimental but we include them in case you'd like the additional expressivity. Within this prompt, we refer to our specific hardware setup (2-pronger gripper arm) and lightly reference the VLA's imperfection (these are both in the description of `world.act(...)` in the prompt). These would be the only things you'd have to change if you intend to use LITEN on a non-arm setup or with a VLA that somehow doesn't fit this generic description.
    * `pizerofive_droid_instruction_space.txt`: This prompt contains guidelines to the VLM on how to phrase instructions for the VLA. This prompt is specific to the complexity and wording of DROID tasks. If you are not using a DROID-finetuned VLA (specifically, pi05-DROID), you should write a new version of this prompt that characterizes as accurately as possible the instructions that your VLA can reasonably expect based on its pretraining (or finetuning) data.
    * `format_task.txt`: This prompt formats the provided task instruction and should be hardware-agnostic.
        * `droid_specific_instruction_space.txt`: This prompt contains task setup-specific details for what the general space of subtask instructions should include. Our version of this prompt is very general, since we are only interested in pick-and-place-like subtasks. If you have a broader set of possible subtasks this prompt should be adjusted accordingly. If you fine-tuned your model on a specific set of instructions, and want to use only those exact instructions, you can provide them line-separated in this file, and the VLM will produce similar (if not the same) instructions.
    * `generate_uids.txt`: This prompt asks the VLM to identify a list of objects, that is then provided to the program-plan generator. We found that this grounding step is very important for correctness and consistency in plan generation. As such, our prompt does mention setup-specific details (a tabletop robot arm), so if not on a tabletop arm this prompt should change accordingly.

### Assessing executions
We use the following prompts for assessment, all of which can be found in `vlm_hl/prompts/assessment`. In our paper's experimental evaluations, we perform tasks and subtasks that are (mostly) able to be assessed just from the start and end images of their executions. We emphasize that if you want to use LITEN for more complicated tasks and subtasks, start and end images *may be insufficient.* In this case, you'll have to provide some form of video to the VLM for assessment, but keep in mind that your VLM may be arbitrarily poor at comprehending these videos. We provide limited support for videos in this implementation if you'd like to try it out.

All of these prompts do make explicit mention of our setup, albeit in a fairly general way ("we use a robot arm on a tabletop"). So each prompt below does need to be adjusted to describe your setup if not a tabletop robot arm. If you need to adjust further, we will mention it below.

* `determine_success_from_images.txt`: This prompt determines whether a given subtask has been successful.
* `critique_failure_from_images.txt` (and `critique_failure_from_video.txt`): This prompt determines what happened instead of the intended subtask instruction, if that subtask instruction was deemed a failure by the previous prompt.
* `reason_about_failure.txt`: This prompt asks the VLM to come up with possible reasons for the intended and actual outcome of a subtask being different. In order to do so, we provide it with a description of VLAs (both in general and what we use):
    * `pizerofive_droid_vla_model_card.txt`: This prompt provides details to the VLM on what VLAs are, how they're trained, and how they're used. The intention is to set the context to be able to reason about how they may fail. We additionally provide a few sentence description about the particular VLA we use, in this case pi05-DROID (which is open-source.) You should modify this prompt to describe your VLA's capabilities, although we expect the description for other similar tabletop VLAs to be quite similar to what we provide.
* `describe_success_scene.txt`: This prompt asks the VLM to briefly describe the environment a subtask was executed in if that subtask instruction was deemed a success by the first prompt.
* `describe_hl_success_scene.txt` and `critique_hl_failure_scene.txt`: These prompts operate at the top-level task and will take as input the results of all the aforementioned prompts, which are subtask-specific. It will additionally use the model card prompt described above, and give a brief assessment as to why the overall task failed or succeeded.

Lastly, we provide the `liten_examples_card.txt` prompt (in `vlm_hl/prompts/plan_reasoning`), which instructs the VLM on how to use the previous experience that is given in-context. This method is broadly general and serves as the fulcrum for connecting the assessment and reasoning phases.

## Try out LITEN on our provided examples

To see how providing different experiences can be used by LITEN to affect its ability to generate plans, we provide a few examples from our experiments for you to try out. You can find these directories in `liten/<experiment_name>/` and can be used in the following way:

```
python3 run.py test-liten <path_to_examples_dir> 
```

For example, you can run the above and set `<path_to_examples_dir>` to `liten_examples/emptying/three_execution_examples` and check out what gets generated. We provide some examples with fewer examples to see what kind of 'incorrect' plans get generated during iterative improvement. The examples we provide that have the most examples for each task will lead to a correct program-plan being generated (most of the time, since VLMs can be a little random it may not always be perfect.) But we encourage you to play around with it and see!

liten-vla

liten-vla Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

liten-vla

liten-vla Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README