IsaacLab-mlx

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 9 Open GitHub ↗

Apple Silicon port of Isaac Lab with MLX training, Metal hot paths, and a mac-sim runtime for robotics research on macOS.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![MLX](https://raw.githubusercontent.com/RobotFlow-Labs/IsaacLab-mlx/codex/mlx-foundation/assets/mlx_logo_dark.png)

---

# IsaacLab-MLX

[![MLX](https://img.shields.io/badge/backend-MLX-black.svg)](https://github.com/ml-explore/mlx)
[![Apple Silicon](https://img.shields.io/badge/platform-Apple%20Silicon-black.svg)](https://www.apple.com/mac/)
[![Python](https://img.shields.io/badge/python-3.11-blue.svg)](https://docs.python.org/3/whatsnew/3.11.html)
[![License](https://img.shields.io/badge/license-BSD--3-yellow.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![License](https://img.shields.io/badge/license-Apache--2.0-yellow.svg)](https://opensource.org/license/apache-2-0)

**IsaacLab-MLX** is a public fork of [Isaac Lab](https://github.com/isaac-sim/IsaacLab) focused on making the Isaac Lab workflow runnable on Apple Silicon macOS by replacing the CUDA-first training/runtime slice with `MLX + mac-sim`.

The fork is not trying to recreate full Omniverse, RTX, or PhysX GPU parity on macOS. The practical goal is narrower and more useful: preserve the high-level Isaac Lab task and training ergonomics wherever possible, introduce clean backend seams for compute and simulation, and build a Mac-native path that can grow task by task.

## Why This Fork Exists

Upstream Isaac Lab is built around NVIDIA Isaac Sim and a CUDA-oriented runtime. That is the right choice for Linux and NVIDIA GPU deployments, but it leaves Apple Silicon users without a realistic local development path.

This fork exists to close that gap:

- keep Isaac Lab recognizable to existing users
- run the learning stack on Apple Silicon with [MLX](https://github.com/ml-explore/mlx)
- replace import-time CUDA and Omniverse crashes with explicit backend capability checks
- build a Mac-native simulator adapter for the first useful environments instead of waiting for full engine parity

## Current Status

What works today:

- runtime/backend selection seam for `torch-cuda|mlx` and `isaacsim|mac-sim`
- lazy import boundaries in `envs`, `sim`, `assets`, `sensors`, `managers`, `controllers`, `devices`, `scene`, and `markers` so the macOS path fails explicitly instead of exploding on missing `omni.*`
- reproducible source bootstrap script for upstream repositories
- a runnable `MLX + mac-sim` cartpole vertical slice
- cartpole showcase variants covering Box, Discrete, MultiDiscrete, Tuple, and Dict spaces
- a runnable `MLX + mac-sim` cart-double-pendulum MARL slice with dict actions/observations/rewards
- a runnable `MLX + mac-sim` quadcopter slice with root-state dynamics
- MLX training, checkpoint save/load, and replay scripts
- portability guards for optional `torch`/`warp` utility imports on macOS
- smoke tests for the backend seam and mac-native task slices

What this does not claim yet:

- full Isaac Sim compatibility on macOS
- RTX sensors, camera pipelines, Warp kernels, cuRobo, or the full task zoo
- Isaac ROS acceleration or CUDA transport parity

## MLX Port Architecture

The fork is organized around two explicit seams:

### Compute backend

The compute backend isolates tensor/runtime concerns:

- device selection
- checkpoint save/load
- RNG and backend metadata
- room for MLX-native custom kernel paths

Current public options:

- `torch-cuda`: upstream Isaac Sim path
- `mlx`: Apple Silicon path

Implementation entrypoint:

- [`source/isaaclab/isaaclab/backends/runtime.py`](source/isaaclab/isaaclab/backends/runtime.py)

### Simulation backend

The simulation backend isolates the minimum simulator contract Isaac Lab environments need:

- `reset(soft=...)`
- `step(render=..., update_fabric=...)`
- joint state reads
- joint effort writes
- root/joint state writes
- backend capability reporting

Current public options:

- `isaacsim`: upstream runtime adapter
- `mac-sim`: Mac-native adapter path

The current `mac-sim` implementation is intentionally narrow. It currently covers cartpole, cart-double-pendulum, and quadcopter slices needed to prove the MLX training/inference path can run without CUDA.

### Kernel backend

The kernel backend isolates Warp and future Metal custom-kernel paths:

- `warp`: upstream Isaac Sim + Warp path
- `metal`: Apple Silicon path for MLX + Metal-backed kernel replacements
- `cpu`: correctness fallback for bring-up and unsupported kernels

Today the public MLX tasks mostly use pure MLX ops. The explicit kernel selection seam exists now so future Metal kernels can land without rewriting task-facing APIs again.

## MLX Quick Start

This path is for Apple Silicon macOS.

### 1. Create the environment with `uv`

```bash
cd IsaacLab
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -e source/isaaclab[macos-mlx,dev]
```

If you want the upstream Isaac Sim runtime on Linux/NVIDIA instead, install the CUDA-side extra:

```bash
uv pip install --python .venv/bin/python -e source/isaaclab[cuda-isaacsim,dev]
```

### 2. Train the MLX cartpole baseline

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/reinforcement_learning/mlx/train_cartpole.py \
  --num-envs 256 \
  --updates 200 \
  --rollout-steps 64 \
  --epochs-per-update 4 \
  --checkpoint logs/mlx/cartpole_policy.npz
```

### 3. Replay a trained checkpoint

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/reinforcement_learning/mlx/play_cartpole.py \
  --checkpoint logs/mlx/cartpole_policy.npz \
  --episodes 3
```

Optional resume flow:

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/reinforcement_learning/mlx/train_cartpole.py \
  --resume-from logs/mlx/cartpole_policy.npz \
  --checkpoint logs/mlx/cartpole_policy_resumed.npz \
  --updates 50
```

The torch-centric RL wrapper extension is now optional. If you need upstream SB3, rl-games, or RSL-RL wrappers, install them separately from the MLX path:

```bash
uv pip install --python .venv/bin/python -e source/isaaclab_rl[sb3]
uv pip install --python .venv/bin/python -e source/isaaclab_rl[rsl-rl]
```

For the MLX/mac-sim task slices documented in this fork, `source/isaaclab_rl` is not required.

### 4. Run cart-double-pendulum MARL smoke

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/reinforcement_learning/mlx/play_cart_double_pendulum.py \
  --num-envs 64 --episodes 3 --max-steps 10000 --random-actions
```

### 5. Run quadcopter smoke

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/reinforcement_learning/mlx/play_quadcopter.py \
  --num-envs 64 --episodes 3 --episode-length-s 0.5 --max-steps 10000 --thrust-action 0.2 --no-random-actions
```

### 6. Run the focused backend test suite

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/pytest \
  scripts/tools/test/test_bootstrap_isaac_sources.py \
  source/isaaclab/test/backends/test_runtime.py \
  source/isaaclab/test/backends/test_portability_utils.py \
  source/isaaclab/test/backends/test_mac_cartpole.py \
  source/isaaclab/test/backends/test_mac_cartpole_showcase.py \
  source/isaaclab/test/backends/test_mac_cart_double_pendulum.py \
  source/isaaclab/test/backends/test_mac_quadcopter.py -q
```

## Runtime Selection

The backend seam is exposed through the app/runtime layer:

- `--compute-backend torch-cuda|mlx`
- `--kernel-backend warp|metal|cpu`
- `--sim-backend isaacsim|mac-sim`

These flags are published through:

- [`source/isaaclab/isaaclab/app/app_launcher.py`](source/isaaclab/isaaclab/app/app_launcher.py)

Examples:

```bash
# Upstream path
python some_script.py --compute-backend torch-cuda --kernel-backend warp --sim-backend isaacsim

# macOS port path
python some_script.py --compute-backend mlx --kernel-backend metal --sim-backend mac-sim
```

Important constraint: `AppLauncher` now accepts `--compute-backend mlx --sim-backend mac-sim` in bootstrap mode, but it does not launch Isaac Sim/Omniverse there. Use the dedicated MLX scripts for task execution.

## M5 Benchmarking

The fork now includes a dedicated MLX benchmark entrypoint for the current mac-native task set:

- [`scripts/benchmarks/mlx/benchmark_mac_tasks.py`](scripts/benchmarks/mlx/benchmark_mac_tasks.py)

Example:

```bash
PYTHONPATH=.:source/isaaclab .venv/bin/python \
  scripts/benchmarks/mlx/benchmark_mac_tasks.py \
  --tasks cartpole cart-double-pendulum quadcopter train-cartpole \
  --num-envs 256 \
  --steps 512 \
  --train-updates 20 \
  --json-out logs/benchmarks/mlx/m5-baseline.json
```

The benchmark emits:

- per-task `env_steps_per_s` for the current MLX/mac-sim env slices
- cartpole `train_frames_per_s` for the first MLX training loop
- runtime metadata including compute, kernel, sensor, and planner backend selection

## Implemented MLX Vertical Slice

Current implemented slices:

- cartpole environment and trainer in [`source/isaaclab/isaaclab/backends/mac_sim/cartpole.py`](source/isaaclab/isaaclab/backends/mac_sim/cartpole.py)
- cartpole showcase space variants in [`source/isaaclab/isaaclab/backends/mac_sim/showcase.py`](source/isaaclab/isaaclab/backends/mac_sim/showcase.py)
- cart-double-pendulum MARL environment in [`source/isaaclab/isaaclab/backends/mac_sim/cart_double_pendulum.py`](source/isaaclab/isaaclab/backends/mac_sim/cart_double_pendulum.py)
- quadcopter environment in [`source/isaaclab/isaaclab/backends/mac_sim/quadcopter.py`](source/isaaclab/isaaclab/backends/mac_sim/quadcopter.py)
- cartpole trainer entrypoint in [`scripts/reinforcement_learning/mlx/train_cartpole.py`](scripts/reinforcement_learning/mlx/train_cartpole.py)
- cartpole replay entrypoint in [`scripts/reinforcement_learning/mlx/play_cartpole.py`](scripts/reinforcement_learning/mlx/play_cartpole.py)
- cart-double-pendulum replay/smoke entrypoint in [`scripts/reinforcement_learning/mlx/play_cart_double_pendulum.py`](scripts/reinforcement_learning/mlx/play_cart_double_pendulum.py)
- quadcopter replay/smoke entrypoint in [`scripts/reinforcement_learning/mlx/play_quadcopter.py`](scripts/reinforcement_learning/mlx/play_quadcopter.py)

The cartpole path preserves the important upstream task semantics:

- observation order remains `pole_pos, pole_vel, cart_pos, cart_vel`
- reward structure matches upstream Isaac Lab cartpole
- termination conditions remain cart out-of-bounds or pole angle exceeding `pi/2`
- reset sampling keeps the pole-angle randomization behavior
- observations returned after done/reset are post-reset observations, matching the upstream direct RL flow
- cart-double-pendulum preserves per-agent dict observations/rewards/dones for `cart` and `pendulum`
- quadcopter preserves a root-state-centric policy observation layout with vectorized thrust/moment control

## Bootstrapping Upstream Sources

This fork also includes a repository bootstrap script so the Isaac Lab / Isaac Sim / Isaac ROS sources can be cloned reproducibly into a workspace.

Script:

- [`scripts/bootstrap_isaac_sources.py`](scripts/bootstrap_isaac_sources.py)

Example:

```bash
uv run scripts/bootstrap_isaac_sources.py \
  --dest ../isaac-sources \
  --with-isaacsim \
  --with-isaac-ros
```

The script:

- clones `IsaacLab` by default
- optionally clones `IsaacSim`
- optionally clones `isaac_ros_common`
- runs `git lfs install` / `git lfs pull` only for `IsaacSim`
- writes a manifest with remotes, requested refs, and resolved SHAs

## What Has Changed Relative To Upstream

The most important fork changes are:

- backend/runtime abstractions in [`source/isaaclab/isaaclab/backends`](source/isaaclab/isaaclab/backends)
- MLX cartpole reference implementation in [`source/isaaclab/isaaclab/backends/mac_sim`](source/isaaclab/isaaclab/backends/mac_sim)
- lazy import handling in `envs`, `sim`, and `utils` so Mac users get explicit unsupported-backend errors
- CUDA device binding moved behind an adapter instead of direct unconditional `torch.cuda.set_device(...)`
- public README and bootstrap path oriented around a publishable MLX fork

## Roadmap

Near-term priorities:

1. Expand `mac-sim` from the current task slices into a more general articulation/scene layer.
2. Add additional locomotion/manipulation tasks with compatible observation/action structure.
3. Push more task code through backend capability checks instead of import-time backend assumptions.
4. Define a stable checkpoint/config story across multiple MLX tasks.
5. Add Apple Silicon CI for the MLX backend slice.

Deferred work:

- Omniverse UI and Kit extension parity
- RTX camera and sensor parity
- Warp and CUDA custom kernel replacements beyond the first task set
- cuRobo and Isaac ROS CUDA transport features

## Relationship To Upstream Isaac Lab

This repository is a fork of the upstream Isaac Lab project and still depends heavily on its architecture, APIs, tasks, and research workflow design.

Upstream remains the reference source for:

- task semantics
- asset conventions
- environment configuration patterns
- Isaac Sim integrations

Original upstream repository:

- [isaac-sim/IsaacLab](https://github.com/isaac-sim/IsaacLab)

Related reference repositories:

- [isaac-sim/IsaacSim](https://github.com/isaac-sim/IsaacSim)
- [NVIDIA-ISAAC-ROS/isaac_ros_common](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common)
- [ml-explore/mlx](https://github.com/ml-explore/mlx)

## Documentation

The upstream Isaac Lab documentation remains the best reference for the broader project model:

- [Installation steps](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html#local-installation)
- [Reinforcement learning](https://isaac-sim.github.io/IsaacLab/main/source/overview/reinforcement-learning/rl_existing_scripts.html)
- [Tutorials](https://isaac-sim.github.io/IsaacLab/main/source/tutorials/index.html)
- [Available environments](https://isaac-sim.github.io/IsaacLab/main/source/overview/environments.html)

## Isaac Sim Version Dependency

Upstream Isaac Lab is built on top of Isaac Sim and requires compatible Isaac Sim versions. This fork still treats Isaac Sim as the reference runtime for the upstream path.

| Isaac Lab Version             | Isaac Sim Version         |
| ----------------------------- | ------------------------- |
| `main` branch                 | Isaac Sim 4.5 / 5.0 / 5.1 |
| `v2.3.X`                      | Isaac Sim 4.5 / 5.0 / 5.1 |
| `v2.2.X`                      | Isaac Sim 4.5 / 5.0       |
| `v2.1.X`                      | Isaac Sim 4.5             |
| `v2.0.X`                      | Isaac Sim 4.5             |

## Contributing

Contributions are welcome, especially in these areas:

- additional MLX-backed task ports
- generalized `mac-sim` asset and articulation support
- capability gating around CUDA-only integrations
- Apple Silicon verification and CI
- documentation for MLX and macOS users

For general upstream contribution guidance, see:

- [Upstream contributing guide](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)

## Troubleshooting

Common current pitfalls:

- If `mlx` is missing, install it into the active `uv` environment.
- If imports fail on `omni.*` or `isaacsim.*`, you are probably invoking an upstream Isaac Sim path rather than the `mac-sim` path.
- If you use `AppLauncher` with `--sim-backend mac-sim`, it runs in bootstrap mode and returns a placeholder app object. Use the dedicated MLX scripts for environment stepping/training.
- If a task depends on cameras, Warp, cuRobo, or Omniverse-only features, it should currently be treated as unsupported on the macOS path.

## Support

- Use GitHub Discussions for design discussions, roadmap ideas, and supported-task requests.
- Use GitHub Issues for concrete bugs, missing capability checks, broken MLX scripts, or backend portability problems.

## License

The Isaac Lab framework is released under [BSD-3 License](LICENSE). The `isaaclab_mimic` extension and its corresponding standalone scripts are released under [Apache 2.0](LICENSE-mimic). The license files of dependencies and assets are present in the [`docs/licenses`](docs/licenses) directory.

Isaac Lab still requires Isaac Sim for the upstream runtime path, and Isaac Sim includes components under proprietary licensing terms. Please see the [Isaac Sim license](docs/licenses/dependencies/isaacsim-license.txt) for details.

The `isaaclab_mimic` extension requires cuRobo, which has proprietary licensing terms listed in [`docs/licenses/dependencies/cuRobo-license.txt`](docs/licenses/dependencies/cuRobo-license.txt).

## Citation

If you use Isaac Lab or this MLX fork in your research, please cite the technical report:

```bibtex
@article{mittal2025isaaclab,
  title={Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning},
  author={Mayank Mittal and Pascal Roth and James Tigue and Antoine Richard and Octi Zhang and Peter Du and Antonio Serrano-Muñoz and Xinjie Yao and René Zurbrügg and Nikita Rudin and Lukasz Wawrzyniak and Milad Rakhsha and Alain Denzler and Eric Heiden and Ales Borovicka and Ossama Ahmed and Iretiayo Akinola and Abrar Anwar and Mark T. Carlson and Ji Yuan Feng and Animesh Garg and Renato Gasoto and Lionel Gulich and Yijie Guo and M. Gussert and Alex Hansen and Mihir Kulkarni and Chenran Li and Wei Liu and Viktor Makoviychuk and Grzegorz Malczyk and Hammad Mazhar and Masoud Moghani and Adithyavairavan Murali and Michael Noseworthy and Alexander Poddubny and Nathan Ratliff and Welf Rehberg and Clemens Schwarke and Ritvik Singh and James Latham Smith and Bingjie Tang and Ruchik Thaker and Matthew Trepte and Karl Van Wyk and Fangzhou Yu and Alex Millane and Vikram Ramasamy and Remo Steiner and Sangeeta Subramanian and Clemens Volk and CY Chen and Neel Jawale and Ashwin Varghese Kuruttukulam and Michael A. Lin and Ajay Mandlekar and Karsten Patzwaldt and John Welsh and Huihua Zhao and Fatima Anes and Jean-Francois Lafleche and Nicolas Moënne-Loccoz and Soowan Park and Rob Stepinski and Dirk Van Gelder and Chris Amevor and Jan Carius and Jumyung Chang and Anka He Chen and Pablo de Heras Ciechomski and Gilles Daviet and Mohammad Mohajerani and Julia von Muralt and Viktor Reutskyy and Michael Sauter and Simon Schirm and Eric L. Shi and Pierre Terdiman and Kenny Vilella and Tobias Widmer and Gordon Yeoman and Tiffany Chen and Sergey Grizan and Cathy Li and Lotus Li and Connor Smith and Rafael Wiltz and Kostas Alexis and Yan Chang and David Chu and Linxi "Jim" Fan and Farbod Farshidian and Ankur Handa and Spencer Huang and Marco Hutter and Yashraj Narang and Soha Pouya and Shiwei Sheng and Yuke Zhu and Miles Macklin and Adam Moravanszky and Philipp Reist and Yunrong Guo and David Hoeller and Gavriel State},
  journal={arXiv preprint arXiv:2511.04831},
  year={2025},
  url={https://arxiv.org/abs/2511.04831}
}
```

## Acknowledgement

Isaac Lab development initiated from the [Orbit](https://isaac-orbit.github.io/) framework. This fork builds on that work and on the upstream Isaac Lab architecture.

IsaacLab-mlx

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README