mylangrobot

KO | EN

gitlite — search

#python #typescript #llm #android #machine-learning #rust #deep-learning #linux #awesome #kubernetes #awesome-list #pytorch #security #openai

(★ 24)

Language instructions to mycobot using GPT-4V

mylangrobot 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Introduction

This is a repository for conducting interactions with robots using language. Using myCobot, pick and place tasks are achieved through verbal interactions with humans.
By utilizing ChatGPT for image interpretation, generating pick and place programs, and creating additional questions for ambiguous instructions, we accomplish the targeted tasks.

For image interpretation, [SoM](https://som-gpt4v.github.io/) is used. This enables more specific object designation by GPT-4V by using images that overlay numbers on segmented images.

## Dependencies

* openai
* openai-whisper
* segment-anything
* pymycobot

## Hardware

* [myCobot](https://shop.elephantrobotics.com/collections/mycobot/products/mycobot-worlds-smallest-and-lightest-six-axis-collaborative-robot)
* [suction-pump](https://shop.elephantrobotics.com/collections/suction-pumps/products/suction-pump-2-0)
* [camera-flange](https://shop.elephantrobotics.com/collections/camera-modules/products/camera-flange-2-0)

## Installation

```sh
sudo apt install portaudio19-dev
git clone https://github.com/neka-nat/mylangrobot.git
cd mylangrobot
pip install -e .  # or poetry install
```

## Environment Settings

```sh
cp .env.sample .env
# Edit .env
# OPENAI_API_KEY=<your api key>
```

## Demo

Interactive demonstration with microphone.

```sh
sudo chmod 666 /dev/ttyUSBxxx
cd scripts
python demo.py
```

Demonstration of sending a single text command.

```sh
sudo chmod 666 /dev/ttyUSBxxx
cd scripts
python oneshot_demo.py --prompt チョコレートの箱取って。
```

## Results

### Captured image and Annotated image

![capture](https://raw.githubusercontent.com/neka-nat/mylangrobot/main/assets/capture.png)
![annotated](https://raw.githubusercontent.com/neka-nat/mylangrobot/main/assets/annotated.png)

### Generated program

```
move_to_object(4)
grab()
move_to_place('drop')
release()
```

### Generated motion

https://github.com/neka-nat/mylangrobot/assets/991515/858ae25a-405a-4ad1-afc2-fbf6934adac3

## Settings

You can configure the robot using the `configs/settings.yml` file.
Please set the connection port with the robot, the camera ID, the suction pin number, and other hardware-related settings according to your own environment.

```yaml
pixel_size_on_capture_position: 0.00043  # [m/pixel]
interface_type: "AUDIO"
camera_id: 0
language: "Japanese"
mycobot_settings:
  urdf_path: "../data/mycobot/mycobot.urdf"
  end_effector_name: "camera_flange"
  port: "/dev/ttyACM0"
  baud: 115200
  default_speed: 40
  default_z_speed: 20
  suction_pin: 5
  command_timeout: 5
  use_gravity_compensation: false
  end_effector_height: 0.065  # pump head offset
  object_height: 0.01
  release_height: 0.05
  positions:
    home: [0, 20, -130, 20, 0, 0]
    capture: [0, 0, -30, -60, 0, -45]
    drop: [-45, 20, -130, 20, 0, 0]
```

## Related links

* [Set-of-Mark-Visual-Prompting-for-GPT-4V](https://github.com/microsoft/SoM)
* [SoM demo](https://github.com/SkalskiP/SoM)
* [Segment Anything](https://github.com/facebookresearch/segment-anything)
* [PromptCraft-Robotics](https://github.com/microsoft/PromptCraft-Robotics)

mylangrobot

mylangrobot 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

mylangrobot

mylangrobot 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README