therobotrix

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 73)

No description available.

therobotrix Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# The RobotriX

[c0]: https://placehold.it/15/000000/000000?text=+
[c1]: https://placehold.it/15/0000ff/000000?text=+
[c2]: https://placehold.it/15/00ff00/000000?text=+
[c3]: https://placehold.it/15/00ffff/000000?text=+
[c4]: https://placehold.it/15/ff0000/000000?text=+
[c5]: https://placehold.it/15/ff00ff/000000?text=+
[c6]: https://placehold.it/15/ffff00/000000?text=+
[c7]: https://placehold.it/15/ffffff/000000?text=+
[c8]: https://placehold.it/15/000080/000000?text=+
[c9]: https://placehold.it/15/008000/000000?text=+
[c10]: https://placehold.it/15/008080/000000?text=+
[c11]: https://placehold.it/15/800000/000000?text=+
[c12]: https://placehold.it/15/800080/000000?text=+

[c13]: https://placehold.it/15/808000/000000?text=+
[c14]: https://placehold.it/15/808080/000000?text=+
[c15]: https://placehold.it/15/ff6000/000000?text=+
[c16]: https://placehold.it/15/ffd7b4/000000?text=+
[c17]: https://placehold.it/15/aa6e28/000000?text=+
[c18]: https://placehold.it/15/e6beff/000000?text=+
[c19]: https://placehold.it/15/d2f53c/000000?text=+
[c20]: https://placehold.it/15/aaffc3/000000?text=+
[c21]: https://placehold.it/15/e6194b/000000?text=+
[c22]: https://placehold.it/15/fffac8/000000?text=+
[c23]: https://placehold.it/15/0082c8/000000?text=+
[c24]: https://placehold.it/15/3cb94b/000000?text=+
[c25]: https://placehold.it/15/f58230/000000?text=+

[c26]: https://placehold.it/15/cdcd00/000000?text=+
[c27]: https://placehold.it/15/ffc125/000000?text=+

[c28]: https://placehold.it/15/ff6000/000000?text=+

[c29]: https://placehold.it/15/717162/000000?text=+
[c30]: https://placehold.it/15/cae1ff/000000?text=+
[c31]: https://placehold.it/15/5f9ea0/000000?text=+
[c32]: https://placehold.it/15/00fa9a/000000?text=+
[c33]: https://placehold.it/15/cdba96/000000?text=+
[c34]: https://placehold.it/15/8b4500/000000?text=+
[c35]: https://placehold.it/15/ff4500/000000?text=+
[c36]: https://placehold.it/15/8e388e/000000?text=+
[c37]: https://placehold.it/15/c67185/000000?text=+
[c38]: https://placehold.it/15/c5c1aa/000000?text=+

[seq0]: ./img/seq0.jpg
[seq0_depth]: ./img/seq0_depth.jpg
[seq0_mask]: ./img/seq0_mask.jpg
[seq1]: ./img/seq1.jpg
[seq1_depth]: ./img/seq1_depth.jpg
[seq1_mask]: ./img/seq1_mask.jpg
[seq2]: ./img/seq2.jpg
[seq2_depth]: ./img/seq2_depth.jpg
[seq2_mask]: ./img/seq2_mask.jpg
[video_preview]: ./img/video_demo_2.jpg

[![video_preview]](https://www.youtube.com/watch?v=CiRc5tCtCak)

Enter the RobotriX, an extremely photorealistic indoor dataset designed to enable the application of deep learning techniques to a wide variety of robotic vision problems. The RobotriX consists of hyperrealistic indoor scenes which are explored by robot agents which also interact with objects in a visually realistic manner in that simulated world. Photorealistic scenes and robots are rendered by Unreal Engine into a virtual reality headset which captures gaze so that a human operator can move the robot and use controllers for the robotic hands; scene information is dumped on a per-frame basis so that it can be reproduced offline using UnrealCV to generate raw data and ground truth labels. By taking this approach we were able to generate a dataset of 38 semantic classes across 512 sequences totaling 8M stills recorded at +60 frames per second with full HD resolution. For each frame, RGB-D and 3D information is provided with full annotations in both spaces. Thanks to the high quality and quantity of both raw information and annotations, the RobotriX will serve as a new milestone for investigating 2D and 3D robotic vision tasks with large-scale data-driven techniques.

![seq0]
![seq0_depth]
![seq0_mask]
![seq1]
![seq1_depth]
![seq1_mask]
![seq2]
![seq2_depth]
![seq2_mask]

## Contents

1. [Data](#data)
2. [UnrealROX](#unrealrox)
3. [Assets](#assets)
4. [Troubleshooting](#troubleshooting)
5. [License](#license)
6. [Contact](#contact)

## Data

We generated a dataset of 512 sequences recorded on 16 different indoor layouts at +60 FPS with a duration that spans between one and five minutes each. That adds up to a total of approximately eight million individual frames. This initial release of the dataset contains 32 detection and 39 semantic classes. The categories were selected from the most common and useful household goods in indoor environments for social robots.

| Type | 00 ![c0] | 01 ![c1] | 02 ![c2] | 03 ![c3] | 04 ![c4] | 05 ![c5] | 06 ![c6] | 07 ![c7] | 08 ![c8] | 09 ![c9] | 10 ![c10] | 11 ![c11] | 12 ![c12] |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| Semantic | void | wall | floor | ceiling | window | door | table | chair | lamp | sofa | cupboard | screen | hand |
| Detection | - | - | - | - | - | - | table | chair | lamp | sofa | cupboard | screen | hand |

| Type | 13 ![c13] | 14 ![c14] | 15 ![c15] | 16 ![c16] | 17 ![c17] | 18 ![c18] | 19 ![c19] | 20 ![c20] | 21 ![c21] | 22 ![c22] | 23 ![c23] | 24 ![c24] | 25 ![c25] |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| Semantic | frame | bed | fridge | whiteboard | book | bottle | plant | furniture | toilet | phone | bathtub | cup | mat |
| Detection | frame | bed | fridge | whiteboard | book | bottle | plant | - | lamp | toilet | phone | bathtub | cup | mat |

| Type | 26 ![c26] | 27 ![c27] | 28 ![c28] | 29 ![c29] | 30 ![c30] | 31 ![c31] | 32 ![c32] | 33 ![c33] | 34 ![c34] | 35 ![c35] | 36 ![c36] | 37 ![c37] | 38 ![c38] |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| Semantic | mirror | sink | box | mouse | keyboard | bin | cushion | shelf | bag | curtain | kitchen_stuff | bath_stuff | prop |
| Detection | mirror | sink | box | mouse | keyboard | bin | cushion | shelf | bag | - | kitchen_stuff | bath_stuff | prop |

Due to the huge size of the data (~7 TiB), we are currently distributing part of it via private links to our FTP server to avoid excessive traffic (drop a mail to [agarcia@dtic.ua.es](mailto:agarcia@dtic.ua.es) for them). However, half of the dataset is already available (and increasing every day) through [OSF](https://osf.io/b3g2y/) at https://osf.io/b3g2y/ (OSF is a free, open source web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers use OSF to collaborate, document, archive, share, and register research projects, materials, and data. OSF is the flagship product of the non-profit Center for Open Science).

### IMPORTANT: Instructions to unzip data from OSF
The data uploaded to OSF was zipped in chunks. For example, suppose we are trying to unzip rgb files of a scene, so you have rgb.z01, rgb.z02, rgb.z03... and rgb.zip main file. In order to unzip the data you can do the following:
1. Use 7-zip software which should be working and unzip rgb.zip. 
2. Use Linux unzip, but firstly you should:
  - Put all the parts together doing the following: `zip -F rgb.zip --out rgb_full.zip`
  - Then you can proceed and unzip the big file rgb_full.zip doing: `unzip rgb_full.zip`
  
If you don't follow the instructions above you probably get bad zipfile offset errors.

The following data is available at the OSF project page:

| **ID** | **Scene** | **Robot** | **Interactable Objects** | **Cameras** | **Duration** | **Frames** | **Total** |
| --------- | -------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------ | ----------- | ------------------- |
| **TOTAL** | **TOTAL** | **TOTAL** | **TOTAL** | **TOTAL** | | | **3.039.252** |
| **000** | **HamburgHaus** | | | | **11:48** | **63.714** | **318.570** |
| 000 | HamburgHaus | Mannequin | 0 | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:27 | 7838 | 39190 |
| 001 | HamburgHaus | Mannequin | 1 (Moka_Chaleira_Moka_10) | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:18 | 7022 | 35110 |
| 002 | HamburgHaus | Mannequin | 1 (Vase_Rounded_46) | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:02 | 5554 | 27770 |
| 003 | HamburgHaus | Mannequin | 3 (Vase_Rounded_46, DEC_living_book_6, and DEC_living_book_4) | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:49 | 9786 | 48930 |
| 004 | HamburgHaus | Mannequin | 5 (Cadeira_Eames_Cadeira_Eames_137, DEC_living_book_6, DEC_living_book_4, Vase_Rounded_46, Moka_Chaleira_Moka_10) | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:59 | 10704 | 53520 |
| 005 | HamburgHaus | Mannequin | 2 (Moka_Chaleira_Moka_10, Vase_Rounded_46) | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:54 | 10269 | 51345 |
| 006 | HamburgHaus | Mannequin | 0 | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:12 | 6482 | 32410 |
| 007 | HamburgHaus | Mannequin | 0 | 5 (Head, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:07 | 6059 | 30295 |
| | | | | | | | |
| **001** | **Viennese** | | | | | **127.521** | **679.****7****36** |
| 000 | Viennese | Mannequin | 3 (Fruit_Pear, Fruit_Apple, Fruit_Orange2) | 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) | 04:18 | 23202 | 116.010 |
| 001 | Viennese | Mannequin | 7 (Fruit_Pear, Fruit_Apple, Fruit_Orange2, Pot, Pot2, Fruit_Apple2, Fruit_Orange) | 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) | 07:09 | 38627 | 193.135 |
| 002 | Viennese | Mannequin | 8 (Moka_Coffe, Pot, Fruit_Pear, Fruit_Apple, Fruit_Orange2, Fruit_Apple2, Fruit_Orange) | 5 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0) | 04:29 | 23661 | 118.305 |
| 003 | Viennese | Mannequin | 14 (Flow_chair, Flow_chair2, chair_table_corners2, Fruit_Pear, Fruit_Apple, Fruit_Orange2, Fruit_Apple2, Fruit_Orange, Moka_Coffe, table_decoration1_, table_decoration_2, Pot, Pot2, DEC_dining_table_vase) | 6 (Head, LeftHand, RightHand, TopViewCamera, CornerCamera0, CornerCamera1) | 07:57 | 42031 | 252.186 |
| | | | | | | | |
| **002** | **InteractiveHouse** | | | | **20:08** | **103.352** | **516.760** |
| 000 | InteractiveHouse | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:39 | 8587 | 42935 |
| 001 | InteractiveHouse | Mannequin | 4 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Chair_93) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:40 | 8510 | 42550 |
| 002 | InteractiveHouse | Mannequin | 7 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Chair_93, Chair2, Chair3, Chair4) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 02:00 | 10800 | 54000 |
| 003 | InteractiveHouse | Mannequin | 6 (Vase_13, Flower_Pot2, SM_MERGED_Plant_Bromelia_10, DEC_dining_vase_332, DEC_dining_vase_320) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 02:31 | 13564 | 67820 |
| 004 | InteractiveHouse | Mannequin | 7 (Vase_13, DEC_dining_vase_332, DEC_dining_vase_320, DEC_living_book_6, Moka_Coffe_27, table_decoration_91, DEC_living_vase_410) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 02:47 | 14649 | 73245 |
| 005 | InteractiveHouse | Mannequin | 9 (Flower_Pot2, SM_MERGED_Plant_Bromelia_10, Vase_13, DEC_dining_vase_332, DEC_dining_vase_320, DEC_living_book_6, Moka_Coffe_27, table_decoration_91, DEC_living_vase_410) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 02:56 | 14091 | 70455 |
| 006 | InteractiveHouse | Mannequin | 4 (Chair_93, Chair2, Chair3, Chair4) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:37 | 8727 | 43635 |
| 007 | InteractiveHouse | Mannequin | 3 (DEC_dining_table_vase_302, DEC_dining_table_sphere_004_302, DEC_dining_table_sphere_300) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 02:20 | 10888 | 54440 |
| 008 | InteractiveHouse | Mannequin | 6 (DEC_dining_table_vase_302, DEC_dining_table_sphere_004_302, DEC_dining_table_sphere_300, Vase_13, Moka_Coffe_27,table_decoration_91) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:35 | 8485 | 42425 |
| 009 | InteractiveHouse | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:06 | 5051 | 25255 |
| | | | | | | | |
| **003** | **StudioApartment** | | | | **05:23** | **28.768** | **143.480** |
| 000 | StudioApartment | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) | 00:38 | 3366 | 16380 |
| 001 | StudioApartment | Mannequin | | 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) | 01:51 | 9793 | 48965 |
| 002 | StudioApartment | Mannequin | 5 (Pot_91, Fruit_Orange_130, Fruit_Apple_133, Moka_Coffee_144, SM_Kitchen_Deco_04_0) | 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) | 01:28 | 7829 | 39145 |
| 003 | StudioApartment | Mannequin | 5 (SM_Kitchen_Deco_05, SM_Kitchen_Deco_04, SM_Kitchen_Deco_02, SM_Kitchen_Deco_04_0, Moka_Coffee_144) | 5 (FirstPerson, LeftHand, RightHand, Entrance, Bedroom) | 01:26 | 7798 | 38990 |
| | | | | | | | |
| **004** | **BerlinFlat** | | | | **16:32** | **88.356** | **441.780** |
| 000 | BerlinFlat | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:33 | 7847 | 39235 |
| 001 | BerlinFlat | Mannequin | 5 (glass1, glass3, glass4, glass5, glass6) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:45 | 9414 | 47070 |
| 002 | BerlinFlat | Mannequin | | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:53 | 9961 | 49805 |
| 003 | BerlinFlat | Mannequin | | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:46 | 9429 | 47145 |
| 004 | BerlinFlat | Mannequin | | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:42 | 9204 | 46020 |
| 005 | BerlinFlat | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:13 | 6531 | 32655 |
| 006 | BerlinFlat | Mannequin | 6 (glass1, glass2, glass3, glass4, glass5, glass6) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:50 | 9852 | 49260 |
| 007 | BerlinFlat | Mannequin | 5 (chair5, chair6, chair7, chair8, chair9) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:20 | 7170 | 35850 |
| 008 | BerlinFlat | Mannequin | 6 (glass1, glass2, glass3, glass4, glass5, glass6, ) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:48 | 9795 | 48975 |
| 009 | BerlinFlat | Mannequin | 9 (glass1, glass2, glass3, glass4, glass5, glass6, table_decoration_44, book1) | 5 (FirstPerson, LeftHand, RightHand, MainRoom, SecondaryRoom) | 01:42 | 9153 | 45765 |
| | | | | | | | |
| **005** | **Singapore** | | | | | **87.951** | **439.755** |
| 000 | Singapore | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 01:58 | 9923 | 49615 |
| 001 | Singapore | Mannequin | 0 | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 01:40 | 8992 | 44960 |
| 002 | Singapore | Mannequin | 5 (SM_Dining_Glass_01_31, SM_Dining_Glass_02_34, SM_Dining_Glass_3, SM_Dining_Glass_4, SM_Dining_Glass_8, SM_Dining_Glass_9, SM_Dining_Glass_10) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 02:26 | 13119 | 65595 |
| 003 | Singapore | Mannequin | 5 (SM_Kitchen_Tableware_14, SM_Kitchen_Tableware_13, SM_Kitchen_Tableware_12, SM_Cactus_4, SM_Cactus_5) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 01:34 | 7890 | 39450 |
| 004 | Singapore | Mannequin | 5 (SM_Dining_Glass_01_31, SM_Dining_Glass_02_34, SM_Dining_Glass_3, SM_Dining_Glass_4, SM_Dining_Glass_8, SM_Dining_Glass_9, SM_Dining_Glass_10) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 02:17 | 12112 | 60560 |
| 005 | Singapore | Mannequin | 6 (SM_Dinning_Chair_9, SM_Dinning_Chair_7, SM_Dinning_Chair_6, SM_Dinning_Chair_3, SM_Dinning_Chair_2, SM_Dinning_Chair_01_8) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 02:32 | 13601 | 68005 |
| 006 | Singapore | Mannequin | 4 (SM_Dinning_Chair_3, SM_MBook_8, SM_Dinning_Glass_02_34, SM_Kitchen_Tableware_13) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 01:41 | 9147 | 45735 |
| 007 | Singapore | Mannequin | 17 (SM_Dinning_Chair_9, SM_Dinning_Chair_3, SM_Dinning_Chair_2, SM_Dinning_Chair_01_8, SM_MBook_8, SM_Kitchen_Tableware_14, SM_Kitchen_Tableware_13, SM_Kitchen_Tableware_12, SM_Kitchen_D7, SM_Kitchen_D5_90, SM_Cactus_5, SM_Cactus_4, SM_Dinning_Glass_01_31, SM_Dinning_Glass_02_34, SM_Dinning_Glass_3, SM_Dinning_Glass_4, SM_Dinning_Glass_8) | 5 (FirstPerson, LeftHand, RightHand, DiningRoom, LivingRoom) | 02:27 | 13167 | 65835 |
| | | | | | | | |
| **006** | **WarmHarbor** | | | | **17:35** | | **499.171** |
| 000 | WarmHarbor | Mannequin | Exploration secuence of the entire scene | 7 (FirstPerson, LeftHand, RightHand, Kitchen, Lounge, Lounge2, Lounge3) | 03:37 | 19175 | 134225 |
| 001 | WarmHarbor | Mannequin | 5 (SM_SoupPot1, SM_MokaCoffe01, SM_Ginger01, SM_Wok1, SM_Decoration29) | 4 (FirstPerson, LeftHand, RightHand, Kitchen) | 04:23 | 23395 | 93580 |
| 002 | WarmHarbor | Mannequin | 6 (SM_Wok1, SM_Melon01, SM_Decoration29, SM_SoupPot1, SM_MokaCoffe01, SM_BarChair2) | 4 (FirstPerson, LeftHand, RightHand, Kitchen) | 03:16 | 17389 | 69556 |
| 003 | WarmHarbor | Mannequin | 6 (SM_Decoration28, SM_Decoration23, SM_Side1, SM_Decoration26, SM_Drink5, SM_DiningChair6) | 6(FirstPerson, LeftHand, RightHand, Lounge, Lounge2, Lounge3) | 03:37 | 19163 | 114978 |
| 004 | WarmHarbor | Mannequin | 7 (SM_Decoration3, SM_Decoration4, SM_Side1, SM_Decoration27, SM_Drink5, SM_Decoration23, SM_Decoration28) | 6(FirstPerson, LeftHand, RightHand, Lounge, Lounge2, Lounge3) | 02:42 | 14472 | 86832 |
| | | | | | | | |
| **007** | **Wooden** | | | | | | **473.110** |
| 000 | Wooden | Mannequin | 0 (Exploration sequence) | 5 (FirstPerson, LeftHand, RightHand, SecondaryRoomCamera, SecondaryRoomCamera2) | 01:25 | 7339 | 36695 |
| 001 | Wooden | Mannequin | 0 (Exploration sequence) | 5 (FirstPerson, LeftHand, RightHand, SecondaryRoomCamera, SecondaryRoomCamera2) | 01:16 | 6860 | 34300 |
| 002 | Wooden | Mannequin | 0 (Exploration sequence) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:31 | 7915 | 39575 |
| 003 | Wooden | Mannequin | 0 (Exploration sequence) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:34 | 8407 | 42035 |
| 004 | Wooden | Mannequin | 3 (SM_Moka_Coffe_01, SM_Dinning_Glass_207, SM_Dinning_Glass_20) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:35 | 8452 | 42260 |
| 005 | Wooden | Mannequin | 5 (SM_Pepper_1, SM_KW_00, SM_KSauce_44, SM_KW_02, SM_KW_01) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:50 | 9759 | 48795 |
| 006 | Wooden | Mannequin | 9 (SM_Chair_08, SM_Chair_05, SM_Chair_06, SM_Chair_07, SM_Dinning_Glass_07, SM_Dinning_Glass_11, SM_Dinning_Glass_12, SM_Dinning_Glass_04, SM_Dinning_Glass_02_138) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:28 | 7858 | 39290 |
| 007 | Wooden | Mannequin | 18 (SM_Dinning_Glass_01_141, SM_Dinning_Glass_02_138, SM_Dinning_Glass_03, SM_Dinning_Glass_05, SM_Dinning_Glass_06, SM_Dinning_Glass_08, SM_Dinning_Glass_09, SM_Dinning_Glass_10, SM_Dinning_Glass_174, SM_Dinning_Glas_13, SM_Dinning_Glass_14, SM_Dinning_Glass_15, SM_Dinning_Glass_16, SM_Dinning_Glass_189, SM_Dinning_Glass_19, SM_Dinning_Glass_07, SM_Dinning_Glass_11, SM_Dinning_Glass_12, SM_Dinning_Glass_04) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 04:15 | 22788 | 113940 |
| 008 | Wooden | Mannequin | 7 (SM_Moka_Coffe_01, SM_Dinning_Glass_207, SM_Dinning_Glass_20, SM_KW_00, SM_KSauce_44, SM_KW_02, SM_KW_01) | 5 (FirstPerson, LeftHand, RightHand, HallCamera, DiningCamera) | 01:55 | 9500 | 47500 |
| 009 | Wooden | Mannequin | 1 (SM_Deco_00) | 5 (FirstPerson, LeftHand, RightHand, CorridorCamera, DiningCamera) | 01:05 | 5744 | 28720 |
| | | | | | | | |
| **008** | **ModernCozy** | | | | | | **339.097** |
| 000 | ModernCozy | Mannequin | | | 01:40 | 8866 | 44330 |
| 001 | ModernCozy | Mannequin | | | 02:10 | 11617 | 58085 |
| 002 | ModernCozy | Mannequin | | | 01:38 | 8791 | 43955 |
| 003 | ModernCozy | Mannequin | | | 02:30 | 12729 | 63645 |
| 004 | ModernCozy | Mannequin | | | 01:34 | 8306 | 41530 |
| 005 | ModernCozy | Mannequin | | | 00:39 | 3567 | 14268 |
| 006 | ModernCozy | Mannequin | | | 01:24 | 7582 | 22746 |
| 007 | ModernCozy | Mannequin | | | 03:09 | 16846 | 50538 |

### Sequences

Each sequence is recorded as a TXT file that describes it and allows its offline playback to generate the aforementioned data. As a matter of fact, those TXT files are processed and converted into JSON files for improved readability and to make them easier to parse. Those sequence descriptor files contain all the information needed to generate the images and extract ground truth.

### Raw Data

For each frame, we provide the following data:

* 3D poses for the cameras, objects, and robot joints.
* RGB image at 1920x1080 resolution in 24-bit JPG(95%) format (instead of PNG for reduced size).
* Depth map at 1920x1080 resolution in 16-bit grayscale PNG format.
* 2D instance mask at 1920x1080 resolution in RGB 24-bit PNG format.

### Ground Truth

For each frame, we provide the tools to generate the following annotations:

* 2D class mask at 1920x1080 resolution in RGB 24-bit PNG format.
* 2D/3D object instance bounding boxes in XML format.
* 3D point cloud in PLY format with RGB color.
* 3D instance/class mask in PLY format with RGB color.

Due to the excessive size of that large-scale ground truth, we provide the needed tools and instructions so that anyone can generate the annotations they need locally using the aforementioned raw data.

## UnrealROX

**IMPORTANT**

The data for this paper was generated using a deprecated tool which extended UnrealCV to generate all the data we needed. This tool has been superseded by UnrealROX, a compatible and home-brewed C++ solution for UnrealEngine that allows efficient and flexible data recording in virtual reality and offline generation with annotations. Since this tool is better suited for our purposes (more efficient, flexible, and complete), we have removed all previous references in this repository to old RobotriX tools that we mention in the paper (although they can be obtained through commit history).

UnrealROX is described in detail at the following [arXiv](https://arxiv.org/abs/1810.06936) paper and it is released at the following [GitHub/3dperceptionlab/unrealrox](https://github.com/3dperceptionlab/unrealrox) repository.

**IMPORTANT**

## Assets

Assets for this project are originated from two sources: [UE4Arch](https://ue4arch.com/) and [Unreal Engine Marketplace](https://www.unrealengine.com/marketplace/store) so they can be acquired from there. At this moment, we are still in conversations with both parties to release the modified assets as we used them in our scenes.

## Troubleshooting

For any kind of problems, configurations, or instructions, please read carefully [UnrealROX's documentation](https://unrealrox.readthedocs.io/en/master/).

We encourage any user to submit any issue related to the data itself using GitHub's built-in issue system within this repository. For any other issue related to the data generation process or tool, please submit the adequate issue to [UnrealROX's repository](https://github.com/3dperceptionlab/unrealrox). Improvements and critics are welcome at all fronts!

## License

Both the data for The RobotriX and the code for UnrealROX are released under the [MIT license](LICENSE).

## Citation

If you use this dataset or the UnrealROX tool, please cite:

- Garcia-Garcia, A., Martinez-Gonzalez, P., Oprea, S., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Jover-Alvarez, A. (2018, October). The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6790-6797). IEEE.

```
@inproceedings{garcia2018robotrix,
  title={The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions},
  author={Garcia-Garcia, Alberto and Martinez-Gonzalez, Pablo and Oprea, Sergiu and Castro-Vargas, John Alejandro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose and Jover-Alvarez, Alvaro},
  booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={6790--6797},
  year={2018},
  organization={IEEE}
}
```

- Martinez-Gonzalez, P., Oprea, S., Garcia-Garcia, A., Jover-Alvarez, A., Orts-Escolano, S., & Garcia-Rodriguez, J. (2018). UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation. arXiv preprint arXiv:1810.06936.

```
@article{martinez2018unrealrox,
  title={UnrealROX: An eXtremely Photorealistic Virtual Reality Environment for Robotics Simulations and Synthetic Data Generation},
  author={Martinez-Gonzalez, Pablo and Oprea, Sergiu and Garcia-Garcia, Alberto and Jover-Alvarez, Alvaro and Orts-Escolano, Sergio and Garcia-Rodriguez, Jose},
  journal={arXiv preprint arXiv:1810.06936},
  year={2018}
}
```

## Contact / Authors

Please contact the authors if you have any questions or requests.

- Alberto Garcia-Garcia [Design, Prototyping, Data Generation, Project Lead] ([agarcia@dtic.ua.es](mailto:agarcia@dtic.ua.es))
- Pablo Martinez-Gonzalez [Design, UE4 Backend, Lead Programmer] ([pmartinez@dtic.ua.es](mailto:pmartinez@dtic.ua.es))
- Sergiu Oprea [Grasping, Data Generation] ([soprea@dtic.ua.es](mailto:soprea@dtic.ua.es))
- John A. Castro-Vargas [Support Programmer, Data Generation] ([jacastro@dtic.ua.es](mailto:jacastro@dtic.ua.es))
- Alvaro Jover-Alvarez [UE4 Expert, Support Programmer] ([ajover@dtic.ua.es](mailto:ajover@dtic.ua.es))
- Sergio Orts-Escolano [Design, Technical Advice] ([sorts@ua.es](mailto:sorts@ua.es))
- Jose Garcia-Rodriguez [Technical Advice] ([jgarcia@dtic.ua.es](mailto:jgarcia@dtic.ua.es))

therobotrix

therobotrix Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

therobotrix

therobotrix Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README