From 9389791889e01dc4af4d0951358824bc4e2d5918 Mon Sep 17 00:00:00 2001 From: Guillaume Duret <guillaume.duret@ec-lyon.fr> Date: Wed, 14 Jun 2023 19:16:56 +0000 Subject: [PATCH] clean readme --- README.md | 195 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 109 insertions(+), 86 deletions(-) diff --git a/README.md b/README.md index 954f087..9d90124 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,18 @@ -# FruitBin -FruitBin is the repository used to precess the dataset from the PickSim genaration (https://gitlab.liris.cnrs.fr/gduret/PickSim ). +## FruitBIn -## Getting started +The early released dataset can be accesible to the link : https://datasets.liris.cnrs.fr/fruitbin-version1 + +The datasets for the 8 scenarios are currently available in compressed zip files under the "scenarios" section. The release of the large-scale dataset, which encompasses a broader range of data, will be made available at the same location as soon as possible, within a few weeks. Considering the substantial size of the dataset, users will have the option to selectively download specific data subsets based on their requirements, such as RGB, depth, and other features, allowing for more efficient and targeted data retrieval. + +Detailed information regarding the dataset generation process can be accessed through the following links: "https://gitlab.liris.cnrs.fr/gduret/PickSim" for insights into the PickSim generation, and this repositiry for access to the processing codes and procedures involved in creating FruitBin, as well as the scenarios designed for training purposes. These resources provide valuable documentation and code references for a better understanding of the dataset's origins and the steps taken to prepare it for various applications. -The expected use is using the followinf command : +To facilitate the reproduction of the benchmark results, we have provided the code for PVnet and Densefusion along with detailed instructions on how to apply them to the FruitBin dataset. You can access the code at the following links: "https://gitlab.liris.cnrs.fr/gduret/pvnet_fruitbin" and "https://gitlab.liris.cnrs.fr/gduret/DenseFusion". These resources will guide you through the implementation of pvnet and Densefusion on the FruitBin dataset, enabling you to replicate the benchmark and further explore the capabilities of the dataset. + + +## Getting started +The expected usage involves utilizing the following command: ``` python main.py --World_begin="$id_begin" --Nb_world="$Nb" --dataset_id="$id_dataset" --occlusion_target_min=$occlusion_min --occlusion_target_max=$occlusion_max --rearrange=$rearrange --compute=$compute @@ -15,41 +22,40 @@ as the exemple : python main.py --World_begin=1 --Nb_world=10000 --dataset_id=1 --occlusion_target_min=0.7 --occlusion_target_max=1.0 --rearrange=yes --compute=yes ``` +The following table presents the different input parameters: -The following table present the different input parameters : -| Parameters | Information | +| Parameter | Description | | :---: | :---: | -| Parameters | Information | -| World_begin | the id of the first scene id that we want to process | -| Nb_world | The number of scene we want to process | -| dataset_id | The id of the dataset we want to process, it is used to give the path of the data we want to process | -| rearrange | This parameter with expected value of "no or yes", decide if the script is doing a rearangement of the data from the data generated from PickSim | -| compute | This parameter with expected value of "no or yes", decide if the script is doing a post processing for a specific scenario | -| occlusion_target_min | In the case of a scenario preprossesing, the value give the lower band of visibility rate that we want the filtering | -| occlusion_target_max | In the case of a scenario preprossesing, the value give the uper band of visibility rate that we want the filtering | +| World_begin | The ID of the first scene to be processed. | +| Nb_world | The number of scenes to be processed. | +| dataset_id | The ID of the dataset to be processed. It is used to determine the path of the data to be processed. | +| rearrange | Determines whether the script should perform data rearrangement from the data generated by PickSim. | +| compute | Determines whether the script should perform post-processing for a specific scenario.| +| occlusion_target_min | In the case of scenario pre-processing, specifies the lower bound of the desired visibility rate for filtering. | +| occlusion_target_max | In the case of scenario pre-processing, specifies the upper bound of the desired visibility rate for filtering. | -Additionally, more parameters can be changes in the main.py as : +Furthermore, additional parameters can be modified in the main.py file, such as: -| Parameters | Information | +| Parameters | Description | | :---: | :---: | -| Nb_camera | The number of camera | -| dataset_src | The path directed to the data generated by PickSim that we want to process | -| dataset_path | The path destination for the rearanged dataset | -| dataset_id | The id of the dataset we want to process, it is used to give the path of the data we want to process | -| choice | This parameter with expected value of "low or high", decide if the script is doing a post processing for ground_truth_depth or ground_truth_rgb | -| list_categories | This the list of category present in the dataset that we want to consider | -| new_size | new size nedeed for resizing features for the futur training | +| Nb_camera | The number of cameras in the dataset | +| dataset_src | The path to the data generated by PickSim that needs to be processed | +| dataset_path | The destination path for the rearranged dataset | +| choice | Determines whether the script performs post-processing for depth sensor data or RGB sensor data | +| list_categories | The list of categories to be considered in the dataset| +| new_size | The desired size for resizing features for future training | ## Rearrange step -The rearange step will move data from the arrangement of Picksim as : + +The rearrange step involves moving data from the original organization of Picksim, such as: ``` ├── [Scene Id] | ├── Meta.json -│  ├── [camera_i] +│  ├── [Camera ID] | | ├── color | | │  └── image | | ├── depth @@ -83,8 +89,7 @@ The rearange step will move data from the arrangement of Picksim as : ``` -to the following organisation : - +to the following structured organization: ``` ├── Bbox_2d @@ -99,44 +104,44 @@ to the following organisation : ├── Semantic_Segmentation ``` -The following table present informations about the different folder of the PickSim generation that actually correspond to different sensors in the simulation within Gazebo to match the different sensors of the realsense camera D415. - -| Parameters | Information | -| :---: | :---: | -| Meta.json | scene oriented meta file, it enumerate all the data recorded during the scene while giving to the user the list of category simulated in the scen and the corresponding id instances. -| Scene Id | The scene id, the data generated is scene oriented | -| camera_i | The camera id, the data generated is camera oriented | -| color | The name of the sensor recording rgb image with a resolution of 1920x1080 (matching the data of the camera realsense D415) | -| depth | The name of the sensor recording depth data such as depth or point cloud with a resolution of 1280x720 (matching the data of the camera realsense D415 ) | -| ground_truth_depth | Recorded features from a new vision plugin for the depth sensor with a resolution of 1920x1080 | -| ground_truth_rgb | Recorded features from a new vision plugin for the depth sensor with a resolution of 1280x720 | -| infra1 | black and white infra chanel 1 with a resolution of | -| infra2 | black and white infra chanel 1 with a resolution of | +The table below provides information about the folders in the PickSim generation, which correspond to different sensors in the Gazebo simulation, aligning with the various sensors of the RealSense camera D415. +| Parameters | Description | +| :---: | :---: | +| Meta.json | Scene-oriented meta file that enumerates all the recorded data and provides the list of categories and their corresponding instance IDs. +| Scene ID | The ID of the scene for which the data was generated. | +| Camera ID | The ID of the camera for which the data was generated. | +| color | The name of the sensor recording RGB images with a resolution of 1920x1080, matching the data from the RealSense camera D415. | +| depth | The name of the sensor recording depth data or point cloud with a resolution of 1280x720, matching the data from the RealSense camera D415. | +| ground_truth_rgb | Recorded features from a new vision plugin for the color sensor with a resolution of 1920x1080. | +| ground_truth_depth | Recorded features from a new vision plugin for the depth sensor with a resolution of 1280x720. | +| infra1 | Black and white infra channel 1 with a resolution of 1280x720. | +| infra2 | Black and white infra channel 2 with a resolution of 1280x720. | -The resulted data is organized by features type. The following table present some information about the resulted data and the corresponding data from the raw data of PickSim : -| Parameters | equivalent PickSim | Information | -| :---: | :---: | :---: | -| Meta | [Scene Id]/Meta.json | -| Bbox_2d | [Scene Id]/[camera_i]/[ground_truth_depth]/2d_detection |In the pre prossessing part, and for the training of 6D pose estiamtion, all data are fruits based : | -| Bbox_2d_loose | [Scene Id]/[camera_i]/[ground_truth_depth]/2d_detection | Liste of meta data describing fruit based meta data such as Scene Id, Camera Id, list of instance id realted to the fruit or the list of occlsion rate associated. -| Bbox_3d | [Scene Id]/[camera_i]/[ground_truth_depth]/3d_detection | Bboxes | -| Depth | [Scene Id]/[camera_i]/[depth]/images | 3D Bboxes | -| Instance_Segmentation | [Scene Id]/[camera_i]/ground_truth_depth/id_map | The id of the dataset we want to process, it is used to give the path of the data we want to process | -| Occlusion | [Scene Id]/[camera_i]/ground_truth_depth/occlusion | This parameter with expected value of "no or yes", decide if the script is doing a rearangement of the data from the data generated from PickSim | -| Pose | [Scene Id]/[camera_i]/ground_truth_depth/3d_pose | This parameter with expected value of "no or yes", decide if the script is doing a post processing for a specific scenario | -| RGB | [Scene Id]/[camera_i]/color/image | In the case of a scenario preprossesing, the value give the lower band of occlusion rate that we want the filtering | -| Semantic_Segmentation | [Scene Id]/[camera_i]/ground_truth_depth/sematic_map | In the case of a scenario preprossesing, the value give the uper band of occlusion rate that we want the filtering | +The resulting data is organized based on the type of features. The following table presents information about the resulting data and its corresponding raw data from PickSim: -The ground truth annotation only take ground_truth_depth to consider RGB-D data with the same resolution. However all the features will be added into the dataset for potential use of the community (https://datasets.liris.cnrs.fr/fruitbin-version1 ) +| Parameters | Equivalent PickSim | Description | +| :---: | :---: | :---: | +| Meta | [Scene Id]/Meta.json | Scene-oriented metadata file that enumerates all recorded data and provides a list of categories and instance IDs.| +| Bbox_2d | [Scene Id]/[camera_i]/[ground_truth_depth]/2d_detection | 2D bounding boxes of objects in the scene. | | +| Bbox_2d_loose | [Scene Id]/[camera_i]/[ground_truth_depth]/2d_detection | Loose 2D bounding boxes of objects in the scene. | +| Bbox_3d | [Scene Id]/[camera_i]/[ground_truth_depth]/3d_detection | 3D bounding boxes of objects in the scene. | +| Depth | [Scene Id]/[camera_i]/[depth]/images | Depth map captured by the depth sensor. | +| Instance_Segmentation | [Scene Id]/[camera_i]/ground_truth_depth/id_map | Instance segmentation mask of objects in the scene. | +| Semantic_Segmentation | [Scene Id]/[camera_i]/ground_truth_depth/sematic_map | Semantic segmentation mask of objects in the scene. | +| Occlusion | [Scene Id]/[camera_i]/ground_truth_depth/occlusion | Occlusion rate of each instance in the scene. | +| Pose | [Scene Id]/[camera_i]/ground_truth_depth/3d_pose | 6D pose (position and orientation) of each instance in the scene. | +| RGB | [Scene Id]/[camera_i]/depth/image | RGB image captured by the depth sensor. | + +The ground truth annotation exclusively utilizes the ground_truth_depth data, which ensures consistency with RGB-D data of the same resolution. However, all features will be included in the dataset to cater to the potential needs of the community. You can access the dataset, including these features, at the following link: [https://datasets.liris.cnrs.fr/fruitbin-version1]. ## Compute step for PVnet and the the scenarios -The compute step will take as input the rearanged data and process it for the futur training. It will create and process the data into the nedeed format of Pvnet 6D pose estiamtion model. Moroever for the training, all data is category/fruit realted as presented in the following architectures: +The compute step takes the rearranged data as input and processes it for future training purposes. It formats the data into the required structure for training the Pvnet 6D pose estimation model. Additionally, for training purposes, all the data is categorized and fruit-related, as illustrated in the following architectures: ``` ├── Fruit_i @@ -157,55 +162,73 @@ The compute step will take as input the rearanged data and process it for the fu │ └── Splitting ``` -The following table give information about the different features generated. Moroever this processed data take into account the filtering parameters given into the main.py script such as the wanted occlusion. Also, for the purpose of FruitBin, the splitting for the scene scenario is done for 6000 scenes for training, 2000 scenes for evalution and testing for a total of 10000. Additionally, camera take 9 camera as training and 3 for evaluating and testing for a total of 15. - -| Parameters | Information | +The table below provides information about the various generated features. Furthermore, this processed data takes into account the filtering parameters specified in the main.py script, such as the desired level of occlusion. Regarding FruitBin, the scene scenarios are divided into 6000 scenes for training, 2000 scenes for evaluation, and 10000 scenes for testing. In addition, 9 cameras are allocated for training, while 3 cameras are assigned for evaluation and testing, resulting in a total of 15 cameras. + + +Fruit_i The fruit category being considered +Meta_Gen Metadata describing fruit-specific information such as Scene ID, Camera ID, a list of instance IDs related to the fruit, and associated occlusion rates +BBox 2D bounding boxes +Bbox_3d_Gen 3D bounding boxes +Depth_Gen Depth map data with a resolution of 1280x720 +Depth_resized Resized depth map data with a resolution of 640x480 for training +FPS Farthest Point Sampling (FPS) key points for the 1280x720 image used in Pvnet +FPS_resized Resized FPS data with a resolution of 640x480 for training in Pvnet +Instance_Mask Instance mask data with a resolution of 1280x720 +Instance_Mask_resized Resized instance mask data with a resolution of 640x480 for training +Labels Instance mask in the Yolov8 format (generated using the 'compute label' script explained below) +Models Meshes of the 8 fruits in a common PLY format +Pose_transformed 6D pose annotations in the PVNet format +RGB_Gen RGB image data with a resolution of 1280x720 +RGB_resized Resized RGB image data with a resolution of 640x480 +Splitting This folder is only available when the dataset is downloaded online. It contains a list of .txt splitting files for different scenarios, describing the train/eval/test split. + +| Parameters | Description | | :---: | :---: | -| Fruit_i | The fruit we are considering, the following data are category oriented | -| Meta_Gen | Liste of meta data describing fruit based meta data such as Scene Id, Camera Id, list of instance id realted to the fruit or the list of occlsion rate associated. -| BBox | Bboxes | -| Bbox_3d_Gen | 3D Bboxes | +| Fruit_i | The fruit category being considered | +| Meta_Gen | Metadata describing fruit-specific information such as Scene ID, Camera ID, a list of instance IDs related to the fruit, and associated occlusion rates | +| BBox | Bboxes | 2D bounding boxes | +| Bbox_3d_Gen | 3D Bboxes | 3D bounding boxes | | Depth_Gen | Depth map data with a resolution of 1280x720 | -| Depth_resized | Depth map data with a resolution of 640x480 for training | -| FPS | FPS(farthest point sampling) key point for the image of 1280x720 for Pvnet | -| FPS_resized | FPS data with a resolution of 640x480 for training for Pvnet | +| Depth_resized | Resized depth map data with a resolution of 640x480 for training | +| FPS | Farthest Point Sampling (FPS) key points for the 1280x720 image used in Pvnet | +| FPS_resized | Resized FPS data with a resolution of 640x480 for training in Pvnet | | Instance_Mask | Instance mask data with a resolution of 1280x720 | -| Instance_Mask_resized | Instance mask data with a resolution of 640x480 for training | -| Labels | Instance mask in the Yolov8 format (it is actully generates with the script compute label as explain bellow) | -| Models | The meshes of the 8 fruit in a common ply format | -| Pose_transformed | 6D pose annotation in the format of PVNet | -| RGB_Gen | RGB image data with a resolution of 1280x720 | +| Instance_Mask_resized | Resized instance mask data with a resolution of 640x480 for training | +| Labels | Instance mask in the Yolov8 format (generated using the 'compute label' script explained below) | +| Models | Meshes of the 8 fruits in a common PLY format | +| Pose_transformed | 6D pose annotations in the PVNet format | +| RGB_Gen | RGB image data with a resolution of 1280x720 | | RGB_resized | RGB image data with a resolution of 1280x720 | -| Splitting | Folder only available when dataset downloaded online, it gives a list of .txt splutting files for different scenarios and for describing the split train/eval/test | - -The following step fully prepare the data for PVnet training. More information can be found into the link : TODO +| Splitting | This folder is only available when the dataset is downloaded online. It contains a list of .txt splitting files for different scenarios, describing the train/eval/test split. | +The following step fully prepares the data for PVNet training. For more detailed information, please refer to the following link: https://gitlab.liris.cnrs.fr/gduret/pvnet_fruitbin ## Compute step for Densefusion - -The last step is to prepare the data for the training of Densefusion. To do so the command is as exemple : - +The final step involves preparing the data for DenseFusion training. To accomplish this, you can use the following command as an example: ``` python3 compute_label.py --path_dataset=/gpfsscratch/rech/uli/ubn15wo/FruitBin1/FruitBin_low_1_0.7_1.0/ --target_folder=Generated_Cameras --path_DF_data=/gpfsscratch/rech/uli/ubn15wo/DenseFusion01_Cameras/datasets/linemod/Linemod_preprocessed/data --occ_data="" ``` +Parameters Description +path_dataset The path to the preprocessed dataset generated during the previous compute step. +target_folder The specific scenario to be considered for DenseFusion training. +path_DF_data The path to the Densefusion folder where the training data will be placed. +occ_data An additional parameter that modifies the name of the resulting .txt DenseFusion splitting file when multiple scenarios are considered within the same folder. + The following table present the different input parameters : -| Parameters | Information | +| Parameters | Description | | :---: | :---: | -| path_dataset | The Path of the preprocessed dataset generated during the compute previous step. | -| target_folder | The exact scerario we want to consider for densefusion | -| path_DF_data | The path of the Densefusion folder where we want to put the data for training | -| occ_data | This additional parameter will change the name of the resulted .txt densefusion splitting in the case of consideration of multiple scenario in the same folder | - -With this step, the preprocessing for DenseFusion is done. See the following link for more information : https://gitlab.liris.cnrs.fr/gduret/DenseFusion - +| path_dataset | The path to the preprocessed dataset generated during the previous compute step. | +| target_folder | The specific scenario to be considered for DenseFusion training. | +| path_DF_data | The path to the Densefusion folder where the training data will be placed.| +| occ_data | An additional parameter that modifies the name of the resulting .txt DenseFusion splitting file when multiple scenarios are considered within the same folder. | +The preprocessing for DenseFusion is now complete. For more detailed information, please refer to the following link: https://gitlab.liris.cnrs.fr/gduret/DenseFusion . ## Authors and acknowledgment Show your appreciation to those who have contributed to the project. ## License -For open source projects, say how it is licensed. - +The dataset is licensed under the CC BY-NC-SA license. -- GitLab