From 5add065f8a7d4dc29ab28de4f9548d44eadc2aca Mon Sep 17 00:00:00 2001 From: David Novotny Date: Wed, 30 Nov 2022 07:02:01 -0800 Subject: [PATCH] Readme updates Summary: Running: - clearly points users to experiment.py/visualize_reconstruction.py Reproducing: - Adds NeRF training on Blender - Adds CO3Dv2 configs Reviewed By: bottler Differential Revision: D41534315 fbshipit-source-id: e85f5f1eafed8c35c9e91d748a04f238509cf8ec --- projects/implicitron_trainer/README.md | 136 ++++++++++++++++-- .../configs/repro_singleseq_nerf_blender.yaml | 2 +- projects/implicitron_trainer/experiment.py | 4 - 3 files changed, 127 insertions(+), 15 deletions(-) diff --git a/projects/implicitron_trainer/README.md b/projects/implicitron_trainer/README.md index fd98648c..d8e2fd9c 100644 --- a/projects/implicitron_trainer/README.md +++ b/projects/implicitron_trainer/README.md @@ -33,6 +33,7 @@ pip install "hydra-core>=1.1" visdom lpips matplotlib accelerate Runner executable is available as `pytorch3d_implicitron_runner` shell command. See [Running](#running) section below for examples of training and evaluation commands. + ## [Option 2] Supporting custom implementations To plug in custom implementations, for example, of renderer or implicit-function protocols, you need to create your own runner script and import the plug-in implementations there. @@ -55,11 +56,31 @@ pip install "hydra-core>=1.1" visdom lpips matplotlib accelerate You are still encouraged to implement custom plugins as above where possible as it makes reusing the code easier. The executable is located in `pytorch3d/projects/implicitron_trainer`. +> **_NOTE:_** Both `pytorch3d_implicitron_runner` and `pytorch3d_implicitron_visualizer` +executables (mentioned below) are not available when using local clone. +Instead users should use the python scripts `experiment.py` and `visualize_reconstruction.py` (see the [Running](Running) section below). + # Running -This section assumes that you use the executable provided by the installed package. -If you have a custom `experiment.py` script (as in the Option 2 above), replace the executable with the path to your script. +This section assumes that you use the executable provided by the installed package +(Option 1 / Option 2 in [#Installation](Installation) above), +i.e. `pytorch3d_implicitron_runner` and `pytorch3d_implicitron_visualizer` are available. + +> **_NOTE:_** If the executables are not available (e.g. when using a local clone - Option 3 in [#Installation](Installation)), +users should directly use the `experiment.py` and `visualize_reconstruction.py` python scripts +which correspond to the executables as follows: +- `pytorch3d_implicitron_runner` corresponds to `/projects/implicitron_trainer/experiment.py` +- `pytorch3d_implicitron_visualizer` corresponds to `/projects/implicitron_trainer/visualize_reconstruction.py` + +For instance, in order to directly execute training with the python script, users can call: +```shell +cd /projects/ \ +python -m implicitron_trainer.experiment ` +``` + +If you have a custom `experiment.py` or `visualize_reconstruction.py` script +(as in the Option 2 [above](#Installation)), replace the executable with the path to your script. ## Training @@ -80,6 +101,13 @@ and `` with a directory where checkpoints will be dumped during Other configuration parameters can be overridden in the same way. See [Configuration system](#configuration-system) section for more information on this. +### Visdom logging + +Note that the training script logs its progress to Visdom. Make sure to start a visdom server before the training commences: +``` +python -m visdom.server +``` +> In case a Visdom server is not started, the console will get flooded with `requests.exceptions.ConnectionError` errors signalling that a Visdom server is not available. Note that these errors will NOT interrupt the program and the training will still continue without issues. ## Evaluation @@ -105,14 +133,14 @@ conda install ffmpeg Here is an example of calling the script: ```shell -projects/implicitron_trainer/visualize_reconstruction.py exp_dir= \ +pytorch3d_implicitron_visualizer exp_dir= \ visdom_show_preds=True n_eval_cameras=40 render_size="[64,64]" video_size="[256,256]" ``` The argument `n_eval_cameras` sets the number of renderring viewpoints sampled on a trajectory, which defaults to a circular fly-around; `render_size` sets the size of a render passed to the model, which can be resized to `video_size` before writing. -Rendered videos of images, masks, and depth maps will be saved to `/vis`. +Rendered videos of images, masks, and depth maps will be saved to `/video`. # Configuration system @@ -127,8 +155,11 @@ Configurables can form hierarchies. For example, `GenericModel` has a field `raysampler: RaySampler`, which is also Configurable. In the config, inner parameters can be propagated using `_args` postfix, e.g. to change `raysampler.n_pts_per_ray_training` (the number of sampled points per ray), the node `raysampler_args.n_pts_per_ray_training` should be specified. -The root of the hierarchy is defined by `ExperimentConfig` dataclass. -It has top-level fields like `eval_only` which was used above for running evaluation by adding a CLI override. +### Top-level configuration class: `Experiment` + +The root of the hierarchy is defined by `Experiment` Configurable in `/projects/implicitron_trainer/experiment.py`. + +It has top-level fields like `seed`, which seeds the random number generator. Additionally, it has non-leaf nodes like `model_factory_ImplicitronModelFactory_args.model_GenericModel_args`, which dispatches the config parameters to `GenericModel`. Thus, changing the model parameters may be achieved in two ways: either by editing the config file, e.g. ```yaml @@ -266,14 +297,66 @@ model_GenericModel_args: GenericModel Please look at the annotations of the respective classes or functions for the lists of hyperparameters. `tests/experiment.yaml` shows every possible option if you have no user-defined classes. -# Reproducing CO3D experiments + +# Implementations of existing methods + +We provide configuration files that implement several existing works. + +The configuration files live in `pytorch3d/projects/implicitron_trainer/configs`. + + +## NeRF + +The following config file corresponds to training of a vanilla NeRF on Blender Synthetic dataset +(see https://arxiv.org/abs/2003.08934 for details of the method): + +`./configs/repro_singleseq_nerf_blender.yaml` + + +### Downloading Blender-Synthetic +Training requires the Blender Synthetic dataset. +To download the dataset, visit the [gdrive bucket](https://drive.google.com/file/d/18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG/view?usp=share_link) +and click Download. +Then unpack the downloaded .zip file to a folder which we call ``. + + +### Launching NeRF training +In order to train NeRF on the "drums" scene, execute the following command: +```shell +export BLENDER_DATASET_ROOT="" \ +export BLENDER_SINGLESEQ_CLASS="drums" \ +pytorch3d_implicitron_runner --config-path ./configs/ --config-name repro_singleseq_nerf_blender +``` + +Note that the training scene is selected by setting the environment variable `BLENDER_SINGLESEQ_CLASS` +appropriately (one of `"chair"`, `"drums"`, `"ficus"`, `"hotdog"`, `"lego"`, `"materials"`, `"mic"`, `"ship"`). + +By default, the training outputs will be stored to `"./data/nerf_blender_repro/$BLENDER_SINGLESEQ_CLASS/"` + + +### Visualizing trained NeRF +```shell +pytorch3d_implicitron_visualizer exp_dir= \ + visdom_show_preds=True n_eval_cameras=40 render_size="[64,64]" video_size="[256,256]" +``` +where `` corresponds to the directory with the training outputs (defaults to `"./data/nerf_blender_repro/$BLENDER_SINGLESEQ_CLASS/"`). + +The script will output a rendered video of the learned radiance field to `"./data/nerf_blender_repro/$BLENDER_SINGLESEQ_CLASS/"` (requires `ffmpeg`). + +> **_NOTE:_** Recall that, if `pytorch3d_implicitron_runner`/`pytorch3d_implicitron_visualizer` are not available, replace the calls +with `cd /projects/; python -m implicitron_trainer.[experiment|visualize_reconstruction]` + + +## CO3D experiments Common Objects in 3D (CO3D) is a large-scale dataset of videos of rigid objects grouped into 50 common categories. Implicitron provides implementations and config files to reproduce the results from [the paper](https://arxiv.org/abs/2109.00512). Please follow [the link](https://github.com/facebookresearch/co3d#automatic-batch-download) for the instructions to download the dataset. In training and evaluation scripts, use the download location as ``. It is also possible to define environment variable `CO3D_DATASET_ROOT` instead of specifying it. -To reproduce the experiments from the paper, use the following configs. For single-sequence experiments: +To reproduce the experiments from the paper, use the following configs. + +For single-sequence experiments: | Method | config file | |-----------------|-------------------------------------| @@ -286,7 +369,7 @@ To reproduce the experiments from the paper, use the following configs. For sing | SRN + WCE | repro_singleseq_srn_wce_noharm.yaml | | SRN + WCE + γ | repro_singleseq_srn_wce_noharm.yaml | -For multi-sequence experiments (without generalisation to new sequences): +For multi-sequence autodecoder experiments (without generalization to new sequences): | Method | config file | |-----------------|--------------------------------------------| @@ -294,7 +377,7 @@ For multi-sequence experiments (without generalisation to new sequences): | SRN + AD | repro_multiseq_srn_ad_hypernet_noharm.yaml | | SRN + γ + AD | repro_multiseq_srn_ad_hypernet.yaml | -For multi-sequence experiments (with generalisation to new sequences): +For multi-sequence experiments (with generalization to new sequences): | Method | config file | |-----------------|--------------------------------------| @@ -302,3 +385,36 @@ For multi-sequence experiments (with generalisation to new sequences): | NerFormer | repro_multiseq_nerformer.yaml | | SRN + WCE | repro_multiseq_srn_wce_noharm.yaml | | SRN + WCE + γ | repro_multiseq_srn_wce.yaml | + + +## CO3Dv2 experiments + +The following config files implement training on the second version of CO3D, `CO3Dv2`. + +In order to launch trainings, set the `CO3DV2_DATASET_ROOT` environment variable +to the root folder of the dataset (note that the name of the env. variable differs from the CO3Dv1 version). + +Single-sequence experiments: + +| Method | config file | +|-----------------|-------------------------------------| +| NeRF | repro_singleseq_v2_nerf.yaml | +| NerFormer | repro_singleseq_v2_nerformer.yaml | +| IDR | repro_singleseq_v2_idr.yaml | +| SRN | repro_singleseq_v2_srn_noharm.yaml | + +Multi-sequence autodecoder experiments (without generalization to new sequences): + +| Method | config file | +|-----------------|--------------------------------------------| +| NeRF + AD | repro_multiseq_v2_nerf_ad.yaml | +| SRN + γ + AD | repro_multiseq_v2_srn_ad_hypernet.yaml | + +Multi-sequence experiments (with generalization to new sequences): + +| Method | config file | +|-----------------|----------------------------------------| +| NeRF + WCE | repro_multiseq_v2_nerf_wce.yaml | +| NerFormer | repro_multiseq_v2_nerformer.yaml | +| SRN + WCE | repro_multiseq_v2_srn_wce_noharm.yaml | +| SRN + WCE + γ | repro_multiseq_v2_srn_wce.yaml | diff --git a/projects/implicitron_trainer/configs/repro_singleseq_nerf_blender.yaml b/projects/implicitron_trainer/configs/repro_singleseq_nerf_blender.yaml index 1d75b893..2a92a92c 100644 --- a/projects/implicitron_trainer/configs/repro_singleseq_nerf_blender.yaml +++ b/projects/implicitron_trainer/configs/repro_singleseq_nerf_blender.yaml @@ -7,7 +7,7 @@ data_source_ImplicitronDataSource_args: dataset_length_train: 100 dataset_map_provider_class_type: BlenderDatasetMapProvider dataset_map_provider_BlenderDatasetMapProvider_args: - base_dir: ${oc.env:BLENDER_DATASET_ROOT} + base_dir: ${oc.env:BLENDER_DATASET_ROOT}/${oc.env:BLENDER_SINGLESEQ_CLASS} n_known_frames_for_test: null object_name: ${oc.env:BLENDER_SINGLESEQ_CLASS} path_manager_factory_class_type: PathManagerFactory diff --git a/projects/implicitron_trainer/experiment.py b/projects/implicitron_trainer/experiment.py index 4febd389..8f6879ab 100755 --- a/projects/implicitron_trainer/experiment.py +++ b/projects/implicitron_trainer/experiment.py @@ -19,10 +19,6 @@ from the command line, for example: ./experiment.py --config-name base_config.yaml override.param.one=42 override.param.two=84 ``` -To run an experiment on a specific GPU, specify the `gpu_idx` key in the -config file / CLI. To run on a different device, specify the device in -`run_training`. - Main functions --------------- - The Experiment class defines `run` which creates the model, optimizer, and other