Summary:
Adding MeshRasterizerOpenGL, a faster alternative to MeshRasterizer. The new rasterizer follows the ideas from "Differentiable Surface Rendering via non-Differentiable Sampling".
The new rasterizer 20x faster on a 2M face mesh (try pose optimization on Nefertiti from https://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/!). The larger the mesh, the larger the speedup.
There are two main disadvantages:
* The new rasterizer works with an OpenGL backend, so requires pycuda.gl and pyopengl installed (though we avoided writing any C++ code, everything is in Python!)
* The new rasterizer is non-differentiable. However, you can still differentiate the rendering function if you use if with the new SplatterPhongShader which we recently added to PyTorch3D (see the original paper cited above).
Reviewed By: patricklabatut, jcjohnson
Differential Revision: D37698816
fbshipit-source-id: 54d120639d3cb001f096237807e54aced0acda25
Summary:
EGLContext is a utility to render with OpenGL without an attached display (that is, without a monitor).
DeviceContextManager allows us to avoid unnecessary context creations and releases. See docstrings for more info.
Reviewed By: jcjohnson
Differential Revision: D36562551
fbshipit-source-id: eb0d2a2f85555ee110e203d435a44ad243281d2c
Summary: Avoid calculating all_train_cameras before it is needed, because it is slow in some datasets.
Reviewed By: shapovalov
Differential Revision: D38037157
fbshipit-source-id: 95461226655cde2626b680661951ab17ebb0ec75
Summary:
X-link: https://github.com/fairinternal/pytorch3d/pull/39
Blender and LLFF cameras were sending screen space focal length and principal point to a camera init function expecting NDC
Reviewed By: shapovalov
Differential Revision: D37788686
fbshipit-source-id: 2ddf7436248bc0d174eceb04c288b93858138582
Summary:
EPnP fails the test when the number of points is below 6. As suggested, quadratic option is in theory to deal with as few as 4 points (so num_pts_thresh=3 is set). And when num_pts > num_pts_thresh=4, skip_q is False.
To avoid bumping num_pts_thresh while passing all the original tests, check_output is set to False when num_pts < 6, similar to the logic in Line 123-127. It makes sure that the algo doesn't crash.
Reviewed By: shapovalov
Differential Revision: D37804438
fbshipit-source-id: 74576d63a9553e25e3ec344677edb6912b5f9354
Summary:
This fixes a indexing bug in HardDepthShader and adds proper unit tests for both of the DepthShaders. This bug was introduced when updating the shader sizes and discovered when I switched my local model onto pytorch3d trunk instead of the patched copy.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/1252
Test Plan:
Unit test + custom model code
```
pytest tests/test_shader.py
```
Reviewed By: bottler
Differential Revision: D37775767
Pulled By: d4l3k
fbshipit-source-id: 5f001903985976d7067d1fa0a3102d602790e3e8
Summary:
For 3D segmentation problems it's really useful to be able to train the models from multiple viewpoints using Pytorch3D as the renderer. Currently due to hardcoded assumptions in a few spots the mesh renderer only supports rendering RGB (3 dimensional) data. You can encode the classification information as 3 channel data but if you have more than 3 classes you're out of luck.
This relaxes the assumptions to make rendering semantic classes work with `HardFlatShader` and `AmbientLights` with no diffusion/specular. The other shaders/lights don't make any sense for classification since they mutate the texture values in some way.
This only requires changes in `Materials` and `AmbientLights`. The bulk of the code is the unit test.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/1248
Test Plan: Added unit test that renders a 5 dimensional texture and compare dimensions 2-5 to a stored picture.
Reviewed By: bottler
Differential Revision: D37764610
Pulled By: d4l3k
fbshipit-source-id: 031895724d9318a6f6bab5b31055bb3f438176a5
Summary: A new version of json index dataset provider supporting CO3Dv2
Reviewed By: shapovalov
Differential Revision: D37690918
fbshipit-source-id: bf2d5fc9d0f1220259e08661dafc69cdbe6b7f94
Summary:
Implements several changes needed for the CO3Dv2 release:
- FrameData contains crop_bbox_xywh which defines the outline of the image crop corresponding to the image-shaped tensors in FrameData
- revised the definition of a bounding box inside JsonDatasetIndex: bbox_xyxy is [xmin, ymin, xmax, ymax], where xmax, ymax are not inclusive; bbox_xywh = [xmin, ymain, xmax-xmin, ymax-ymin]
- is_filtered for detecting whether the entries of the dataset were somehow filtered
- seq_frame_index_to_dataset_index allows to skip entries that are not present in the dataset
Reviewed By: shapovalov
Differential Revision: D37687547
fbshipit-source-id: 7842756b0517878cc0964fc0935d3c0769454d78
Summary: As part of removing Task, make the dataset code generate the source cameras for itself. There's a small optimization available here, in that the JsonIndexDataset could avoid loading images.
Reviewed By: shapovalov
Differential Revision: D37313423
fbshipit-source-id: 3e5e0b2aabbf9cc51f10547a3523e98c72ad8755
Summary:
Make ViewMetrics easy to replace by putting them into an OmegaConf dataclass.
Also, re-word a few variable names and fix minor TODOs.
Reviewed By: bottler
Differential Revision: D37327157
fbshipit-source-id: 78d8e39bbb3548b952f10abbe05688409fb987cc
Summary: After landing https://github.com/pytorch/pytorch/pull/69607, that made it an error to use indexing with `cpu_tensor[cuda_indices]`. There was one outstanding test in fbcode that incorrectly used indexing in that way, which is fixed here
Reviewed By: bottler, osalpekar
Differential Revision: D37128838
fbshipit-source-id: 611b6f717b5b5d89fa61fd9ebeb513ad7e65a656
Summary:
David had his code crashed when using frame_annot["meta"] dictionary. Turns out we had a typo.
The tests were passing by chance since all the keys were single-character strings.
Reviewed By: bottler
Differential Revision: D37503987
fbshipit-source-id: c12b0df21116cfbbc4675a0182b9b9e6d62bad2e
Summary: small followup to D37172537 (cba26506b6) and D37209012 (81d63c6382): changing default #harmonics and improving a test
Reviewed By: shapovalov
Differential Revision: D37412357
fbshipit-source-id: 1af1005a129425fd24fa6dd213d69c71632099a0
Summary: The blender synthetic dataset contains object masks in the alpha channel. Provide these in the corresponding dataset.
Reviewed By: shapovalov
Differential Revision: D37344380
fbshipit-source-id: 3ddacad9d667c0fa0ae5a61fb1d2ffc806c9abf3
Summary: Images were coming out in the wrong format.
Reviewed By: shapovalov
Differential Revision: D37291278
fbshipit-source-id: c10871c37dd186982e7abf2071ac66ed583df2e6
Summary: Allow specifying a color for non-opaque pixels in LSTMRenderer.
Reviewed By: davnov134
Differential Revision: D37172537
fbshipit-source-id: 6039726678bb7947f7d8cd04035b5023b2d5398c
Summary: Copy code from NeRF for loading LLFF data and blender synthetic data, and create dataset objects for them
Reviewed By: shapovalov
Differential Revision: D35581039
fbshipit-source-id: af7a6f3e9a42499700693381b5b147c991f57e5d
Summary: Make dataset type and args configurable on JsonIndexDatasetMapProvider.
Reviewed By: davnov134
Differential Revision: D36666705
fbshipit-source-id: 4d0a3781d9a956504f51f1c7134c04edf1eb2846
Summary: Changes to JsonIndexDataset to make it fit with OmegaConf.structured. Also match some default values to what the provider defaults to.
Reviewed By: davnov134
Differential Revision: D36666704
fbshipit-source-id: 65b059a1dbaa240ce85c3e8762b7c3db3b5a6e75
Summary: Allow a class to modify its subparts in get_default_args by defining the special function provide_config_hook.
Reviewed By: davnov134
Differential Revision: D36671081
fbshipit-source-id: 3e5b73880cb846c494a209c4479835f6352f45cf
Summary: Our tests fail (https://fburl.com/jmoqo9bz) because test_splatter_blend uses torch.tile, which is not supported in earlier torch versions. Replace it with tensor.extend.
Reviewed By: bottler
Differential Revision: D36796098
fbshipit-source-id: 38d5b40667f98f3163b33f44e53e96b858cfeba2
Summary: Fix divide by zero for empty pointcloud in chamfer. Also for empty batches. In process, needed to regularize num_points_per_cloud for empty batches.
Reviewed By: kjchalup
Differential Revision: D36311330
fbshipit-source-id: 3378ab738bee77ecc286f2110a5c8dc445960340
Summary: A few minor additions I didn't fit into the SplatterBlender diffs, as requested by reviewers.
Reviewed By: jcjohnson
Differential Revision: D36682437
fbshipit-source-id: 57af995e766dfd2674b3984a3ba00aef7ca7db80
Summary: The ImplicitronDataset class corresponds to JsonIndexDatasetMapProvider
Reviewed By: shapovalov
Differential Revision: D36661396
fbshipit-source-id: 80ca2ff81ef9ecc2e3d1f4e1cd14b6f66a7ec34d
Summary: test_viewpool was inactive so missed being fixed in D36547815 (2d1c6d5d93)
Reviewed By: kjchalup
Differential Revision: D36625587
fbshipit-source-id: e7224eadfa5581fe61f10f67d2221071783de04a
Summary:
Benchmarking. We only use num_faces=2 for splatter, because as far as I can see one would never need to use more. Pose optimization and mesh optimization experiments (see next two diffs) showed that Splatter with 2 faces beats Softmax with 50 and 100 faces in terms of accuracy.
Results: We're slower at 64px^2. At 128px and 256px, we're slower than Softmax+50faces, but faster than Softmax+100faces. We're also slower at 10 faces/pix, but expectation as well as results show that more then 2 faces shouldn't be necessary. See also more results in .https://fburl.com/gdoc/ttv7u7hp
Reviewed By: jcjohnson
Differential Revision: D36210575
fbshipit-source-id: c8de28c8a59ce5fe21a47263bd43d2757b15d123
Summary: Splatting shader. See code comments for details. Same API as SoftPhongShader.
Reviewed By: jcjohnson
Differential Revision: D36354301
fbshipit-source-id: 71ee37f7ff6bb9ce028ba42a65741424a427a92d
Summary: PLY with mixture of triangle and quadrilateral faces was failing.
Reviewed By: gkioxari
Differential Revision: D36592981
fbshipit-source-id: 5373edb2f38389ac646a75fd2e1fa7300eb8d054
Summary: Use small image size for test_all_gm_configs
Reviewed By: shapovalov
Differential Revision: D36511528
fbshipit-source-id: 2c65f518a4f23626850343a62d103f85abfabd88
Summary: replace dataset_zoo with a pluggable DatasetMapProvider. The logic is now in annotated_file_dataset_map_provider.
Reviewed By: shapovalov
Differential Revision: D36443965
fbshipit-source-id: 9087649802810055e150b2fbfcc3c197a761f28a
Summary: Separate ImplicitronDatasetBase and FrameData (to be used by all data sources) from ImplicitronDataset (which is specific).
Reviewed By: shapovalov
Differential Revision: D36413111
fbshipit-source-id: 3725744cde2e08baa11aff4048237ba10c7efbc6
Summary: Make ResNetFeatureExtractor be an implementation of FeatureExtractorBase.
Reviewed By: davnov134
Differential Revision: D35433098
fbshipit-source-id: 0664a9166a88e150231cfe2eceba017ae55aed3a
Summary: Allow extra data in a FrameAnnotation. Therefore allow Optional[T] systematically in _dataclass_from_dict
Reviewed By: davnov134
Differential Revision: D36442691
fbshipit-source-id: ba70f6491574c08b0d9c9acb63f35514d29de214
Summary: Fix recently observed case where enable_get_default_args was missing things declared as Optional[something mutable]=None.
Reviewed By: davnov134
Differential Revision: D36440492
fbshipit-source-id: 192ec07564c325b3b24ccc49b003788f67c63a3d
