Summary:
Moving SQL dataset to PyTorch3D. It has been extensively tested in pixar_replay.
It requires SQLAlchemy 2.0, which is not supported in fbcode. So I exclude the sources and tests that depend on it from buck TARGETS.
Reviewed By: bottler
Differential Revision: D45086611
fbshipit-source-id: 0285f03e5824c0478c70ad13731525bb5ec7deef
Summary:
We currently support caching bounding boxes in MaskAnnotation. If present, they are not re-computed from the mask. However, the masks need to be loaded for the bbox to be set.
This diff fixes that. Even if load_masks / load_blobs are unset, the bounding box can be picked up from the metadata.
Reviewed By: bottler
Differential Revision: D45144918
fbshipit-source-id: 8a2e2c115e96070b6fcdc29cbe57e1cee606ddcd
Summary: The code does not crash if depth map/mask are not given.
Reviewed By: bottler
Differential Revision: D45082985
fbshipit-source-id: 3610d8beb4ac897fbbe52f56a6dd012a6365b89b
Summary: Provide an extension point pre_expand to let a configurable class A make sure another class B is registered before A is expanded. This reduces top level imports.
Reviewed By: bottler
Differential Revision: D44504122
fbshipit-source-id: c418bebbe6d33862d239be592d9751378eee3a62
Summary:
Introduces the OverfitModel for NeRF-style training with overfitting to one scene.
It is a specific case of GenericModel. It has been disentangle to ease usage.
## General modification
1. Modularize a minimum GenericModel to introduce OverfitModel
2. Introduce OverfitModel and ensure through unit testing that it behaves like GenericModel.
## Modularization
The following methods have been extracted from GenericModel to allow modularity with ManyViewModel:
- get_objective is now a call to weighted_sum_losses
- log_loss_weights
- prepare_inputs
The generic methods have been moved to an utils.py file.
Simplify the code to introduce OverfitModel.
Private methods like chunk_generator are now public and can now be used by ManyViewModel.
Reviewed By: shapovalov
Differential Revision: D43771992
fbshipit-source-id: 6102aeb21c7fdd56aa2ff9cd1dd23fd9fbf26315
Summary: Indexing with a big matrix now fails with a ValueError, possibly because of pytorch improvements. Remove the testcase for it.
Reviewed By: davidsonic
Differential Revision: D42609741
fbshipit-source-id: 0a5a6632ed199cb942bfc4cc4ed347b72e491125
Summary: For the new API, filtering iterators over sequences by subsets is quite helpful. The change is backwards compatible.
Reviewed By: bottler
Differential Revision: D42739669
fbshipit-source-id: d150a404aeaf42fd04a81304c63a4cba203f897d
Summary:
Fixes some issues with RayBundle plotting:
- allows plotting raybundles on gpu
- view -> reshape since we do not require contiguous raybundle tensors as input
Reviewed By: bottler, shapovalov
Differential Revision: D42665923
fbshipit-source-id: e9c6c7810428365dca4cb5ec80ef15ff28644163
Summary: Use IndexError so that a camera object is an iterable
Reviewed By: shapovalov
Differential Revision: D42312021
fbshipit-source-id: 67c417d5f1398e8b30a6944468eda057b4ceb444
Summary: Make GLB files report their own length correctly. They were off by 28.
Reviewed By: davidsonic
Differential Revision: D41838340
fbshipit-source-id: 9cd66e8337c142298d5ae1d7c27e51fd812d5c7b
Summary: Write the amalgamated mesh from the Mesh module to glb. In this version, the json header and the binary data specified by the buffer are merged into glb. The image texture attributes are added.
Reviewed By: bottler
Differential Revision: D41489778
fbshipit-source-id: 3af0e9a8f9e9098e73737a254177802e0fb6bd3c
Summary:
Rasterize MC was not adapted to heterogeneous bundles.
There are some caveats though:
1) on CO3D, we get up to 18 points per image, which is too few for a reasonable visualisation (see below);
2) rasterising for a batch of 100 is slow.
I also moved the unpacking code close to the bundle to be able to reuse it.
{F789678778}
Reviewed By: bottler, davnov134
Differential Revision: D41008600
fbshipit-source-id: 9f10f1f9f9a174cf8c534b9b9859587d69832b71
Summary: Fix indexing of directions after filtering of points by scaffold.
Reviewed By: shapovalov
Differential Revision: D40853482
fbshipit-source-id: 9cfdb981e97cb82edcd27632c5848537ed2c6837
Summary:
Allows loading of multiple categories.
Multiple categories are provided in a comma-separated list of category names.
Reviewed By: bottler, shapovalov
Differential Revision: D40803297
fbshipit-source-id: 863938be3aa6ffefe9e563aede4a2e9e66aeeaa8
Summary:
According to the profiler trace D40326775, _check_valid_rotation_matrix is slow because of aten::all_close operation and _safe_det_3x3 bottlenecks. Disable the check by default unless environment variable PYTORCH3D_CHECK_ROTATION_MATRICES is set to 1.
Comparison after applying the change:
```
Profiling/Function get_world_to_view (ms) Transform_points(ms) specular(ms)
before 12.751 18.577 21.384
after 4.432 (34.7%) 9.248 (49.8%) 11.507 (53.8%)
```
Profiling trace:
https://pxl.cl/2h687
More details in https://docs.google.com/document/d/1kfhEQfpeQToikr5OH9ZssM39CskxWoJ2p8DO5-t6eWk/edit?usp=sharing
Reviewed By: kjchalup
Differential Revision: D40442503
fbshipit-source-id: 954b58de47de235c9d93af441643c22868b547d0
Summary:
Adds the ability to have different learning rates for different parts of the model. The trainable parts of the implicitron have a new member
param_groups: dictionary where keys are names of individual parameters,
or module’s members and values are the parameter group where the
parameter/member will be sorted to. "self" key is used to denote the
parameter group at the module level. Possible keys, including the "self" key
do not have to be defined. By default all parameters are put into "default"
parameter group and have the learning rate defined in the optimizer,
it can be overriden at the:
- module level with “self” key, all the parameters and child
module s parameters will be put to that parameter group
- member level, which is the same as if the `param_groups` in that
member has key=“self” and value equal to that parameter group.
This is useful if members do not have `param_groups`, for
example torch.nn.Linear.
- parameter level, parameter with the same name as the key
will be put to that parameter group.
And in the optimizer factory, parameters and their learning rates are recursively gathered.
Reviewed By: shapovalov
Differential Revision: D40145802
fbshipit-source-id: 631c02b8d79ee1c0eb4c31e6e42dbd3d2882078a
Summary:
Small config system fix. Allows get_default_args to work on an instance which has been created with a dict (instead of a DictConfig) as an args field. E.g.
```
gm = GenericModel(
raysampler_AdaptiveRaySampler_args={"scene_extent": 4.0}
)
OmegaConf.structured(gm1)
```
Reviewed By: shapovalov
Differential Revision: D40341047
fbshipit-source-id: 587d0e8262e271df442a80858949a48e5d6db3df
Summary: Tensorf does relu or softmax after the density grid. This diff adds the ability to replicate that.
Reviewed By: bottler
Differential Revision: D40023228
fbshipit-source-id: 9f19868cd68460af98ab6e61c7f708158c26dc08
Summary:
TensoRF at step 2000 does volume croping and resizing.
At those steps it calculates part of the voxel grid which has density big enough to have objects and resizes the grid to fit that object.
Change is done on 3 levels:
- implicit function subscribes to epochs and at specific epochs finds the bounding box of the object and calls resizing of the color and density voxel grids to fit it
- VoxelGrid module calls cropping of the underlaying voxel grid and resizing to fit previous size it also adjusts its extends and translation to match wanted size
- Each voxel grid has its own way of cropping the underlaying data
Reviewed By: kjchalup
Differential Revision: D39854548
fbshipit-source-id: 5435b6e599aef1eaab980f5421d3369ee4829c50
Summary: Forward method is sped up using the scaffold, a low resolution voxel grid which is used to filter out the points in empty space. These points will be predicted as having 0 density and (0, 0, 0) color. The points which were not evaluated as empty space will be passed through the steps outlined above.
Reviewed By: kjchalup
Differential Revision: D39579671
fbshipit-source-id: 8eab8bb43ef77c2a73557efdb725e99a6c60d415
Summary:
Torch C++ extension for Marching Cubes
- Add torch C++ extension for marching cubes. Observe a speed up of ~255x-324x speed up (over varying batch sizes and spatial resolutions)
- Add C++ impl in existing unit-tests.
(Note: this ignores all push blocking failures!)
Reviewed By: kjchalup
Differential Revision: D39590638
fbshipit-source-id: e44d2852a24c2c398e5ea9db20f0dfaa1817e457
Summary: Overhaul of marching_cubes_naive for better performance and to avoid relying on unstable hashing. In particular, instead of hashing vertex positions, we index each interpolated vertex with its corresponding edge in the 3d grid.
Reviewed By: kjchalup
Differential Revision: D39419642
fbshipit-source-id: b5fede3525c545d1d374198928dfb216262f0ec0
Summary:
Threaded the for loop:
```
for (int yi = 0; yi < H; ++yi) {...}
```
in function `RasterizeMeshesNaiveCpu()`.
Chunk size is approx equal.
Reviewed By: bottler
Differential Revision: D40063604
fbshipit-source-id: 09150269405538119b0f1b029892179501421e68
Summary:
Changed ray_sampler and metrics to be able to use mixed frame raysampling.
Ray_sampler now has a new member which it passes to the pytorch3d raysampler.
If the raybundle is heterogeneous metrics now samples images by padding xys first. This reduces memory consumption.
Reviewed By: bottler, kjchalup
Differential Revision: D39542221
fbshipit-source-id: a6fec23838d3049ae5c2fd2e1f641c46c7c927e3
Summary: new implicitronRayBundle with added cameraIDs and camera counts. Added to enable a single raybundle inside Implicitron and easier extension in the future. Since RayBundle is named tuple and RayBundleHeterogeneous is dataclass and RayBundleHeterogeneous cannot inherit RayBundle. So if there was no ImplicitronRayBundle every function that uses RayBundle now would have to use Union[RayBundle, RaybundleHeterogeneous] which is confusing and unecessary complicated.
Reviewed By: bottler, kjchalup
Differential Revision: D39262999
fbshipit-source-id: ece160e32f6c88c3977e408e966789bf8307af59
Summary:
Added heterogeneous raysampling to pytorch3d raysampler, different cameras are sampled different number of times.
It now returns RayBundle if heterogeneous raysampling is off and new RayBundleHeterogeneous (with added fields `camera_ids` and `camera_counts`). Heterogeneous raysampling is on if `n_rays_total` is not None.
Reviewed By: bottler
Differential Revision: D39542222
fbshipit-source-id: d3d88d822ec7696e856007c088dc36a1cfa8c625
Summary:
This is quite a thin wrapper – not sure we need it. The motivation is that `Transform3d` is not as matrix-centric now, it can be converted to SE(3) logarithm equally easily.
It simplifies things like averaging cameras and getting axis-angle of camera rotation (previously, one would need to call `se3_log_map(cameras.get_world_to_camera_transform().get_matrix())`), now one fewer thing to call / discover.
Reviewed By: bottler
Differential Revision: D39928000
fbshipit-source-id: 85248d5b8af136618f1d08791af5297ea5179d19
Summary:
`get_rotation_to_best_fit_xy` is useful to expose externally, however there was a bug (which we probably did not care about for our use case): it could return a rotation matrix with det(R) == −1.
The diff fixes that, and also makes centroid optional (it can be computed from points).
Reviewed By: bottler
Differential Revision: D39926791
fbshipit-source-id: 5120c7892815b829f3ddcc23e93d4a5ec0ca0013
Summary: Any module can be subscribed to step updates from the training loop. Once the training loop publishes a step the voxel grid changes its dimensions. During the construction of VoxelGridModule and its parameters it does not know which is the resolution that will be loaded from checkpoint, so before the checkpoint loading a hook runs which changes the VoxelGridModule's parameters to match shapes of the loaded checkpoint.
Reviewed By: bottler
Differential Revision: D39026775
fbshipit-source-id: 0d359ea5c8d2eda11d773d79c7513c83585d5f17
Summary:
User reported that cloned cameras fail to save. The error with latest PyTorch is
```
pickle.PicklingError: Can't pickle ~T_destination: attribute lookup T_destination on torch.nn.modules.module failed
```
This fixes it.
Reviewed By: btgraham
Differential Revision: D39692258
fbshipit-source-id: 75bbf3b8dfa0023dc28bf7d4cc253ca96e46a64d
Summary:
We need to make packing/unpacking in 2 places for mixed frame raysampling (metrics and raysampler) but those tensors that need to be unpacked/packed have more than two dimensions.
I could have reshaped and stored dimensions but this seems to just complicate code there with something which packed_to_padded should support.
I could have made a separate function for implicitron but it would confusing to have two different padded_to_packed functions inside pytorch3d codebase one of which does packing for (b, max) and (b, max, f) and the other for (b, max, …)
Reviewed By: bottler
Differential Revision: D39729026
fbshipit-source-id: 2bdebf290dcc6c316b7fe1aeee49bbb5255e508c
Summary: Add the ability to process arbitrary point shapes `[n_grids, ..., 3]` instead of only `[n_grids, n_points, 3]`.
Reviewed By: bottler
Differential Revision: D39574373
fbshipit-source-id: 0a9ecafe9ea58cd8f909644de43a1185ecf934f4
Summary:
Added export of UV textures to IO.save_mesh in Pytorch3d
MeshObjFormat now passes verts_uv, faces_uv, and texture_map as input to save_obj
TODO: check if TexturesUV.verts_uv_list or TexturesUV.verts_uv_padded() should be passed to save_obj
IO.save_mesh(obj_file, meshes, decimal_places=2) should be IO().save_mesh(obj_file, meshes, decimal_places=2)
Reviewed By: bottler
Differential Revision: D39617441
fbshipit-source-id: 4628b7f26f70e38c65f235852b990c8edb0ded23
Summary:
- indicate location of OmegaConf.structured failures
- split the data gathering from enable_get_default_args to ease experimenting with it.
- comment fixes.
- nicer error when a_class_type has weird type.
Reviewed By: kjchalup
Differential Revision: D39434447
fbshipit-source-id: b80c7941547ca450e848038ef5be95b7ebbe8f3e
Summary:
Move the flyaround rendering function into core implicitron.
The unblocks an example in the facebookresearch/co3d repo.
Reviewed By: bottler
Differential Revision: D39257801
fbshipit-source-id: 6841a88a43d4aa364dd86ba83ca2d4c3cf0435a4
Summary: Currently some implicit functions in implicitron take a raybundle, others take ray_points_world. raybundle is what they really need. However, the raybundle is going to become a bit more flexible later, as it will contain different numbers of rays for each camera.
Reviewed By: bottler
Differential Revision: D39173751
fbshipit-source-id: ebc038e426d22e831e67a18ba64655d8a61e1eb9
