Summary:
Similar to non square image rasterization for meshes, apply the same updates to the pointcloud rasterizer.
Main API Change:
- PointRasterizationSettings now accepts a tuple/list of (H, W) for the image size.
Reviewed By: jcjohnson
Differential Revision: D25465206
fbshipit-source-id: 7370d83c431af1b972158cecae19d82364623380
Summary:
There are a couple of options for supporting non square images:
1) NDC stays at [-1, 1] in both directions with the distance calculations all modified by (W/H). There are a lot of distance based calculations (e.g. triangle areas for barycentric coordinates etc) so this requires changes in many places.
2) NDC is scaled by (W/H) so the smallest side has [-1, 1]. In this case none of the distance calculations need to be updated and only the pixel to NDC calculation needs to be modified.
I decided to go with option 2 after trying option 1!
API Changes:
- Image size can now be specified optionally as a tuple
TODO:
- add a benchmark test for the non square case.
Reviewed By: jcjohnson
Differential Revision: D24404975
fbshipit-source-id: 545efb67c822d748ec35999b35762bce58db2cf4
Summary: This fixed#442 by declaring two math functions to be device-only.
Reviewed By: bottler
Differential Revision: D24896992
fbshipit-source-id: a15918d06d2a3e6ee5cf250fec7af5f2f50a6164
Summary:
Changes to CI and some minor fixes now that pulsar is part of pytorch3d. Most significantly, add CUB to CI builds.
Make CUB_HOME override the CUB already in cudatoolkit (important for cuda11.0 which uses cub 1.9.9 which pulsar doesn't work well with.
Make imageio available for testing.
Lint fixes.
Fix some test verbosity.
Avoid use of atomicAdd_block on older GPUs.
Reviewed By: nikhilaravi, classner
Differential Revision: D24773716
fbshipit-source-id: 2428356bb2e62735f2bc0c15cbe4cff35b1b24b8
Summary:
Removes the now-unnecessary kernels from point mesh edge file
Migrates all point mesh functionality into one file.
Reviewed By: gkioxari
Differential Revision: D24550086
fbshipit-source-id: f924996cd38a7c2c1cf189d8a01611de4506cfa3
Summary: This diff creates the generic MeshBackwardKernel which can handle distance calculations between point, edge and faces in either direction. Replaces only point_mesh_face code for now.
Reviewed By: gkioxari
Differential Revision: D24549374
fbshipit-source-id: 2853c1da1c2a6b6de8d0e40007ba0735b8959044
Summary: This diff creates the generic MeshForwardKernel which can handle distance calculations between point, edge and faces in either direction. Replaces only point_mesh_face code for now.
Reviewed By: gkioxari
Differential Revision: D24543316
fbshipit-source-id: 302707d7cec2d77a899738adf40481035c240da8
Summary: Added missing include for cstdint for Windows and removed problematic inline assembly.
Reviewed By: bottler
Differential Revision: D24838053
fbshipit-source-id: 95496be841c2c22a82068073d4740e98ee8a02ac
Summary: This diff updates the documentation and tutorials with information about the new pulsar backend. For more information about the pulsar backend, see the release notes and the paper (https://arxiv.org/abs/2004.07484). For information on how to use the backend, see the point cloud rendering notebook and the examples in the folder docs/examples.
Reviewed By: nikhilaravi
Differential Revision: D24498129
fbshipit-source-id: e312b0169a72b13590df6e4db36bfe6190d742f9
Summary:
This diff integrates the pulsar renderer source code into PyTorch3D as an alternative backend for the PyTorch3D point renderer. This diff is the first of a series of three diffs to complete that migration and focuses on the packaging and integration of the source code.
For more information about the pulsar backend, see the release notes and the paper (https://arxiv.org/abs/2004.07484). For information on how to use the backend, see the point cloud rendering notebook and the examples in the folder `docs/examples`.
Tasks addressed in the following diffs:
* Add the PyTorch3D interface,
* Add notebook examples and documentation (or adapt the existing ones to feature both interfaces).
Reviewed By: nikhilaravi
Differential Revision: D23947736
fbshipit-source-id: a5e77b53e6750334db22aefa89b4c079cda1b443
Summary:
faces_uvs_packed and verts_uvs_packed were only used in one place and the definition of the former was ambiguous. This meant that the wrong coordinates could be used for meshes other than the first in the batch. I have therefore removed both functions and build their common result inline. Added a test that a simple batch of two meshes is rendered consistently with the rendering of each alone. This test would have failed before.
I hope this fixes https://github.com/facebookresearch/pytorch3d/issues/283.
Some other small improvements to the textures code.
Reviewed By: nikhilaravi
Differential Revision: D23161936
fbshipit-source-id: f99b560a46f6b30262e07028b049812bc04350a7
Summary: A triangle is culled if any vertex in a triangle is behind the camera. This fixes incorrect rendering of triangles that are partially behind the camera, where screen coordinate calculations are strange. It doesn't work for triangles that are partially behind the camera but still intersect with the view frustum.
Reviewed By: nikhilaravi
Differential Revision: D22856181
fbshipit-source-id: a9cbaa1327d89601b83d0dfd3e4a04f934a4a213
Summary:
Added support for barycentric clipping in the C++/CUDA rasterization kernels which can be switched on/off via a rasterization setting.
Added tests and a benchmark to compare with the current implementation in PyTorch - for some cases of large image size/faces per pixel the cuda version is 10x faster.
Reviewed By: gkioxari
Differential Revision: D21705503
fbshipit-source-id: e835c0f927f1e5088ca89020aef5ff27ac3a8769
Summary:
C++/CUDA implementation of forward and backward passes for the sigmoid alpha blending function.
This is slightly faster than the vectorized implementation in Python, but more importantly uses less memory due to fewer tensors being created.
Reviewed By: gkioxari
Differential Revision: D19980671
fbshipit-source-id: 0779055d2c68b1f20fb0870e60046077ef4613ff
Summary: When rendering meshes with Phong shading, interpolate_face_attributes was taking up a nontrivial fraction of the overall shading time. This diff replaces our Python implementation of this function with a CUDA implementation.
Reviewed By: nikhilaravi
Differential Revision: D21610763
fbshipit-source-id: 2bb362a28f698541812aeab539047264b125ebb8
Summary: Fix the new CPU implementation of point_mesh functionality to be compatible with older C++.
Reviewed By: nikhilaravi
Differential Revision: D22066785
fbshipit-source-id: a245849342019a93ff68e186a10985458b007436
Summary:
point_mesh functions were missing CPU implementations.
The indices returned are not always matching, possibly due to numerical instability.
Reviewed By: gkioxari
Differential Revision: D21594264
fbshipit-source-id: 3016930e2a9a0f3cd8b3ac4c94a92c9411c0989d
Summary:
Fix division by zero when alpha is 1.0
In this case, the nominator is already 0 and we need to make sure division with 0 does not occur which would produce nans
Reviewed By: nikhilaravi
Differential Revision: D21650478
fbshipit-source-id: bc457105b3050fef1c8bd4e58e7d6d15c0c81ffd
Summary:
Update the cuda kernels to:
- remove contiguous checks for the grad tensors and for cpu functions which use accessors
- for cuda implementations call `.contiguous()` on all tensors in the host function before invoking the kernel
Reviewed By: gkioxari
Differential Revision: D21598008
fbshipit-source-id: 9b97bda4582fd4269c8a00999874d4552a1aea2d
Summary: pytorch is adding checks that mean integer tensors with requires_grad=True need to be avoided. Fix accidentally creating them.
Reviewed By: jcjohnson, gkioxari
Differential Revision: D21576712
fbshipit-source-id: 008218997986800a36d93caa1a032ee91f2bffcd
Summary: This has been failing intermittently
Reviewed By: nikhilaravi
Differential Revision: D21403157
fbshipit-source-id: 51b74d6c813b52effe72d14b565e250fcabbb463
Summary:
Updates to:
- enable cuda kernel launches on any GPU (not just the default)
- cuda and contiguous checks for all kernels
- checks to ensure all tensors are on the same device
- error reporting in the cuda kernels
- cuda tests now run on a random device not just the default
Reviewed By: jcjohnson, gkioxari
Differential Revision: D21215280
fbshipit-source-id: 1bedc9fe6c35e9e920bdc4d78ed12865b1005519
Summary:
Use aten instead of torch interface in all cuda code. This allows the cuda build to work with pytorch 1.5 with GCC 5 (e.g. the compiler of ubuntu 16.04LTS). This wasn't working. It has been failing with errors like the below, perhaps due to a bug in nvcc.
```
torch/include/torch/csrc/api/include/torch/nn/cloneable.h:68:61: error: invalid static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, std::shared_ptr<torch::nn::Module> >’ to type ‘torch::OrderedDict<std::basic_string<char>, std::shared_ptr<torch::nn::Module> >
```
Reviewed By: nikhilaravi
Differential Revision: D21204029
fbshipit-source-id: ca6bdbcecf42493365e1c23a33fe35e1759fe8b6
Summary:
We have multiple KNN CUDA implementations. From python, users can currently request a particular implementation via the `version` flag, but they have no way of knowing which implementations can be used for a given problem.
This diff exposes a function `pytorch3d._C.knn_check_version(version, D, K)` that returns whether a particular version can be used.
Reviewed By: nikhilaravi
Differential Revision: D21162573
fbshipit-source-id: 6061960bdcecba454fd920b00036f4e9ff3fdbc0
Summary:
Added backface culling as an option to the `raster_settings`. This is needed for the full forward rendering of shapenet meshes with texture (some meshes contain
multiple overlapping segments which have different textures).
For a triangle (v0, v1, v2) define the vectors A = (v1 - v0) and B = (v2 − v0) and use this to calculate the area of the triangle as:
```
area = 0.5 * A x B
area = 0.5 * ((x1 − x0)(y2 − y0) − (x2 − x0)(y1 − y0))
```
The area will be positive if (v0, v1, v2) are oriented counterclockwise (a front face), and negative if (v0, v1, v2) are oriented clockwise (a back face).
We can reuse the `edge_function` as it already calculates the triangle area.
Reviewed By: jcjohnson
Differential Revision: D20960115
fbshipit-source-id: 2d8a4b9ccfb653df18e79aed8d05c7ec0f057ab1
Summary:
Fix a bug which resulted in a rendering artifacts if the image size was not a multiple of 16.
Fix: Revert coarse rasterization to original implementation and only update fine rasterization to reverse the ordering of Y and X axis. This is much simpler than the previous approach!
Additional changes:
- updated mesh rendering end-end tests to check outputs from both naive and coarse to fine rasterization.
- added pointcloud rendering end-end tests
Reviewed By: gkioxari
Differential Revision: D21102725
fbshipit-source-id: 2e7e1b013dd6dd12b3a00b79eb8167deddb2e89a
Summary: This is mostly replacing the old PackedTensorAccessor with the new PackedTensorAccessor64.
Reviewed By: gkioxari
Differential Revision: D21088773
fbshipit-source-id: 5973e5a29d934eafb7c70ec5ec154ca076b64d27
Summary: A couple of files for the removed nearest_neighbor functionality are left behind.
Reviewed By: nikhilaravi
Differential Revision: D21088624
fbshipit-source-id: 4bb29016b4e5f63102765b384c363733b60032fa
Summary: knn is more general and faster than the nearest_neighbor code, so remove the latter.
Reviewed By: gkioxari
Differential Revision: D20816424
fbshipit-source-id: 75d6c44d17180752d0c9859814bbdf7892558158
Summary: Interface and working implementation of ragged KNN. Benchmarks (which aren't ragged) haven't slowed. New benchmark shows that ragged is faster than non-ragged of the same shape.
Reviewed By: jcjohnson
Differential Revision: D20696507
fbshipit-source-id: 21b80f71343a3475c8d3ee0ce2680f92f0fae4de
Summary: Run linter after recent changes. Fix long comment in knn.h which clang-format has reflowed badly. Add crude test that code doesn't call deprecated `.type()` or `.data()`.
Reviewed By: nikhilaravi
Differential Revision: D20692935
fbshipit-source-id: 28ce0308adae79a870cb41a810b7cf8744f41ab8
Summary:
Implements K-Nearest Neighbors with C++ and CUDA versions.
KNN in CUDA is highly nontrivial. I've implemented a few different versions of the kernel, and we heuristically dispatch to different kernels based on the problem size. Some of the kernels rely on template specialization on either D or K, so we use template metaprogramming to compile specialized versions for ranges of D and K.
These kernels are up to 3x faster than our existing 1-nearest-neighbor kernels, so we should also consider swapping out `nn_points_idx` to use these kernels in the backend.
I've been working mostly on the CUDA kernels, and haven't converged on the correct Python API.
I still want to benchmark against FAISS to see how far away we are from their performance.
Reviewed By: bottler
Differential Revision: D19729286
fbshipit-source-id: 608ffbb7030c21fe4008f330522f4890f0c3c21a
Summary: CPU-only builds should be fixed by this change
Reviewed By: nikhilaravi
Differential Revision: D20598014
fbshipit-source-id: df098ec4c6c93d38515172805fe57cac7463c506
Summary: Comments were describing squared distance as absolute distance in a few places.
Reviewed By: nikhilaravi
Differential Revision: D20426020
fbshipit-source-id: 009946867c4a98f61f5ce7158542d41e22bf8346
Summary:
Applying the changes added for mesh rasterization to ensure that +Y is up and +X is left so that the coordinate system is right handed.
Also updated the diagram in the docs to indicate that (0,0) is in the top left hand corner.
Reviewed By: gkioxari
Differential Revision: D20394849
fbshipit-source-id: cfb7c79090eb1f55ad38b92327a74a70a8dc541e
Summary:
Update the point cloud rasterizer to:
- use the pointcloud datastructure (rebased on top of D19791851.)
- support rasterization of heterogeneous point clouds in the same way as with Meshes.
The main changes to the API will be as follows:
- The input to `rasterize_points` will be a `Pointclouds` object instead of a tensor. This will be easy to update e.g.
```
points = torch.randn(N, P, 3)
idx2, zbuf2, dists2 = rasterize_points(points, image_size, radius, points_per_pixel)
points = torch.randn(N, P, 3)
pointclouds = Pointclouds(points=points)
idx2, zbuf2, dists2 = rasterize_points(pointclouds, image_size, radius, points_per_pixel)
```
- The indices output from rasterization will now refer to points in `poinclouds.points_packed()`.
This may require some changes to the functions which consume the outputs of rasterization if they were previously
assuming that the indices ranged from 0 to P where P is the number of points in each pointcloud.
Making this change now so that Olivia can update her PR accordingly.
Reviewed By: gkioxari
Differential Revision: D20088651
fbshipit-source-id: 833ed659909712bcbbb6a50e2ec0189839f0413a
Summary:
## Updates
- Defined the world and camera coordinates according to this figure. The world coordinates are defined as having +Y up, +X left and +Z in.
{F230888499}
- Removed all flipping from blending functions.
- Updated the rasterizer to return images with +Y up and +X left.
- Updated all the mesh rasterizer tests
- The expected values are now defined in terms of the default +Y up, +X left
- Added tests where the triangles in the meshes are non symmetrical so that it is clear which direction +X and +Y are
## Questions:
- Should we have **scene settings** instead of raster settings?
- To be more correct we should be [z clipping in the rasterizer based on the far/near clipping planes](https://github.com/ShichenLiu/SoftRas/blob/master/soft_renderer/cuda/soft_rasterize_cuda_kernel.cu#L400) - these values are also required in the blending functions so should we make these scene level parameters and have a scene settings tuple which is available to the rasterizer and shader?
Reviewed By: gkioxari
Differential Revision: D20208604
fbshipit-source-id: 55787301b1bffa0afa9618f0a0886cc681da51f3
Summary:
`PointLineDistanceForward()` should return squared distance. However, it seems that it returned non-squared distance when `v0` was near by `v1` in CPU implementation.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/83
Reviewed By: bottler
Differential Revision: D20097181
Pulled By: nikhilaravi
fbshipit-source-id: 7ea851c0837ab89364e42d283c999df21ff5ff02
Summary:
Fixed a few MSVC compiler (visual studio 2019, MSVC 19.16.27034) compatibility issues
1. Replaced long with int64_t. aten::data_ptr\<long\> is not supported in MSVC
2. pytorch3d/csrc/rasterize_points/rasterize_points_cpu.cpp, inline function is not correctly recognized by MSVC.
3. pytorch3d/csrc/rasterize_meshes/geometry_utils.cuh
const auto kEpsilon = 1e-30;
MSVC does not compile this const into both host and device, change to a MACRO.
4. pytorch3d/csrc/rasterize_meshes/geometry_utils.cuh,
const float area2 = pow(area, 2.0);
2.0 is considered as double by MSVC and raised an error
5. pytorch3d/csrc/rasterize_points/rasterize_points_cpu.cpp
std::tuple<torch::Tensor, torch::Tensor> RasterizePointsCoarseCpu() return type does not match the declaration in rasterize_points_cpu.h.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/9
Reviewed By: nikhilaravi
Differential Revision: D19986567
Pulled By: yuanluxu
fbshipit-source-id: f4d98525d088c99c513b85193db6f0fc69c7f017
