bbc7573261
Summary:
There has historically been a lot of duplication between the coarse rasterization logic for point clouds and meshes. This diff factors out the shared logic, so coarse rasterization of point clouds and meshes share the same core logic.
Previously the only difference between the coarse rasterization kernels for points and meshes was the logic for checking whether a {point / triangle} intersects a tile in the image. We implement a generic coarse rasterization kernel that takes a set of 2D bounding boxes rather than geometric primitives; we then implement separate kernels that compute 2D bounding boxes for points and triangles.
This change does not affect the Python API at all. It also should not change any rasterization behavior, since this diff is just a refactoring of the existing logic.
I see this diff as the first in a few pieces of rasterizer refactoring. Followup diffs should do the following:
- Add a check for bin overflow in the generic coarse rasterizer kernel: allocate a global scalar to flag bin overflow which kernel worker threads can write to in case they detect bin overflow. The C++ launcher function can then check this flag after the kernel returns and issue a warning to the user in case of overflow.
- As a slightly more involved mechanism, if bin overflow is detected then the coarse kernel can continue running in order to count how many elements fall into each bin, without actually writing out their indices to the coarse output tensor. Then the actual number of entries per bin can be used to re-allocate the output tensor and re-run the coarse rasterization kernel so that bin overflow can be automatically avoided.
- The unification of the coarse and fine rasterization kernels also allows us to insert an extra CUDA kernel prior to coarse rasterization that filters out primitives outside the view frustum. This would be helpful for rendering full scenes (e.g. Matterport data) where only a small piece of the mesh is actually visible at any one time.
Reviewed By: bottler
Differential Revision: D25710361
fbshipit-source-id: 9c9dea512cb339c42adb3c92e7733fedd586ce1b
Introduction
PyTorch3D provides efficient, reusable components for 3D Computer Vision research with PyTorch.
Key features include:
- Data structure for storing and manipulating triangle meshes
- Efficient operations on triangle meshes (projective transformations, graph convolution, sampling, loss functions)
- A differentiable mesh renderer
PyTorch3D is designed to integrate smoothly with deep learning methods for predicting and manipulating 3D data. For this reason, all operators in PyTorch3D:
- Are implemented using PyTorch tensors
- Can handle minibatches of hetereogenous data
- Can be differentiated
- Can utilize GPUs for acceleration
Within FAIR, PyTorch3D has been used to power research projects such as Mesh R-CNN.
Installation
For detailed instructions refer to INSTALL.md.
License
PyTorch3D is released under the BSD License.
Tutorials
Get started with PyTorch3D by trying one of the tutorial notebooks.
 |
 |
|---|---|
| Deform a sphere mesh to dolphin | Bundle adjustment |
 |
 |
|---|---|
| Render textured meshes | Camera position optimization |
 |
 |
|---|---|
| Render textured pointclouds | Fit a mesh with texture |
 |
 |
|---|---|
| Render DensePose data | Load & Render ShapeNet data |
 |
 |
|---|---|
| Fit Textured Volume | Fit A Simple Neural Radiance Field |
Documentation
Learn more about the API by reading the PyTorch3D documentation.
We also have deep dive notes on several API components:
Overview Video
We have created a short (~14 min) video tutorial providing an overview of the PyTorch3D codebase including several code examples. Click on the image below to watch the video on YouTube:
Development
We welcome new contributions to PyTorch3D and we will be actively maintaining this library! Please refer to CONTRIBUTING.md for full instructions on how to run the code, tests and linter, and submit your pull requests.
Contributors
PyTorch3D is written and maintained by the Facebook AI Research Computer Vision Team.
In alphabetical order:
- Amitav Baruah
- Steve Branson
- Luya Gao
- Georgia Gkioxari
- Taylor Gordon
- Justin Johnson
- Patrick Labtut
- Christoph Lassner
- Wan-Yen Lo
- David Novotny
- Nikhila Ravi
- Jeremy Reizenstein
- Dave Schnizlein
- Roman Shapovalov
- Olivia Wiles
Citation
If you find PyTorch3D useful in your research, please cite our tech report:
@article{ravi2020pytorch3d,
author = {Nikhila Ravi and Jeremy Reizenstein and David Novotny and Taylor Gordon
and Wan-Yen Lo and Justin Johnson and Georgia Gkioxari},
title = {Accelerating 3D Deep Learning with PyTorch3D},
journal = {arXiv:2007.08501},
year = {2020},
}
If you are using the pulsar backend for sphere-rendering (the PulsarPointRenderer or pytorch3d.renderer.points.pulsar.Renderer), please cite the tech report:
@article{lassner2020pulsar,
author = {Christoph Lassner and Michael Zollh\"ofer},
title = {Pulsar: Efficient Sphere-based Neural Rendering},
journal = {arXiv:2004.07484},
year = {2020},
}
News
Please see below for a timeline of the codebase updates in reverse chronological order. We are sharing updates on the releases as well as research projects which are built with PyTorch3D. The changelogs for the releases are available under Releases, and the builds can be installed using conda as per the instructions in INSTALL.md.
[Feb 9th 2021]: PyTorch3D v0.4.0 released with support for implicit functions, volume rendering and a reimplementation of NeRF.
[November 2nd 2020]: PyTorch3D v0.3.0 released, integrating the pulsar backend.
[Aug 28th 2020]: PyTorch3D v0.2.5 released
[July 17th 2020]: PyTorch3D tech report published on ArXiv: https://arxiv.org/abs/2007.08501
[April 24th 2020]: PyTorch3D v0.2.0 released
[March 25th 2020]: SynSin codebase released using PyTorch3D: https://github.com/facebookresearch/synsin
[March 8th 2020]: PyTorch3D v0.1.1 bug fix release
[Jan 23rd 2020]: PyTorch3D v0.1.0 released. Mesh R-CNN codebase released: https://github.com/facebookresearch/meshrcnn