Website and docs updates

Summary:
- Added sbranson's fit mesh tutorial to the website
- Updated rendering docs with info about texturing and new shader types.

TODO:
- add pointcloud rendering tutorial to the website as well (https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/render_colored_points.ipynb)
- docs for camera
- update some tutorials which depended on the Textures from structures.

Reviewed By: gkioxari

Differential Revision: D23143977

fbshipit-source-id: 6843c9bf3ce11115c459c64da5b0ad778dc92177

docs/notes/datasets.md

@@ -8,10 +8,8 @@ The PyTorch3D [ShapeNetCore data loader](https://github.com/facebookresearch/pyt
 
 The loaded dataset can be passed to `torch.utils.data.DataLoader` with PyTorch3D's customized collate_fn: `collate_batched_meshes` from the `pytorch3d.dataset.utils` module. The `vertices` and `faces` of the models are used to construct a [Meshes](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py) object representing the batched meshes. This `Meshes` representation can be easily used with other ops and rendering in PyTorch3D.
 
-
 ### R2N2
 
 The R2N2 dataset contains 13 categories that are a subset of the ShapeNetCore v.1 dataset. The R2N2 dataset also contains its own 24 renderings of each object and voxelized models. The R2N2 Dataset can be downloaded following the instructions [here](http://3d-r2n2.stanford.edu/).
 
-
 The PyTorch3D [R2N2 data loader](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/datasets/r2n2/r2n2.py) is initialized with the paths to the ShapeNet dataset, the R2N2 dataset and the splits file for R2N2. Just like `ShapeNetCore`, it can be passed to `torch.utils.data.DataLoader` with a customized collate_fn: `collate_batched_R2N2` from the `pytorch3d.dataset.r2n2.utils` module. It returns all the data that `ShapeNetCore` returns, and in addition, it returns the R2N2 renderings (24 views for each model) along with the camera calibration matrices and a voxel representation for each model. Similar to `ShapeNetCore`, it has a customized `render` function that supports rendering specified models with the PyTorch3D differentiable renderer. In addition, it supports rendering models with the same orientations as R2N2's original renderings.

docs/notes/renderer.md

@@ -11,13 +11,11 @@ For example, by rendering an image from a 3D shape predicted by a neural network
 
 We extensively researched existing codebases for differentiable rendering and found that:
 - the rendering pipeline is complex with more than 7 separate components which need to interoperate and be differentiable
-- popular existing approaches [[1](#1), [2](#2)] are based on the same core implementation which bundles many of the key components into large CUDA kernels which require significant expertise to understand, and has limited scope for extensions  
+- popular existing approaches [[1](#1), [2](#2)] are based on the same core implementation which bundles many of the key components into large CUDA kernels which require significant expertise to understand, and has limited scope for extensions
 - existing methods either do not support batching or assume that meshes in a batch have the same number of vertices and faces
 - existing projects only provide CUDA implementations so they cannot be used without GPUs
 
-In order to experiment with different approaches, we wanted a modular implementation that is easy to use and extend, and supports [heterogeneous batching](batching.md).
+In order to experiment with different approaches, we wanted a modular implementation that is easy to use and extend, and supports [heterogeneous batching](batching.md). Taking inspiration from existing work [[1](#1), [2](#2)], we have created a new, modular, differentiable renderer with **parallel implementations in PyTorch, C++ and CUDA**, as well as comprehensive documentation and tests, with the aim of helping to further research in this field.
-
-Taking inspiration from existing work [[1](#1), [2](#2)], we have created a new, modular, differentiable renderer with **parallel implementations in PyTorch, C++ and CUDA**, as well as comprehensive documentation and tests, with the aim of helping to further research in this field.
 
 Our implementation decouples the rasterization and shading steps of rendering. The core rasterization step (based on [[2]](#2)) returns several intermediate variables and has an optimized implementation in CUDA. The rest of the pipeline is implemented purely in PyTorch, and is designed to be customized and extended. With this approach, the PyTorch3D differentiable renderer can be imported as a library.
 
@@ -25,65 +23,17 @@ Our implementation decouples the rasterization and shading steps of rendering. T
 
 To learn about more the implementation and start using the renderer refer to [getting started with renderer](renderer_getting_started.md), which also contains the [architecture overview](assets/architecture_overview.png) and [coordinate transformation conventions](assets/transformations_overview.png).
 
+## <u>Tech Report</u>
 
-## <u>Key features</u>
+For an in depth explanation of the renderer design, key features and benchmarks please refer to the PyTorch3D Technical Report on ArXiv: [Accelerating 3D Deep Learning with PyTorch3D](https://arxiv.org/abs/2007.08501)
-
-### 1. CUDA support for fast rasterization of large meshes
-
-We implemented modular CUDA kernels for the forward and backward pass of rasterization, adaptating a traditional graphics approach known as "coarse-to-fine" rasterization.
-
-First, the image is divided into a coarse grid and mesh faces are allocated to the grid cell in which they occur. This is followed by a refinement step which does pixel wise rasterization of the reduced subset of faces per grid cell. The grid cell size is a parameter which can be varied (`bin_size`).
-
-We additionally introduce a parameter `faces_per_pixel` which allows users to specify the top K faces which should be returned per pixel in the image (as opposed to traditional rasterization which returns only the index of the closest face in the mesh per pixel). The top K face properties can then be aggregated using different methods (such as the sigmoid/softmax approach proposed by Li et at in SoftRasterizer [[2]](#2)).
-
-We compared PyTorch3D with SoftRasterizer to measure the effect of both these design changes on the speed of rasterization. We selected a set of meshes of different sizes from ShapeNetV1 core, and rasterized one mesh in each batch to produce images of different sizes. We report the speed of the forward and backward passes.
-
-**Fig 1: PyTorch3D Naive vs Coarse-to-fine**
-
-This figure shows how the coarse-to-fine strategy for rasterization results in significant speed up compared to naive rasterization for large image size and large mesh sizes.
-
-<img src="assets/p3d_naive_vs_coarse.png" width="1000">
-
-
-For small mesh and image sizes, the naive approach is slightly faster. We advise that you understand the data you are using and choose the rasterization setting which suits your performance requirements. It is easy to switch between the naive and coarse-to-fine options by adjusting the `bin_size` value when initializing the [rasterization settings](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/rasterizer.py#L26).
-
-Setting `bin_size = 0` will enable naive rasterization. If `bin_size > 0`, the coarse-to-fine approach is used. The default is `bin_size = None` in which case we set the bin size based on [heuristics](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/rasterize_meshes.py#L92).
-
-**Fig 2: PyTorch3D Coarse-to-fine vs SoftRasterizer**
-
-This figure shows the effect of the _combination_ of coarse-to-fine rasterization and caching the faces rasterized per pixel returned from the forward pass. For large meshes and image sizes, we again observe that the PyTorch3D rasterizer is significantly faster, noting that the speed is dominated by the forward pass and the backward pass is very fast.
-
-In the SoftRasterizer implementation, in both the forward and backward pass, there is a loop over every single face in the mesh for every pixel in the image. Therefore, the time for the full forward plus backward pass is ~2x the time for the forward pass. For small mesh and image sizes, the SoftRasterizer approach is slightly faster.
-
-<img src="assets/p3d_vs_softras.png" width="1000">
-
-
-
-### 2. Support for Heterogeneous Batches
-
-PyTorch3D supports efficient rendering of batches of meshes where each mesh has different numbers of vertices and faces. This is done without using padded inputs.
-
-We again compare with SoftRasterizer which only supports batches of homogeneous meshes and test two cases: 1) a for loop over meshes in the batch, 2) padded inputs, and compare with the native heterogeneous batching support in PyTorch3D.
-
-We group meshes from ShapeNet into bins based on the number of faces in the mesh, and sample to compose a batch. We then render images of fixed size and measure the speed of the forward and backward passes.
-
-We tested with a range of increasingly large meshes and bin sizes.
-
-**Fig 3: PyTorch3D heterogeneous batching compared with SoftRasterizer**
-
-<img src="assets/fullset_batch_size_16.png" width="700"/>
-
-This shows that for large meshes and large bin width (i.e. more variation in mesh size in the batch) the heterogeneous batching approach in PyTorch3D is faster than either of the workarounds with SoftRasterizer.
-
-(settings: batch size = 16, mesh sizes in bins ranging from 500-350k faces, image size = 64, faces per pixel = 100)
 
 ---
 
 **NOTE: CUDA Memory usage**
 
-The SoftRasterizer forward CUDA kernel only outputs one `(N, H, W, 4)` FloatTensor compared with the PyTorch3D rasterizer forward CUDA kernel which outputs 4 tensors:
+The main comparison in the Technical Report is with SoftRasterizer [[2](#2)]. The SoftRasterizer forward CUDA kernel only outputs one `(N, H, W, 4)` FloatTensor compared with the PyTorch3D rasterizer forward CUDA kernel which outputs 4 tensors:
 
-  - `pix_to_face`, LongTensor `(N, H, W, K)`  
+  - `pix_to_face`, LongTensor `(N, H, W, K)`
   - `zbuf`, FloatTensor `(N, H, W, K)`
   - `dist`, FloatTensor `(N, H, W, K)`
   - `bary_coords`, FloatTensor `(N, H, W, K, 3)`
@@ -107,12 +57,6 @@ We need 48 bytes per face per pixel of the rasterized output. In order to remain
 
 ---
 
-
-### 3. Modular design for easy experimentation and extensibility.
-
-We redesigned the rendering pipeline from the ground up to be modular and extensible and challenged many of the limitations in existing libraries. Refer to [renderer_getting_started.md](renderer_getting_started.md) for a detailed description of the architecture.
-
-
 ### References
 
 <a id="1">[1]</a> Kato et al, 'Neural 3D Mesh Renderer', CVPR 2018

docs/notes/renderer_getting_started.md

@@ -9,7 +9,7 @@ sidebar_label: Getting Started
 
 The renderer is designed to be modular, extensible and support batching and gradients for all inputs. The following figure describes all the components of the rendering pipeline.
 
-<img src="assets/architecture_overview.png" width="1000">
+<img src="assets/architecture_renderer.jpg" width="1000">
 
 ##### Fragments
 
@@ -36,7 +36,7 @@ The differentiable renderer API is experimental and subject to change!.
 
 Rendering requires transformations between several different coordinate frames: world space, view/camera space, NDC space and screen space. At each step it is important to know where the camera is located, how the +X, +Y, +Z axes are aligned and the possible range of values. The following figure outlines the conventions used PyTorch3D.
 
-<img src="assets/transformations_overview.png" width="1000">
+<img src="assets/transforms_overview.jpg" width="1000">
 
 
 For example, given a teapot mesh, the world coordinate frame, camera coordiante frame and image are show in the figure below. Note that the world and camera coordinate frames have the +z direction pointing in to the page.
@@ -54,6 +54,17 @@ While we tried to emulate several aspects of OpenGL, there are differences in th
 <img align="center" src="assets/opengl_coordframes.png" width="300">
 
 ---
+
+### Texturing options
+
+For mesh texturing we offer several options (in `pytorch3d/renderer/mesh/texturing.py`):
+
+1. **Vertex Textures**: D dimensional textures for each vertex (for example an RGB color) which can be interpolated across the face. This can be represented as an `(N, V, D)` tensor. This is a fairly simple representation though and cannot model complex textures if the mesh faces are large.
+2. **UV Textures**: vertex UV coordinates and **one** texture map for the whole face. For a point on a face with given barycentric coordinates, the face color can be computed by interpolating the vertex uv coordinates and then sampling from the texture map. This representation requires two tensors (UVs: `(N, V, 2), Texture map: `(N, H, W, 3)`), and is limited to only support one texture map per mesh.
+3. **Face Textures**: In more complex cases such as ShapeNet meshes, there are multiple texture maps per mesh and some faces have texture while other do not. For these cases, a more flexible representation is a texture atlas, where each face is represented as an `(RxR)` texture map where R is the texture resolution. For a given point on the face, the texture value can be sampled from the per face texture map using the barycentric coordinates of the point. This representation requires one tensor of shape `(N, F, R, R, 3)`. This texturing method is inspired by the SoftRasterizer implementation. For more details refer to the [`make_material_atlas`](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/io/mtl_io.py#L123) and [`sample_textures`](https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/textures.py#L452) functions.
+
+<img src="assets/texturing.jpg" width="1000">
+
 ### A simple renderer
 
 A renderer in PyTorch3D is composed of a **rasterizer** and a **shader**. Create a renderer in a few simple steps:
@@ -77,7 +88,6 @@ raster_settings = RasterizationSettings(
     image_size=512,
     blur_radius=0.0,
     faces_per_pixel=1,
-    bin_size=0
 )
 
 # Create a phong renderer by composing a rasterizer and a shader. Here we can use a predefined
@@ -99,13 +109,11 @@ A shader can incorporate several steps:
 
  We have examples of several combinations of these functions based on the texturing/shading/blending support we have currently. These are summarised in this table below. Many other combinations are possible and we plan to expand the options available for texturing, shading and blending.
 
-
+|Example Shaders  | Vertex Textures| UV Textures | Textures Atlas | Flat Shading| Gouraud Shading| Phong Shading | Hard blending | Soft Blending |
-|Example Shaders  | Vertex Textures| Texture Map| Flat Shading| Gouraud Shading| Phong Shading | Hard blending | Soft Blending |
+| ------------- |:-------------:  | :--------------:| :--------------:| :--------------:| :--------------:| :--------------:|:--------------:|:--------------:|
-| ------------- |:-------------:  | :--------------:| :--------------:| :--------------:| :--------------:|:--------------:|:--------------:|
+| HardPhongShader |  ✔️ |✔️|✔️|||  ✔️ |  ✔️||
-| HardPhongShader |  :heavy_check_mark: ||||  :heavy_check_mark: |  :heavy_check_mark:||
+| SoftPhongShader |  ✔️ |✔️|✔️|||  ✔️ | | ✔️|
-| SoftPhongShader |  :heavy_check_mark: ||||  :heavy_check_mark: | | :heavy_check_mark:|
+| HardGouraudShader |  ✔️ |✔️|✔️||  ✔️ ||  ✔️||
-| HardGouraudShader |  :heavy_check_mark: |||  :heavy_check_mark: ||  :heavy_check_mark:||
+| SoftGouraudShader |  ✔️ |✔️|✔️||  ✔️ |||  ✔️|
-| SoftGouraudShader |  :heavy_check_mark: |||  :heavy_check_mark: |||  :heavy_check_mark:|
+| HardFlatShader |  ✔️ |✔️|✔️|  ✔️ |||  ✔️||
-| TexturedSoftPhongShader ||  :heavy_check_mark: |||  :heavy_check_mark: ||  :heavy_check_mark:|
+| SoftSilhouetteShader ||||||||  ✔️|
-| HardFlatShader |  :heavy_check_mark: ||  :heavy_check_mark: |||  :heavy_check_mark:||
-| SoftSilhouetteShader |||||||  :heavy_check_mark:|

website/README.md

@@ -2,7 +2,7 @@ This website was created with [Docusaurus](https://docusaurus.io/).
 
 # Building the PyTorch3D website
 
-## Install 
+## Install
 
 1. Make sure all the dependencies for the website are installed:
 
@@ -10,7 +10,7 @@ This website was created with [Docusaurus](https://docusaurus.io/).
 # Install dependencies
 $ yarn
 
-or 
+or
 
 $ npm install docusaurus-init
 ```
@@ -40,7 +40,7 @@ Install yarn:
 ```
 brew install yarn
 
-# or 
+# or
 
 curl -o- -L https://yarnpkg.com/install.sh | bash
 ```
@@ -57,11 +57,11 @@ This will build the docusaurus website and run a script to parse the tutorials a
 - `.py`/`.ipynb` files in the `website/static/files` folder
 
 
-TODO: Add support for latex in markdown in jupyter notebooks and embedded images. 
+TODO: Add support for latex in markdown in jupyter notebooks and embedded images.
 
 ## Build and publish the website
 
-The following script will build the tutorials and the website and push to the gh-pages 
+The following script will build the tutorials and the website and push to the gh-pages
 branch of `github.com/facebookresearch/pytorch3d`.
 
 ```
@@ -71,17 +71,17 @@ bash scripts/publish_website.sh
 
 ## Add a new tutorial
 
-The tutorials to include in the website are listed in `website/tutorials.json`. If you create a new tutorial add an entry to the list in this file. This is needed in order to generate the sidebar for the tutorials page. 
+The tutorials to include in the website are listed in `website/tutorials.json`. If you create a new tutorial add an entry to the list in this file. This is needed in order to generate the sidebar for the tutorials page.
 
 
 ## Edit the landing page
 
-To change the content of the landing page modify: `website/pages/en/index.js`. 
+To change the content of the landing page modify: `website/pages/en/index.js`.
 
 
 ## Edit the tutorials page
 
-To change the content of the tutorials home page modify: `website/pages/tutorials/index.js`. 
+To change the content of the tutorials home page modify: `website/pages/tutorials/index.js`.
 
 
 ---------------------------------------------------------

website/sidebars.json

@@ -1,7 +1,8 @@
 {
   "docs": {
     "Introduction": ["why_pytorch3d"],
-    "Meshes": ["batching", "meshes_io"],
+    "Data": ["meshes_io", "datasets", "batching"],
-    "Differentiable Renderer": ["renderer", "renderer_getting_started"]
+    "Ops": ["cubify"],
+    "Renderer": ["renderer", "renderer_getting_started", "cameras"]
   }
 }

website/tutorials.json

@@ -12,9 +12,18 @@
      {
       "id": "render_textured_meshes",
       "title": "Render Textured Meshes"
+     },{
+      "id": "fit_textured_mesh",
+      "title": "Fit a mesh with texture via rendering"
      }, {
       "id": "camera_position_optimization_with_differentiable_rendering",
-      "title": "Camera Position Optimization"         
+      "title": "Camera Position Optimization"
      }
-  ]
+  ],
-}
+  "Dataloaders": [
+      {
+         "id": "dataloaders_ShapeNetCore_R2N2",
+         "title": "Data loaders for ShapeNetCore and R2N2"
+      }
+   ]
+}