Summary:
This diff integrates the pulsar renderer source code into PyTorch3D as an alternative backend for the PyTorch3D point renderer. This diff is the first of a series of three diffs to complete that migration and focuses on the packaging and integration of the source code.
For more information about the pulsar backend, see the release notes and the paper (https://arxiv.org/abs/2004.07484). For information on how to use the backend, see the point cloud rendering notebook and the examples in the folder `docs/examples`.
Tasks addressed in the following diffs:
* Add the PyTorch3D interface,
* Add notebook examples and documentation (or adapt the existing ones to feature both interfaces).
Reviewed By: nikhilaravi
Differential Revision: D23947736
fbshipit-source-id: a5e77b53e6750334db22aefa89b4c079cda1b443
Summary: To initialize the Cameras class currently we require the principal point, focal length and other parameters to be specified from which we calculate the intrinsic matrix. In some cases the matrix might be directly available e.g. from a dataset and the associated metadata for an image.
Reviewed By: nikhilaravi
Differential Revision: D24489509
fbshipit-source-id: 1b411f19c5f6c8074bcfbf613f3339d5e242c119
Summary: We envision `pytorch3d.vis` to contain submodules with different dependencies. Allow (and require) them to be imported independently.
Reviewed By: theschnitz
Differential Revision: D24622519
fbshipit-source-id: 44840f70f5fd2bd410405bf09546024e48238744
Summary: New methods to directly plot a TexturesUV map with its used points, using PIL and matplotlib.
Reviewed By: gkioxari
Differential Revision: D23782968
fbshipit-source-id: 692970857b5be13a35a3175dc82ac03963a73555
Summary: Fix axis_angle conversions where I used torch.square which doesn't work with pytorch 1.4
Reviewed By: nikhilaravi
Differential Revision: D24451546
fbshipit-source-id: ba26f7dad5fa991f0a8f7d3d09ee7151163aecf4
Summary: We can represent a rotation as a vector in the axis direction, whose length is the rotation anticlockwise in radians around that axis.
Reviewed By: gkioxari
Differential Revision: D24306293
fbshipit-source-id: 2e0f138eda8329f6cceff600a6e5f17a00e4deb7
Summary: Take in a renderer with camera(s) and render the cameras as wireframes in the corresponding plotly plots
Reviewed By: nikhilaravi
Differential Revision: D24151706
fbshipit-source-id: f8e86d61f3d991500bafc0533738c79b96bda630
Summary:
Use a provided renderer's camera positions to render a plotly plot to match what the renderer would render for pointclouds and meshes.
- takes in a Cameras object for viewpoints
- for each subplot, will index into the Cameras object (or use the Cameras object, if len(viewpoint_cameras) == 1 and use the Cameras' eye and at vectors to set plotly's camera's corresponding values, the eye and center values.
Reviewed By: nikhilaravi
Differential Revision: D24094934
fbshipit-source-id: 48abcdb04c6909a172ba9f721522c3446952a089
Summary:
- adds plot_batch_individually
- for each batched object, plots each object in its own subplot with other same-indexed elements of the other batched objects provided as input
Reviewed By: nikhilaravi
Differential Revision: D24258389
fbshipit-source-id: a80128e6e7a03a44c257b0598569159afadb2d39
Summary: Defines a function plot_scene that takes in a dictionary defining subplot and trace layouts for Mesh/Pointcloud objects and plots them. Also supports other plotly axis arguments and mesh lighting. Plot_batch_individually is a wrapper function that takes in one or multiple batched Meshes/Pointclouds and uses plot_scene to plot each element within a batch in an individual subplot, possibly sharing that subplot with traces of other individual elements of the other batched structures passed in.
Reviewed By: nikhilaravi
Differential Revision: D24235479
fbshipit-source-id: 9f669f1b186d55fe5c75552083316c0cf1387472
Summary:
As pointed out in #328, we had an indexing operation where a reshape would do and be faster. The resulting faces_packed_to_edges_packed is no longer contiguous.
Also fix a use of faces_packed_to_edges_packed which might modify the object unintentionally.
Reviewed By: theschnitz
Differential Revision: D24390292
fbshipit-source-id: 225677d8fcc1d6b76efad7706718ecdb5182ffe1
Summary: The linter has become stricter about the indenting of comments and docstrings. This was accompanied by a codemod. In a few places we can fix the problem nicer than the codemod has.
Reviewed By: gkioxari
Differential Revision: D24363880
fbshipit-source-id: 4cff3bbe3d2a834bc92a490469a2b24fa376e6ab
Summary:
Small fix and updated tests for multigpu rendering case.
This resolves the issue seen in: https://github.com/facebookresearch/pytorch3d/issues/401
Reviewed By: gkioxari
Differential Revision: D24314681
fbshipit-source-id: 84c5a5359844c77518b48044001daa9a86f3c43a
Summary: Issue #119. The function `sqrt(max(x, 0))` is not convex and has infinite gradient at 0, but 0 is a subgradient at 0. Here we implement it in such a way as to give 0 as the gradient.
Reviewed By: gkioxari
Differential Revision: D24306294
fbshipit-source-id: 48d136faca083babad4d64970be7ea522dbe9e09
Summary: As mentioned in a comment on https://github.com/facebookresearch/pytorch3d/issues/77, we can't use this function in Python 3.6. It's easy to write our own version.
Reviewed By: gkioxari
Differential Revision: D24249915
fbshipit-source-id: 4c70a3efb03daa115041d082e616511297eab8fa
Summary:
Fix for GitHub issue #381.
The example mesh provided in the issue only had material properties but no texture image. The current implementation of texture atlassing generated an atlas using both the material properties and the texture image but only worked if there was a texture image and associated vertex uv coordinates. I have now modified the texture atlas creation so that it doesn't require an image and can work with materials which only have material properties.
Reviewed By: gkioxari
Differential Revision: D24153068
fbshipit-source-id: 63e9d325db09a84b336b83369d5342ce588a9932
Summary: Importing from pytorch3d.visualization is wordy, so shortened the path to the vis module and updated the relevant imports.
Reviewed By: nikhilaravi
Differential Revision: D24116527
fbshipit-source-id: e0e4da7d48c5afedec07482d7be43362b6822445
Summary: Enhance every texture type with `faces_verts_textures_packed` that allows users to query the texture of each vertex in mesh
Reviewed By: nikhilaravi
Differential Revision: D24058778
fbshipit-source-id: 19d0e3a244fa96aae462c47bf52e07dfd3b7c6f0
Summary:
Enhanced `sample_points_from_meshes` with texture sampling
* This new feature is used to return textures corresponding to the sampled points in `sample_points_from_meshes`
Reviewed By: nikhilaravi
Differential Revision: D24031525
fbshipit-source-id: 8e5d8f784cc38aa391aa8e84e54423bd9fad7ad1
Summary:
Visualize a pointcloud in plotly.
- customize lighting and light position
- customizable axis arguments
- customizable height and width of plotly figure
- render batches in subplots or the same plot
Reviewed By: nikhilaravi
Differential Revision: D23872391
fbshipit-source-id: 9b1e1fd417500521be9d0eb85d71c77a538fa77c
Summary:
Visualize a mesh in a plotly figure.
- customize lighting and light position
- customizable axis arguments (x, y, z)
- customizable height and width of plotly figure
- render batches of meshes in subplots or in a singular plot
Reviewed By: nikhilaravi
Differential Revision: D22611960
fbshipit-source-id: 5dc5c55e599d5b0d9c38f22e156c662654099e11
Summary: Small documentation fix for FOVPerspectiveCamera. The error pointed out in the GitHub issue was due to the docs referring to the OpenGL z parameterization from [-1, 1] whereas we use the [0, 1] range.
Reviewed By: gkioxari
Differential Revision: D23857312
fbshipit-source-id: 2d20677ec614c935e76f67e100a41a23df708d59
Summary:
Support for moving all the tensors of the renderer to another device by calling `renderer.to(new_device)`
Currently the `MeshRenderer`, `MeshRasterizer` and `SoftPhongShader` (and other shaders) are all of type `nn.Module` which already supports easily moving tensors of submodules (defined as class attributes) to a different device. However the class attributes of the rasterizer and shader (e.g. cameras, lights, materials), are of type `TensorProperties`, not nn.Module so we need to explicity create a `to` method to move these tensors to device. Note that the `TensorProperties` class already has a `to` method so we only need to call `cameras.to(device)` and don't need to worry about the internal tensors.
The other option is of course making these other classes (cameras, lights etc) also of type nn.Module.
Reviewed By: gkioxari
Differential Revision: D23885107
fbshipit-source-id: d71565c442181f739de4d797076ed5d00fb67f8e
Summary:
I'm constantly encountering 3D models with resources that have spaces in their filenames (especially on Windows) and therefore they can't be loaded in pytorch3d. Let me know what you think.
Thanks
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/358
Reviewed By: bottler
Differential Revision: D23798492
Pulled By: nikhilaravi
fbshipit-source-id: 4d85b7ee05339486d2e5ef53a531f8e6052251c5
Summary:
Make save_ply save to binary instead of ascii. An option makes the previous functionality available. save_ply's API accepts a stream, but this is undocumented; that stream must now be a binary stream not a text stream.
Avoiding warnings about making tensors from immutable numpy arrays.
Possible performance improvement when reading binary files.
Fix reading zero-length binary lists.
Reviewed By: nikhilaravi
Differential Revision: D22333118
fbshipit-source-id: b423dfd3da46e047bead200255f47a7707306811
Summary: Support rendering different color backgrounds for pointclouds for both compositors
Reviewed By: nikhilaravi
Differential Revision: D23611043
fbshipit-source-id: ab029650d51349340372c5bd66700e6577d48851
Summary: Fixes a bug that initializes default SfMCameras with a default _R instead of default _T
Reviewed By: gkioxari
Differential Revision: D23654583
fbshipit-source-id: ccfb7235b2fb6df5a2e402b9fb4b194e97d78dc6
Summary: When the camera is vertically oriented, calculating the look_at x-axis (also known as the "right" vector) does not succeed, resulting in the x-axis being placed at the origin. Adds a check to correctly calculate the x-axis if this case occurs.
Reviewed By: nikhilaravi, sbranson
Differential Revision: D23511859
fbshipit-source-id: ee5145cdbecdbe2f7c7d288588bd0899480cb327
Summary:
This fixes two small issues with blending.py:softmax_rgb_blend():
1) zfar and znear attributes are propagated from the camera settings instead of just using default settings of znear=1.0 and zfar=100.0
2) A check is added to prevent arithmetic overflow in softmax_rgb_blend()
This is a fix in response to https://github.com/facebookresearch/pytorch3d/issues/334
where meshes rendererd using a SoftPhongShader with faces_per_pixel=1 appear black. This only occurs when the scale of the mesh is large (vertex values > 100, where 100 is the default value of zfar). This fix allows the caller to increase the value of cameras.zfar to match the scale of her/his mesh.
Reviewed By: nikhilaravi
Differential Revision: D23517541
fbshipit-source-id: ab8631ce9e5f2149f140b67b13eff857771b8807
Summary: When raster_settings.clip_barycentric_coords is unspecified, clip_barycentric_coords is turned on if blur_radius is greater than 0. This matches the behavior prior to D21705503 (cc70950f40).
Reviewed By: gkioxari
Differential Revision: D23375257
fbshipit-source-id: 4b87588aabb69d4d835d4dcceb11153628121d30
Summary:
Update the installation cells to import torch.
Replace use of deprecated Textures in deform tutorial.
Correct the deform tutorial's idea of its own name.
Fix typo in batched part of render_textured_meshes which meant only one mesh was being rendered with a batch of cameras.
Add an error check in the rasterizer to make the error friendly from such a mistake elsewhere.
Reviewed By: gkioxari
Differential Revision: D23345462
fbshipit-source-id: 1d5bd25db052f7ef687b7168d7aee5cc4dce8952
Summary:
Add a join_scene method to all the textures to allow the join_mesh function to include textures. Rename the join_mesh function to join_meshes_as_scene.
For TexturesAtlas, we now interpolate if the user attempts to have the resolution vary across the batch. This doesn't look great if the resolution is already very low.
For TexturesUV, a rectangle packing function is required, this does something simple.
Reviewed By: gkioxari
Differential Revision: D23188773
fbshipit-source-id: c013db061a04076e13e90ccc168a7913e933a9c5
Summary:
Allow, and make default, align_corners=True for texture maps. Allow changing the padding_mode and set the default to be "border" which produces more logical results. Some new documentation.
The previous behavior corresponds to padding_mode="zeros" and align_corners=False.
Reviewed By: gkioxari
Differential Revision: D23268775
fbshipit-source-id: 58d6229baa591baa69705bcf97471c80ba3651de
Summary:
The look_at_view_transform did not give the correct results when the object location `at` was not (0,0,0).
The problem was on computing the cameras' location in world's coordinate `C`. It only took into account the camera position from spherical angles, but ignored the object location in the world's coordinate system. I simply modified the C tensor to take into account the object's location which is not necessarily in the origin.
I ran unit tests and all but 4 failed with the same error message: `RuntimeError: CUDA error: invalid device ordinal`. However the same happens before this patch, so I believe these errors are unrelated.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/230
Reviewed By: gkioxari
Differential Revision: D23278126
Pulled By: nikhilaravi
fbshipit-source-id: c06e891bc46de8222325ee7b37aa43cde44648e8
Summary:
- Add support for loading textures from ShapeNet Obj files as a texture atlas.
- Support textured rendering of shapenet models
Reviewed By: gkioxari
Differential Revision: D23141143
fbshipit-source-id: 26eb81758d4cdbd6d820b072b58f5c6c08cb90bc
Summary:
Found a bug in extending textures with vertex uv coordinates. This was due to the padded -> list conversion of vertex uv coordinates i.e. The number of vertices in the mesh and in verts_uvs can differ
e.g. if a vertex is shared between 3 faces, it can
have up to 3 different uv coordinates. Therefore we cannot convert directly from padded to list using _num_verts_per_mesh
Reviewed By: bottler
Differential Revision: D23233595
fbshipit-source-id: 0c66d15baae697ead0bdc384f74c27d4c6539fc9
Summary:
faces_uvs_packed and verts_uvs_packed were only used in one place and the definition of the former was ambiguous. This meant that the wrong coordinates could be used for meshes other than the first in the batch. I have therefore removed both functions and build their common result inline. Added a test that a simple batch of two meshes is rendered consistently with the rendering of each alone. This test would have failed before.
I hope this fixes https://github.com/facebookresearch/pytorch3d/issues/283.
Some other small improvements to the textures code.
Reviewed By: nikhilaravi
Differential Revision: D23161936
fbshipit-source-id: f99b560a46f6b30262e07028b049812bc04350a7
