423A35C7/pytorch3d
1
0
Fork 0
mirror of https://github.com/facebookresearch/pytorch3d.git synced 2026-04-09 14:06:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

update for version 0.5.0

Browse Source
This commit is contained in:
Jeremy Francis Reizenstein
2021-08-10 07:51:01 -07:00
parent 13e2fd2b5b
commit 3ccabda50d
65 changed files with 1660 additions and 1150 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
files/bundle_adjustment.ipynb
View File

@@ -41,7 +41,7 @@
"Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:\n",
"![Solution](https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_final.png?raw=1)\n",
"\n",
"In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \\in SO(3); T \\in \\mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exponential_map`) of the axis-angle representation of the rotation `log_R_absolute`.\n",
"In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \\in SO(3); T \\in \\mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exp_map`) of the axis-angle representation of the rotation `log_R_absolute`.\n",
"\n",
"Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \\in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \\vec{0})$.\n"
]
@@ -63,7 +63,7 @@
"id": "WAHR1LMJmP-h"
},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -80,19 +80,25 @@
},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -116,7 +122,7 @@
"# imports\n",
"import torch\n",
"from pytorch3d.transforms.so3 import (\n",
" so3_exponential_map,\n",
" so3_exp_map,\n",
" so3_relative_angle,\n",
")\n",
"from pytorch3d.renderer.cameras import (\n",
@@ -322,7 +328,7 @@
"\n",
"As mentioned earlier, `log_R_absolute` is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix `R_absolute` that corresponds to `log_R_absolute` with:\n",
"\n",
"`R_absolute = so3_exponential_map(log_R_absolute)`\n"
"`R_absolute = so3_exp_map(log_R_absolute)`\n"
]
},
{
@@ -372,7 +378,7 @@
" # compute the absolute camera rotations as \n",
" # an exponential map of the logarithms (=axis-angles)\n",
" # of the absolute rotations\n",
" R_absolute = so3_exponential_map(log_R_absolute * camera_mask)\n",
" R_absolute = so3_exp_map(log_R_absolute * camera_mask)\n",
"\n",
" # get the current absolute cameras\n",
" cameras_absolute = SfMPerspectiveCameras(\n",
@@ -381,7 +387,7 @@
" device = device,\n",
" )\n",
"\n",
" # compute the relative cameras as a compositon of the absolute cameras\n",
" # compute the relative cameras as a composition of the absolute cameras\n",
" cameras_relative_composed = \\\n",
" get_relative_camera(cameras_absolute, relative_edges)\n",
"\n",

40
files/bundle_adjustment.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -28,31 +28,37 @@
# Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:
# ![Solution](https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_final.png?raw=1)
#
# In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exponential_map`) of the axis-angle representation of the rotation `log_R_absolute`.
# In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exp_map`) of the axis-angle representation of the rotation `log_R_absolute`.
#
# Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \vec{0})$.
#
# ## 0. Install and Import Modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -65,7 +71,7 @@ else:
# imports
import torch
from pytorch3d.transforms.so3 import (
so3_exponential_map,
so3_exp_map,
so3_relative_angle,
)
from pytorch3d.renderer.cameras import (
@@ -197,7 +203,7 @@ def get_relative_camera(cams, edges):
#
# As mentioned earlier, `log_R_absolute` is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix `R_absolute` that corresponds to `log_R_absolute` with:
#
# `R_absolute = so3_exponential_map(log_R_absolute)`
# `R_absolute = so3_exp_map(log_R_absolute)`
#
# In[ ]:
@@ -236,7 +242,7 @@ for it in range(n_iter):
# compute the absolute camera rotations as
# an exponential map of the logarithms (=axis-angles)
# of the absolute rotations
R_absolute = so3_exponential_map(log_R_absolute * camera_mask)
R_absolute = so3_exp_map(log_R_absolute * camera_mask)
# get the current absolute cameras
cameras_absolute = SfMPerspectiveCameras(
@@ -245,7 +251,7 @@ for it in range(n_iter):
device = device,
)
# compute the relative cameras as a compositon of the absolute cameras
# compute the relative cameras as a composition of the absolute cameras
cameras_relative_composed = get_relative_camera(cameras_absolute, relative_edges)
# compare the composed cameras with the ground truth relative cameras

43
files/camera_position_optimization_with_differentiable_rendering.ipynb
View File

@@ -50,7 +50,7 @@
"id": "qkX7DiM6rmeM"
},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -67,19 +67,25 @@
},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -217,9 +223,9 @@
"source": [
"### Create a renderer\n",
"\n",
"A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest. \n",
"A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest. \n",
"\n",
"For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **phong shading** and use this for visualizing the outputs. "
"For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **Phong shading** and use this for visualizing the outputs. "
]
},
{
@@ -260,7 +266,7 @@
")\n",
"\n",
"\n",
"# We will also create a phong renderer. This is simpler and only needs to render one face per pixel.\n",
"# We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.\n",
"raster_settings = RasterizationSettings(\n",
" image_size=256, \n",
" blur_radius=0.0, \n",
@@ -316,15 +322,15 @@
"R, T = look_at_view_transform(distance, elevation, azimuth, device=device)\n",
"\n",
"# Render the teapot providing the values of R and T. \n",
"silhouete = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
"silhouette = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
"image_ref = phong_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
"\n",
"silhouete = silhouete.cpu().numpy()\n",
"silhouette = silhouette.cpu().numpy()\n",
"image_ref = image_ref.cpu().numpy()\n",
"\n",
"plt.figure(figsize=(10, 10))\n",
"plt.subplot(1, 2, 1)\n",
"plt.imshow(silhouete.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image\n",
"plt.imshow(silhouette.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image\n",
"plt.grid(False)\n",
"plt.subplot(1, 2, 2)\n",
"plt.imshow(image_ref.squeeze())\n",
@@ -371,7 +377,7 @@
" def forward(self):\n",
" \n",
" # Render the image using the updated camera position. Based on the new position of the \n",
" # camer we calculate the rotation and translation matrices\n",
" # camera we calculate the rotation and translation matrices\n",
" R = look_at_rotation(self.camera_position[None, :], device=self.device) # (1, 3, 3)\n",
" T = -torch.bmm(R.transpose(1, 2), self.camera_position[None, :, None])[:, :, 0] # (1, 3)\n",
" \n",
@@ -514,7 +520,6 @@
" plt.figure()\n",
" plt.imshow(image[..., :3])\n",
" plt.title(\"iter: %d, loss: %0.2f\" % (i, loss.data))\n",
" plt.grid(\"off\")\n",
" plt.axis(\"off\")\n",
" \n",
"writer.close()"

45
files/camera_position_optimization_with_differentiable_rendering.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -22,24 +22,30 @@
# ## 0. Install and import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -121,9 +127,9 @@ teapot_mesh = Meshes(
# ### Create a renderer
#
# A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.
# A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
#
# For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **phong shading** and use this for visualizing the outputs.
# For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **Phong shading** and use this for visualizing the outputs.
# In[ ]:
@@ -156,7 +162,7 @@ silhouette_renderer = MeshRenderer(
)
# We will also create a phong renderer. This is simpler and only needs to render one face per pixel.
# We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.
raster_settings = RasterizationSettings(
image_size=256,
blur_radius=0.0,
@@ -193,15 +199,15 @@ azimuth = 0.0 # No rotation so the camera is positioned on the +Z axis.
R, T = look_at_view_transform(distance, elevation, azimuth, device=device)
# Render the teapot providing the values of R and T.
silhouete = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)
silhouette = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)
image_ref = phong_renderer(meshes_world=teapot_mesh, R=R, T=T)
silhouete = silhouete.cpu().numpy()
silhouette = silhouette.cpu().numpy()
image_ref = image_ref.cpu().numpy()
plt.figure(figsize=(10, 10))
plt.subplot(1, 2, 1)
plt.imshow(silhouete.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image
plt.imshow(silhouette.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image
plt.grid(False)
plt.subplot(1, 2, 2)
plt.imshow(image_ref.squeeze())
@@ -233,7 +239,7 @@ class Model(nn.Module):
def forward(self):
# Render the image using the updated camera position. Based on the new position of the
# camer we calculate the rotation and translation matrices
# camera we calculate the rotation and translation matrices
R = look_at_rotation(self.camera_position[None, :], device=self.device) # (1, 3, 3)
T = -torch.bmm(R.transpose(1, 2), self.camera_position[None, :, None])[:, :, 0] # (1, 3)
@@ -313,7 +319,6 @@ for i in loop:
plt.figure()
plt.imshow(image[..., :3])
plt.title("iter: %d, loss: %0.2f" % (i, loss.data))
plt.grid("off")
plt.axis("off")
writer.close()

38
files/dataloaders_ShapeNetCore_R2N2.ipynb
View File

@@ -33,7 +33,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -42,19 +42,25 @@
"metadata": {},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -184,7 +190,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces."
"We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces."
]
},
{
@@ -248,11 +254,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from Pytorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).\n",
"Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from PyTorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).\n",
"\n",
"For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.\n",
"\n",
"**NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting `return_all_views = False`."
"**NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting `return_all_views = False`."
]
},
{
@@ -289,7 +295,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Render ShapeNetCore models with PyTorch3D's differntiable renderer"
"## 3. Render ShapeNetCore models with PyTorch3D's differentiable renderer"
]
},
{
@@ -444,7 +450,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we will visualize PyTorch3d's renderings:"
"Next, we will visualize PyTorch3D's renderings:"
]
},
{

40
files/dataloaders_ShapeNetCore_R2N2.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -17,24 +17,30 @@
# ## 0. Install and import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -120,7 +126,7 @@ SPLITS_PATH = "None"
r2n2_dataset = R2N2("train", SHAPENET_PATH, R2N2_PATH, SPLITS_PATH, return_voxels=True)
# We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces.
# We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces.
# In[ ]:
@@ -155,11 +161,11 @@ image_grid(r2n2_renderings.numpy(), rows=1, cols=2, rgb=True)
# ## 2. Use the datasets with `torch.utils.data.DataLoader`
# Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from Pytorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).
# Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from PyTorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).
#
# For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.
#
# **NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting `return_all_views = False`.
# **NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting `return_all_views = False`.
# In[ ]:
@@ -180,7 +186,7 @@ batch_renderings = r2n2_batch["images"] # (N, V, H, W, 3), and in this case V is
image_grid(batch_renderings.squeeze().numpy(), rows=3, cols=4, rgb=True)
# ## 3. Render ShapeNetCore models with PyTorch3D's differntiable renderer
# ## 3. Render ShapeNetCore models with PyTorch3D's differentiable renderer
# Both `ShapeNetCore` and `R2N2` dataloaders have customized `render` functions that support rendering models by specifying their model ids, categories or indices using PyTorch3D's differentiable renderer implementation.
@@ -272,7 +278,7 @@ original_rendering = r2n2_dataset[6,[1,2]]["images"]
image_grid(original_rendering.numpy(), rows=1, cols=2, rgb=True)
# Next, we will visualize PyTorch3d's renderings:
# Next, we will visualize PyTorch3D's renderings:
# In[ ]:

36
files/deform_source_mesh_to_target_mesh.ipynb
View File

@@ -68,7 +68,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -81,19 +81,25 @@
},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -400,10 +406,10 @@
" loop.set_description('total_loss = %.6f' % loss)\n",
" \n",
" # Save the losses for plotting\n",
" chamfer_losses.append(loss_chamfer)\n",
" edge_losses.append(loss_edge)\n",
" normal_losses.append(loss_normal)\n",
" laplacian_losses.append(loss_laplacian)\n",
" chamfer_losses.append(float(loss_chamfer.detach().cpu()))\n",
" edge_losses.append(float(loss_edge.detach().cpu()))\n",
" normal_losses.append(float(loss_normal.detach().cpu()))\n",
" laplacian_losses.append(float(loss_laplacian.detach().cpu()))\n",
" \n",
" # Plot mesh\n",
" if i % plot_period == 0:\n",

38
files/deform_source_mesh_to_target_mesh.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -34,24 +34,30 @@
# ## 0. Install and Import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -237,10 +243,10 @@ for i in loop:
loop.set_description('total_loss = %.6f' % loss)
# Save the losses for plotting
chamfer_losses.append(loss_chamfer)
edge_losses.append(loss_edge)
normal_losses.append(loss_normal)
laplacian_losses.append(loss_laplacian)
chamfer_losses.append(float(loss_chamfer.detach().cpu()))
edge_losses.append(float(loss_edge.detach().cpu()))
normal_losses.append(float(loss_normal.detach().cpu()))
laplacian_losses.append(float(loss_laplacian.detach().cpu()))
# Plot mesh
if i % plot_period == 0:

54
files/fit_textured_mesh.ipynb
View File

@@ -46,7 +46,7 @@
"id": "okLalbR_g7NS"
},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -59,19 +59,25 @@
},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -91,7 +97,6 @@
"import os\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"from skimage.io import imread\n",
"\n",
"from pytorch3d.utils import ico_sphere\n",
"import numpy as np\n",
@@ -187,11 +192,11 @@
"source": [
"### 1. Load a mesh and texture file\n",
"\n",
"Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
"Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
"\n",
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes. \n",
"\n",
"**TexturesVertex** is an auxillary datastructure for storing vertex rgb texture information about meshes. \n",
"**TexturesVertex** is an auxiliary datastructure for storing vertex rgb texture information about meshes. \n",
"\n",
"**Meshes** has several class methods which are used throughout the rendering pipeline."
]
@@ -255,7 +260,7 @@
"N = verts.shape[0]\n",
"center = verts.mean(0)\n",
"scale = max((verts - center).abs().max(0)[0])\n",
"mesh.offset_verts_(-center.expand(N, 3))\n",
"mesh.offset_verts_(-center)\n",
"mesh.scale_verts_((1.0 / float(scale)));"
]
},
@@ -309,7 +314,7 @@
"# purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to \n",
"# rasterize_meshes.py for explanations of these parameters. We also leave \n",
"# bin_size and max_faces_per_bin to their default values of None, which sets \n",
"# their values using huristics and ensures that the faster coarse-to-fine \n",
"# their values using heuristics and ensures that the faster coarse-to-fine \n",
"# rasterization method is used. Refer to docs/notes/renderer.md for an \n",
"# explanation of the difference between naive and coarse-to-fine rasterization. \n",
"raster_settings = RasterizationSettings(\n",
@@ -318,8 +323,8 @@
" faces_per_pixel=1, \n",
")\n",
"\n",
"# Create a phong renderer by composing a rasterizer and a shader. The textured \n",
"# phong shader will interpolate the texture uv coordinates for each vertex, \n",
"# Create a Phong renderer by composing a rasterizer and a shader. The textured \n",
"# Phong shader will interpolate the texture uv coordinates for each vertex, \n",
"# sample from a texture image and apply the Phong lighting model\n",
"renderer = MeshRenderer(\n",
" rasterizer=MeshRasterizer(\n",
@@ -380,7 +385,7 @@
"id": "gOb4rYx65E8z"
},
"source": [
"Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel."
"Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel."
]
},
{
@@ -449,7 +454,8 @@
" target_image=target_rgb[1], title='', \n",
" silhouette=False):\n",
" inds = 3 if silhouette else range(3)\n",
" predicted_images = renderer(predicted_mesh)\n",
" with torch.no_grad():\n",
" predicted_images = renderer(predicted_mesh)\n",
" plt.figure(figsize=(20, 10))\n",
" plt.subplot(1, 2, 1)\n",
" plt.imshow(predicted_images[0, ..., inds].cpu().detach().numpy())\n",
@@ -457,7 +463,6 @@
" plt.subplot(1, 2, 2)\n",
" plt.imshow(target_image.cpu().detach().numpy())\n",
" plt.title(title)\n",
" plt.grid(\"off\")\n",
" plt.axis(\"off\")\n",
"\n",
"# Plot losses as a function of optimization iteration\n",
@@ -601,7 +606,7 @@
"id": "QLc9zK8lEqFS"
},
"source": [
"We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:"
"We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:"
]
},
{
@@ -640,7 +645,8 @@
" sum_loss = torch.tensor(0.0, device=device)\n",
" for k, l in loss.items():\n",
" sum_loss += l * losses[k][\"weight\"]\n",
" losses[k][\"values\"].append(l)\n",
" losses[k][\"values\"].append(float(l.detach().cpu()))\n",
"\n",
" \n",
" # Print the losses\n",
" loop.set_description(\"total_loss = %.6f\" % sum_loss)\n",
@@ -824,7 +830,7 @@
" sum_loss = torch.tensor(0.0, device=device)\n",
" for k, l in loss.items():\n",
" sum_loss += l * losses[k][\"weight\"]\n",
" losses[k][\"values\"].append(l)\n",
" losses[k][\"values\"].append(float(l.detach().cpu()))\n",
" \n",
" # Print the losses\n",
" loop.set_description(\"total_loss = %.6f\" % sum_loss)\n",

56
files/fit_textured_mesh.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -17,24 +17,30 @@
# ## 0. Install and Import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -47,7 +53,6 @@ else:
import os
import torch
import matplotlib.pyplot as plt
from skimage.io import imread
from pytorch3d.utils import ico_sphere
import numpy as np
@@ -105,11 +110,11 @@ from plot_image_grid import image_grid
# ### 1. Load a mesh and texture file
#
# Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object.
# Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object.
#
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
#
# **TexturesVertex** is an auxillary datastructure for storing vertex rgb texture information about meshes.
# **TexturesVertex** is an auxiliary datastructure for storing vertex rgb texture information about meshes.
#
# **Meshes** has several class methods which are used throughout the rendering pipeline.
@@ -150,7 +155,7 @@ verts = mesh.verts_packed()
N = verts.shape[0]
center = verts.mean(0)
scale = max((verts - center).abs().max(0)[0])
mesh.offset_verts_(-center.expand(N, 3))
mesh.offset_verts_(-center)
mesh.scale_verts_((1.0 / float(scale)));
@@ -190,7 +195,7 @@ camera = OpenGLPerspectiveCameras(device=device, R=R[None, 1, ...],
# purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to
# rasterize_meshes.py for explanations of these parameters. We also leave
# bin_size and max_faces_per_bin to their default values of None, which sets
# their values using huristics and ensures that the faster coarse-to-fine
# their values using heuristics and ensures that the faster coarse-to-fine
# rasterization method is used. Refer to docs/notes/renderer.md for an
# explanation of the difference between naive and coarse-to-fine rasterization.
raster_settings = RasterizationSettings(
@@ -199,8 +204,8 @@ raster_settings = RasterizationSettings(
faces_per_pixel=1,
)
# Create a phong renderer by composing a rasterizer and a shader. The textured
# phong shader will interpolate the texture uv coordinates for each vertex,
# Create a Phong renderer by composing a rasterizer and a shader. The textured
# Phong shader will interpolate the texture uv coordinates for each vertex,
# sample from a texture image and apply the Phong lighting model
renderer = MeshRenderer(
rasterizer=MeshRasterizer(
@@ -239,7 +244,7 @@ image_grid(target_images.cpu().numpy(), rows=4, cols=5, rgb=True)
plt.show()
# Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel.
# Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel.
# In[ ]:
@@ -285,7 +290,8 @@ def visualize_prediction(predicted_mesh, renderer=renderer_silhouette,
target_image=target_rgb[1], title='',
silhouette=False):
inds = 3 if silhouette else range(3)
predicted_images = renderer(predicted_mesh)
with torch.no_grad():
predicted_images = renderer(predicted_mesh)
plt.figure(figsize=(20, 10))
plt.subplot(1, 2, 1)
plt.imshow(predicted_images[0, ..., inds].cpu().detach().numpy())
@@ -293,7 +299,6 @@ def visualize_prediction(predicted_mesh, renderer=renderer_silhouette,
plt.subplot(1, 2, 2)
plt.imshow(target_image.cpu().detach().numpy())
plt.title(title)
plt.grid("off")
plt.axis("off")
# Plot losses as a function of optimization iteration
@@ -385,7 +390,7 @@ deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)
optimizer = torch.optim.SGD([deform_verts], lr=1.0, momentum=0.9)
# We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:
# We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:
# In[ ]:
@@ -416,7 +421,8 @@ for i in loop:
sum_loss = torch.tensor(0.0, device=device)
for k, l in loss.items():
sum_loss += l * losses[k]["weight"]
losses[k]["values"].append(l)
losses[k]["values"].append(float(l.detach().cpu()))
# Print the losses
loop.set_description("total_loss = %.6f" % sum_loss)
@@ -547,7 +553,7 @@ for i in loop:
sum_loss = torch.tensor(0.0, device=device)
for k, l in loss.items():
sum_loss += l * losses[k]["weight"]
losses[k]["values"].append(l)
losses[k]["values"].append(float(l.detach().cpu()))
# Print the losses
loop.set_description("total_loss = %.6f" % sum_loss)

36
files/render_colored_points.ipynb
View File

@@ -32,7 +32,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -41,19 +41,25 @@
"metadata": {},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -70,7 +76,6 @@
"import torch\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt\n",
"from skimage.io import imread\n",
"\n",
"# Util function for loading point clouds|\n",
"import numpy as np\n",
@@ -151,7 +156,7 @@
"source": [
"## Create a renderer\n",
"\n",
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.\n",
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.\n",
"\n",
"In this example we will first create a **renderer** which uses an **orthographic camera**, and applies **alpha compositing**. Then we learn how to vary different components using the modular API. \n",
"\n",
@@ -196,7 +201,6 @@
"images = renderer(point_cloud)\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\")\n",
"plt.axis(\"off\");"
]
},
@@ -223,7 +227,6 @@
"\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\")\n",
"plt.axis(\"off\");"
]
},
@@ -271,7 +274,6 @@
"images = renderer(point_cloud)\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\")\n",
"plt.axis(\"off\");"
]
},
@@ -297,7 +299,6 @@
"images = renderer(point_cloud)\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\")\n",
"plt.axis(\"off\");"
]
},
@@ -330,7 +331,6 @@
" bg_col=torch.tensor([0.0, 1.0, 0.0, 1.0], dtype=torch.float32, device=device))\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\")\n",
"plt.axis(\"off\");"
]
},

38
files/render_colored_points.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -16,24 +16,30 @@
# ## Import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -47,7 +53,6 @@ import os
import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt
from skimage.io import imread
# Util function for loading point clouds|
import numpy as np
@@ -108,7 +113,7 @@ point_cloud = Pointclouds(points=[verts], features=[rgb])
# ## Create a renderer
#
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
#
# In this example we will first create a **renderer** which uses an **orthographic camera**, and applies **alpha compositing**. Then we learn how to vary different components using the modular API.
#
@@ -146,7 +151,6 @@ renderer = PointsRenderer(
images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off")
plt.axis("off");
@@ -165,7 +169,6 @@ images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off")
plt.axis("off");
@@ -202,7 +205,6 @@ renderer = PointsRenderer(
images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off")
plt.axis("off");
@@ -220,7 +222,6 @@ renderer = PointsRenderer(
images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off")
plt.axis("off");
@@ -245,7 +246,6 @@ images = renderer(point_cloud, gamma=(1e-4,),
bg_col=torch.tensor([0.0, 1.0, 0.0, 1.0], dtype=torch.float32, device=device))
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off")
plt.axis("off");

64
files/render_densepose.ipynb
View File

@@ -39,7 +39,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If torch, torchvision and PyTorch3D are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -48,19 +48,25 @@
"metadata": {},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -86,12 +92,11 @@
"import os\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"from skimage.io import imread\n",
"import numpy as np\n",
"\n",
"# libraries for reading data from files\n",
"from scipy.io import loadmat\n",
"from pytorch3d.io.utils import _read_image\n",
"from PIL import Image\n",
"import pickle\n",
"\n",
"# Data structures and functions for rendering\n",
@@ -178,13 +183,15 @@
" data = pickle.load(f, encoding='latin1') \n",
" v_template = torch.Tensor(data['v_template']).to(device) # (6890, 3)\n",
"ALP_UV = loadmat(data_filename)\n",
"tex = torch.from_numpy(_read_image(file_name=tex_filename, format='RGB') / 255. ).unsqueeze(0).to(device)\n",
"with Image.open(tex_filename) as image:\n",
" np_image = np.asarray(image.convert(\"RGB\")).astype(np.float32)\n",
"tex = torch.from_numpy(np_image / 255.)[None].to(device)\n",
"\n",
"verts = torch.from_numpy((ALP_UV[\"All_vertices\"]).astype(int)).squeeze().to(device) # (7829, 1)\n",
"verts = torch.from_numpy((ALP_UV[\"All_vertices\"]).astype(int)).squeeze().to(device) # (7829,)\n",
"U = torch.Tensor(ALP_UV['All_U_norm']).to(device) # (7829, 1)\n",
"V = torch.Tensor(ALP_UV['All_V_norm']).to(device) # (7829, 1)\n",
"faces = torch.from_numpy((ALP_UV['All_Faces'] - 1).astype(int)).to(device) # (13774, 3)\n",
"face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()"
"face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze() # (13774,)"
]
},
{
@@ -196,7 +203,6 @@
"# Display the texture image\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(tex.squeeze(0).cpu())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -222,6 +228,9 @@
" part = rows * i + j + 1 # parts are 1-indexed in face_indices\n",
" offset_per_part[part] = (u, v)\n",
"\n",
"U_norm = U.clone()\n",
"V_norm = V.clone()\n",
"\n",
"# iterate over faces and offset the corresponding vertex u and v values\n",
"for i in range(len(faces)):\n",
" face_vert_idxs = faces[i]\n",
@@ -232,15 +241,15 @@
" # vertices are reused, but we don't want to offset multiple times\n",
" if vert_idx.item() not in already_offset:\n",
" # offset u value\n",
" U[vert_idx] = U[vert_idx] / cols + offset_u\n",
" U_norm[vert_idx] = U[vert_idx] / cols + offset_u\n",
" # offset v value\n",
" # this also flips each part locally, as each part is upside down\n",
" V[vert_idx] = (1 - V[vert_idx]) / rows + offset_v\n",
" V_norm[vert_idx] = (1 - V[vert_idx]) / rows + offset_v\n",
" # add vertex to our set tracking offsetted vertices\n",
" already_offset.add(vert_idx.item())\n",
"\n",
"# invert V values\n",
"U_norm, V_norm = U, 1 - V"
"V_norm = 1 - V_norm"
]
},
{
@@ -257,10 +266,7 @@
"# Therefore when initializing the Meshes class,\n",
"# we need to map each of the vertices referenced by the DensePose faces (in verts, which is the \"All_vertices\" field)\n",
"# to the correct xyz coordinate in the SMPL template mesh.\n",
"v_template_extended = torch.stack(list(map(lambda vert: v_template[vert-1], verts))).unsqueeze(0).to(device) # (1, 7829, 3)\n",
"\n",
"# add a batch dimension to faces\n",
"faces = faces.unsqueeze(0)"
"v_template_extended = v_template[verts-1][None] # (1, 7829, 3)"
]
},
{
@@ -271,7 +277,7 @@
"\n",
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.\n",
"\n",
"**TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes."
"**TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes."
]
},
{
@@ -280,8 +286,8 @@
"metadata": {},
"outputs": [],
"source": [
"texture = TexturesUV(maps=tex, faces_uvs=faces, verts_uvs=verts_uv)\n",
"mesh = Meshes(v_template_extended, faces, texture)"
"texture = TexturesUV(maps=tex, faces_uvs=faces[None], verts_uvs=verts_uv)\n",
"mesh = Meshes(v_template_extended, faces[None], texture)"
]
},
{
@@ -314,7 +320,7 @@
"# Place a point light in front of the person. \n",
"lights = PointLights(device=device, location=[[0.0, 0.0, 2.0]])\n",
"\n",
"# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will \n",
"# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will \n",
"# interpolate the texture uv coordinates for each vertex, sample from a texture image and \n",
"# apply the Phong lighting model\n",
"renderer = MeshRenderer(\n",
@@ -346,7 +352,6 @@
"images = renderer(mesh)\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -387,7 +392,6 @@
"source": [
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},

66
files/render_densepose.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -20,24 +20,30 @@
# ## Import modules
# If torch, torchvision and PyTorch3D are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -57,12 +63,11 @@ get_ipython().system('pip install chumpy')
import os
import torch
import matplotlib.pyplot as plt
from skimage.io import imread
import numpy as np
# libraries for reading data from files
from scipy.io import loadmat
from pytorch3d.io.utils import _read_image
from PIL import Image
import pickle
# Data structures and functions for rendering
@@ -133,13 +138,15 @@ with open(verts_filename, 'rb') as f:
data = pickle.load(f, encoding='latin1')
v_template = torch.Tensor(data['v_template']).to(device) # (6890, 3)
ALP_UV = loadmat(data_filename)
tex = torch.from_numpy(_read_image(file_name=tex_filename, format='RGB') / 255. ).unsqueeze(0).to(device)
with Image.open(tex_filename) as image:
np_image = np.asarray(image.convert("RGB")).astype(np.float32)
tex = torch.from_numpy(np_image / 255.)[None].to(device)
verts = torch.from_numpy((ALP_UV["All_vertices"]).astype(int)).squeeze().to(device) # (7829, 1)
verts = torch.from_numpy((ALP_UV["All_vertices"]).astype(int)).squeeze().to(device) # (7829,)
U = torch.Tensor(ALP_UV['All_U_norm']).to(device) # (7829, 1)
V = torch.Tensor(ALP_UV['All_V_norm']).to(device) # (7829, 1)
faces = torch.from_numpy((ALP_UV['All_Faces'] - 1).astype(int)).to(device) # (13774, 3)
face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()
face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze() # (13774,)
# In[ ]:
@@ -148,7 +155,6 @@ face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()
# Display the texture image
plt.figure(figsize=(10, 10))
plt.imshow(tex.squeeze(0).cpu())
plt.grid("off");
plt.axis("off");
@@ -166,6 +172,9 @@ for i, u in enumerate(np.linspace(0, 1, cols, endpoint=False)):
part = rows * i + j + 1 # parts are 1-indexed in face_indices
offset_per_part[part] = (u, v)
U_norm = U.clone()
V_norm = V.clone()
# iterate over faces and offset the corresponding vertex u and v values
for i in range(len(faces)):
face_vert_idxs = faces[i]
@@ -176,15 +185,15 @@ for i in range(len(faces)):
# vertices are reused, but we don't want to offset multiple times
if vert_idx.item() not in already_offset:
# offset u value
U[vert_idx] = U[vert_idx] / cols + offset_u
U_norm[vert_idx] = U[vert_idx] / cols + offset_u
# offset v value
# this also flips each part locally, as each part is upside down
V[vert_idx] = (1 - V[vert_idx]) / rows + offset_v
V_norm[vert_idx] = (1 - V[vert_idx]) / rows + offset_v
# add vertex to our set tracking offsetted vertices
already_offset.add(vert_idx.item())
# invert V values
U_norm, V_norm = U, 1 - V
V_norm = 1 - V_norm
# In[ ]:
@@ -198,23 +207,20 @@ verts_uv = torch.cat([U_norm[None],V_norm[None]], dim=2) # (1, 7829, 2)
# Therefore when initializing the Meshes class,
# we need to map each of the vertices referenced by the DensePose faces (in verts, which is the "All_vertices" field)
# to the correct xyz coordinate in the SMPL template mesh.
v_template_extended = torch.stack(list(map(lambda vert: v_template[vert-1], verts))).unsqueeze(0).to(device) # (1, 7829, 3)
# add a batch dimension to faces
faces = faces.unsqueeze(0)
v_template_extended = v_template[verts-1][None] # (1, 7829, 3)
# ### Create our textured mesh
#
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
#
# **TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes.
# **TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes.
# In[ ]:
texture = TexturesUV(maps=tex, faces_uvs=faces, verts_uvs=verts_uv)
mesh = Meshes(v_template_extended, faces, texture)
texture = TexturesUV(maps=tex, faces_uvs=faces[None], verts_uvs=verts_uv)
mesh = Meshes(v_template_extended, faces[None], texture)
# ## Create a renderer
@@ -239,7 +245,7 @@ raster_settings = RasterizationSettings(
# Place a point light in front of the person.
lights = PointLights(device=device, location=[[0.0, 0.0, 2.0]])
# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will
# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will
# interpolate the texture uv coordinates for each vertex, sample from a texture image and
# apply the Phong lighting model
renderer = MeshRenderer(
@@ -263,7 +269,6 @@ renderer = MeshRenderer(
images = renderer(mesh)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");
@@ -293,7 +298,6 @@ images = renderer(mesh, lights=lights, cameras=cameras)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");

46
files/render_textured_meshes.ipynb
View File

@@ -47,7 +47,7 @@
"id": "okLalbR_g7NS"
},
"source": [
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
]
},
{
@@ -64,19 +64,25 @@
},
"outputs": [],
"source": [
"!pip install torch torchvision\n",
"import os\n",
"import sys\n",
"import torch\n",
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
" !pip install pytorch3d\n",
"else:\n",
" need_pytorch3d=False\n",
" try:\n",
" import pytorch3d\n",
" except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
" if need_pytorch3d:\n",
"need_pytorch3d=False\n",
"try:\n",
" import pytorch3d\n",
"except ModuleNotFoundError:\n",
" need_pytorch3d=True\n",
"if need_pytorch3d:\n",
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
" # We try to install PyTorch3D via a released wheel.\n",
" version_str=\"\".join([\n",
" f\"py3{sys.version_info.minor}_cu\",\n",
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
" ])\n",
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" else:\n",
" # We try to install PyTorch3D from source.\n",
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
" !tar xzf 1.10.0.tar.gz\n",
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
@@ -96,7 +102,6 @@
"import os\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"from skimage.io import imread\n",
"\n",
"# Util function for loading meshes\n",
"from pytorch3d.io import load_objs_as_meshes, load_obj\n",
@@ -185,11 +190,11 @@
"source": [
"### 1. Load a mesh and texture file\n",
"\n",
"Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
"Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
"\n",
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes. \n",
"\n",
"**TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes. \n",
"**TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes. \n",
"\n",
"**Meshes** has several class methods which are used throughout the rendering pipeline."
]
@@ -277,7 +282,6 @@
"plt.figure(figsize=(7,7))\n",
"texture_image=mesh.textures.maps_padded()\n",
"plt.imshow(texture_image.squeeze().cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -296,7 +300,6 @@
"source": [
"plt.figure(figsize=(7,7))\n",
"texturesuv_image_matplotlib(mesh.textures, subsample=None)\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -311,7 +314,7 @@
"\n",
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.\n",
"\n",
"In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **phong shading**. Then we learn how to vary different components using the modular API. "
"In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **Phong shading**. Then we learn how to vary different components using the modular API. "
]
},
{
@@ -346,7 +349,7 @@
"# -z direction. \n",
"lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])\n",
"\n",
"# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will \n",
"# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will \n",
"# interpolate the texture uv coordinates for each vertex, sample from a texture image and \n",
"# apply the Phong lighting model\n",
"renderer = MeshRenderer(\n",
@@ -399,7 +402,6 @@
"images = renderer(mesh)\n",
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -412,7 +414,7 @@
"source": [
"## 4. Move the light behind the object and re-render\n",
"\n",
"We can pass arbirary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/\n",
"We can pass arbitrary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/\n",
"\n",
"In this case, we can simply update the location of the lights and pass them into the call to the renderer. \n",
"\n",
@@ -450,7 +452,6 @@
"source": [
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -514,7 +515,6 @@
"source": [
"plt.figure(figsize=(10, 10))\n",
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
"plt.grid(\"off\");\n",
"plt.axis(\"off\");"
]
},
@@ -573,7 +573,7 @@
},
"outputs": [],
"source": [
"# We can pass arbirary keyword arguments to the rasterizer/shader via the renderer\n",
"# We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer\n",
"# so the renderer does not need to be reinitialized if any of the settings change.\n",
"images = renderer(meshes, cameras=cameras, lights=lights)"
]

48
files/render_textured_meshes.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env python
# coding: utf-8
# In[ ]:
@@ -18,24 +18,30 @@
# ## 0. Install and Import modules
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
# In[ ]:
get_ipython().system('pip install torch torchvision')
import os
import sys
import torch
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
get_ipython().system('pip install pytorch3d')
else:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
need_pytorch3d=False
try:
import pytorch3d
except ModuleNotFoundError:
need_pytorch3d=True
if need_pytorch3d:
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
# We try to install PyTorch3D via a released wheel.
version_str="".join([
f"py3{sys.version_info.minor}_cu",
torch.version.cuda.replace(".",""),
f"_pyt{torch.__version__[0:5:2]}"
])
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
else:
# We try to install PyTorch3D from source.
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
get_ipython().system('tar xzf 1.10.0.tar.gz')
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
@@ -48,7 +54,6 @@ else:
import os
import torch
import matplotlib.pyplot as plt
from skimage.io import imread
# Util function for loading meshes
from pytorch3d.io import load_objs_as_meshes, load_obj
@@ -96,11 +101,11 @@ from plot_image_grid import image_grid
# ### 1. Load a mesh and texture file
#
# Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object.
# Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object.
#
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
#
# **TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes.
# **TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes.
#
# **Meshes** has several class methods which are used throughout the rendering pipeline.
@@ -142,7 +147,6 @@ mesh = load_objs_as_meshes([obj_filename], device=device)
plt.figure(figsize=(7,7))
texture_image=mesh.textures.maps_padded()
plt.imshow(texture_image.squeeze().cpu().numpy())
plt.grid("off");
plt.axis("off");
@@ -153,7 +157,6 @@ plt.axis("off");
plt.figure(figsize=(7,7))
texturesuv_image_matplotlib(mesh.textures, subsample=None)
plt.grid("off");
plt.axis("off");
@@ -161,7 +164,7 @@ plt.axis("off");
#
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
#
# In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **phong shading**. Then we learn how to vary different components using the modular API.
# In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **Phong shading**. Then we learn how to vary different components using the modular API.
# In[ ]:
@@ -188,7 +191,7 @@ raster_settings = RasterizationSettings(
# -z direction.
lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will
# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will
# interpolate the texture uv coordinates for each vertex, sample from a texture image and
# apply the Phong lighting model
renderer = MeshRenderer(
@@ -214,13 +217,12 @@ renderer = MeshRenderer(
images = renderer(mesh)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");
# ## 4. Move the light behind the object and re-render
#
# We can pass arbirary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/
# We can pass arbitrary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/
#
# In this case, we can simply update the location of the lights and pass them into the call to the renderer.
#
@@ -239,7 +241,6 @@ images = renderer(mesh, lights=lights)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");
@@ -277,7 +278,6 @@ images = renderer(mesh, lights=lights, materials=materials, cameras=cameras)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");
@@ -315,7 +315,7 @@ lights.location = torch.tensor([[0.0, 0.0, -3.0]], device=device)
# In[ ]:
# We can pass arbirary keyword arguments to the rasterizer/shader via the renderer
# We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer
# so the renderer does not need to be reinitialized if any of the settings change.
images = renderer(meshes, cameras=cameras, lights=lights)