mirror of
https://github.com/facebookresearch/pytorch3d.git
synced 2025-08-02 03:42:50 +08:00
update for version 0.5.0
This commit is contained in:
Jeremy Francis Reizenstein
parent
13e2fd2b5b
commit
3ccabda50d
File diff suppressed because one or more lines are too long
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>batching · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Batching"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="batching · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Batching"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>batching · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Batching"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="batching · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Batching"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -63,7 +63,7 @@
|
||||
}
|
||||
});
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="batching"></a><a href="#batching" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Batching</h1>
|
||||
<p>In deep learning, every optimization step operates on multiple input examples for robust training. Thus, efficient batching is crucial. For image inputs, batching is straighforward; N images are resized to the same height and width and stacked as a 4 dimensional tensor of shape <code>N x 3 x H x W</code>. For meshes, batching is less straighforward.</p>
|
||||
<p>In deep learning, every optimization step operates on multiple input examples for robust training. Thus, efficient batching is crucial. For image inputs, batching is straightforward; N images are resized to the same height and width and stacked as a 4 dimensional tensor of shape <code>N x 3 x H x W</code>. For meshes, batching is less straightforward.</p>
|
||||
<p><img src="assets/batch_intro.png" alt="batch_intro" align="middle"/></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="batch-modes-for-meshes"></a><a href="#batch-modes-for-meshes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Batch modes for meshes</h2>
|
||||
<p>Assume you want to construct a batch containing two meshes, with <code>mesh1 = (v1: V1 x 3, f1: F1 x 3)</code> containing <code>V1</code> vertices and <code>F1</code> faces, and <code>mesh2 = (v2: V2 x 3, f2: F2 x 3)</code> with <code>V2 (!= V1)</code> vertices and <code>F2 (!= F1)</code> faces. The <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure provides three different ways to batch <em>heterogeneous</em> meshes. If <code>meshes = Meshes(verts = [v1, v2], faces = [f1, f2])</code> is an instantiation of the data structure, then</p>
|
||||
@ -74,6 +74,6 @@
|
||||
</ul>
|
||||
<p><img src="assets/batch_modes.gif" alt="batch_modes" height="450" align="middle" /></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="use-cases-for-batch-modes"></a><a href="#use-cases-for-batch-modes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Use cases for batch modes</h2>
|
||||
<p>The need for different mesh batch modes is inherent to the way pytorch operators are implemented. To fully utilize the optimized pytorch ops, the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure allows for efficient conversion between the different batch modes. This is crucial when aiming for a fast and efficient training cycle. An example of this is <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a>. Here, in the same forward pass different parts of the network assume different inputs, which are computed by converting between the different batch modes. In particular, <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/vert_align.py">vert_align</a> assumes a <em>padded</em> input tensor while immediately after <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/graph_conv.py">graph_conv</a> assumes a <em>packed</em> input tensor.</p>
|
||||
<p>The need for different mesh batch modes is inherent to the way PyTorch operators are implemented. To fully utilize the optimized PyTorch ops, the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure allows for efficient conversion between the different batch modes. This is crucial when aiming for a fast and efficient training cycle. An example of this is <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a>. Here, in the same forward pass different parts of the network assume different inputs, which are computed by converting between the different batch modes. In particular, <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/vert_align.py">vert_align</a> assumes a <em>padded</em> input tensor while immediately after <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/graph_conv.py">graph_conv</a> assumes a <em>packed</em> input tensor.</p>
|
||||
<p><img src="assets/meshrcnn.png" alt="meshrcnn" width="700" align="middle" /></p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/datasets"><span class="arrow-prev">← </span><span>Data loaders</span></a><a class="docs-next button" href="/docs/cubify"><span>Cubify</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#batch-modes-for-meshes">Batch modes for meshes</a></li><li><a href="#use-cases-for-batch-modes">Use cases for batch modes</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/datasets"><span class="arrow-prev">← </span><span>Data loaders</span></a><a class="docs-next button" href="/docs/cubify"><span>Cubify</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#batch-modes-for-meshes">Batch modes for meshes</a></li><li><a href="#use-cases-for-batch-modes">Use cases for batch modes</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>batching · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Batching"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="batching · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Batching"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>batching · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Batching"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="batching · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Batching"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -63,7 +63,7 @@
|
||||
}
|
||||
});
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="batching"></a><a href="#batching" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Batching</h1>
|
||||
<p>In deep learning, every optimization step operates on multiple input examples for robust training. Thus, efficient batching is crucial. For image inputs, batching is straighforward; N images are resized to the same height and width and stacked as a 4 dimensional tensor of shape <code>N x 3 x H x W</code>. For meshes, batching is less straighforward.</p>
|
||||
<p>In deep learning, every optimization step operates on multiple input examples for robust training. Thus, efficient batching is crucial. For image inputs, batching is straightforward; N images are resized to the same height and width and stacked as a 4 dimensional tensor of shape <code>N x 3 x H x W</code>. For meshes, batching is less straightforward.</p>
|
||||
<p><img src="assets/batch_intro.png" alt="batch_intro" align="middle"/></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="batch-modes-for-meshes"></a><a href="#batch-modes-for-meshes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Batch modes for meshes</h2>
|
||||
<p>Assume you want to construct a batch containing two meshes, with <code>mesh1 = (v1: V1 x 3, f1: F1 x 3)</code> containing <code>V1</code> vertices and <code>F1</code> faces, and <code>mesh2 = (v2: V2 x 3, f2: F2 x 3)</code> with <code>V2 (!= V1)</code> vertices and <code>F2 (!= F1)</code> faces. The <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure provides three different ways to batch <em>heterogeneous</em> meshes. If <code>meshes = Meshes(verts = [v1, v2], faces = [f1, f2])</code> is an instantiation of the data structure, then</p>
|
||||
@ -74,6 +74,6 @@
|
||||
</ul>
|
||||
<p><img src="assets/batch_modes.gif" alt="batch_modes" height="450" align="middle" /></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="use-cases-for-batch-modes"></a><a href="#use-cases-for-batch-modes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Use cases for batch modes</h2>
|
||||
<p>The need for different mesh batch modes is inherent to the way pytorch operators are implemented. To fully utilize the optimized pytorch ops, the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure allows for efficient conversion between the different batch modes. This is crucial when aiming for a fast and efficient training cycle. An example of this is <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a>. Here, in the same forward pass different parts of the network assume different inputs, which are computed by converting between the different batch modes. In particular, <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/vert_align.py">vert_align</a> assumes a <em>padded</em> input tensor while immediately after <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/graph_conv.py">graph_conv</a> assumes a <em>packed</em> input tensor.</p>
|
||||
<p>The need for different mesh batch modes is inherent to the way PyTorch operators are implemented. To fully utilize the optimized PyTorch ops, the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure allows for efficient conversion between the different batch modes. This is crucial when aiming for a fast and efficient training cycle. An example of this is <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a>. Here, in the same forward pass different parts of the network assume different inputs, which are computed by converting between the different batch modes. In particular, <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/vert_align.py">vert_align</a> assumes a <em>padded</em> input tensor while immediately after <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/graph_conv.py">graph_conv</a> assumes a <em>packed</em> input tensor.</p>
|
||||
<p><img src="assets/meshrcnn.png" alt="meshrcnn" width="700" align="middle" /></p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/datasets"><span class="arrow-prev">← </span><span>Data loaders</span></a><a class="docs-next button" href="/docs/cubify"><span>Cubify</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#batch-modes-for-meshes">Batch modes for meshes</a></li><li><a href="#use-cases-for-batch-modes">Use cases for batch modes</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/datasets"><span class="arrow-prev">← </span><span>Data loaders</span></a><a class="docs-next button" href="/docs/cubify"><span>Cubify</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#batch-modes-for-meshes">Batch modes for meshes</a></li><li><a href="#use-cases-for-batch-modes">Use cases for batch modes</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cameras · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cameras"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cameras · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cameras"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cameras · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cameras"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cameras · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cameras"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -71,45 +71,68 @@ This is the system the object/scene lives - the world.</li>
|
||||
<li><strong>Camera view coordinate system</strong>
|
||||
This is the system that has its origin on the image plane and the <code>Z</code>-axis perpendicular to the image plane. In PyTorch3D, we assume that <code>+X</code> points left, and <code>+Y</code> points up and <code>+Z</code> points out from the image plane. The transformation from world to view happens after applying a rotation (<code>R</code>) and translation (<code>T</code>).</li>
|
||||
<li><strong>NDC coordinate system</strong>
|
||||
This is the normalized coordinate system that confines in a volume the renderered part of the object/scene. Also known as view volume. Under the PyTorch3D convention, <code>(+1, +1, znear)</code> is the top left near corner, and <code>(-1, -1, zfar)</code> is the bottom right far corner of the volume. The transformation from view to NDC happens after applying the camera projection matrix (<code>P</code>).</li>
|
||||
This is the normalized coordinate system that confines in a volume the rendered part of the object/scene. Also known as view volume. Under the PyTorch3D convention, <code>(+1, +1, znear)</code> is the top left near corner, and <code>(-1, -1, zfar)</code> is the bottom right far corner of the volume. For non-square volumes, the side of the volume in <code>XY</code> with the smallest length ranges from <code>[-1, 1]</code> while the larger side from <code>[-s, s]</code>, where <code>s</code> is the aspect ratio and <code>s > 1</code> (larger divided by smaller side).
|
||||
The transformation from view to NDC happens after applying the camera projection matrix (<code>P</code>).</li>
|
||||
<li><strong>Screen coordinate system</strong>
|
||||
This is another representation of the view volume with the <code>XY</code> coordinates defined in pixel space instead of a normalized space.</li>
|
||||
</ul>
|
||||
<p>An illustration of the 4 coordinate systems is shown below
|
||||
<img src="https://user-images.githubusercontent.com/4369065/90317960-d9b8db80-dee1-11ea-8088-39c414b1e2fa.png" alt="cameras"></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="defining-cameras-in-pytorch3d"></a><a href="#defining-cameras-in-pytorch3d" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Defining Cameras in PyTorch3D</h2>
|
||||
<p>Cameras in PyTorch3D transform an object/scene from world to NDC by first transforming the object/scene to view (via transforms <code>R</code> and <code>T</code>) and then projecting the 3D object/scene to NDC (via the projection matrix <code>P</code>, else known as camera matrix). Thus, the camera parameters in <code>P</code> are assumed to be in NDC space. If the user has camera parameters in screen space, which is a common use case, the parameters should transformed to NDC (see below for an example)</p>
|
||||
<p>We describe the camera types in PyTorch3D and the convention for the camera parameters provided at construction time.</p>
|
||||
<p>Cameras in PyTorch3D transform an object/scene from world to view by first transforming the object/scene to view (via transforms <code>R</code> and <code>T</code>) and then projecting the 3D object/scene to a normalized space via the projection matrix <code>P = K[R | T]</code>, where <code>K</code> is the intrinsic matrix. The camera parameters in <code>K</code> define the normalized space. If users define the camera parameters in NDC space, then the transform projects points to NDC. If the camera parameters are defined in screen space, the transformed points are in screen space.</p>
|
||||
<p>Note that the base <code>CamerasBase</code> class makes no assumptions about the coordinate systems. All the above transforms are geometric transforms defined purely by <code>R</code>, <code>T</code> and <code>K</code>. This means that users can define cameras in any coordinate system and for any transforms. The method <code>transform_points</code> will apply <code>K</code> , <code>R</code> and <code>T</code> to the input points as a simple matrix transformation. However, if users wish to use cameras with the PyTorch3D renderer, they need to abide to PyTorch3D's coordinate system assumptions (read below).</p>
|
||||
<p>We provide instantiations of common camera types in PyTorch3D and how users can flexibly define the projection space below.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="interfacing-with-the-pytorch3d-renderer"></a><a href="#interfacing-with-the-pytorch3d-renderer" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Interfacing with the PyTorch3D Renderer</h2>
|
||||
<p>The PyTorch3D renderer for both meshes and point clouds assumes that the camera transformed points, meaning the points passed as input to the rasterizer, are in PyTorch3D's NDC space. So to get the expected rendering outcome, users need to make sure that their 3D input data and cameras abide by these PyTorch3D coordinate system assumptions. The PyTorch3D coordinate system assumes <code>+X:left</code>, <code>+Y: up</code> and <code>+Z: from us to scene</code> (right-handed) . Confusions regarding coordinate systems are common so we advise that you spend some time understanding your data and the coordinate system they live in and transform them accordingly before using the PyTorch3D renderer.</p>
|
||||
<p>Examples of cameras and how they interface with the PyTorch3D renderer can be found in our tutorials.</p>
|
||||
<h3><a class="anchor" aria-hidden="true" id="camera-types"></a><a href="#camera-types" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Camera Types</h3>
|
||||
<p>All cameras inherit from <code>CamerasBase</code> which is a base class for all cameras. PyTorch3D provides four different camera types. The <code>CamerasBase</code> defines methods that are common to all camera models:</p>
|
||||
<ul>
|
||||
<li><code>get_camera_center</code> that returns the optical center of the camera in world coordinates</li>
|
||||
<li><code>get_world_to_view_transform</code> which returns a 3D transform from world coordinates to the camera view coordinates (R, T)</li>
|
||||
<li><code>get_full_projection_transform</code> which composes the projection transform (P) with the world-to-view transform (R, T)</li>
|
||||
<li><code>get_world_to_view_transform</code> which returns a 3D transform from world coordinates to the camera view coordinates <code>(R, T)</code></li>
|
||||
<li><code>get_full_projection_transform</code> which composes the projection transform (<code>K</code>) with the world-to-view transform <code>(R, T)</code></li>
|
||||
<li><code>transform_points</code> which takes a set of input points in world coordinates and projects to NDC coordinates ranging from [-1, -1, znear] to [+1, +1, zfar].</li>
|
||||
<li><code>get_ndc_camera_transform</code> which defines the conversion to PyTorch3D's NDC space and is called when interfacing with the PyTorch3D renderer. If the camera is defined in NDC space, then the identity transform is returned. If the cameras is defined in screen space, the conversion from screen to NDC is returned. If users define their own camera in screen space, they need to think of the screen to NDC conversion. We provide examples for the <code>PerspectiveCameras</code> and <code>OrthographicCameras</code>.</li>
|
||||
<li><code>transform_points_ndc</code> which takes a set of points in world coordinates and projects them to PyTorch3D's NDC space</li>
|
||||
<li><code>transform_points_screen</code> which takes a set of input points in world coordinates and projects them to the screen coordinates ranging from [0, 0, znear] to [W-1, H-1, zfar]</li>
|
||||
</ul>
|
||||
<p>Users can easily customize their own cameras. For each new camera, users should implement the <code>get_projection_transform</code> routine that returns the mapping <code>P</code> from camera view coordinates to NDC coordinates.</p>
|
||||
<h4><a class="anchor" aria-hidden="true" id="fovperspectivecameras-fovorthographiccameras"></a><a href="#fovperspectivecameras-fovorthographiccameras" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FoVPerspectiveCameras, FoVOrthographicCameras</h4>
|
||||
<p>These two cameras follow the OpenGL convention for perspective and orthographic cameras respectively. The user provides the near <code>znear</code> and far <code>zfar</code> field which confines the view volume in the <code>Z</code> axis. The view volume in the <code>XY</code> plane is defined by field of view angle (<code>fov</code>) in the case of <code>FoVPerspectiveCameras</code> and by <code>min_x, min_y, max_x, max_y</code> in the case of <code>FoVOrthographicCameras</code>.</p>
|
||||
<p>These two cameras follow the OpenGL convention for perspective and orthographic cameras respectively. The user provides the near <code>znear</code> and far <code>zfar</code> field which confines the view volume in the <code>Z</code> axis. The view volume in the <code>XY</code> plane is defined by field of view angle (<code>fov</code>) in the case of <code>FoVPerspectiveCameras</code> and by <code>min_x, min_y, max_x, max_y</code> in the case of <code>FoVOrthographicCameras</code>.
|
||||
These cameras are by default in NDC space.</p>
|
||||
<h4><a class="anchor" aria-hidden="true" id="perspectivecameras-orthographiccameras"></a><a href="#perspectivecameras-orthographiccameras" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>PerspectiveCameras, OrthographicCameras</h4>
|
||||
<p>These two cameras follow the Multi-View Geometry convention for cameras. The user provides the focal length (<code>fx</code>, <code>fy</code>) and the principal point (<code>px</code>, <code>py</code>). For example, <code>camera = PerspectiveCameras(focal_length=((fx, fy),), principal_point=((px, py),))</code></p>
|
||||
<p>As mentioned above, the focal length and principal point are used to convert a point <code>(X, Y, Z)</code> from view coordinates to NDC coordinates, as follows</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># for perspective</span>
|
||||
<span class="hljs-attr">x_ndc</span> = fx * X / Z + px
|
||||
<span class="hljs-attr">y_ndc</span> = fy * Y / Z + py
|
||||
<span class="hljs-attr">z_ndc</span> = <span class="hljs-number">1</span> / Z
|
||||
<p>The camera projection of a 3D point <code>(X, Y, Z)</code> in view coordinates to a point <code>(x, y, z)</code> in projection space (either NDC or screen) is</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># for perspective camera</span>
|
||||
<span class="hljs-attr">x</span> = fx * X / Z + px
|
||||
<span class="hljs-attr">y</span> = fy * Y / Z + py
|
||||
<span class="hljs-attr">z</span> = <span class="hljs-number">1</span> / Z
|
||||
|
||||
<span class="hljs-comment"># for orthographic</span>
|
||||
<span class="hljs-attr">x_ndc</span> = fx * X + px
|
||||
<span class="hljs-attr">y_ndc</span> = fy * Y + py
|
||||
<span class="hljs-attr">z_ndc</span> = Z
|
||||
<span class="hljs-comment"># for orthographic camera</span>
|
||||
<span class="hljs-attr">x</span> = fx * X + px
|
||||
<span class="hljs-attr">y</span> = fy * Y + py
|
||||
<span class="hljs-attr">z</span> = Z
|
||||
</code></pre>
|
||||
<p>Commonly, users have access to the focal length (<code>fx_screen</code>, <code>fy_screen</code>) and the principal point (<code>px_screen</code>, <code>py_screen</code>) in screen space. In that case, to construct the camera the user needs to additionally provide the <code>image_size = ((image_width, image_height),)</code>. More precisely, <code>camera = PerspectiveCameras(focal_length=((fx_screen, fy_screen),), principal_point=((px_screen, py_screen),), image_size = ((image_width, image_height),))</code>. Internally, the camera parameters are converted from screen to NDC as follows:</p>
|
||||
<pre><code class="hljs"><span class="hljs-attr">fx</span> = fx_screen * <span class="hljs-number">2.0</span> / image_width
|
||||
<span class="hljs-attr">fy</span> = fy_screen * <span class="hljs-number">2.0</span> / image_height
|
||||
<p>The user can define the camera parameters in NDC or in screen space. Screen space camera parameters are common and for that case the user needs to set <code>in_ndc</code> to <code>False</code> and also provide the <code>image_size=(height, width)</code> of the screen, aka the image.</p>
|
||||
<p>The <code>get_ndc_camera_transform</code> provides the transform from screen to NDC space in PyTorch3D. Note that the screen space assumes that the principal point is provided in the space with <code>+X left</code>, <code>+Y down</code> and origin at the top left corner of the image. To convert to NDC we need to account for the scaling of the normalized space as well as the change in <code>XY</code> direction.</p>
|
||||
<p>Below are example of equivalent <code>PerspectiveCameras</code> instantiations in NDC and screen space, respectively.</p>
|
||||
<pre><code class="hljs css language-python"><span class="hljs-comment"># NDC space camera</span>
|
||||
fcl_ndc = (<span class="hljs-number">1.2</span>,)
|
||||
prp_ndc = ((<span class="hljs-number">0.2</span>, <span class="hljs-number">0.5</span>),)
|
||||
cameras_ndc = PerspectiveCameras(focal_length=fcl_ndc, principal_point=prp_ndc)
|
||||
|
||||
<span class="hljs-attr">px</span> = - (px_screen - image_width / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / image_width
|
||||
<span class="hljs-attr">py</span> = - (py_screen - image_height / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span>/ image_height
|
||||
<span class="hljs-comment"># Screen space camera</span>
|
||||
image_size = ((<span class="hljs-number">128</span>, <span class="hljs-number">256</span>),) <span class="hljs-comment"># (h, w)</span>
|
||||
fcl_screen = (<span class="hljs-number">76.2</span>,) <span class="hljs-comment"># fcl_ndc * (min(image_size) - 1) / 2</span>
|
||||
prp_screen = ((<span class="hljs-number">114.8</span>, <span class="hljs-number">31.75</span>), ) <span class="hljs-comment"># (w - 1) / 2 - px_ndc * (min(image_size) - 1) / 2, (h - 1) / 2 - py_ndc * (min(image_size) - 1) / 2</span>
|
||||
cameras_screen = PerspectiveCameras(focal_length=fcl_screen, principal_point=prp_screen, in_ndc=<span class="hljs-literal">False</span>, image_size=image_size)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer_getting_started"><span class="arrow-prev">← </span><span>Getting Started</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#camera-coordinate-systems">Camera Coordinate Systems</a></li><li><a href="#defining-cameras-in-pytorch3d">Defining Cameras in PyTorch3D</a><ul class="toc-headings"><li><a href="#camera-types">Camera Types</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
<p>The relationship between screen and NDC specifications of a camera's <code>focal_length</code> and <code>principal_point</code> is given by the following equations, where <code>s = min(image_width, image_height)</code>.
|
||||
The transformation of x and y coordinates between screen and NDC is exactly the same as for px and py.</p>
|
||||
<pre><code class="hljs">fx_ndc = fx_screen * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
fy_ndc = fy_screen * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
|
||||
px_ndc = - (<span class="hljs-name">px_screen</span> - (<span class="hljs-name">image_width</span> - <span class="hljs-number">1</span>) / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
py_ndc = - (<span class="hljs-name">py_screen</span> - (<span class="hljs-name">image_height</span> - <span class="hljs-number">1</span>) / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Georgia Gkioxari</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer_getting_started"><span class="arrow-prev">← </span><span>Getting Started</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#camera-coordinate-systems">Camera Coordinate Systems</a></li><li><a href="#defining-cameras-in-pytorch3d">Defining Cameras in PyTorch3D</a></li><li><a href="#interfacing-with-the-pytorch3d-renderer">Interfacing with the PyTorch3D Renderer</a><ul class="toc-headings"><li><a href="#camera-types">Camera Types</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cameras · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cameras"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cameras · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cameras"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cameras · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cameras"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cameras · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cameras"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -71,45 +71,68 @@ This is the system the object/scene lives - the world.</li>
|
||||
<li><strong>Camera view coordinate system</strong>
|
||||
This is the system that has its origin on the image plane and the <code>Z</code>-axis perpendicular to the image plane. In PyTorch3D, we assume that <code>+X</code> points left, and <code>+Y</code> points up and <code>+Z</code> points out from the image plane. The transformation from world to view happens after applying a rotation (<code>R</code>) and translation (<code>T</code>).</li>
|
||||
<li><strong>NDC coordinate system</strong>
|
||||
This is the normalized coordinate system that confines in a volume the renderered part of the object/scene. Also known as view volume. Under the PyTorch3D convention, <code>(+1, +1, znear)</code> is the top left near corner, and <code>(-1, -1, zfar)</code> is the bottom right far corner of the volume. The transformation from view to NDC happens after applying the camera projection matrix (<code>P</code>).</li>
|
||||
This is the normalized coordinate system that confines in a volume the rendered part of the object/scene. Also known as view volume. Under the PyTorch3D convention, <code>(+1, +1, znear)</code> is the top left near corner, and <code>(-1, -1, zfar)</code> is the bottom right far corner of the volume. For non-square volumes, the side of the volume in <code>XY</code> with the smallest length ranges from <code>[-1, 1]</code> while the larger side from <code>[-s, s]</code>, where <code>s</code> is the aspect ratio and <code>s > 1</code> (larger divided by smaller side).
|
||||
The transformation from view to NDC happens after applying the camera projection matrix (<code>P</code>).</li>
|
||||
<li><strong>Screen coordinate system</strong>
|
||||
This is another representation of the view volume with the <code>XY</code> coordinates defined in pixel space instead of a normalized space.</li>
|
||||
</ul>
|
||||
<p>An illustration of the 4 coordinate systems is shown below
|
||||
<img src="https://user-images.githubusercontent.com/4369065/90317960-d9b8db80-dee1-11ea-8088-39c414b1e2fa.png" alt="cameras"></p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="defining-cameras-in-pytorch3d"></a><a href="#defining-cameras-in-pytorch3d" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Defining Cameras in PyTorch3D</h2>
|
||||
<p>Cameras in PyTorch3D transform an object/scene from world to NDC by first transforming the object/scene to view (via transforms <code>R</code> and <code>T</code>) and then projecting the 3D object/scene to NDC (via the projection matrix <code>P</code>, else known as camera matrix). Thus, the camera parameters in <code>P</code> are assumed to be in NDC space. If the user has camera parameters in screen space, which is a common use case, the parameters should transformed to NDC (see below for an example)</p>
|
||||
<p>We describe the camera types in PyTorch3D and the convention for the camera parameters provided at construction time.</p>
|
||||
<p>Cameras in PyTorch3D transform an object/scene from world to view by first transforming the object/scene to view (via transforms <code>R</code> and <code>T</code>) and then projecting the 3D object/scene to a normalized space via the projection matrix <code>P = K[R | T]</code>, where <code>K</code> is the intrinsic matrix. The camera parameters in <code>K</code> define the normalized space. If users define the camera parameters in NDC space, then the transform projects points to NDC. If the camera parameters are defined in screen space, the transformed points are in screen space.</p>
|
||||
<p>Note that the base <code>CamerasBase</code> class makes no assumptions about the coordinate systems. All the above transforms are geometric transforms defined purely by <code>R</code>, <code>T</code> and <code>K</code>. This means that users can define cameras in any coordinate system and for any transforms. The method <code>transform_points</code> will apply <code>K</code> , <code>R</code> and <code>T</code> to the input points as a simple matrix transformation. However, if users wish to use cameras with the PyTorch3D renderer, they need to abide to PyTorch3D's coordinate system assumptions (read below).</p>
|
||||
<p>We provide instantiations of common camera types in PyTorch3D and how users can flexibly define the projection space below.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="interfacing-with-the-pytorch3d-renderer"></a><a href="#interfacing-with-the-pytorch3d-renderer" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Interfacing with the PyTorch3D Renderer</h2>
|
||||
<p>The PyTorch3D renderer for both meshes and point clouds assumes that the camera transformed points, meaning the points passed as input to the rasterizer, are in PyTorch3D's NDC space. So to get the expected rendering outcome, users need to make sure that their 3D input data and cameras abide by these PyTorch3D coordinate system assumptions. The PyTorch3D coordinate system assumes <code>+X:left</code>, <code>+Y: up</code> and <code>+Z: from us to scene</code> (right-handed) . Confusions regarding coordinate systems are common so we advise that you spend some time understanding your data and the coordinate system they live in and transform them accordingly before using the PyTorch3D renderer.</p>
|
||||
<p>Examples of cameras and how they interface with the PyTorch3D renderer can be found in our tutorials.</p>
|
||||
<h3><a class="anchor" aria-hidden="true" id="camera-types"></a><a href="#camera-types" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Camera Types</h3>
|
||||
<p>All cameras inherit from <code>CamerasBase</code> which is a base class for all cameras. PyTorch3D provides four different camera types. The <code>CamerasBase</code> defines methods that are common to all camera models:</p>
|
||||
<ul>
|
||||
<li><code>get_camera_center</code> that returns the optical center of the camera in world coordinates</li>
|
||||
<li><code>get_world_to_view_transform</code> which returns a 3D transform from world coordinates to the camera view coordinates (R, T)</li>
|
||||
<li><code>get_full_projection_transform</code> which composes the projection transform (P) with the world-to-view transform (R, T)</li>
|
||||
<li><code>get_world_to_view_transform</code> which returns a 3D transform from world coordinates to the camera view coordinates <code>(R, T)</code></li>
|
||||
<li><code>get_full_projection_transform</code> which composes the projection transform (<code>K</code>) with the world-to-view transform <code>(R, T)</code></li>
|
||||
<li><code>transform_points</code> which takes a set of input points in world coordinates and projects to NDC coordinates ranging from [-1, -1, znear] to [+1, +1, zfar].</li>
|
||||
<li><code>get_ndc_camera_transform</code> which defines the conversion to PyTorch3D's NDC space and is called when interfacing with the PyTorch3D renderer. If the camera is defined in NDC space, then the identity transform is returned. If the cameras is defined in screen space, the conversion from screen to NDC is returned. If users define their own camera in screen space, they need to think of the screen to NDC conversion. We provide examples for the <code>PerspectiveCameras</code> and <code>OrthographicCameras</code>.</li>
|
||||
<li><code>transform_points_ndc</code> which takes a set of points in world coordinates and projects them to PyTorch3D's NDC space</li>
|
||||
<li><code>transform_points_screen</code> which takes a set of input points in world coordinates and projects them to the screen coordinates ranging from [0, 0, znear] to [W-1, H-1, zfar]</li>
|
||||
</ul>
|
||||
<p>Users can easily customize their own cameras. For each new camera, users should implement the <code>get_projection_transform</code> routine that returns the mapping <code>P</code> from camera view coordinates to NDC coordinates.</p>
|
||||
<h4><a class="anchor" aria-hidden="true" id="fovperspectivecameras-fovorthographiccameras"></a><a href="#fovperspectivecameras-fovorthographiccameras" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FoVPerspectiveCameras, FoVOrthographicCameras</h4>
|
||||
<p>These two cameras follow the OpenGL convention for perspective and orthographic cameras respectively. The user provides the near <code>znear</code> and far <code>zfar</code> field which confines the view volume in the <code>Z</code> axis. The view volume in the <code>XY</code> plane is defined by field of view angle (<code>fov</code>) in the case of <code>FoVPerspectiveCameras</code> and by <code>min_x, min_y, max_x, max_y</code> in the case of <code>FoVOrthographicCameras</code>.</p>
|
||||
<p>These two cameras follow the OpenGL convention for perspective and orthographic cameras respectively. The user provides the near <code>znear</code> and far <code>zfar</code> field which confines the view volume in the <code>Z</code> axis. The view volume in the <code>XY</code> plane is defined by field of view angle (<code>fov</code>) in the case of <code>FoVPerspectiveCameras</code> and by <code>min_x, min_y, max_x, max_y</code> in the case of <code>FoVOrthographicCameras</code>.
|
||||
These cameras are by default in NDC space.</p>
|
||||
<h4><a class="anchor" aria-hidden="true" id="perspectivecameras-orthographiccameras"></a><a href="#perspectivecameras-orthographiccameras" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>PerspectiveCameras, OrthographicCameras</h4>
|
||||
<p>These two cameras follow the Multi-View Geometry convention for cameras. The user provides the focal length (<code>fx</code>, <code>fy</code>) and the principal point (<code>px</code>, <code>py</code>). For example, <code>camera = PerspectiveCameras(focal_length=((fx, fy),), principal_point=((px, py),))</code></p>
|
||||
<p>As mentioned above, the focal length and principal point are used to convert a point <code>(X, Y, Z)</code> from view coordinates to NDC coordinates, as follows</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># for perspective</span>
|
||||
<span class="hljs-attr">x_ndc</span> = fx * X / Z + px
|
||||
<span class="hljs-attr">y_ndc</span> = fy * Y / Z + py
|
||||
<span class="hljs-attr">z_ndc</span> = <span class="hljs-number">1</span> / Z
|
||||
<p>The camera projection of a 3D point <code>(X, Y, Z)</code> in view coordinates to a point <code>(x, y, z)</code> in projection space (either NDC or screen) is</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># for perspective camera</span>
|
||||
<span class="hljs-attr">x</span> = fx * X / Z + px
|
||||
<span class="hljs-attr">y</span> = fy * Y / Z + py
|
||||
<span class="hljs-attr">z</span> = <span class="hljs-number">1</span> / Z
|
||||
|
||||
<span class="hljs-comment"># for orthographic</span>
|
||||
<span class="hljs-attr">x_ndc</span> = fx * X + px
|
||||
<span class="hljs-attr">y_ndc</span> = fy * Y + py
|
||||
<span class="hljs-attr">z_ndc</span> = Z
|
||||
<span class="hljs-comment"># for orthographic camera</span>
|
||||
<span class="hljs-attr">x</span> = fx * X + px
|
||||
<span class="hljs-attr">y</span> = fy * Y + py
|
||||
<span class="hljs-attr">z</span> = Z
|
||||
</code></pre>
|
||||
<p>Commonly, users have access to the focal length (<code>fx_screen</code>, <code>fy_screen</code>) and the principal point (<code>px_screen</code>, <code>py_screen</code>) in screen space. In that case, to construct the camera the user needs to additionally provide the <code>image_size = ((image_width, image_height),)</code>. More precisely, <code>camera = PerspectiveCameras(focal_length=((fx_screen, fy_screen),), principal_point=((px_screen, py_screen),), image_size = ((image_width, image_height),))</code>. Internally, the camera parameters are converted from screen to NDC as follows:</p>
|
||||
<pre><code class="hljs"><span class="hljs-attr">fx</span> = fx_screen * <span class="hljs-number">2.0</span> / image_width
|
||||
<span class="hljs-attr">fy</span> = fy_screen * <span class="hljs-number">2.0</span> / image_height
|
||||
<p>The user can define the camera parameters in NDC or in screen space. Screen space camera parameters are common and for that case the user needs to set <code>in_ndc</code> to <code>False</code> and also provide the <code>image_size=(height, width)</code> of the screen, aka the image.</p>
|
||||
<p>The <code>get_ndc_camera_transform</code> provides the transform from screen to NDC space in PyTorch3D. Note that the screen space assumes that the principal point is provided in the space with <code>+X left</code>, <code>+Y down</code> and origin at the top left corner of the image. To convert to NDC we need to account for the scaling of the normalized space as well as the change in <code>XY</code> direction.</p>
|
||||
<p>Below are example of equivalent <code>PerspectiveCameras</code> instantiations in NDC and screen space, respectively.</p>
|
||||
<pre><code class="hljs css language-python"><span class="hljs-comment"># NDC space camera</span>
|
||||
fcl_ndc = (<span class="hljs-number">1.2</span>,)
|
||||
prp_ndc = ((<span class="hljs-number">0.2</span>, <span class="hljs-number">0.5</span>),)
|
||||
cameras_ndc = PerspectiveCameras(focal_length=fcl_ndc, principal_point=prp_ndc)
|
||||
|
||||
<span class="hljs-attr">px</span> = - (px_screen - image_width / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / image_width
|
||||
<span class="hljs-attr">py</span> = - (py_screen - image_height / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span>/ image_height
|
||||
<span class="hljs-comment"># Screen space camera</span>
|
||||
image_size = ((<span class="hljs-number">128</span>, <span class="hljs-number">256</span>),) <span class="hljs-comment"># (h, w)</span>
|
||||
fcl_screen = (<span class="hljs-number">76.2</span>,) <span class="hljs-comment"># fcl_ndc * (min(image_size) - 1) / 2</span>
|
||||
prp_screen = ((<span class="hljs-number">114.8</span>, <span class="hljs-number">31.75</span>), ) <span class="hljs-comment"># (w - 1) / 2 - px_ndc * (min(image_size) - 1) / 2, (h - 1) / 2 - py_ndc * (min(image_size) - 1) / 2</span>
|
||||
cameras_screen = PerspectiveCameras(focal_length=fcl_screen, principal_point=prp_screen, in_ndc=<span class="hljs-literal">False</span>, image_size=image_size)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer_getting_started"><span class="arrow-prev">← </span><span>Getting Started</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#camera-coordinate-systems">Camera Coordinate Systems</a></li><li><a href="#defining-cameras-in-pytorch3d">Defining Cameras in PyTorch3D</a><ul class="toc-headings"><li><a href="#camera-types">Camera Types</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
<p>The relationship between screen and NDC specifications of a camera's <code>focal_length</code> and <code>principal_point</code> is given by the following equations, where <code>s = min(image_width, image_height)</code>.
|
||||
The transformation of x and y coordinates between screen and NDC is exactly the same as for px and py.</p>
|
||||
<pre><code class="hljs">fx_ndc = fx_screen * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
fy_ndc = fy_screen * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
|
||||
px_ndc = - (<span class="hljs-name">px_screen</span> - (<span class="hljs-name">image_width</span> - <span class="hljs-number">1</span>) / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
py_ndc = - (<span class="hljs-name">py_screen</span> - (<span class="hljs-name">image_height</span> - <span class="hljs-number">1</span>) / <span class="hljs-number">2.0</span>) * <span class="hljs-number">2.0</span> / (<span class="hljs-name">s</span> - <span class="hljs-number">1</span>)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Georgia Gkioxari</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer_getting_started"><span class="arrow-prev">← </span><span>Getting Started</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#camera-coordinate-systems">Camera Coordinate Systems</a></li><li><a href="#defining-cameras-in-pytorch3d">Defining Cameras in PyTorch3D</a></li><li><a href="#interfacing-with-the-pytorch3d-renderer">Interfacing with the PyTorch3D Renderer</a><ul class="toc-headings"><li><a href="#camera-types">Camera Types</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cubify · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cubify"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cubify · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cubify"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cubify · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cubify"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cubify · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cubify"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -66,4 +66,4 @@
|
||||
<p>The <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/cubify.py">cubify operator</a> converts an 3D occupancy grid of shape <code>BxDxHxW</code>, where <code>B</code> is the batch size, into a mesh instantiated as a <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure of <code>B</code> elements. The operator replaces every occupied voxel (if its occupancy probability is greater than a user defined threshold) with a cuboid of 12 faces and 8 vertices. Shared vertices are merged, and internal faces are removed resulting in a <strong>watertight</strong> mesh.</p>
|
||||
<p>The operator provides three alignment modes {<em>topleft</em>, <em>corner</em>, <em>center</em>} which define the span of the mesh vertices with respect to the voxel grid. The alignment modes are described in the figure below for a 2D grid.</p>
|
||||
<p><img src="https://user-images.githubusercontent.com/4369065/81032959-af697380-8e46-11ea-91a8-fae89597f988.png" alt="input"></p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/batching"><span class="arrow-prev">← </span><span>Batching</span></a><a class="docs-next button" href="/docs/visualization"><span>Plotly Visualization</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/batching"><span class="arrow-prev">← </span><span>Batching</span></a><a class="docs-next button" href="/docs/visualization"><span>Plotly Visualization</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cubify · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cubify"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cubify · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cubify"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>cubify · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Cubify"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="cubify · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Cubify"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -66,4 +66,4 @@
|
||||
<p>The <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/ops/cubify.py">cubify operator</a> converts an 3D occupancy grid of shape <code>BxDxHxW</code>, where <code>B</code> is the batch size, into a mesh instantiated as a <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/structures/meshes.py">Meshes</a> data structure of <code>B</code> elements. The operator replaces every occupied voxel (if its occupancy probability is greater than a user defined threshold) with a cuboid of 12 faces and 8 vertices. Shared vertices are merged, and internal faces are removed resulting in a <strong>watertight</strong> mesh.</p>
|
||||
<p>The operator provides three alignment modes {<em>topleft</em>, <em>corner</em>, <em>center</em>} which define the span of the mesh vertices with respect to the voxel grid. The alignment modes are described in the figure below for a 2D grid.</p>
|
||||
<p><img src="https://user-images.githubusercontent.com/4369065/81032959-af697380-8e46-11ea-91a8-fae89597f988.png" alt="input"></p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/batching"><span class="arrow-prev">← </span><span>Batching</span></a><a class="docs-next button" href="/docs/visualization"><span>Plotly Visualization</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/batching"><span class="arrow-prev">← </span><span>Batching</span></a><a class="docs-next button" href="/docs/visualization"><span>Plotly Visualization</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>datasets · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Data loaders for common 3D Datasets"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="datasets · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Data loaders for common 3D Datasets"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>datasets · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Data loaders for common 3D Datasets"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="datasets · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Data loaders for common 3D Datasets"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -70,4 +70,4 @@
|
||||
<h3><a class="anchor" aria-hidden="true" id="r2n2"></a><a href="#r2n2" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>R2N2</h3>
|
||||
<p>The R2N2 dataset contains 13 categories that are a subset of the ShapeNetCore v.1 dataset. The R2N2 dataset also contains its own 24 renderings of each object and voxelized models. The R2N2 Dataset can be downloaded following the instructions <a href="http://3d-r2n2.stanford.edu/">here</a>.</p>
|
||||
<p>The PyTorch3D <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/datasets/r2n2/r2n2.py">R2N2 data loader</a> is initialized with the paths to the ShapeNet dataset, the R2N2 dataset and the splits file for R2N2. Just like <code>ShapeNetCore</code>, it can be passed to <code>torch.utils.data.DataLoader</code> with a customized collate_fn: <code>collate_batched_R2N2</code> from the <code>pytorch3d.dataset.r2n2.utils</code> module. It returns all the data that <code>ShapeNetCore</code> returns, and in addition, it returns the R2N2 renderings (24 views for each model) along with the camera calibration matrices and a voxel representation for each model. Similar to <code>ShapeNetCore</code>, it has a customized <code>render</code> function that supports rendering specified models with the PyTorch3D differentiable renderer. In addition, it supports rendering models with the same orientations as R2N2's original renderings.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/meshes_io"><span class="arrow-prev">← </span><span>Loading from file</span></a><a class="docs-next button" href="/docs/batching"><span>Batching</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/meshes_io"><span class="arrow-prev">← </span><span>Loading from file</span></a><a class="docs-next button" href="/docs/batching"><span>Batching</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>datasets · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Data loaders for common 3D Datasets"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="datasets · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Data loaders for common 3D Datasets"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>datasets · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Data loaders for common 3D Datasets"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="datasets · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Data loaders for common 3D Datasets"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -70,4 +70,4 @@
|
||||
<h3><a class="anchor" aria-hidden="true" id="r2n2"></a><a href="#r2n2" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>R2N2</h3>
|
||||
<p>The R2N2 dataset contains 13 categories that are a subset of the ShapeNetCore v.1 dataset. The R2N2 dataset also contains its own 24 renderings of each object and voxelized models. The R2N2 Dataset can be downloaded following the instructions <a href="http://3d-r2n2.stanford.edu/">here</a>.</p>
|
||||
<p>The PyTorch3D <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/datasets/r2n2/r2n2.py">R2N2 data loader</a> is initialized with the paths to the ShapeNet dataset, the R2N2 dataset and the splits file for R2N2. Just like <code>ShapeNetCore</code>, it can be passed to <code>torch.utils.data.DataLoader</code> with a customized collate_fn: <code>collate_batched_R2N2</code> from the <code>pytorch3d.dataset.r2n2.utils</code> module. It returns all the data that <code>ShapeNetCore</code> returns, and in addition, it returns the R2N2 renderings (24 views for each model) along with the camera calibration matrices and a voxel representation for each model. Similar to <code>ShapeNetCore</code>, it has a customized <code>render</code> function that supports rendering specified models with the PyTorch3D differentiable renderer. In addition, it supports rendering models with the same orientations as R2N2's original renderings.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/meshes_io"><span class="arrow-prev">← </span><span>Loading from file</span></a><a class="docs-next button" href="/docs/batching"><span>Batching</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Nikhila Ravi</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/meshes_io"><span class="arrow-prev">← </span><span>Loading from file</span></a><a class="docs-next button" href="/docs/batching"><span>Batching</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
30
docs/io.html
Normal file
30
docs/io.html
Normal file
@ -0,0 +1,30 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>io · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# File IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# File IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body class="sideNavVisible separateOnPageNav"><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="docMainWrapper wrapper"><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="file-io"></a><a href="#file-io" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>File IO</h1>
|
||||
<p>There is a flexible interface for loading and saving point clouds and meshes from different formats.</p>
|
||||
<p>The main usage is via the <code>pytorch3d.io.IO</code> object, and its methods
|
||||
<code>load_mesh</code>, <code>save_mesh</code>, <code>load_point_cloud</code> and <code>save_point_cloud</code>.</p>
|
||||
<p>For example, to load a mesh you might do</p>
|
||||
<pre><code class="hljs">from pytorch3d.io import IO
|
||||
|
||||
device=torch.device(<span class="hljs-string">"cuda:0"</span>)
|
||||
mesh = <span class="hljs-constructor">IO()</span>.load<span class="hljs-constructor">_mesh(<span class="hljs-string">"mymesh.ply"</span>, <span class="hljs-params">device</span>=<span class="hljs-params">device</span>)</span>
|
||||
</code></pre>
|
||||
<p>and to save a pointcloud you might do</p>
|
||||
<pre><code class="hljs">pcl = <span class="hljs-constructor">Pointclouds(<span class="hljs-operator">...</span>)</span>
|
||||
<span class="hljs-constructor">IO()</span>.save<span class="hljs-constructor">_point_cloud(<span class="hljs-params">pcl</span>, <span class="hljs-string">"output_pointcloud.obj"</span>)</span>
|
||||
</code></pre>
|
||||
<p>For meshes, this supports OBJ, PLY and OFF files.</p>
|
||||
<p>For pointclouds, this supports PLY files.</p>
|
||||
<p>In addition, there is experimental support for loading meshes from
|
||||
<a href="https://github.com/KhronosGroup/glTF/tree/master/specification/2.0">glTF 2 assets</a>
|
||||
stored either in a GLB container file or a glTF JSON file with embedded binary data.
|
||||
This must be enabled explicitly, as described in
|
||||
<code>pytorch3d/io/experimental_gltf_io.ply</code>.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
30
docs/io/index.html
Normal file
30
docs/io/index.html
Normal file
@ -0,0 +1,30 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>io · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# File IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# File IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body class="sideNavVisible separateOnPageNav"><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="docMainWrapper wrapper"><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="file-io"></a><a href="#file-io" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>File IO</h1>
|
||||
<p>There is a flexible interface for loading and saving point clouds and meshes from different formats.</p>
|
||||
<p>The main usage is via the <code>pytorch3d.io.IO</code> object, and its methods
|
||||
<code>load_mesh</code>, <code>save_mesh</code>, <code>load_point_cloud</code> and <code>save_point_cloud</code>.</p>
|
||||
<p>For example, to load a mesh you might do</p>
|
||||
<pre><code class="hljs">from pytorch3d.io import IO
|
||||
|
||||
device=torch.device(<span class="hljs-string">"cuda:0"</span>)
|
||||
mesh = <span class="hljs-constructor">IO()</span>.load<span class="hljs-constructor">_mesh(<span class="hljs-string">"mymesh.ply"</span>, <span class="hljs-params">device</span>=<span class="hljs-params">device</span>)</span>
|
||||
</code></pre>
|
||||
<p>and to save a pointcloud you might do</p>
|
||||
<pre><code class="hljs">pcl = <span class="hljs-constructor">Pointclouds(<span class="hljs-operator">...</span>)</span>
|
||||
<span class="hljs-constructor">IO()</span>.save<span class="hljs-constructor">_point_cloud(<span class="hljs-params">pcl</span>, <span class="hljs-string">"output_pointcloud.obj"</span>)</span>
|
||||
</code></pre>
|
||||
<p>For meshes, this supports OBJ, PLY and OFF files.</p>
|
||||
<p>For pointclouds, this supports PLY files.</p>
|
||||
<p>In addition, there is experimental support for loading meshes from
|
||||
<a href="https://github.com/KhronosGroup/glTF/tree/master/specification/2.0">glTF 2 assets</a>
|
||||
stored either in a GLB container file or a glTF JSON file with embedded binary data.
|
||||
This must be enabled explicitly, as described in
|
||||
<code>pytorch3d/io/experimental_gltf_io.ply</code>.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>meshes_io · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Meshes and IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="meshes_io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Meshes and IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>meshes_io · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Meshes and IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="meshes_io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Meshes and IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -64,12 +64,12 @@
|
||||
});
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="meshes-and-io"></a><a href="#meshes-and-io" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Meshes and IO</h1>
|
||||
<p>The Meshes object represents a batch of triangulated meshes, and is central to
|
||||
much of the functionality of pytorch3d. There is no insistence that each mesh in
|
||||
much of the functionality of PyTorch3D. There is no insistence that each mesh in
|
||||
the batch has the same number of vertices or faces. When available, it can store
|
||||
other data which pertains to the mesh, for example face normals, face areas
|
||||
and textures.</p>
|
||||
<p>Two common file formats for storing single meshes are ".obj" and ".ply" files,
|
||||
and pytorch3d has functions for reading these.</p>
|
||||
and PyTorch3D has functions for reading these.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="obj"></a><a href="#obj" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>OBJ</h2>
|
||||
<p>Obj files have a standard way to store extra information about a mesh. Given an
|
||||
obj file, it can be read with</p>
|
||||
@ -105,7 +105,7 @@ entire mesh e.g.</p>
|
||||
</code></pre>
|
||||
<p>The <code>load_objs_as_meshes</code> function provides this procedure.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="ply"></a><a href="#ply" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>PLY</h2>
|
||||
<p>Ply files are flexible in the way they store additional information, pytorch3d
|
||||
<p>Ply files are flexible in the way they store additional information. PyTorch3D
|
||||
provides a function just to read the vertices and faces from a ply file.
|
||||
The call</p>
|
||||
<pre><code class="hljs"> verts, faces = load<span class="hljs-constructor">_ply(<span class="hljs-params">filename</span>)</span>
|
||||
@ -116,4 +116,4 @@ are not triangles will be split into triangles. A Meshes object containing a
|
||||
single mesh can be created from this data using</p>
|
||||
<pre><code class="hljs"> meshes = <span class="hljs-constructor">Meshes(<span class="hljs-params">verts</span>=[<span class="hljs-params">verts</span>], <span class="hljs-params">faces</span>=[<span class="hljs-params">faces</span>])</span>
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/why_pytorch3d"><span class="arrow-prev">← </span><span class="function-name-prevnext">Why PyTorch3D</span></a><a class="docs-next button" href="/docs/datasets"><span>Data loaders</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#obj">OBJ</a></li><li><a href="#ply">PLY</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/why_pytorch3d"><span class="arrow-prev">← </span><span class="function-name-prevnext">Why PyTorch3D</span></a><a class="docs-next button" href="/docs/datasets"><span>Data loaders</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#obj">OBJ</a></li><li><a href="#ply">PLY</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>meshes_io · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Meshes and IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="meshes_io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Meshes and IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>meshes_io · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Meshes and IO"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="meshes_io · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Meshes and IO"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -64,12 +64,12 @@
|
||||
});
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="meshes-and-io"></a><a href="#meshes-and-io" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Meshes and IO</h1>
|
||||
<p>The Meshes object represents a batch of triangulated meshes, and is central to
|
||||
much of the functionality of pytorch3d. There is no insistence that each mesh in
|
||||
much of the functionality of PyTorch3D. There is no insistence that each mesh in
|
||||
the batch has the same number of vertices or faces. When available, it can store
|
||||
other data which pertains to the mesh, for example face normals, face areas
|
||||
and textures.</p>
|
||||
<p>Two common file formats for storing single meshes are ".obj" and ".ply" files,
|
||||
and pytorch3d has functions for reading these.</p>
|
||||
and PyTorch3D has functions for reading these.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="obj"></a><a href="#obj" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>OBJ</h2>
|
||||
<p>Obj files have a standard way to store extra information about a mesh. Given an
|
||||
obj file, it can be read with</p>
|
||||
@ -105,7 +105,7 @@ entire mesh e.g.</p>
|
||||
</code></pre>
|
||||
<p>The <code>load_objs_as_meshes</code> function provides this procedure.</p>
|
||||
<h2><a class="anchor" aria-hidden="true" id="ply"></a><a href="#ply" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>PLY</h2>
|
||||
<p>Ply files are flexible in the way they store additional information, pytorch3d
|
||||
<p>Ply files are flexible in the way they store additional information. PyTorch3D
|
||||
provides a function just to read the vertices and faces from a ply file.
|
||||
The call</p>
|
||||
<pre><code class="hljs"> verts, faces = load<span class="hljs-constructor">_ply(<span class="hljs-params">filename</span>)</span>
|
||||
@ -116,4 +116,4 @@ are not triangles will be split into triangles. A Meshes object containing a
|
||||
single mesh can be created from this data using</p>
|
||||
<pre><code class="hljs"> meshes = <span class="hljs-constructor">Meshes(<span class="hljs-params">verts</span>=[<span class="hljs-params">verts</span>], <span class="hljs-params">faces</span>=[<span class="hljs-params">faces</span>])</span>
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/why_pytorch3d"><span class="arrow-prev">← </span><span class="function-name-prevnext">Why PyTorch3D</span></a><a class="docs-next button" href="/docs/datasets"><span>Data loaders</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#obj">OBJ</a></li><li><a href="#ply">PLY</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/why_pytorch3d"><span class="arrow-prev">← </span><span class="function-name-prevnext">Why PyTorch3D</span></a><a class="docs-next button" href="/docs/datasets"><span>Data loaders</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#obj">OBJ</a></li><li><a href="#ply">PLY</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Rendering Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Rendering Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Rendering Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Rendering Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -109,4 +109,4 @@ total_memory = memory_forward_pass + memory_backward_pass
|
||||
<p><a id="6">[6]</a> Yifan et al, 'Differentiable Surface Splatting for Point-based Geometry Processing', SIGGRAPH Asia 2019</p>
|
||||
<p><a id="7">[7]</a> Loubet et al, 'Reparameterizing Discontinuous Integrands for Differentiable Rendering', SIGGRAPH Asia 2019</p>
|
||||
<p><a id="8">[8]</a> Chen et al, 'Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer', NeurIPS 2019</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/visualization"><span class="arrow-prev">← </span><span>Plotly Visualization</span></a><a class="docs-next button" href="/docs/renderer_getting_started"><span>Getting Started</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#uget-startedu"><u>Get started</u></a></li><li><a href="#utech-reportu"><u>Tech Report</u></a><ul class="toc-headings"><li><a href="#references">References</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/visualization"><span class="arrow-prev">← </span><span>Plotly Visualization</span></a><a class="docs-next button" href="/docs/renderer_getting_started"><span>Getting Started</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#uget-startedu"><u>Get started</u></a></li><li><a href="#utech-reportu"><u>Tech Report</u></a><ul class="toc-headings"><li><a href="#references">References</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Rendering Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Rendering Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Rendering Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Rendering Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -109,4 +109,4 @@ total_memory = memory_forward_pass + memory_backward_pass
|
||||
<p><a id="6">[6]</a> Yifan et al, 'Differentiable Surface Splatting for Point-based Geometry Processing', SIGGRAPH Asia 2019</p>
|
||||
<p><a id="7">[7]</a> Loubet et al, 'Reparameterizing Discontinuous Integrands for Differentiable Rendering', SIGGRAPH Asia 2019</p>
|
||||
<p><a id="8">[8]</a> Chen et al, 'Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer', NeurIPS 2019</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/visualization"><span class="arrow-prev">← </span><span>Plotly Visualization</span></a><a class="docs-next button" href="/docs/renderer_getting_started"><span>Getting Started</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#uget-startedu"><u>Get started</u></a></li><li><a href="#utech-reportu"><u>Tech Report</u></a><ul class="toc-headings"><li><a href="#references">References</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/visualization"><span class="arrow-prev">← </span><span>Plotly Visualization</span></a><a class="docs-next button" href="/docs/renderer_getting_started"><span>Getting Started</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#uget-startedu"><u>Get started</u></a></li><li><a href="#utech-reportu"><u>Tech Report</u></a><ul class="toc-headings"><li><a href="#references">References</a></li></ul></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer_getting_started · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Getting Started With Renderer"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer_getting_started · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Getting Started With Renderer"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer_getting_started · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Getting Started With Renderer"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer_getting_started · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Getting Started With Renderer"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -94,6 +94,14 @@ giving the barycentric coordinates in NDC units of the nearest faces at each pix
|
||||
</ul>
|
||||
<p><img align="center" src="assets/opengl_coordframes.png" width="300"></p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="rasterizing-non-square-images"></a><a href="#rasterizing-non-square-images" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rasterizing Non Square Images</h3>
|
||||
<p>To rasterize an image where H != W, you can specify the <code>image_size</code> in the <code>RasterizationSettings</code> as a tuple of (H, W).</p>
|
||||
<p>The aspect ratio needs special consideration. There are two aspect ratios to be aware of:
|
||||
- the aspect ratio of each pixel
|
||||
- the aspect ratio of the output image
|
||||
In the cameras e.g. <code>FoVPerspectiveCameras</code>, the <code>aspect_ratio</code> argument can be used to set the pixel aspect ratio. In the rasterizer, we assume square pixels, but variable image aspect ratio (i.e rectangle images).</p>
|
||||
<p>In most cases you will want to set the camera aspect ratio to 1.0 (i.e. square pixels) and only vary the <code>image_size</code> in the <code>RasterizationSettings</code>(i.e. the output image dimensions in pixels).</p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="the-pulsar-backend"></a><a href="#the-pulsar-backend" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The pulsar backend</h3>
|
||||
<p>Since v0.3, <a href="https://arxiv.org/abs/2004.07484">pulsar</a> can be used as a backend for point-rendering. It has a focus on efficiency, which comes with pros and cons: it is highly optimized and all rendering stages are integrated in the CUDA kernels. This leads to significantly higher speed and better scaling behavior. We use it at Facebook Reality Labs to render and optimize scenes with millions of spheres in resolutions up to 4K. You can find a runtime comparison plot below (settings: <code>bin_size=None</code>, <code>points_per_pixel=5</code>, <code>image_size=1024</code>, <code>radius=1e-2</code>, <code>composite_params.radius=1e-4</code>; benchmarked on an RTX 2070 GPU).</p>
|
||||
<p><img align="center" src="assets/pulsar_bm.png" width="300"></p>
|
||||
@ -104,9 +112,10 @@ giving the barycentric coordinates in NDC units of the nearest faces at each pix
|
||||
<ol>
|
||||
<li><strong>Vertex Textures</strong>: D dimensional textures for each vertex (for example an RGB color) which can be interpolated across the face. This can be represented as an <code>(N, V, D)</code> tensor. This is a fairly simple representation though and cannot model complex textures if the mesh faces are large.</li>
|
||||
<li><strong>UV Textures</strong>: vertex UV coordinates and <strong>one</strong> texture map for the whole mesh. For a point on a face with given barycentric coordinates, the face color can be computed by interpolating the vertex uv coordinates and then sampling from the texture map. This representation requires two tensors (UVs: <code>(N, V, 2), Texture map:</code>(N, H, W, 3)`), and is limited to only support one texture map per mesh.</li>
|
||||
<li><strong>Face Textures</strong>: In more complex cases such as ShapeNet meshes, there are multiple texture maps per mesh and some faces have texture while other do not. For these cases, a more flexible representation is a texture atlas, where each face is represented as an <code>(RxR)</code> texture map where R is the texture resolution. For a given point on the face, the texture value can be sampled from the per face texture map using the barycentric coordinates of the point. This representation requires one tensor of shape <code>(N, F, R, R, 3)</code>. This texturing method is inspired by the SoftRasterizer implementation. For more details refer to the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/io/mtl_io.py#L123"><code>make_material_atlas</code></a> and <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/textures.py#L452"><code>sample_textures</code></a> functions.</li>
|
||||
<li><strong>Face Textures</strong>: In more complex cases such as ShapeNet meshes, there are multiple texture maps per mesh and some faces have texture while other do not. For these cases, a more flexible representation is a texture atlas, where each face is represented as an <code>(RxR)</code> texture map where R is the texture resolution. For a given point on the face, the texture value can be sampled from the per face texture map using the barycentric coordinates of the point. This representation requires one tensor of shape <code>(N, F, R, R, 3)</code>. This texturing method is inspired by the SoftRasterizer implementation. For more details refer to the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/io/mtl_io.py#L123"><code>make_material_atlas</code></a> and <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/textures.py#L452"><code>sample_textures</code></a> functions. <strong>NOTE:</strong>: The <code>TexturesAtlas</code> texture sampling is only differentiable with respect to the texture atlas but not differentiable with respect to the barycentric coordinates.</li>
|
||||
</ol>
|
||||
<p><img src="assets/texturing.jpg" width="1000"></p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="a-simple-renderer"></a><a href="#a-simple-renderer" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A simple renderer</h3>
|
||||
<p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong>. Create a renderer in a few simple steps:</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># Imports</span>
|
||||
@ -129,13 +138,14 @@ raster_settings = RasterizationSettings(
|
||||
<span class="hljs-attribute">faces_per_pixel</span>=1,
|
||||
)
|
||||
|
||||
<span class="hljs-comment"># Create a phong renderer by composing a rasterizer and a shader. Here we can use a predefined</span>
|
||||
<span class="hljs-comment"># Create a Phong renderer by composing a rasterizer and a shader. Here we can use a predefined</span>
|
||||
<span class="hljs-comment"># PhongShader, passing in the device on which to initialize the default parameters</span>
|
||||
renderer = MeshRenderer(
|
||||
<span class="hljs-attribute">rasterizer</span>=MeshRasterizer(cameras=cameras, <span class="hljs-attribute">raster_settings</span>=raster_settings),
|
||||
<span class="hljs-attribute">shader</span>=HardPhongShader(device=device, <span class="hljs-attribute">cameras</span>=cameras)
|
||||
)
|
||||
</code></pre>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="a-custom-shader"></a><a href="#a-custom-shader" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A custom shader</h3>
|
||||
<p>Shaders are the most flexible part of the PyTorch3D rendering API. We have created some examples of shaders in <code>shaders.py</code> but this is a non exhaustive set.</p>
|
||||
<p>A shader can incorporate several steps:</p>
|
||||
@ -158,4 +168,4 @@ renderer = MeshRenderer(
|
||||
<tr><td>SoftSilhouetteShader</td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center">✔️</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer"><span class="arrow-prev">← </span><span>Overview</span></a><a class="docs-next button" href="/docs/cameras"><span>Cameras</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer"><span class="arrow-prev">← </span><span>Overview</span></a><a class="docs-next button" href="/docs/cameras"><span>Cameras</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer_getting_started · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Getting Started With Renderer"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer_getting_started · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Getting Started With Renderer"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>renderer_getting_started · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Getting Started With Renderer"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="renderer_getting_started · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Getting Started With Renderer"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -94,6 +94,14 @@ giving the barycentric coordinates in NDC units of the nearest faces at each pix
|
||||
</ul>
|
||||
<p><img align="center" src="assets/opengl_coordframes.png" width="300"></p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="rasterizing-non-square-images"></a><a href="#rasterizing-non-square-images" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rasterizing Non Square Images</h3>
|
||||
<p>To rasterize an image where H != W, you can specify the <code>image_size</code> in the <code>RasterizationSettings</code> as a tuple of (H, W).</p>
|
||||
<p>The aspect ratio needs special consideration. There are two aspect ratios to be aware of:
|
||||
- the aspect ratio of each pixel
|
||||
- the aspect ratio of the output image
|
||||
In the cameras e.g. <code>FoVPerspectiveCameras</code>, the <code>aspect_ratio</code> argument can be used to set the pixel aspect ratio. In the rasterizer, we assume square pixels, but variable image aspect ratio (i.e rectangle images).</p>
|
||||
<p>In most cases you will want to set the camera aspect ratio to 1.0 (i.e. square pixels) and only vary the <code>image_size</code> in the <code>RasterizationSettings</code>(i.e. the output image dimensions in pixels).</p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="the-pulsar-backend"></a><a href="#the-pulsar-backend" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The pulsar backend</h3>
|
||||
<p>Since v0.3, <a href="https://arxiv.org/abs/2004.07484">pulsar</a> can be used as a backend for point-rendering. It has a focus on efficiency, which comes with pros and cons: it is highly optimized and all rendering stages are integrated in the CUDA kernels. This leads to significantly higher speed and better scaling behavior. We use it at Facebook Reality Labs to render and optimize scenes with millions of spheres in resolutions up to 4K. You can find a runtime comparison plot below (settings: <code>bin_size=None</code>, <code>points_per_pixel=5</code>, <code>image_size=1024</code>, <code>radius=1e-2</code>, <code>composite_params.radius=1e-4</code>; benchmarked on an RTX 2070 GPU).</p>
|
||||
<p><img align="center" src="assets/pulsar_bm.png" width="300"></p>
|
||||
@ -104,9 +112,10 @@ giving the barycentric coordinates in NDC units of the nearest faces at each pix
|
||||
<ol>
|
||||
<li><strong>Vertex Textures</strong>: D dimensional textures for each vertex (for example an RGB color) which can be interpolated across the face. This can be represented as an <code>(N, V, D)</code> tensor. This is a fairly simple representation though and cannot model complex textures if the mesh faces are large.</li>
|
||||
<li><strong>UV Textures</strong>: vertex UV coordinates and <strong>one</strong> texture map for the whole mesh. For a point on a face with given barycentric coordinates, the face color can be computed by interpolating the vertex uv coordinates and then sampling from the texture map. This representation requires two tensors (UVs: <code>(N, V, 2), Texture map:</code>(N, H, W, 3)`), and is limited to only support one texture map per mesh.</li>
|
||||
<li><strong>Face Textures</strong>: In more complex cases such as ShapeNet meshes, there are multiple texture maps per mesh and some faces have texture while other do not. For these cases, a more flexible representation is a texture atlas, where each face is represented as an <code>(RxR)</code> texture map where R is the texture resolution. For a given point on the face, the texture value can be sampled from the per face texture map using the barycentric coordinates of the point. This representation requires one tensor of shape <code>(N, F, R, R, 3)</code>. This texturing method is inspired by the SoftRasterizer implementation. For more details refer to the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/io/mtl_io.py#L123"><code>make_material_atlas</code></a> and <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/textures.py#L452"><code>sample_textures</code></a> functions.</li>
|
||||
<li><strong>Face Textures</strong>: In more complex cases such as ShapeNet meshes, there are multiple texture maps per mesh and some faces have texture while other do not. For these cases, a more flexible representation is a texture atlas, where each face is represented as an <code>(RxR)</code> texture map where R is the texture resolution. For a given point on the face, the texture value can be sampled from the per face texture map using the barycentric coordinates of the point. This representation requires one tensor of shape <code>(N, F, R, R, 3)</code>. This texturing method is inspired by the SoftRasterizer implementation. For more details refer to the <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/io/mtl_io.py#L123"><code>make_material_atlas</code></a> and <a href="https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/textures.py#L452"><code>sample_textures</code></a> functions. <strong>NOTE:</strong>: The <code>TexturesAtlas</code> texture sampling is only differentiable with respect to the texture atlas but not differentiable with respect to the barycentric coordinates.</li>
|
||||
</ol>
|
||||
<p><img src="assets/texturing.jpg" width="1000"></p>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="a-simple-renderer"></a><a href="#a-simple-renderer" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A simple renderer</h3>
|
||||
<p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong>. Create a renderer in a few simple steps:</p>
|
||||
<pre><code class="hljs"><span class="hljs-comment"># Imports</span>
|
||||
@ -129,13 +138,14 @@ raster_settings = RasterizationSettings(
|
||||
<span class="hljs-attribute">faces_per_pixel</span>=1,
|
||||
)
|
||||
|
||||
<span class="hljs-comment"># Create a phong renderer by composing a rasterizer and a shader. Here we can use a predefined</span>
|
||||
<span class="hljs-comment"># Create a Phong renderer by composing a rasterizer and a shader. Here we can use a predefined</span>
|
||||
<span class="hljs-comment"># PhongShader, passing in the device on which to initialize the default parameters</span>
|
||||
renderer = MeshRenderer(
|
||||
<span class="hljs-attribute">rasterizer</span>=MeshRasterizer(cameras=cameras, <span class="hljs-attribute">raster_settings</span>=raster_settings),
|
||||
<span class="hljs-attribute">shader</span>=HardPhongShader(device=device, <span class="hljs-attribute">cameras</span>=cameras)
|
||||
)
|
||||
</code></pre>
|
||||
<hr>
|
||||
<h3><a class="anchor" aria-hidden="true" id="a-custom-shader"></a><a href="#a-custom-shader" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A custom shader</h3>
|
||||
<p>Shaders are the most flexible part of the PyTorch3D rendering API. We have created some examples of shaders in <code>shaders.py</code> but this is a non exhaustive set.</p>
|
||||
<p>A shader can incorporate several steps:</p>
|
||||
@ -158,4 +168,4 @@ renderer = MeshRenderer(
|
||||
<tr><td>SoftSilhouetteShader</td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center"></td><td style="text-align:center">✔️</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Christoph Lassner</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer"><span class="arrow-prev">← </span><span>Overview</span></a><a class="docs-next button" href="/docs/cameras"><span>Cameras</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Jeremy Reizenstein</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/renderer"><span class="arrow-prev">← </span><span>Overview</span></a><a class="docs-next button" href="/docs/cameras"><span>Cameras</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>visualization · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="visualization · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>visualization · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="visualization · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -75,4 +75,4 @@
|
||||
<pre><code class="hljs"><span class="hljs-attribute">fig</span> = ...
|
||||
fig.write_image(<span class="hljs-string">"image_name.png"</span>)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Amitav Baruah</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/cubify"><span class="arrow-prev">← </span><span>Cubify</span></a><a class="docs-next button" href="/docs/renderer"><span>Overview</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Amitav Baruah</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/cubify"><span class="arrow-prev">← </span><span>Cubify</span></a><a class="docs-next button" href="/docs/renderer"><span>Overview</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>visualization · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="visualization · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>visualization · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Overview"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="visualization · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Overview"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -75,4 +75,4 @@
|
||||
<pre><code class="hljs"><span class="hljs-attribute">fig</span> = ...
|
||||
fig.write_image(<span class="hljs-string">"image_name.png"</span>)
|
||||
</code></pre>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Amitav Baruah</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/cubify"><span class="arrow-prev">← </span><span>Cubify</span></a><a class="docs-next button" href="/docs/renderer"><span>Overview</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Amitav Baruah</em></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/cubify"><span class="arrow-prev">← </span><span>Cubify</span></a><a class="docs-next button" href="/docs/renderer"><span>Overview</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>why_pytorch3d · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Why PyTorch3D"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="why_pytorch3d · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Why PyTorch3D"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>why_pytorch3d · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Why PyTorch3D"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="why_pytorch3d · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Why PyTorch3D"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -65,4 +65,4 @@
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="why-pytorch3d"></a><a href="#why-pytorch3d" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why PyTorch3D</h1>
|
||||
<p>Our goal with PyTorch3D is to help accelerate research at the intersection of deep learning and 3D. 3D data is more complex than 2D images and while working on projects such as <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a> and <a href="https://github.com/facebookresearch/c3dpo_nrsfm">C3DPO</a>, we encountered several challenges including 3D data representation, batching, and speed. We have developed many useful operators and abstractions for working on 3D deep learning and want to share this with the community to drive novel research in this area.</p>
|
||||
<p>In PyTorch3D we have included efficient 3D operators, heterogeneous batching capabilities, and a modular differentiable rendering API, to equip researchers in this field with a much needed toolkit to implement cutting-edge research with complex 3D inputs.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Patrick Labatut</em></div><div class="docs-prevnext"><a class="docs-next button" href="/docs/meshes_io"><span>Loading from file</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Patrick Labatut</em></div><div class="docs-prevnext"><a class="docs-next button" href="/docs/meshes_io"><span>Loading from file</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>why_pytorch3d · PyTorch3D</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Why PyTorch3D"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="why_pytorch3d · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Why PyTorch3D"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>why_pytorch3d · PyTorch3D</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="# Why PyTorch3D"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="why_pytorch3d · PyTorch3D"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="# Why PyTorch3D"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -65,4 +65,4 @@
|
||||
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"></header><article><div><span><h1><a class="anchor" aria-hidden="true" id="why-pytorch3d"></a><a href="#why-pytorch3d" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why PyTorch3D</h1>
|
||||
<p>Our goal with PyTorch3D is to help accelerate research at the intersection of deep learning and 3D. 3D data is more complex than 2D images and while working on projects such as <a href="https://github.com/facebookresearch/meshrcnn">Mesh R-CNN</a> and <a href="https://github.com/facebookresearch/c3dpo_nrsfm">C3DPO</a>, we encountered several challenges including 3D data representation, batching, and speed. We have developed many useful operators and abstractions for working on 3D deep learning and want to share this with the community to drive novel research in this area.</p>
|
||||
<p>In PyTorch3D we have included efficient 3D operators, heterogeneous batching capabilities, and a modular differentiable rendering API, to equip researchers in this field with a much needed toolkit to implement cutting-edge research with complex 3D inputs.</p>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Patrick Labatut</em></div><div class="docs-prevnext"><a class="docs-next button" href="/docs/meshes_io"><span>Loading from file</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></article></div><div class="docLastUpdate"><em>Last updated by Patrick Labatut</em></div><div class="docs-prevnext"><a class="docs-next button" href="/docs/meshes_io"><span>Loading from file</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -12,4 +12,4 @@
|
||||
</span></div></h2><div><span><p>Ask questions about the documentation and project</p>
|
||||
</span></div></div></div><div class="blockElement threeByGridBlock"><div class="blockContent"><h2><div><span><p>Stay up to date</p>
|
||||
</span></div></h2><div><span><p>Find out what's new with this project</p>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -12,4 +12,4 @@
|
||||
</span></div></h2><div><span><p>Ask questions about the documentation and project</p>
|
||||
</span></div></div></div><div class="blockElement threeByGridBlock"><div class="blockContent"><h2><div><span><p>Stay up to date</p>
|
||||
</span></div></h2><div><span><p>Find out what's new with this project</p>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -28,4 +28,4 @@ sample_sphere = sample_points_from_meshes(sphere_mesh, <span class="hljs-number"
|
||||
sample_test = sample_points_from_meshes(test_mesh, <span class="hljs-number">5000</span>)
|
||||
loss_chamfer, _ = chamfer_distance(sample_sphere, sample_test)
|
||||
</code></pre>
|
||||
</span></div></li></ol></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></li></ol></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -6,4 +6,4 @@
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -6,4 +6,4 @@
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -41,7 +41,7 @@
|
||||
"Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \\in SO(3); T \\in \\mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exponential_map`) of the axis-angle representation of the rotation `log_R_absolute`.\n",
|
||||
"In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \\in SO(3); T \\in \\mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exp_map`) of the axis-angle representation of the rotation `log_R_absolute`.\n",
|
||||
"\n",
|
||||
"Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \\in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \\vec{0})$.\n"
|
||||
]
|
||||
@ -63,7 +63,7 @@
|
||||
"id": "WAHR1LMJmP-h"
|
||||
},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -80,19 +80,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -116,7 +122,7 @@
|
||||
"# imports\n",
|
||||
"import torch\n",
|
||||
"from pytorch3d.transforms.so3 import (\n",
|
||||
" so3_exponential_map,\n",
|
||||
" so3_exp_map,\n",
|
||||
" so3_relative_angle,\n",
|
||||
")\n",
|
||||
"from pytorch3d.renderer.cameras import (\n",
|
||||
@ -322,7 +328,7 @@
|
||||
"\n",
|
||||
"As mentioned earlier, `log_R_absolute` is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix `R_absolute` that corresponds to `log_R_absolute` with:\n",
|
||||
"\n",
|
||||
"`R_absolute = so3_exponential_map(log_R_absolute)`\n"
|
||||
"`R_absolute = so3_exp_map(log_R_absolute)`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -372,7 +378,7 @@
|
||||
" # compute the absolute camera rotations as \n",
|
||||
" # an exponential map of the logarithms (=axis-angles)\n",
|
||||
" # of the absolute rotations\n",
|
||||
" R_absolute = so3_exponential_map(log_R_absolute * camera_mask)\n",
|
||||
" R_absolute = so3_exp_map(log_R_absolute * camera_mask)\n",
|
||||
"\n",
|
||||
" # get the current absolute cameras\n",
|
||||
" cameras_absolute = SfMPerspectiveCameras(\n",
|
||||
@ -381,7 +387,7 @@
|
||||
" device = device,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # compute the relative cameras as a compositon of the absolute cameras\n",
|
||||
" # compute the relative cameras as a composition of the absolute cameras\n",
|
||||
" cameras_relative_composed = \\\n",
|
||||
" get_relative_camera(cameras_absolute, relative_edges)\n",
|
||||
"\n",
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -28,31 +28,37 @@
|
||||
# Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:
|
||||
# 
|
||||
#
|
||||
# In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exponential_map`) of the axis-angle representation of the rotation `log_R_absolute`.
|
||||
# In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the `SfMPerspectiveCameras` class initialized with the corresponding rotation and translation matrices `R_absolute` and `T_absolute` that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that `R_absolute` is a valid rotation matrix, we represent it using an exponential map (implemented with `so3_exp_map`) of the axis-angle representation of the rotation `log_R_absolute`.
|
||||
#
|
||||
# Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \vec{0})$.
|
||||
#
|
||||
|
||||
# ## 0. Install and Import Modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -65,7 +71,7 @@ else:
|
||||
# imports
|
||||
import torch
|
||||
from pytorch3d.transforms.so3 import (
|
||||
so3_exponential_map,
|
||||
so3_exp_map,
|
||||
so3_relative_angle,
|
||||
)
|
||||
from pytorch3d.renderer.cameras import (
|
||||
@ -197,7 +203,7 @@ def get_relative_camera(cams, edges):
|
||||
#
|
||||
# As mentioned earlier, `log_R_absolute` is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix `R_absolute` that corresponds to `log_R_absolute` with:
|
||||
#
|
||||
# `R_absolute = so3_exponential_map(log_R_absolute)`
|
||||
# `R_absolute = so3_exp_map(log_R_absolute)`
|
||||
#
|
||||
|
||||
# In[ ]:
|
||||
@ -236,7 +242,7 @@ for it in range(n_iter):
|
||||
# compute the absolute camera rotations as
|
||||
# an exponential map of the logarithms (=axis-angles)
|
||||
# of the absolute rotations
|
||||
R_absolute = so3_exponential_map(log_R_absolute * camera_mask)
|
||||
R_absolute = so3_exp_map(log_R_absolute * camera_mask)
|
||||
|
||||
# get the current absolute cameras
|
||||
cameras_absolute = SfMPerspectiveCameras(
|
||||
@ -245,7 +251,7 @@ for it in range(n_iter):
|
||||
device = device,
|
||||
)
|
||||
|
||||
# compute the relative cameras as a compositon of the absolute cameras
|
||||
# compute the relative cameras as a composition of the absolute cameras
|
||||
cameras_relative_composed = get_relative_camera(cameras_absolute, relative_edges)
|
||||
|
||||
# compare the composed cameras with the ground truth relative cameras
|
||||
|
||||
@ -50,7 +50,7 @@
|
||||
"id": "qkX7DiM6rmeM"
|
||||
},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -67,19 +67,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -217,9 +223,9 @@
|
||||
"source": [
|
||||
"### Create a renderer\n",
|
||||
"\n",
|
||||
"A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest. \n",
|
||||
"A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest. \n",
|
||||
"\n",
|
||||
"For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **phong shading** and use this for visualizing the outputs. "
|
||||
"For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **Phong shading** and use this for visualizing the outputs. "
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -260,7 +266,7 @@
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# We will also create a phong renderer. This is simpler and only needs to render one face per pixel.\n",
|
||||
"# We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.\n",
|
||||
"raster_settings = RasterizationSettings(\n",
|
||||
" image_size=256, \n",
|
||||
" blur_radius=0.0, \n",
|
||||
@ -316,15 +322,15 @@
|
||||
"R, T = look_at_view_transform(distance, elevation, azimuth, device=device)\n",
|
||||
"\n",
|
||||
"# Render the teapot providing the values of R and T. \n",
|
||||
"silhouete = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
|
||||
"silhouette = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
|
||||
"image_ref = phong_renderer(meshes_world=teapot_mesh, R=R, T=T)\n",
|
||||
"\n",
|
||||
"silhouete = silhouete.cpu().numpy()\n",
|
||||
"silhouette = silhouette.cpu().numpy()\n",
|
||||
"image_ref = image_ref.cpu().numpy()\n",
|
||||
"\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.subplot(1, 2, 1)\n",
|
||||
"plt.imshow(silhouete.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image\n",
|
||||
"plt.imshow(silhouette.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image\n",
|
||||
"plt.grid(False)\n",
|
||||
"plt.subplot(1, 2, 2)\n",
|
||||
"plt.imshow(image_ref.squeeze())\n",
|
||||
@ -371,7 +377,7 @@
|
||||
" def forward(self):\n",
|
||||
" \n",
|
||||
" # Render the image using the updated camera position. Based on the new position of the \n",
|
||||
" # camer we calculate the rotation and translation matrices\n",
|
||||
" # camera we calculate the rotation and translation matrices\n",
|
||||
" R = look_at_rotation(self.camera_position[None, :], device=self.device) # (1, 3, 3)\n",
|
||||
" T = -torch.bmm(R.transpose(1, 2), self.camera_position[None, :, None])[:, :, 0] # (1, 3)\n",
|
||||
" \n",
|
||||
@ -514,7 +520,6 @@
|
||||
" plt.figure()\n",
|
||||
" plt.imshow(image[..., :3])\n",
|
||||
" plt.title(\"iter: %d, loss: %0.2f\" % (i, loss.data))\n",
|
||||
" plt.grid(\"off\")\n",
|
||||
" plt.axis(\"off\")\n",
|
||||
" \n",
|
||||
"writer.close()"
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -22,24 +22,30 @@
|
||||
|
||||
# ## 0. Install and import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -121,9 +127,9 @@ teapot_mesh = Meshes(
|
||||
|
||||
# ### Create a renderer
|
||||
#
|
||||
# A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.
|
||||
# A **renderer** in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
|
||||
#
|
||||
# For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **phong shading** and use this for visualizing the outputs.
|
||||
# For optimizing the camera position we will use a renderer which produces a **silhouette** of the object only and does not apply any **lighting** or **shading**. We will also initialize another renderer which applies full **Phong shading** and use this for visualizing the outputs.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -156,7 +162,7 @@ silhouette_renderer = MeshRenderer(
|
||||
)
|
||||
|
||||
|
||||
# We will also create a phong renderer. This is simpler and only needs to render one face per pixel.
|
||||
# We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.
|
||||
raster_settings = RasterizationSettings(
|
||||
image_size=256,
|
||||
blur_radius=0.0,
|
||||
@ -193,15 +199,15 @@ azimuth = 0.0 # No rotation so the camera is positioned on the +Z axis.
|
||||
R, T = look_at_view_transform(distance, elevation, azimuth, device=device)
|
||||
|
||||
# Render the teapot providing the values of R and T.
|
||||
silhouete = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)
|
||||
silhouette = silhouette_renderer(meshes_world=teapot_mesh, R=R, T=T)
|
||||
image_ref = phong_renderer(meshes_world=teapot_mesh, R=R, T=T)
|
||||
|
||||
silhouete = silhouete.cpu().numpy()
|
||||
silhouette = silhouette.cpu().numpy()
|
||||
image_ref = image_ref.cpu().numpy()
|
||||
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.subplot(1, 2, 1)
|
||||
plt.imshow(silhouete.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image
|
||||
plt.imshow(silhouette.squeeze()[..., 3]) # only plot the alpha channel of the RGBA image
|
||||
plt.grid(False)
|
||||
plt.subplot(1, 2, 2)
|
||||
plt.imshow(image_ref.squeeze())
|
||||
@ -233,7 +239,7 @@ class Model(nn.Module):
|
||||
def forward(self):
|
||||
|
||||
# Render the image using the updated camera position. Based on the new position of the
|
||||
# camer we calculate the rotation and translation matrices
|
||||
# camera we calculate the rotation and translation matrices
|
||||
R = look_at_rotation(self.camera_position[None, :], device=self.device) # (1, 3, 3)
|
||||
T = -torch.bmm(R.transpose(1, 2), self.camera_position[None, :, None])[:, :, 0] # (1, 3)
|
||||
|
||||
@ -313,7 +319,6 @@ for i in loop:
|
||||
plt.figure()
|
||||
plt.imshow(image[..., :3])
|
||||
plt.title("iter: %d, loss: %0.2f" % (i, loss.data))
|
||||
plt.grid("off")
|
||||
plt.axis("off")
|
||||
|
||||
writer.close()
|
||||
|
||||
@ -33,7 +33,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -42,19 +42,25 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -184,7 +190,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces."
|
||||
"We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -248,11 +254,11 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from Pytorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).\n",
|
||||
"Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from PyTorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).\n",
|
||||
"\n",
|
||||
"For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.\n",
|
||||
"\n",
|
||||
"**NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting `return_all_views = False`."
|
||||
"**NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting `return_all_views = False`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -289,7 +295,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Render ShapeNetCore models with PyTorch3D's differntiable renderer"
|
||||
"## 3. Render ShapeNetCore models with PyTorch3D's differentiable renderer"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -444,7 +450,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, we will visualize PyTorch3d's renderings:"
|
||||
"Next, we will visualize PyTorch3D's renderings:"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -17,24 +17,30 @@
|
||||
|
||||
# ## 0. Install and import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -120,7 +126,7 @@ SPLITS_PATH = "None"
|
||||
r2n2_dataset = R2N2("train", SHAPENET_PATH, R2N2_PATH, SPLITS_PATH, return_voxels=True)
|
||||
|
||||
|
||||
# We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces.
|
||||
# We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: http://image-net.org/download-API), its model id, and its vertices and faces.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -155,11 +161,11 @@ image_grid(r2n2_renderings.numpy(), rows=1, cols=2, rgb=True)
|
||||
|
||||
# ## 2. Use the datasets with `torch.utils.data.DataLoader`
|
||||
|
||||
# Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from Pytorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).
|
||||
# Training deep learning models, usually requires passing in batches of inputs. The `torch.utils.data.DataLoader` from PyTorch helps us do this. PyTorch3D provides a function `collate_batched_meshes` to group the input meshes into a single `Meshes` object which represents the batch. The `Meshes` datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. `graph_conv`).
|
||||
#
|
||||
# For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.
|
||||
#
|
||||
# **NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting `return_all_views = False`.
|
||||
# **NOTE**: All models in the `val` split of R2N2 have 24 views, but there are 8 models that split their 24 views between `train` and `test` splits, in which case `collate_batched_meshes` will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting `return_all_views = False`.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -180,7 +186,7 @@ batch_renderings = r2n2_batch["images"] # (N, V, H, W, 3), and in this case V is
|
||||
image_grid(batch_renderings.squeeze().numpy(), rows=3, cols=4, rgb=True)
|
||||
|
||||
|
||||
# ## 3. Render ShapeNetCore models with PyTorch3D's differntiable renderer
|
||||
# ## 3. Render ShapeNetCore models with PyTorch3D's differentiable renderer
|
||||
|
||||
# Both `ShapeNetCore` and `R2N2` dataloaders have customized `render` functions that support rendering models by specifying their model ids, categories or indices using PyTorch3D's differentiable renderer implementation.
|
||||
|
||||
@ -272,7 +278,7 @@ original_rendering = r2n2_dataset[6,[1,2]]["images"]
|
||||
image_grid(original_rendering.numpy(), rows=1, cols=2, rgb=True)
|
||||
|
||||
|
||||
# Next, we will visualize PyTorch3d's renderings:
|
||||
# Next, we will visualize PyTorch3D's renderings:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
@ -68,7 +68,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -81,19 +81,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -400,10 +406,10 @@
|
||||
" loop.set_description('total_loss = %.6f' % loss)\n",
|
||||
" \n",
|
||||
" # Save the losses for plotting\n",
|
||||
" chamfer_losses.append(loss_chamfer)\n",
|
||||
" edge_losses.append(loss_edge)\n",
|
||||
" normal_losses.append(loss_normal)\n",
|
||||
" laplacian_losses.append(loss_laplacian)\n",
|
||||
" chamfer_losses.append(float(loss_chamfer.detach().cpu()))\n",
|
||||
" edge_losses.append(float(loss_edge.detach().cpu()))\n",
|
||||
" normal_losses.append(float(loss_normal.detach().cpu()))\n",
|
||||
" laplacian_losses.append(float(loss_laplacian.detach().cpu()))\n",
|
||||
" \n",
|
||||
" # Plot mesh\n",
|
||||
" if i % plot_period == 0:\n",
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -34,24 +34,30 @@
|
||||
|
||||
# ## 0. Install and Import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -237,10 +243,10 @@ for i in loop:
|
||||
loop.set_description('total_loss = %.6f' % loss)
|
||||
|
||||
# Save the losses for plotting
|
||||
chamfer_losses.append(loss_chamfer)
|
||||
edge_losses.append(loss_edge)
|
||||
normal_losses.append(loss_normal)
|
||||
laplacian_losses.append(loss_laplacian)
|
||||
chamfer_losses.append(float(loss_chamfer.detach().cpu()))
|
||||
edge_losses.append(float(loss_edge.detach().cpu()))
|
||||
normal_losses.append(float(loss_normal.detach().cpu()))
|
||||
laplacian_losses.append(float(loss_laplacian.detach().cpu()))
|
||||
|
||||
# Plot mesh
|
||||
if i % plot_period == 0:
|
||||
|
||||
@ -46,7 +46,7 @@
|
||||
"id": "okLalbR_g7NS"
|
||||
},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -59,19 +59,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -91,7 +97,6 @@
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from skimage.io import imread\n",
|
||||
"\n",
|
||||
"from pytorch3d.utils import ico_sphere\n",
|
||||
"import numpy as np\n",
|
||||
@ -187,11 +192,11 @@
|
||||
"source": [
|
||||
"### 1. Load a mesh and texture file\n",
|
||||
"\n",
|
||||
"Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
|
||||
"Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
|
||||
"\n",
|
||||
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes. \n",
|
||||
"\n",
|
||||
"**TexturesVertex** is an auxillary datastructure for storing vertex rgb texture information about meshes. \n",
|
||||
"**TexturesVertex** is an auxiliary datastructure for storing vertex rgb texture information about meshes. \n",
|
||||
"\n",
|
||||
"**Meshes** has several class methods which are used throughout the rendering pipeline."
|
||||
]
|
||||
@ -255,7 +260,7 @@
|
||||
"N = verts.shape[0]\n",
|
||||
"center = verts.mean(0)\n",
|
||||
"scale = max((verts - center).abs().max(0)[0])\n",
|
||||
"mesh.offset_verts_(-center.expand(N, 3))\n",
|
||||
"mesh.offset_verts_(-center)\n",
|
||||
"mesh.scale_verts_((1.0 / float(scale)));"
|
||||
]
|
||||
},
|
||||
@ -309,7 +314,7 @@
|
||||
"# purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to \n",
|
||||
"# rasterize_meshes.py for explanations of these parameters. We also leave \n",
|
||||
"# bin_size and max_faces_per_bin to their default values of None, which sets \n",
|
||||
"# their values using huristics and ensures that the faster coarse-to-fine \n",
|
||||
"# their values using heuristics and ensures that the faster coarse-to-fine \n",
|
||||
"# rasterization method is used. Refer to docs/notes/renderer.md for an \n",
|
||||
"# explanation of the difference between naive and coarse-to-fine rasterization. \n",
|
||||
"raster_settings = RasterizationSettings(\n",
|
||||
@ -318,8 +323,8 @@
|
||||
" faces_per_pixel=1, \n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Create a phong renderer by composing a rasterizer and a shader. The textured \n",
|
||||
"# phong shader will interpolate the texture uv coordinates for each vertex, \n",
|
||||
"# Create a Phong renderer by composing a rasterizer and a shader. The textured \n",
|
||||
"# Phong shader will interpolate the texture uv coordinates for each vertex, \n",
|
||||
"# sample from a texture image and apply the Phong lighting model\n",
|
||||
"renderer = MeshRenderer(\n",
|
||||
" rasterizer=MeshRasterizer(\n",
|
||||
@ -380,7 +385,7 @@
|
||||
"id": "gOb4rYx65E8z"
|
||||
},
|
||||
"source": [
|
||||
"Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel."
|
||||
"Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -449,7 +454,8 @@
|
||||
" target_image=target_rgb[1], title='', \n",
|
||||
" silhouette=False):\n",
|
||||
" inds = 3 if silhouette else range(3)\n",
|
||||
" predicted_images = renderer(predicted_mesh)\n",
|
||||
" with torch.no_grad():\n",
|
||||
" predicted_images = renderer(predicted_mesh)\n",
|
||||
" plt.figure(figsize=(20, 10))\n",
|
||||
" plt.subplot(1, 2, 1)\n",
|
||||
" plt.imshow(predicted_images[0, ..., inds].cpu().detach().numpy())\n",
|
||||
@ -457,7 +463,6 @@
|
||||
" plt.subplot(1, 2, 2)\n",
|
||||
" plt.imshow(target_image.cpu().detach().numpy())\n",
|
||||
" plt.title(title)\n",
|
||||
" plt.grid(\"off\")\n",
|
||||
" plt.axis(\"off\")\n",
|
||||
"\n",
|
||||
"# Plot losses as a function of optimization iteration\n",
|
||||
@ -601,7 +606,7 @@
|
||||
"id": "QLc9zK8lEqFS"
|
||||
},
|
||||
"source": [
|
||||
"We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:"
|
||||
"We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -640,7 +645,8 @@
|
||||
" sum_loss = torch.tensor(0.0, device=device)\n",
|
||||
" for k, l in loss.items():\n",
|
||||
" sum_loss += l * losses[k][\"weight\"]\n",
|
||||
" losses[k][\"values\"].append(l)\n",
|
||||
" losses[k][\"values\"].append(float(l.detach().cpu()))\n",
|
||||
"\n",
|
||||
" \n",
|
||||
" # Print the losses\n",
|
||||
" loop.set_description(\"total_loss = %.6f\" % sum_loss)\n",
|
||||
@ -824,7 +830,7 @@
|
||||
" sum_loss = torch.tensor(0.0, device=device)\n",
|
||||
" for k, l in loss.items():\n",
|
||||
" sum_loss += l * losses[k][\"weight\"]\n",
|
||||
" losses[k][\"values\"].append(l)\n",
|
||||
" losses[k][\"values\"].append(float(l.detach().cpu()))\n",
|
||||
" \n",
|
||||
" # Print the losses\n",
|
||||
" loop.set_description(\"total_loss = %.6f\" % sum_loss)\n",
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -17,24 +17,30 @@
|
||||
|
||||
# ## 0. Install and Import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -47,7 +53,6 @@ else:
|
||||
import os
|
||||
import torch
|
||||
import matplotlib.pyplot as plt
|
||||
from skimage.io import imread
|
||||
|
||||
from pytorch3d.utils import ico_sphere
|
||||
import numpy as np
|
||||
@ -105,11 +110,11 @@ from plot_image_grid import image_grid
|
||||
|
||||
# ### 1. Load a mesh and texture file
|
||||
#
|
||||
# Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object.
|
||||
# Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object.
|
||||
#
|
||||
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
|
||||
#
|
||||
# **TexturesVertex** is an auxillary datastructure for storing vertex rgb texture information about meshes.
|
||||
# **TexturesVertex** is an auxiliary datastructure for storing vertex rgb texture information about meshes.
|
||||
#
|
||||
# **Meshes** has several class methods which are used throughout the rendering pipeline.
|
||||
|
||||
@ -150,7 +155,7 @@ verts = mesh.verts_packed()
|
||||
N = verts.shape[0]
|
||||
center = verts.mean(0)
|
||||
scale = max((verts - center).abs().max(0)[0])
|
||||
mesh.offset_verts_(-center.expand(N, 3))
|
||||
mesh.offset_verts_(-center)
|
||||
mesh.scale_verts_((1.0 / float(scale)));
|
||||
|
||||
|
||||
@ -190,7 +195,7 @@ camera = OpenGLPerspectiveCameras(device=device, R=R[None, 1, ...],
|
||||
# purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to
|
||||
# rasterize_meshes.py for explanations of these parameters. We also leave
|
||||
# bin_size and max_faces_per_bin to their default values of None, which sets
|
||||
# their values using huristics and ensures that the faster coarse-to-fine
|
||||
# their values using heuristics and ensures that the faster coarse-to-fine
|
||||
# rasterization method is used. Refer to docs/notes/renderer.md for an
|
||||
# explanation of the difference between naive and coarse-to-fine rasterization.
|
||||
raster_settings = RasterizationSettings(
|
||||
@ -199,8 +204,8 @@ raster_settings = RasterizationSettings(
|
||||
faces_per_pixel=1,
|
||||
)
|
||||
|
||||
# Create a phong renderer by composing a rasterizer and a shader. The textured
|
||||
# phong shader will interpolate the texture uv coordinates for each vertex,
|
||||
# Create a Phong renderer by composing a rasterizer and a shader. The textured
|
||||
# Phong shader will interpolate the texture uv coordinates for each vertex,
|
||||
# sample from a texture image and apply the Phong lighting model
|
||||
renderer = MeshRenderer(
|
||||
rasterizer=MeshRasterizer(
|
||||
@ -239,7 +244,7 @@ image_grid(target_images.cpu().numpy(), rows=4, cols=5, rgb=True)
|
||||
plt.show()
|
||||
|
||||
|
||||
# Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel.
|
||||
# Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -285,7 +290,8 @@ def visualize_prediction(predicted_mesh, renderer=renderer_silhouette,
|
||||
target_image=target_rgb[1], title='',
|
||||
silhouette=False):
|
||||
inds = 3 if silhouette else range(3)
|
||||
predicted_images = renderer(predicted_mesh)
|
||||
with torch.no_grad():
|
||||
predicted_images = renderer(predicted_mesh)
|
||||
plt.figure(figsize=(20, 10))
|
||||
plt.subplot(1, 2, 1)
|
||||
plt.imshow(predicted_images[0, ..., inds].cpu().detach().numpy())
|
||||
@ -293,7 +299,6 @@ def visualize_prediction(predicted_mesh, renderer=renderer_silhouette,
|
||||
plt.subplot(1, 2, 2)
|
||||
plt.imshow(target_image.cpu().detach().numpy())
|
||||
plt.title(title)
|
||||
plt.grid("off")
|
||||
plt.axis("off")
|
||||
|
||||
# Plot losses as a function of optimization iteration
|
||||
@ -385,7 +390,7 @@ deform_verts = torch.full(verts_shape, 0.0, device=device, requires_grad=True)
|
||||
optimizer = torch.optim.SGD([deform_verts], lr=1.0, momentum=0.9)
|
||||
|
||||
|
||||
# We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:
|
||||
# We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -416,7 +421,8 @@ for i in loop:
|
||||
sum_loss = torch.tensor(0.0, device=device)
|
||||
for k, l in loss.items():
|
||||
sum_loss += l * losses[k]["weight"]
|
||||
losses[k]["values"].append(l)
|
||||
losses[k]["values"].append(float(l.detach().cpu()))
|
||||
|
||||
|
||||
# Print the losses
|
||||
loop.set_description("total_loss = %.6f" % sum_loss)
|
||||
@ -547,7 +553,7 @@ for i in loop:
|
||||
sum_loss = torch.tensor(0.0, device=device)
|
||||
for k, l in loss.items():
|
||||
sum_loss += l * losses[k]["weight"]
|
||||
losses[k]["values"].append(l)
|
||||
losses[k]["values"].append(float(l.detach().cpu()))
|
||||
|
||||
# Print the losses
|
||||
loop.set_description("total_loss = %.6f" % sum_loss)
|
||||
|
||||
@ -32,7 +32,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -41,19 +41,25 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -70,7 +76,6 @@
|
||||
"import torch\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from skimage.io import imread\n",
|
||||
"\n",
|
||||
"# Util function for loading point clouds|\n",
|
||||
"import numpy as np\n",
|
||||
@ -151,7 +156,7 @@
|
||||
"source": [
|
||||
"## Create a renderer\n",
|
||||
"\n",
|
||||
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.\n",
|
||||
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.\n",
|
||||
"\n",
|
||||
"In this example we will first create a **renderer** which uses an **orthographic camera**, and applies **alpha compositing**. Then we learn how to vary different components using the modular API. \n",
|
||||
"\n",
|
||||
@ -196,7 +201,6 @@
|
||||
"images = renderer(point_cloud)\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\")\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -223,7 +227,6 @@
|
||||
"\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\")\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -271,7 +274,6 @@
|
||||
"images = renderer(point_cloud)\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\")\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -297,7 +299,6 @@
|
||||
"images = renderer(point_cloud)\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\")\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -330,7 +331,6 @@
|
||||
" bg_col=torch.tensor([0.0, 1.0, 0.0, 1.0], dtype=torch.float32, device=device))\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\")\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -16,24 +16,30 @@
|
||||
|
||||
# ## Import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -47,7 +53,6 @@ import os
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
import matplotlib.pyplot as plt
|
||||
from skimage.io import imread
|
||||
|
||||
# Util function for loading point clouds|
|
||||
import numpy as np
|
||||
@ -108,7 +113,7 @@ point_cloud = Pointclouds(points=[verts], features=[rgb])
|
||||
|
||||
# ## Create a renderer
|
||||
#
|
||||
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.
|
||||
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
|
||||
#
|
||||
# In this example we will first create a **renderer** which uses an **orthographic camera**, and applies **alpha compositing**. Then we learn how to vary different components using the modular API.
|
||||
#
|
||||
@ -146,7 +151,6 @@ renderer = PointsRenderer(
|
||||
images = renderer(point_cloud)
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off")
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -165,7 +169,6 @@ images = renderer(point_cloud)
|
||||
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off")
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -202,7 +205,6 @@ renderer = PointsRenderer(
|
||||
images = renderer(point_cloud)
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off")
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -220,7 +222,6 @@ renderer = PointsRenderer(
|
||||
images = renderer(point_cloud)
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off")
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -245,7 +246,6 @@ images = renderer(point_cloud, gamma=(1e-4,),
|
||||
bg_col=torch.tensor([0.0, 1.0, 0.0, 1.0], dtype=torch.float32, device=device))
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off")
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
|
||||
@ -39,7 +39,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If torch, torchvision and PyTorch3D are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -48,19 +48,25 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -86,12 +92,11 @@
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from skimage.io import imread\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# libraries for reading data from files\n",
|
||||
"from scipy.io import loadmat\n",
|
||||
"from pytorch3d.io.utils import _read_image\n",
|
||||
"from PIL import Image\n",
|
||||
"import pickle\n",
|
||||
"\n",
|
||||
"# Data structures and functions for rendering\n",
|
||||
@ -178,13 +183,15 @@
|
||||
" data = pickle.load(f, encoding='latin1') \n",
|
||||
" v_template = torch.Tensor(data['v_template']).to(device) # (6890, 3)\n",
|
||||
"ALP_UV = loadmat(data_filename)\n",
|
||||
"tex = torch.from_numpy(_read_image(file_name=tex_filename, format='RGB') / 255. ).unsqueeze(0).to(device)\n",
|
||||
"with Image.open(tex_filename) as image:\n",
|
||||
" np_image = np.asarray(image.convert(\"RGB\")).astype(np.float32)\n",
|
||||
"tex = torch.from_numpy(np_image / 255.)[None].to(device)\n",
|
||||
"\n",
|
||||
"verts = torch.from_numpy((ALP_UV[\"All_vertices\"]).astype(int)).squeeze().to(device) # (7829, 1)\n",
|
||||
"verts = torch.from_numpy((ALP_UV[\"All_vertices\"]).astype(int)).squeeze().to(device) # (7829,)\n",
|
||||
"U = torch.Tensor(ALP_UV['All_U_norm']).to(device) # (7829, 1)\n",
|
||||
"V = torch.Tensor(ALP_UV['All_V_norm']).to(device) # (7829, 1)\n",
|
||||
"faces = torch.from_numpy((ALP_UV['All_Faces'] - 1).astype(int)).to(device) # (13774, 3)\n",
|
||||
"face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()"
|
||||
"face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze() # (13774,)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -196,7 +203,6 @@
|
||||
"# Display the texture image\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(tex.squeeze(0).cpu())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -222,6 +228,9 @@
|
||||
" part = rows * i + j + 1 # parts are 1-indexed in face_indices\n",
|
||||
" offset_per_part[part] = (u, v)\n",
|
||||
"\n",
|
||||
"U_norm = U.clone()\n",
|
||||
"V_norm = V.clone()\n",
|
||||
"\n",
|
||||
"# iterate over faces and offset the corresponding vertex u and v values\n",
|
||||
"for i in range(len(faces)):\n",
|
||||
" face_vert_idxs = faces[i]\n",
|
||||
@ -232,15 +241,15 @@
|
||||
" # vertices are reused, but we don't want to offset multiple times\n",
|
||||
" if vert_idx.item() not in already_offset:\n",
|
||||
" # offset u value\n",
|
||||
" U[vert_idx] = U[vert_idx] / cols + offset_u\n",
|
||||
" U_norm[vert_idx] = U[vert_idx] / cols + offset_u\n",
|
||||
" # offset v value\n",
|
||||
" # this also flips each part locally, as each part is upside down\n",
|
||||
" V[vert_idx] = (1 - V[vert_idx]) / rows + offset_v\n",
|
||||
" V_norm[vert_idx] = (1 - V[vert_idx]) / rows + offset_v\n",
|
||||
" # add vertex to our set tracking offsetted vertices\n",
|
||||
" already_offset.add(vert_idx.item())\n",
|
||||
"\n",
|
||||
"# invert V values\n",
|
||||
"U_norm, V_norm = U, 1 - V"
|
||||
"V_norm = 1 - V_norm"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -257,10 +266,7 @@
|
||||
"# Therefore when initializing the Meshes class,\n",
|
||||
"# we need to map each of the vertices referenced by the DensePose faces (in verts, which is the \"All_vertices\" field)\n",
|
||||
"# to the correct xyz coordinate in the SMPL template mesh.\n",
|
||||
"v_template_extended = torch.stack(list(map(lambda vert: v_template[vert-1], verts))).unsqueeze(0).to(device) # (1, 7829, 3)\n",
|
||||
"\n",
|
||||
"# add a batch dimension to faces\n",
|
||||
"faces = faces.unsqueeze(0)"
|
||||
"v_template_extended = v_template[verts-1][None] # (1, 7829, 3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -271,7 +277,7 @@
|
||||
"\n",
|
||||
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.\n",
|
||||
"\n",
|
||||
"**TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes."
|
||||
"**TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -280,8 +286,8 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"texture = TexturesUV(maps=tex, faces_uvs=faces, verts_uvs=verts_uv)\n",
|
||||
"mesh = Meshes(v_template_extended, faces, texture)"
|
||||
"texture = TexturesUV(maps=tex, faces_uvs=faces[None], verts_uvs=verts_uv)\n",
|
||||
"mesh = Meshes(v_template_extended, faces[None], texture)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -314,7 +320,7 @@
|
||||
"# Place a point light in front of the person. \n",
|
||||
"lights = PointLights(device=device, location=[[0.0, 0.0, 2.0]])\n",
|
||||
"\n",
|
||||
"# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will \n",
|
||||
"# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will \n",
|
||||
"# interpolate the texture uv coordinates for each vertex, sample from a texture image and \n",
|
||||
"# apply the Phong lighting model\n",
|
||||
"renderer = MeshRenderer(\n",
|
||||
@ -346,7 +352,6 @@
|
||||
"images = renderer(mesh)\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -387,7 +392,6 @@
|
||||
"source": [
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -20,24 +20,30 @@
|
||||
|
||||
# ## Import modules
|
||||
|
||||
# If torch, torchvision and PyTorch3D are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -57,12 +63,11 @@ get_ipython().system('pip install chumpy')
|
||||
import os
|
||||
import torch
|
||||
import matplotlib.pyplot as plt
|
||||
from skimage.io import imread
|
||||
import numpy as np
|
||||
|
||||
# libraries for reading data from files
|
||||
from scipy.io import loadmat
|
||||
from pytorch3d.io.utils import _read_image
|
||||
from PIL import Image
|
||||
import pickle
|
||||
|
||||
# Data structures and functions for rendering
|
||||
@ -133,13 +138,15 @@ with open(verts_filename, 'rb') as f:
|
||||
data = pickle.load(f, encoding='latin1')
|
||||
v_template = torch.Tensor(data['v_template']).to(device) # (6890, 3)
|
||||
ALP_UV = loadmat(data_filename)
|
||||
tex = torch.from_numpy(_read_image(file_name=tex_filename, format='RGB') / 255. ).unsqueeze(0).to(device)
|
||||
with Image.open(tex_filename) as image:
|
||||
np_image = np.asarray(image.convert("RGB")).astype(np.float32)
|
||||
tex = torch.from_numpy(np_image / 255.)[None].to(device)
|
||||
|
||||
verts = torch.from_numpy((ALP_UV["All_vertices"]).astype(int)).squeeze().to(device) # (7829, 1)
|
||||
verts = torch.from_numpy((ALP_UV["All_vertices"]).astype(int)).squeeze().to(device) # (7829,)
|
||||
U = torch.Tensor(ALP_UV['All_U_norm']).to(device) # (7829, 1)
|
||||
V = torch.Tensor(ALP_UV['All_V_norm']).to(device) # (7829, 1)
|
||||
faces = torch.from_numpy((ALP_UV['All_Faces'] - 1).astype(int)).to(device) # (13774, 3)
|
||||
face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()
|
||||
face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze() # (13774,)
|
||||
|
||||
|
||||
# In[ ]:
|
||||
@ -148,7 +155,6 @@ face_indices = torch.Tensor(ALP_UV['All_FaceIndices']).squeeze()
|
||||
# Display the texture image
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(tex.squeeze(0).cpu())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -166,6 +172,9 @@ for i, u in enumerate(np.linspace(0, 1, cols, endpoint=False)):
|
||||
part = rows * i + j + 1 # parts are 1-indexed in face_indices
|
||||
offset_per_part[part] = (u, v)
|
||||
|
||||
U_norm = U.clone()
|
||||
V_norm = V.clone()
|
||||
|
||||
# iterate over faces and offset the corresponding vertex u and v values
|
||||
for i in range(len(faces)):
|
||||
face_vert_idxs = faces[i]
|
||||
@ -176,15 +185,15 @@ for i in range(len(faces)):
|
||||
# vertices are reused, but we don't want to offset multiple times
|
||||
if vert_idx.item() not in already_offset:
|
||||
# offset u value
|
||||
U[vert_idx] = U[vert_idx] / cols + offset_u
|
||||
U_norm[vert_idx] = U[vert_idx] / cols + offset_u
|
||||
# offset v value
|
||||
# this also flips each part locally, as each part is upside down
|
||||
V[vert_idx] = (1 - V[vert_idx]) / rows + offset_v
|
||||
V_norm[vert_idx] = (1 - V[vert_idx]) / rows + offset_v
|
||||
# add vertex to our set tracking offsetted vertices
|
||||
already_offset.add(vert_idx.item())
|
||||
|
||||
# invert V values
|
||||
U_norm, V_norm = U, 1 - V
|
||||
V_norm = 1 - V_norm
|
||||
|
||||
|
||||
# In[ ]:
|
||||
@ -198,23 +207,20 @@ verts_uv = torch.cat([U_norm[None],V_norm[None]], dim=2) # (1, 7829, 2)
|
||||
# Therefore when initializing the Meshes class,
|
||||
# we need to map each of the vertices referenced by the DensePose faces (in verts, which is the "All_vertices" field)
|
||||
# to the correct xyz coordinate in the SMPL template mesh.
|
||||
v_template_extended = torch.stack(list(map(lambda vert: v_template[vert-1], verts))).unsqueeze(0).to(device) # (1, 7829, 3)
|
||||
|
||||
# add a batch dimension to faces
|
||||
faces = faces.unsqueeze(0)
|
||||
v_template_extended = v_template[verts-1][None] # (1, 7829, 3)
|
||||
|
||||
|
||||
# ### Create our textured mesh
|
||||
#
|
||||
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
|
||||
#
|
||||
# **TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes.
|
||||
# **TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
texture = TexturesUV(maps=tex, faces_uvs=faces, verts_uvs=verts_uv)
|
||||
mesh = Meshes(v_template_extended, faces, texture)
|
||||
texture = TexturesUV(maps=tex, faces_uvs=faces[None], verts_uvs=verts_uv)
|
||||
mesh = Meshes(v_template_extended, faces[None], texture)
|
||||
|
||||
|
||||
# ## Create a renderer
|
||||
@ -239,7 +245,7 @@ raster_settings = RasterizationSettings(
|
||||
# Place a point light in front of the person.
|
||||
lights = PointLights(device=device, location=[[0.0, 0.0, 2.0]])
|
||||
|
||||
# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will
|
||||
# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will
|
||||
# interpolate the texture uv coordinates for each vertex, sample from a texture image and
|
||||
# apply the Phong lighting model
|
||||
renderer = MeshRenderer(
|
||||
@ -263,7 +269,6 @@ renderer = MeshRenderer(
|
||||
images = renderer(mesh)
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -293,7 +298,6 @@ images = renderer(mesh, lights=lights, cameras=cameras)
|
||||
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
|
||||
@ -47,7 +47,7 @@
|
||||
"id": "okLalbR_g7NS"
|
||||
},
|
||||
"source": [
|
||||
"If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:"
|
||||
"Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -64,19 +64,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install torch torchvision\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"import torch\n",
|
||||
"if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):\n",
|
||||
" !pip install pytorch3d\n",
|
||||
"else:\n",
|
||||
" need_pytorch3d=False\n",
|
||||
" try:\n",
|
||||
" import pytorch3d\n",
|
||||
" except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
" if need_pytorch3d:\n",
|
||||
"need_pytorch3d=False\n",
|
||||
"try:\n",
|
||||
" import pytorch3d\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" need_pytorch3d=True\n",
|
||||
"if need_pytorch3d:\n",
|
||||
" if torch.__version__.startswith(\"1.9\") and sys.platform.startswith(\"linux\"):\n",
|
||||
" # We try to install PyTorch3D via a released wheel.\n",
|
||||
" version_str=\"\".join([\n",
|
||||
" f\"py3{sys.version_info.minor}_cu\",\n",
|
||||
" torch.version.cuda.replace(\".\",\"\"),\n",
|
||||
" f\"_pyt{torch.__version__[0:5:2]}\"\n",
|
||||
" ])\n",
|
||||
" !pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
|
||||
" else:\n",
|
||||
" # We try to install PyTorch3D from source.\n",
|
||||
" !curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz\n",
|
||||
" !tar xzf 1.10.0.tar.gz\n",
|
||||
" os.environ[\"CUB_HOME\"] = os.getcwd() + \"/cub-1.10.0\"\n",
|
||||
@ -96,7 +102,6 @@
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from skimage.io import imread\n",
|
||||
"\n",
|
||||
"# Util function for loading meshes\n",
|
||||
"from pytorch3d.io import load_objs_as_meshes, load_obj\n",
|
||||
@ -185,11 +190,11 @@
|
||||
"source": [
|
||||
"### 1. Load a mesh and texture file\n",
|
||||
"\n",
|
||||
"Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
|
||||
"Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object. \n",
|
||||
"\n",
|
||||
"**Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes. \n",
|
||||
"\n",
|
||||
"**TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes. \n",
|
||||
"**TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes. \n",
|
||||
"\n",
|
||||
"**Meshes** has several class methods which are used throughout the rendering pipeline."
|
||||
]
|
||||
@ -277,7 +282,6 @@
|
||||
"plt.figure(figsize=(7,7))\n",
|
||||
"texture_image=mesh.textures.maps_padded()\n",
|
||||
"plt.imshow(texture_image.squeeze().cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -296,7 +300,6 @@
|
||||
"source": [
|
||||
"plt.figure(figsize=(7,7))\n",
|
||||
"texturesuv_image_matplotlib(mesh.textures, subsample=None)\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -311,7 +314,7 @@
|
||||
"\n",
|
||||
"A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.\n",
|
||||
"\n",
|
||||
"In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **phong shading**. Then we learn how to vary different components using the modular API. "
|
||||
"In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **Phong shading**. Then we learn how to vary different components using the modular API. "
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -346,7 +349,7 @@
|
||||
"# -z direction. \n",
|
||||
"lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])\n",
|
||||
"\n",
|
||||
"# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will \n",
|
||||
"# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will \n",
|
||||
"# interpolate the texture uv coordinates for each vertex, sample from a texture image and \n",
|
||||
"# apply the Phong lighting model\n",
|
||||
"renderer = MeshRenderer(\n",
|
||||
@ -399,7 +402,6 @@
|
||||
"images = renderer(mesh)\n",
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -412,7 +414,7 @@
|
||||
"source": [
|
||||
"## 4. Move the light behind the object and re-render\n",
|
||||
"\n",
|
||||
"We can pass arbirary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/\n",
|
||||
"We can pass arbitrary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/\n",
|
||||
"\n",
|
||||
"In this case, we can simply update the location of the lights and pass them into the call to the renderer. \n",
|
||||
"\n",
|
||||
@ -450,7 +452,6 @@
|
||||
"source": [
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -514,7 +515,6 @@
|
||||
"source": [
|
||||
"plt.figure(figsize=(10, 10))\n",
|
||||
"plt.imshow(images[0, ..., :3].cpu().numpy())\n",
|
||||
"plt.grid(\"off\");\n",
|
||||
"plt.axis(\"off\");"
|
||||
]
|
||||
},
|
||||
@ -573,7 +573,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# We can pass arbirary keyword arguments to the rasterizer/shader via the renderer\n",
|
||||
"# We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer\n",
|
||||
"# so the renderer does not need to be reinitialized if any of the settings change.\n",
|
||||
"images = renderer(meshes, cameras=cameras, lights=lights)"
|
||||
]
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# coding: utf-8
|
||||
|
||||
# In[ ]:
|
||||
@ -18,24 +18,30 @@
|
||||
|
||||
# ## 0. Install and Import modules
|
||||
|
||||
# If `torch`, `torchvision` and `pytorch3d` are not installed, run the following cell:
|
||||
# Ensure `torch` and `torchvision` are installed. If `pytorch3d` is not installed, install it using the following cell:
|
||||
|
||||
# In[ ]:
|
||||
|
||||
|
||||
get_ipython().system('pip install torch torchvision')
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
if torch.__version__=='1.6.0+cu101' and sys.platform.startswith('linux'):
|
||||
get_ipython().system('pip install pytorch3d')
|
||||
else:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
need_pytorch3d=False
|
||||
try:
|
||||
import pytorch3d
|
||||
except ModuleNotFoundError:
|
||||
need_pytorch3d=True
|
||||
if need_pytorch3d:
|
||||
if torch.__version__.startswith("1.9") and sys.platform.startswith("linux"):
|
||||
# We try to install PyTorch3D via a released wheel.
|
||||
version_str="".join([
|
||||
f"py3{sys.version_info.minor}_cu",
|
||||
torch.version.cuda.replace(".",""),
|
||||
f"_pyt{torch.__version__[0:5:2]}"
|
||||
])
|
||||
get_ipython().system('pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html')
|
||||
else:
|
||||
# We try to install PyTorch3D from source.
|
||||
get_ipython().system('curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz')
|
||||
get_ipython().system('tar xzf 1.10.0.tar.gz')
|
||||
os.environ["CUB_HOME"] = os.getcwd() + "/cub-1.10.0"
|
||||
@ -48,7 +54,6 @@ else:
|
||||
import os
|
||||
import torch
|
||||
import matplotlib.pyplot as plt
|
||||
from skimage.io import imread
|
||||
|
||||
# Util function for loading meshes
|
||||
from pytorch3d.io import load_objs_as_meshes, load_obj
|
||||
@ -96,11 +101,11 @@ from plot_image_grid import image_grid
|
||||
|
||||
# ### 1. Load a mesh and texture file
|
||||
#
|
||||
# Load an `.obj` file and it's associated `.mtl` file and create a **Textures** and **Meshes** object.
|
||||
# Load an `.obj` file and its associated `.mtl` file and create a **Textures** and **Meshes** object.
|
||||
#
|
||||
# **Meshes** is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.
|
||||
#
|
||||
# **TexturesUV** is an auxillary datastructure for storing vertex uv and texture maps for meshes.
|
||||
# **TexturesUV** is an auxiliary datastructure for storing vertex uv and texture maps for meshes.
|
||||
#
|
||||
# **Meshes** has several class methods which are used throughout the rendering pipeline.
|
||||
|
||||
@ -142,7 +147,6 @@ mesh = load_objs_as_meshes([obj_filename], device=device)
|
||||
plt.figure(figsize=(7,7))
|
||||
texture_image=mesh.textures.maps_padded()
|
||||
plt.imshow(texture_image.squeeze().cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -153,7 +157,6 @@ plt.axis("off");
|
||||
|
||||
plt.figure(figsize=(7,7))
|
||||
texturesuv_image_matplotlib(mesh.textures, subsample=None)
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -161,7 +164,7 @@ plt.axis("off");
|
||||
#
|
||||
# A renderer in PyTorch3D is composed of a **rasterizer** and a **shader** which each have a number of subcomponents such as a **camera** (orthographic/perspective). Here we initialize some of these components and use default values for the rest.
|
||||
#
|
||||
# In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **phong shading**. Then we learn how to vary different components using the modular API.
|
||||
# In this example we will first create a **renderer** which uses a **perspective camera**, a **point light** and applies **Phong shading**. Then we learn how to vary different components using the modular API.
|
||||
|
||||
# In[ ]:
|
||||
|
||||
@ -188,7 +191,7 @@ raster_settings = RasterizationSettings(
|
||||
# -z direction.
|
||||
lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
|
||||
|
||||
# Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will
|
||||
# Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will
|
||||
# interpolate the texture uv coordinates for each vertex, sample from a texture image and
|
||||
# apply the Phong lighting model
|
||||
renderer = MeshRenderer(
|
||||
@ -214,13 +217,12 @@ renderer = MeshRenderer(
|
||||
images = renderer(mesh)
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
# ## 4. Move the light behind the object and re-render
|
||||
#
|
||||
# We can pass arbirary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/
|
||||
# We can pass arbitrary keyword arguments to the `rasterizer`/`shader` via the call to the `renderer` so the renderer does not need to be reinitialized if any of the settings change/
|
||||
#
|
||||
# In this case, we can simply update the location of the lights and pass them into the call to the renderer.
|
||||
#
|
||||
@ -239,7 +241,6 @@ images = renderer(mesh, lights=lights)
|
||||
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -277,7 +278,6 @@ images = renderer(mesh, lights=lights, materials=materials, cameras=cameras)
|
||||
|
||||
plt.figure(figsize=(10, 10))
|
||||
plt.imshow(images[0, ..., :3].cpu().numpy())
|
||||
plt.grid("off");
|
||||
plt.axis("off");
|
||||
|
||||
|
||||
@ -315,7 +315,7 @@ lights.location = torch.tensor([[0.0, 0.0, -3.0]], device=device)
|
||||
# In[ ]:
|
||||
|
||||
|
||||
# We can pass arbirary keyword arguments to the rasterizer/shader via the renderer
|
||||
# We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer
|
||||
# so the renderer does not need to be reinitialized if any of the settings change.
|
||||
images = renderer(meshes, cameras=cameras, lights=lights)
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -12,4 +12,4 @@
|
||||
</span></div></h2><div><span><p>Ask questions about the documentation and project</p>
|
||||
</span></div></div></div><div class="blockElement threeByGridBlock"><div class="blockContent"><h2><div><span><p>Stay up to date</p>
|
||||
</span></div></h2><div><span><p>Find out what's new with this project</p>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -12,4 +12,4 @@
|
||||
</span></div></h2><div><span><p>Ask questions about the documentation and project</p>
|
||||
</span></div></div></div><div class="blockElement threeByGridBlock"><div class="blockContent"><h2><div><span><p>Stay up to date</p>
|
||||
</span></div></h2><div><span><p>Find out what's new with this project</p>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></div></div></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -28,4 +28,4 @@ sample_sphere = sample_points_from_meshes(sphere_mesh, <span class="hljs-number"
|
||||
sample_test = sample_points_from_meshes(test_mesh, <span class="hljs-number">5000</span>)
|
||||
loss_chamfer, _ = chamfer_distance(sample_sphere, sample_test)
|
||||
</code></pre>
|
||||
</span></div></li></ol></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</span></div></li></ol></div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1 +1 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url><loc>https://pytorch3d.org/help</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/help"/></url><url><loc>https://pytorch3d.org/index</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/index"/></url><url><loc>https://pytorch3d.org/users</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/users"/></url><url><loc>https://pytorch3d.org/docs/batching</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/batching"/></url><url><loc>https://pytorch3d.org/docs/cameras</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/cameras"/></url><url><loc>https://pytorch3d.org/docs/cubify</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/cubify"/></url><url><loc>https://pytorch3d.org/docs/datasets</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/datasets"/></url><url><loc>https://pytorch3d.org/docs/meshes_io</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/meshes_io"/></url><url><loc>https://pytorch3d.org/docs/renderer_getting_started</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/renderer_getting_started"/></url><url><loc>https://pytorch3d.org/docs/renderer</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/renderer"/></url><url><loc>https://pytorch3d.org/docs/visualization</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/visualization"/></url><url><loc>https://pytorch3d.org/docs/why_pytorch3d</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/why_pytorch3d"/></url></urlset>
|
||||
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url><loc>https://pytorch3d.org/help</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/help"/></url><url><loc>https://pytorch3d.org/index</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/index"/></url><url><loc>https://pytorch3d.org/users</loc><changefreq>weekly</changefreq><priority>0.5</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/en/users"/></url><url><loc>https://pytorch3d.org/docs/batching</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/batching"/></url><url><loc>https://pytorch3d.org/docs/cameras</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/cameras"/></url><url><loc>https://pytorch3d.org/docs/cubify</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/cubify"/></url><url><loc>https://pytorch3d.org/docs/datasets</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/datasets"/></url><url><loc>https://pytorch3d.org/docs/io</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/io"/></url><url><loc>https://pytorch3d.org/docs/meshes_io</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/meshes_io"/></url><url><loc>https://pytorch3d.org/docs/renderer_getting_started</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/renderer_getting_started"/></url><url><loc>https://pytorch3d.org/docs/renderer</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/renderer"/></url><url><loc>https://pytorch3d.org/docs/visualization</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/visualization"/></url><url><loc>https://pytorch3d.org/docs/why_pytorch3d</loc><changefreq>hourly</changefreq><priority>1.0</priority><xhtml:link rel="alternate" hreflang="en" href="https://pytorch3d.org/docs/why_pytorch3d"/></url></urlset>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Absolute-camera-orientation-given-set-of-relative-camera-pairs">Absolute camera orientation given set of relative camera pairs<a class="anchor-link" href="#Absolute-camera-orientation-given-set-of-relative-camera-pairs">¶</a></h1><p>This tutorial showcases the <code>cameras</code>, <code>transforms</code> and <code>so3</code> API.</p>
|
||||
<p>The problem we deal with is defined as follows:</p>
|
||||
@ -97,22 +98,24 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<img alt="Initialization" src="https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_initialization.png?raw=1"/></p>
|
||||
<p>Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:
|
||||
<img alt="Solution" src="https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_final.png?raw=1"/></p>
|
||||
<p>In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the <code>SfMPerspectiveCameras</code> class initialized with the corresponding rotation and translation matrices <code>R_absolute</code> and <code>T_absolute</code> that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that <code>R_absolute</code> is a valid rotation matrix, we represent it using an exponential map (implemented with <code>so3_exponential_map</code>) of the axis-angle representation of the rotation <code>log_R_absolute</code>.</p>
|
||||
<p>In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the <code>SfMPerspectiveCameras</code> class initialized with the corresponding rotation and translation matrices <code>R_absolute</code> and <code>T_absolute</code> that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that <code>R_absolute</code> is a valid rotation matrix, we represent it using an exponential map (implemented with <code>so3_exp_map</code>) of the axis-angle representation of the rotation <code>log_R_absolute</code>.</p>
|
||||
<p>Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \vec{0})$.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-Modules">0. Install and Import Modules<a class="anchor-link" href="#0.-Install-and-Import-Modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -121,19 +124,25 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -150,11 +159,11 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># imports</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms.so3</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="n">so3_exponential_map</span><span class="p">,</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms.so3</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">so3_exp_map</span><span class="p">,</span>
|
||||
<span class="n">so3_relative_angle</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer.cameras</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer.cameras</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">SfMPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
@ -176,7 +185,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting the camera scene, and the ground truth camera positions:</p>
|
||||
</div>
|
||||
@ -188,7 +198,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/camera_visualization.py
|
||||
<span class="kn">from</span> <span class="nn">camera_visualization</span> <span class="k">import</span> <span class="n">plot_camera_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">camera_visualization</span> <span class="kn">import</span> <span class="n">plot_camera_scene</span>
|
||||
|
||||
<span class="o">!</span>mkdir data
|
||||
<span class="o">!</span>wget -P data https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/data/camera_graph.pth
|
||||
@ -198,7 +208,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -216,7 +227,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Set-up-Cameras-and-load-ground-truth-positions">1. Set up Cameras and load ground truth positions<a class="anchor-link" href="#1.-Set-up-Cameras-and-load-ground-truth-positions">¶</a></h2>
|
||||
</div>
|
||||
@ -256,7 +268,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Define-optimization-functions">2. Define optimization functions<a class="anchor-link" href="#2.-Define-optimization-functions">¶</a></h2><h3 id="Relative-cameras-and-camera-distance">Relative cameras and camera distance<a class="anchor-link" href="#Relative-cameras-and-camera-distance">¶</a></h3><p>We now define two functions crucial for the optimization.</p>
|
||||
<p><strong><code>calc_camera_distance</code></strong> compares a pair of cameras. This function is important as it defines the loss that we are minimizing. The method utilizes the <code>so3_relative_angle</code> function from the SO3 API.</p>
|
||||
@ -318,12 +331,13 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Optimization">3. Optimization<a class="anchor-link" href="#3.-Optimization">¶</a></h2><p>Finally, we start the optimization of the absolute cameras.</p>
|
||||
<p>We use SGD with momentum and optimize over <code>log_R_absolute</code> and <code>T_absolute</code>.</p>
|
||||
<p>As mentioned earlier, <code>log_R_absolute</code> is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix <code>R_absolute</code> that corresponds to <code>log_R_absolute</code> with:</p>
|
||||
<p><code>R_absolute = so3_exponential_map(log_R_absolute)</code></p>
|
||||
<p><code>R_absolute = so3_exp_map(log_R_absolute)</code></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -354,7 +368,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="n">camera_mask</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">0.</span>
|
||||
|
||||
<span class="c1"># init the optimizer</span>
|
||||
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">([</span><span class="n">log_R_absolute</span><span class="p">,</span> <span class="n">T_absolute</span><span class="p">],</span> <span class="n">lr</span><span class="o">=.</span><span class="mi">1</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">)</span>
|
||||
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">([</span><span class="n">log_R_absolute</span><span class="p">,</span> <span class="n">T_absolute</span><span class="p">],</span> <span class="n">lr</span><span class="o">=</span><span class="mf">.1</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># run the optimization</span>
|
||||
<span class="n">n_iter</span> <span class="o">=</span> <span class="mi">2000</span> <span class="c1"># fix the number of iterations</span>
|
||||
@ -365,7 +379,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="c1"># compute the absolute camera rotations as </span>
|
||||
<span class="c1"># an exponential map of the logarithms (=axis-angles)</span>
|
||||
<span class="c1"># of the absolute rotations</span>
|
||||
<span class="n">R_absolute</span> <span class="o">=</span> <span class="n">so3_exponential_map</span><span class="p">(</span><span class="n">log_R_absolute</span> <span class="o">*</span> <span class="n">camera_mask</span><span class="p">)</span>
|
||||
<span class="n">R_absolute</span> <span class="o">=</span> <span class="n">so3_exp_map</span><span class="p">(</span><span class="n">log_R_absolute</span> <span class="o">*</span> <span class="n">camera_mask</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># get the current absolute cameras</span>
|
||||
<span class="n">cameras_absolute</span> <span class="o">=</span> <span class="n">SfMPerspectiveCameras</span><span class="p">(</span>
|
||||
@ -374,7 +388,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="n">device</span> <span class="o">=</span> <span class="n">device</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># compute the relative cameras as a compositon of the absolute cameras</span>
|
||||
<span class="c1"># compute the relative cameras as a composition of the absolute cameras</span>
|
||||
<span class="n">cameras_relative_composed</span> <span class="o">=</span> \
|
||||
<span class="n">get_relative_camera</span><span class="p">(</span><span class="n">cameras_absolute</span><span class="p">,</span> <span class="n">relative_edges</span><span class="p">)</span>
|
||||
|
||||
@ -401,10 +415,11 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Conclusion">4. Conclusion<a class="anchor-link" href="#4.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to initialize a batch of SfM Cameras, set up loss functions for bundle adjustment, and run an optimization loop.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Absolute-camera-orientation-given-set-of-relative-camera-pairs">Absolute camera orientation given set of relative camera pairs<a class="anchor-link" href="#Absolute-camera-orientation-given-set-of-relative-camera-pairs">¶</a></h1><p>This tutorial showcases the <code>cameras</code>, <code>transforms</code> and <code>so3</code> API.</p>
|
||||
<p>The problem we deal with is defined as follows:</p>
|
||||
@ -97,22 +98,24 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<img alt="Initialization" src="https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_initialization.png?raw=1"/></p>
|
||||
<p>Our optimization seeks to align the estimated (orange) cameras with the ground truth (purple) cameras, by minimizing the discrepancies between pairs of relative cameras. Thus, the solution to the problem should look as follows:
|
||||
<img alt="Solution" src="https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/data/bundle_adjustment_final.png?raw=1"/></p>
|
||||
<p>In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the <code>SfMPerspectiveCameras</code> class initialized with the corresponding rotation and translation matrices <code>R_absolute</code> and <code>T_absolute</code> that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that <code>R_absolute</code> is a valid rotation matrix, we represent it using an exponential map (implemented with <code>so3_exponential_map</code>) of the axis-angle representation of the rotation <code>log_R_absolute</code>.</p>
|
||||
<p>In practice, the camera extrinsics $g_{ij}$ and $g_i$ are represented using objects from the <code>SfMPerspectiveCameras</code> class initialized with the corresponding rotation and translation matrices <code>R_absolute</code> and <code>T_absolute</code> that define the extrinsic parameters $g = (R, T); R \in SO(3); T \in \mathbb{R}^3$. In order to ensure that <code>R_absolute</code> is a valid rotation matrix, we represent it using an exponential map (implemented with <code>so3_exp_map</code>) of the axis-angle representation of the rotation <code>log_R_absolute</code>.</p>
|
||||
<p>Note that the solution to this problem could only be recovered up to an unknown global rigid transformation $g_{glob} \in SE(3)$. Thus, for simplicity, we assume knowledge of the absolute extrinsics of the first camera $g_0$. We set $g_0$ as a trivial camera $g_0 = (I, \vec{0})$.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-Modules">0. Install and Import Modules<a class="anchor-link" href="#0.-Install-and-Import-Modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -121,19 +124,25 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -150,11 +159,11 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># imports</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms.so3</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="n">so3_exponential_map</span><span class="p">,</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms.so3</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">so3_exp_map</span><span class="p">,</span>
|
||||
<span class="n">so3_relative_angle</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer.cameras</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer.cameras</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">SfMPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
@ -176,7 +185,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting the camera scene, and the ground truth camera positions:</p>
|
||||
</div>
|
||||
@ -188,7 +198,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/camera_visualization.py
|
||||
<span class="kn">from</span> <span class="nn">camera_visualization</span> <span class="k">import</span> <span class="n">plot_camera_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">camera_visualization</span> <span class="kn">import</span> <span class="n">plot_camera_scene</span>
|
||||
|
||||
<span class="o">!</span>mkdir data
|
||||
<span class="o">!</span>wget -P data https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/data/camera_graph.pth
|
||||
@ -198,7 +208,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -216,7 +227,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Set-up-Cameras-and-load-ground-truth-positions">1. Set up Cameras and load ground truth positions<a class="anchor-link" href="#1.-Set-up-Cameras-and-load-ground-truth-positions">¶</a></h2>
|
||||
</div>
|
||||
@ -256,7 +268,8 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Define-optimization-functions">2. Define optimization functions<a class="anchor-link" href="#2.-Define-optimization-functions">¶</a></h2><h3 id="Relative-cameras-and-camera-distance">Relative cameras and camera distance<a class="anchor-link" href="#Relative-cameras-and-camera-distance">¶</a></h3><p>We now define two functions crucial for the optimization.</p>
|
||||
<p><strong><code>calc_camera_distance</code></strong> compares a pair of cameras. This function is important as it defines the loss that we are minimizing. The method utilizes the <code>so3_relative_angle</code> function from the SO3 API.</p>
|
||||
@ -318,12 +331,13 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Optimization">3. Optimization<a class="anchor-link" href="#3.-Optimization">¶</a></h2><p>Finally, we start the optimization of the absolute cameras.</p>
|
||||
<p>We use SGD with momentum and optimize over <code>log_R_absolute</code> and <code>T_absolute</code>.</p>
|
||||
<p>As mentioned earlier, <code>log_R_absolute</code> is the axis angle representation of the rotation part of our absolute cameras. We can obtain the 3x3 rotation matrix <code>R_absolute</code> that corresponds to <code>log_R_absolute</code> with:</p>
|
||||
<p><code>R_absolute = so3_exponential_map(log_R_absolute)</code></p>
|
||||
<p><code>R_absolute = so3_exp_map(log_R_absolute)</code></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -354,7 +368,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="n">camera_mask</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">0.</span>
|
||||
|
||||
<span class="c1"># init the optimizer</span>
|
||||
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">([</span><span class="n">log_R_absolute</span><span class="p">,</span> <span class="n">T_absolute</span><span class="p">],</span> <span class="n">lr</span><span class="o">=.</span><span class="mi">1</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">)</span>
|
||||
<span class="n">optimizer</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">([</span><span class="n">log_R_absolute</span><span class="p">,</span> <span class="n">T_absolute</span><span class="p">],</span> <span class="n">lr</span><span class="o">=</span><span class="mf">.1</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># run the optimization</span>
|
||||
<span class="n">n_iter</span> <span class="o">=</span> <span class="mi">2000</span> <span class="c1"># fix the number of iterations</span>
|
||||
@ -365,7 +379,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="c1"># compute the absolute camera rotations as </span>
|
||||
<span class="c1"># an exponential map of the logarithms (=axis-angles)</span>
|
||||
<span class="c1"># of the absolute rotations</span>
|
||||
<span class="n">R_absolute</span> <span class="o">=</span> <span class="n">so3_exponential_map</span><span class="p">(</span><span class="n">log_R_absolute</span> <span class="o">*</span> <span class="n">camera_mask</span><span class="p">)</span>
|
||||
<span class="n">R_absolute</span> <span class="o">=</span> <span class="n">so3_exp_map</span><span class="p">(</span><span class="n">log_R_absolute</span> <span class="o">*</span> <span class="n">camera_mask</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># get the current absolute cameras</span>
|
||||
<span class="n">cameras_absolute</span> <span class="o">=</span> <span class="n">SfMPerspectiveCameras</span><span class="p">(</span>
|
||||
@ -374,7 +388,7 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
<span class="n">device</span> <span class="o">=</span> <span class="n">device</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># compute the relative cameras as a compositon of the absolute cameras</span>
|
||||
<span class="c1"># compute the relative cameras as a composition of the absolute cameras</span>
|
||||
<span class="n">cameras_relative_composed</span> <span class="o">=</span> \
|
||||
<span class="n">get_relative_camera</span><span class="p">(</span><span class="n">cameras_absolute</span><span class="p">,</span> <span class="n">relative_edges</span><span class="p">)</span>
|
||||
|
||||
@ -401,10 +415,11 @@ where $d(g_i, g_j)$ is a suitable metric that compares the extrinsics of cameras
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Conclusion">4. Conclusion<a class="anchor-link" href="#4.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to initialize a batch of SfM Cameras, set up loss functions for bundle adjustment, and run an optimization loop.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Camera-position-optimization-using-differentiable-rendering">Camera position optimization using differentiable rendering<a class="anchor-link" href="#Camera-position-optimization-using-differentiable-rendering">¶</a></h1><p>In this tutorial we will learn the [x, y, z] position of a camera given a reference image using differentiable rendering.</p>
|
||||
<p>We will first initialize a renderer with a starting position for the camera. We will then use this to generate an image, compute a loss with the reference image, and finally backpropagate through the entire pipeline to update the position of the camera.</p>
|
||||
@ -97,16 +98,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-import-modules">0. Install and import modules<a class="anchor-link" href="#0.-Install-and-import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -115,19 +118,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -145,24 +154,24 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">import</span> <span class="nn">imageio</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage</span> <span class="k">import</span> <span class="n">img_as_ubyte</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage</span> <span class="kn">import</span> <span class="n">img_as_ubyte</span>
|
||||
|
||||
<span class="c1"># io utils</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_obj</span>
|
||||
|
||||
<span class="c1"># datastructures</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
|
||||
<span class="c1"># 3D transformations functions</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms</span> <span class="k">import</span> <span class="n">Rotate</span><span class="p">,</span> <span class="n">Translate</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms</span> <span class="kn">import</span> <span class="n">Rotate</span><span class="p">,</span> <span class="n">Translate</span>
|
||||
|
||||
<span class="c1"># rendering components</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span> <span class="n">look_at_view_transform</span><span class="p">,</span> <span class="n">look_at_rotation</span><span class="p">,</span>
|
||||
<span class="n">RasterizationSettings</span><span class="p">,</span> <span class="n">MeshRenderer</span><span class="p">,</span> <span class="n">MeshRasterizer</span><span class="p">,</span> <span class="n">BlendParams</span><span class="p">,</span>
|
||||
<span class="n">SoftSilhouetteShader</span><span class="p">,</span> <span class="n">HardPhongShader</span><span class="p">,</span> <span class="n">PointLights</span><span class="p">,</span> <span class="n">TexturesVertex</span><span class="p">,</span>
|
||||
@ -173,14 +182,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-the-Obj">1. Load the Obj<a class="anchor-link" href="#1.-Load-the-Obj">¶</a></h2><p>We will load an obj file and create a <strong>Meshes</strong> object. <strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with <strong>batches of meshes of different sizes</strong>. It has several useful class methods which are used in the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If you are running this notebook locally after cloning the PyTorch3D repository, the mesh will already be available. <strong>If using Google Colab, fetch the mesh and save it at the path <code>data/</code></strong>:</p>
|
||||
</div>
|
||||
@ -230,17 +241,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Optimization-setup">2. Optimization setup<a class="anchor-link" href="#2.-Optimization-setup">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h3><p>A <strong>renderer</strong> in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>For optimizing the camera position we will use a renderer which produces a <strong>silhouette</strong> of the object only and does not apply any <strong>lighting</strong> or <strong>shading</strong>. We will also initialize another renderer which applies full <strong>phong shading</strong> and use this for visualizing the outputs.</p>
|
||||
<h3 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h3><p>A <strong>renderer</strong> in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>For optimizing the camera position we will use a renderer which produces a <strong>silhouette</strong> of the object only and does not apply any <strong>lighting</strong> or <strong>shading</strong>. We will also initialize another renderer which applies full <strong>Phong shading</strong> and use this for visualizing the outputs.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -277,7 +290,7 @@
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># We will also create a phong renderer. This is simpler and only needs to render one face per pixel.</span>
|
||||
<span class="c1"># We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.</span>
|
||||
<span class="n">raster_settings</span> <span class="o">=</span> <span class="n">RasterizationSettings</span><span class="p">(</span>
|
||||
<span class="n">image_size</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span>
|
||||
<span class="n">blur_radius</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span>
|
||||
@ -298,7 +311,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-a-reference-image">Create a reference image<a class="anchor-link" href="#Create-a-reference-image">¶</a></h3><p>We will first position the teapot and generate an image. We use helper functions to rotate the teapot to a desired viewpoint. Then we can use the renderers to produce an image. Here we will use both renderers and visualize the silhouette and full shaded image.</p>
|
||||
<p>The world coordinate system is defined as +Y up, +X left and +Z in. The teapot in world coordinates has the spout pointing to the left.</p>
|
||||
@ -320,15 +334,15 @@
|
||||
<span class="n">R</span><span class="p">,</span> <span class="n">T</span> <span class="o">=</span> <span class="n">look_at_view_transform</span><span class="p">(</span><span class="n">distance</span><span class="p">,</span> <span class="n">elevation</span><span class="p">,</span> <span class="n">azimuth</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Render the teapot providing the values of R and T. </span>
|
||||
<span class="n">silhouete</span> <span class="o">=</span> <span class="n">silhouette_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
<span class="n">silhouette</span> <span class="o">=</span> <span class="n">silhouette_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
<span class="n">image_ref</span> <span class="o">=</span> <span class="n">phong_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
|
||||
<span class="n">silhouete</span> <span class="o">=</span> <span class="n">silhouete</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
<span class="n">silhouette</span> <span class="o">=</span> <span class="n">silhouette</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
<span class="n">image_ref</span> <span class="o">=</span> <span class="n">image_ref</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">silhouete</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()[</span><span class="o">...</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="c1"># only plot the alpha channel of the RGBA image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">silhouette</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()[</span><span class="o">...</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="c1"># only plot the alpha channel of the RGBA image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="kc">False</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">image_ref</span><span class="o">.</span><span class="n">squeeze</span><span class="p">())</span>
|
||||
@ -339,7 +353,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Set-up-a-basic-model">Set up a basic model<a class="anchor-link" href="#Set-up-a-basic-model">¶</a></h3><p>Here we create a simple model class and initialize a parameter for the camera position.</p>
|
||||
</div>
|
||||
@ -351,7 +366,7 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">Model</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
|
||||
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">meshes</span><span class="p">,</span> <span class="n">renderer</span><span class="p">,</span> <span class="n">image_ref</span><span class="p">):</span>
|
||||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">meshes</span><span class="p">,</span> <span class="n">renderer</span><span class="p">,</span> <span class="n">image_ref</span><span class="p">):</span>
|
||||
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
|
||||
<span class="bp">self</span><span class="o">.</span><span class="n">meshes</span> <span class="o">=</span> <span class="n">meshes</span>
|
||||
<span class="bp">self</span><span class="o">.</span><span class="n">device</span> <span class="o">=</span> <span class="n">meshes</span><span class="o">.</span><span class="n">device</span>
|
||||
@ -368,7 +383,7 @@
|
||||
<span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||||
|
||||
<span class="c1"># Render the image using the updated camera position. Based on the new position of the </span>
|
||||
<span class="c1"># camer we calculate the rotation and translation matrices</span>
|
||||
<span class="c1"># camera we calculate the rotation and translation matrices</span>
|
||||
<span class="n">R</span> <span class="o">=</span> <span class="n">look_at_rotation</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">camera_position</span><span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="p">:],</span> <span class="n">device</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (1, 3, 3)</span>
|
||||
<span class="n">T</span> <span class="o">=</span> <span class="o">-</span><span class="n">torch</span><span class="o">.</span><span class="n">bmm</span><span class="p">(</span><span class="n">R</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">camera_position</span><span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="p">:,</span> <span class="kc">None</span><span class="p">])[:,</span> <span class="p">:,</span> <span class="mi">0</span><span class="p">]</span> <span class="c1"># (1, 3)</span>
|
||||
|
||||
@ -384,7 +399,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Initialize-the-model-and-optimizer">3. Initialize the model and optimizer<a class="anchor-link" href="#3.-Initialize-the-model-and-optimizer">¶</a></h2><p>Now we can create an instance of the <strong>model</strong> above and set up an <strong>optimizer</strong> for the camera position parameter.</p>
|
||||
</div>
|
||||
@ -410,7 +426,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Visualize-the-starting-position-and-the-reference-position">Visualize the starting position and the reference position<a class="anchor-link" href="#Visualize-the-starting-position-and-the-reference-position">¶</a></h3>
|
||||
</div>
|
||||
@ -439,7 +456,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Run-the-optimization">4. Run the optimization<a class="anchor-link" href="#4.-Run-the-optimization">¶</a></h2><p>We run several iterations of the forward and backward pass and save outputs every 10 iterations. When this has finished take a look at <code>./teapot_optimization_demo.gif</code> for a cool gif of the optimization process!</p>
|
||||
</div>
|
||||
@ -474,7 +492,6 @@
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">image</span><span class="p">[</span><span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">])</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">"iter: </span><span class="si">%d</span><span class="s2">, loss: </span><span class="si">%0.2f</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">loss</span><span class="o">.</span><span class="n">data</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
|
||||
<span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||||
@ -484,10 +501,11 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Conclusion">5. Conclusion<a class="anchor-link" href="#5.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to <strong>load</strong> a mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an <strong>Renderer</strong> consisting of a <strong>Rasterizer</strong> and a <strong>Shader</strong>, set up an optimization loop including a <strong>Model</strong> and a <strong>loss function</strong>, and run the optimization.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Camera-position-optimization-using-differentiable-rendering">Camera position optimization using differentiable rendering<a class="anchor-link" href="#Camera-position-optimization-using-differentiable-rendering">¶</a></h1><p>In this tutorial we will learn the [x, y, z] position of a camera given a reference image using differentiable rendering.</p>
|
||||
<p>We will first initialize a renderer with a starting position for the camera. We will then use this to generate an image, compute a loss with the reference image, and finally backpropagate through the entire pipeline to update the position of the camera.</p>
|
||||
@ -97,16 +98,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-import-modules">0. Install and import modules<a class="anchor-link" href="#0.-Install-and-import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -115,19 +118,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -145,24 +154,24 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">import</span> <span class="nn">imageio</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage</span> <span class="k">import</span> <span class="n">img_as_ubyte</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage</span> <span class="kn">import</span> <span class="n">img_as_ubyte</span>
|
||||
|
||||
<span class="c1"># io utils</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_obj</span>
|
||||
|
||||
<span class="c1"># datastructures</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
|
||||
<span class="c1"># 3D transformations functions</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms</span> <span class="k">import</span> <span class="n">Rotate</span><span class="p">,</span> <span class="n">Translate</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.transforms</span> <span class="kn">import</span> <span class="n">Rotate</span><span class="p">,</span> <span class="n">Translate</span>
|
||||
|
||||
<span class="c1"># rendering components</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span> <span class="n">look_at_view_transform</span><span class="p">,</span> <span class="n">look_at_rotation</span><span class="p">,</span>
|
||||
<span class="n">RasterizationSettings</span><span class="p">,</span> <span class="n">MeshRenderer</span><span class="p">,</span> <span class="n">MeshRasterizer</span><span class="p">,</span> <span class="n">BlendParams</span><span class="p">,</span>
|
||||
<span class="n">SoftSilhouetteShader</span><span class="p">,</span> <span class="n">HardPhongShader</span><span class="p">,</span> <span class="n">PointLights</span><span class="p">,</span> <span class="n">TexturesVertex</span><span class="p">,</span>
|
||||
@ -173,14 +182,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-the-Obj">1. Load the Obj<a class="anchor-link" href="#1.-Load-the-Obj">¶</a></h2><p>We will load an obj file and create a <strong>Meshes</strong> object. <strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with <strong>batches of meshes of different sizes</strong>. It has several useful class methods which are used in the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If you are running this notebook locally after cloning the PyTorch3D repository, the mesh will already be available. <strong>If using Google Colab, fetch the mesh and save it at the path <code>data/</code></strong>:</p>
|
||||
</div>
|
||||
@ -230,17 +241,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Optimization-setup">2. Optimization setup<a class="anchor-link" href="#2.-Optimization-setup">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h3><p>A <strong>renderer</strong> in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>For optimizing the camera position we will use a renderer which produces a <strong>silhouette</strong> of the object only and does not apply any <strong>lighting</strong> or <strong>shading</strong>. We will also initialize another renderer which applies full <strong>phong shading</strong> and use this for visualizing the outputs.</p>
|
||||
<h3 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h3><p>A <strong>renderer</strong> in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>For optimizing the camera position we will use a renderer which produces a <strong>silhouette</strong> of the object only and does not apply any <strong>lighting</strong> or <strong>shading</strong>. We will also initialize another renderer which applies full <strong>Phong shading</strong> and use this for visualizing the outputs.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -277,7 +290,7 @@
|
||||
<span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># We will also create a phong renderer. This is simpler and only needs to render one face per pixel.</span>
|
||||
<span class="c1"># We will also create a Phong renderer. This is simpler and only needs to render one face per pixel.</span>
|
||||
<span class="n">raster_settings</span> <span class="o">=</span> <span class="n">RasterizationSettings</span><span class="p">(</span>
|
||||
<span class="n">image_size</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span>
|
||||
<span class="n">blur_radius</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span>
|
||||
@ -298,7 +311,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-a-reference-image">Create a reference image<a class="anchor-link" href="#Create-a-reference-image">¶</a></h3><p>We will first position the teapot and generate an image. We use helper functions to rotate the teapot to a desired viewpoint. Then we can use the renderers to produce an image. Here we will use both renderers and visualize the silhouette and full shaded image.</p>
|
||||
<p>The world coordinate system is defined as +Y up, +X left and +Z in. The teapot in world coordinates has the spout pointing to the left.</p>
|
||||
@ -320,15 +334,15 @@
|
||||
<span class="n">R</span><span class="p">,</span> <span class="n">T</span> <span class="o">=</span> <span class="n">look_at_view_transform</span><span class="p">(</span><span class="n">distance</span><span class="p">,</span> <span class="n">elevation</span><span class="p">,</span> <span class="n">azimuth</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Render the teapot providing the values of R and T. </span>
|
||||
<span class="n">silhouete</span> <span class="o">=</span> <span class="n">silhouette_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
<span class="n">silhouette</span> <span class="o">=</span> <span class="n">silhouette_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
<span class="n">image_ref</span> <span class="o">=</span> <span class="n">phong_renderer</span><span class="p">(</span><span class="n">meshes_world</span><span class="o">=</span><span class="n">teapot_mesh</span><span class="p">,</span> <span class="n">R</span><span class="o">=</span><span class="n">R</span><span class="p">,</span> <span class="n">T</span><span class="o">=</span><span class="n">T</span><span class="p">)</span>
|
||||
|
||||
<span class="n">silhouete</span> <span class="o">=</span> <span class="n">silhouete</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
<span class="n">silhouette</span> <span class="o">=</span> <span class="n">silhouette</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
<span class="n">image_ref</span> <span class="o">=</span> <span class="n">image_ref</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">silhouete</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()[</span><span class="o">...</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="c1"># only plot the alpha channel of the RGBA image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">silhouette</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()[</span><span class="o">...</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="c1"># only plot the alpha channel of the RGBA image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="kc">False</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">image_ref</span><span class="o">.</span><span class="n">squeeze</span><span class="p">())</span>
|
||||
@ -339,7 +353,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Set-up-a-basic-model">Set up a basic model<a class="anchor-link" href="#Set-up-a-basic-model">¶</a></h3><p>Here we create a simple model class and initialize a parameter for the camera position.</p>
|
||||
</div>
|
||||
@ -351,7 +366,7 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">Model</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
|
||||
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">meshes</span><span class="p">,</span> <span class="n">renderer</span><span class="p">,</span> <span class="n">image_ref</span><span class="p">):</span>
|
||||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">meshes</span><span class="p">,</span> <span class="n">renderer</span><span class="p">,</span> <span class="n">image_ref</span><span class="p">):</span>
|
||||
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
|
||||
<span class="bp">self</span><span class="o">.</span><span class="n">meshes</span> <span class="o">=</span> <span class="n">meshes</span>
|
||||
<span class="bp">self</span><span class="o">.</span><span class="n">device</span> <span class="o">=</span> <span class="n">meshes</span><span class="o">.</span><span class="n">device</span>
|
||||
@ -368,7 +383,7 @@
|
||||
<span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||||
|
||||
<span class="c1"># Render the image using the updated camera position. Based on the new position of the </span>
|
||||
<span class="c1"># camer we calculate the rotation and translation matrices</span>
|
||||
<span class="c1"># camera we calculate the rotation and translation matrices</span>
|
||||
<span class="n">R</span> <span class="o">=</span> <span class="n">look_at_rotation</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">camera_position</span><span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="p">:],</span> <span class="n">device</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (1, 3, 3)</span>
|
||||
<span class="n">T</span> <span class="o">=</span> <span class="o">-</span><span class="n">torch</span><span class="o">.</span><span class="n">bmm</span><span class="p">(</span><span class="n">R</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">camera_position</span><span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="p">:,</span> <span class="kc">None</span><span class="p">])[:,</span> <span class="p">:,</span> <span class="mi">0</span><span class="p">]</span> <span class="c1"># (1, 3)</span>
|
||||
|
||||
@ -384,7 +399,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Initialize-the-model-and-optimizer">3. Initialize the model and optimizer<a class="anchor-link" href="#3.-Initialize-the-model-and-optimizer">¶</a></h2><p>Now we can create an instance of the <strong>model</strong> above and set up an <strong>optimizer</strong> for the camera position parameter.</p>
|
||||
</div>
|
||||
@ -410,7 +426,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Visualize-the-starting-position-and-the-reference-position">Visualize the starting position and the reference position<a class="anchor-link" href="#Visualize-the-starting-position-and-the-reference-position">¶</a></h3>
|
||||
</div>
|
||||
@ -439,7 +456,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Run-the-optimization">4. Run the optimization<a class="anchor-link" href="#4.-Run-the-optimization">¶</a></h2><p>We run several iterations of the forward and backward pass and save outputs every 10 iterations. When this has finished take a look at <code>./teapot_optimization_demo.gif</code> for a cool gif of the optimization process!</p>
|
||||
</div>
|
||||
@ -474,7 +492,6 @@
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">image</span><span class="p">[</span><span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">])</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">"iter: </span><span class="si">%d</span><span class="s2">, loss: </span><span class="si">%0.2f</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">loss</span><span class="o">.</span><span class="n">data</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
|
||||
<span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||||
@ -484,10 +501,11 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Conclusion">5. Conclusion<a class="anchor-link" href="#5.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to <strong>load</strong> a mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an <strong>Renderer</strong> consisting of a <strong>Rasterizer</strong> and a <strong>Shader</strong>, set up an optimization loop including a <strong>Model</strong> and a <strong>loss function</strong>, and run the optimization.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Dataloaders-for-ShapeNetCore-and-R2N2">Dataloaders for ShapeNetCore and R2N2<a class="anchor-link" href="#Dataloaders-for-ShapeNetCore-and-R2N2">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -96,16 +97,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-import-modules">0. Install and import modules<a class="anchor-link" href="#0.-Install-and-import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -114,19 +117,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -144,13 +153,13 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.datasets</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.datasets</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">R2N2</span><span class="p">,</span>
|
||||
<span class="n">ShapeNetCore</span><span class="p">,</span>
|
||||
<span class="n">collate_batched_meshes</span><span class="p">,</span>
|
||||
<span class="n">render_cubified_voxels</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">OpenGLPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
<span class="n">RasterizationSettings</span><span class="p">,</span>
|
||||
@ -158,8 +167,8 @@
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">torch.utils.data</span> <span class="k">import</span> <span class="n">DataLoader</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">torch.utils.data</span> <span class="kn">import</span> <span class="n">DataLoader</span>
|
||||
|
||||
<span class="c1"># add path for demo utils functions </span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
@ -171,7 +180,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -183,14 +193,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running locally uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -208,14 +219,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-the-datasets">1. Load the datasets<a class="anchor-link" href="#1.-Load-the-datasets">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If you haven't already downloaded the ShapeNetCore dataset, first do that following the instructions here: <a href="https://www.shapenet.org/">https://www.shapenet.org/</a>. ShapeNetCore is a subset of the ShapeNet dataset. In PyTorch3D we support both version 1 (57 categories) and version 2 (55 categories).</p>
|
||||
<p>Then modify <code>SHAPENET_PATH</code> below to you local path to the ShapeNetCore dataset folder.</p>
|
||||
@ -242,7 +255,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>The R2N2 dataset can be downloaded using the instructions here: <a href="http://3d-r2n2.stanford.edu/">http://3d-r2n2.stanford.edu/</a>. Look at the links for <code>ShapeNetRendering</code> and <code>ShapeNetVox32</code>. The R2N2 dataset contains 13 categories that are a subset of the ShapeNetCore v.1
|
||||
dataset. The R2N2 dataset also contains its own 24 renderings of each object and voxelized models.</p>
|
||||
@ -264,9 +278,10 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: <a href="http://image-net.org/download-API">http://image-net.org/download-API</a>), its model id, and its vertices and faces.</p>
|
||||
<p>We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: <a href="http://image-net.org/download-API">http://image-net.org/download-API</a>), its model id, and its vertices and faces.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -285,7 +300,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can use its vertices and faces to form a <code>Meshes</code> object which is a PyTorch3D datastructure for working with batched meshes.</p>
|
||||
</div>
|
||||
@ -308,7 +324,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>With R2N2, we can further examine R2N2's original renderings. For instance, if we would like to see the second and third views of the eleventh objects in the R2N2 dataset, we can do the following:</p>
|
||||
</div>
|
||||
@ -327,18 +344,20 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Use-the-datasets-with-torch.utils.data.DataLoader">2. Use the datasets with <code>torch.utils.data.DataLoader</code><a class="anchor-link" href="#2.-Use-the-datasets-with-torch.utils.data.DataLoader">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Training deep learning models, usually requires passing in batches of inputs. The <code>torch.utils.data.DataLoader</code> from Pytorch helps us do this. PyTorch3D provides a function <code>collate_batched_meshes</code> to group the input meshes into a single <code>Meshes</code> object which represents the batch. The <code>Meshes</code> datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. <code>graph_conv</code>).</p>
|
||||
<p>Training deep learning models, usually requires passing in batches of inputs. The <code>torch.utils.data.DataLoader</code> from PyTorch helps us do this. PyTorch3D provides a function <code>collate_batched_meshes</code> to group the input meshes into a single <code>Meshes</code> object which represents the batch. The <code>Meshes</code> datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. <code>graph_conv</code>).</p>
|
||||
<p>For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.</p>
|
||||
<p><strong>NOTE</strong>: All models in the <code>val</code> split of R2N2 have 24 views, but there are 8 models that split their 24 views between <code>train</code> and <code>test</code> splits, in which case <code>collate_batched_meshes</code> will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting <code>return_all_views = False</code>.</p>
|
||||
<p><strong>NOTE</strong>: All models in the <code>val</code> split of R2N2 have 24 views, but there are 8 models that split their 24 views between <code>train</code> and <code>test</code> splits, in which case <code>collate_batched_meshes</code> will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting <code>return_all_views = False</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -356,7 +375,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Let's visualize all the views (one for each model) in the batch:</p>
|
||||
</div>
|
||||
@ -377,14 +397,16 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Render-ShapeNetCore-models-with-PyTorch3D's-differntiable-renderer">3. Render ShapeNetCore models with PyTorch3D's differntiable renderer<a class="anchor-link" href="#3.-Render-ShapeNetCore-models-with-PyTorch3D's-differntiable-renderer">¶</a></h2>
|
||||
<h2 id="3.-Render-ShapeNetCore-models-with-PyTorch3D's-differentiable-renderer">3. Render ShapeNetCore models with PyTorch3D's differentiable renderer<a class="anchor-link" href="#3.-Render-ShapeNetCore-models-with-PyTorch3D's-differentiable-renderer">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Both <code>ShapeNetCore</code> and <code>R2N2</code> dataloaders have customized <code>render</code> functions that support rendering models by specifying their model ids, categories or indices using PyTorch3D's differentiable renderer implementation.</p>
|
||||
</div>
|
||||
@ -406,7 +428,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>First we will try to render three models by their model ids:</p>
|
||||
</div>
|
||||
@ -435,7 +458,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Suppose we would like to render the first three models in the dataset, we can render models by their indices:</p>
|
||||
</div>
|
||||
@ -460,7 +484,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Alternatively, if we are not interested in any particular models but would like see random models from some specific categories, we can do that by specifying <code>categories</code> and <code>sample_nums</code>. For example, if we would like to render 2 models from the category "faucet" and 3 models from the category "chair", we can do the following:</p>
|
||||
</div>
|
||||
@ -486,7 +511,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If we are not interested in any particular categories and just would like to render some random models from the whole dataset, we can set the number of models to be rendered in <code>sample_nums</code> and not specify any <code>categories</code>:</p>
|
||||
</div>
|
||||
@ -511,21 +537,24 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Render-R2N2-models-with-the-same-orientations-as-the-original-renderings-in-the-dataset">4. Render R2N2 models with the same orientations as the original renderings in the dataset<a class="anchor-link" href="#4.-Render-R2N2-models-with-the-same-orientations-as-the-original-renderings-in-the-dataset">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can render R2N2 models the same way as we rendered ShapeNetCore models above. In addition, we can also render R2N2 models with the same orientations as the original renderings in the dataset. For this we will use R2N2's customized <code>render</code> function and a different type of PyTorch3D camera called <code>BlenderCamera</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example, we will render the seventh model with the same orientations as its second and third views. First we will retrieve R2N2's original renderings to compare with the result.</p>
|
||||
</div>
|
||||
@ -544,9 +573,10 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Next, we will visualize PyTorch3d's renderings:</p>
|
||||
<p>Next, we will visualize PyTorch3D's renderings:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -569,21 +599,24 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Visualize-R2N2-models'-voxels">5. Visualize R2N2 models' voxels<a class="anchor-link" href="#5.-Visualize-R2N2-models'-voxels">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>R2N2 dataloader also returns models' voxels. We can visualize them by utilizing R2N2's <code>render_vox_to_mesh</code> function. This will cubify the voxels to a Meshes object, which will then be rendered.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example we will visualize the tenth model in the dataset with the same orientation of its second and third views. First we will retrieve R2N2's original renderings to compare with the result.</p>
|
||||
</div>
|
||||
@ -603,7 +636,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Next, we will pass the voxels to <code>render_vox_to_mesh</code>:</p>
|
||||
</div>
|
||||
@ -621,4 +655,4 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Dataloaders-for-ShapeNetCore-and-R2N2">Dataloaders for ShapeNetCore and R2N2<a class="anchor-link" href="#Dataloaders-for-ShapeNetCore-and-R2N2">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -96,16 +97,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-import-modules">0. Install and import modules<a class="anchor-link" href="#0.-Install-and-import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -114,19 +117,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -144,13 +153,13 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.datasets</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.datasets</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">R2N2</span><span class="p">,</span>
|
||||
<span class="n">ShapeNetCore</span><span class="p">,</span>
|
||||
<span class="n">collate_batched_meshes</span><span class="p">,</span>
|
||||
<span class="n">render_cubified_voxels</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">OpenGLPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
<span class="n">RasterizationSettings</span><span class="p">,</span>
|
||||
@ -158,8 +167,8 @@
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">torch.utils.data</span> <span class="k">import</span> <span class="n">DataLoader</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">torch.utils.data</span> <span class="kn">import</span> <span class="n">DataLoader</span>
|
||||
|
||||
<span class="c1"># add path for demo utils functions </span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
@ -171,7 +180,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -183,14 +193,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running locally uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -208,14 +219,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-the-datasets">1. Load the datasets<a class="anchor-link" href="#1.-Load-the-datasets">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If you haven't already downloaded the ShapeNetCore dataset, first do that following the instructions here: <a href="https://www.shapenet.org/">https://www.shapenet.org/</a>. ShapeNetCore is a subset of the ShapeNet dataset. In PyTorch3D we support both version 1 (57 categories) and version 2 (55 categories).</p>
|
||||
<p>Then modify <code>SHAPENET_PATH</code> below to you local path to the ShapeNetCore dataset folder.</p>
|
||||
@ -242,7 +255,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>The R2N2 dataset can be downloaded using the instructions here: <a href="http://3d-r2n2.stanford.edu/">http://3d-r2n2.stanford.edu/</a>. Look at the links for <code>ShapeNetRendering</code> and <code>ShapeNetVox32</code>. The R2N2 dataset contains 13 categories that are a subset of the ShapeNetCore v.1
|
||||
dataset. The R2N2 dataset also contains its own 24 renderings of each object and voxelized models.</p>
|
||||
@ -264,9 +278,10 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalend to wnid described in ImageNet's API: <a href="http://image-net.org/download-API">http://image-net.org/download-API</a>), its model id, and its vertices and faces.</p>
|
||||
<p>We can retrieve a model by indexing into the loaded dataset. For both ShapeNetCore and R2N2, we can examine the category this model belongs to (in the form of a synset id, equivalent to wnid described in ImageNet's API: <a href="http://image-net.org/download-API">http://image-net.org/download-API</a>), its model id, and its vertices and faces.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -285,7 +300,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can use its vertices and faces to form a <code>Meshes</code> object which is a PyTorch3D datastructure for working with batched meshes.</p>
|
||||
</div>
|
||||
@ -308,7 +324,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>With R2N2, we can further examine R2N2's original renderings. For instance, if we would like to see the second and third views of the eleventh objects in the R2N2 dataset, we can do the following:</p>
|
||||
</div>
|
||||
@ -327,18 +344,20 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Use-the-datasets-with-torch.utils.data.DataLoader">2. Use the datasets with <code>torch.utils.data.DataLoader</code><a class="anchor-link" href="#2.-Use-the-datasets-with-torch.utils.data.DataLoader">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Training deep learning models, usually requires passing in batches of inputs. The <code>torch.utils.data.DataLoader</code> from Pytorch helps us do this. PyTorch3D provides a function <code>collate_batched_meshes</code> to group the input meshes into a single <code>Meshes</code> object which represents the batch. The <code>Meshes</code> datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. <code>graph_conv</code>).</p>
|
||||
<p>Training deep learning models, usually requires passing in batches of inputs. The <code>torch.utils.data.DataLoader</code> from PyTorch helps us do this. PyTorch3D provides a function <code>collate_batched_meshes</code> to group the input meshes into a single <code>Meshes</code> object which represents the batch. The <code>Meshes</code> datastructure can then be used directly by other PyTorch3D ops which might be part of the deep learning model (e.g. <code>graph_conv</code>).</p>
|
||||
<p>For R2N2, if all the models in the batch have the same number of views, the views, rotation matrices, translation matrices, intrinsic matrices and voxels will also be stacked into batched tensors.</p>
|
||||
<p><strong>NOTE</strong>: All models in the <code>val</code> split of R2N2 have 24 views, but there are 8 models that split their 24 views between <code>train</code> and <code>test</code> splits, in which case <code>collate_batched_meshes</code> will only be able to join the matrices, views and voxels as lists. However, this can be avoided by laoding only one view of each model by setting <code>return_all_views = False</code>.</p>
|
||||
<p><strong>NOTE</strong>: All models in the <code>val</code> split of R2N2 have 24 views, but there are 8 models that split their 24 views between <code>train</code> and <code>test</code> splits, in which case <code>collate_batched_meshes</code> will only be able to join the matrices, views and voxels as lists. However, this can be avoided by loading only one view of each model by setting <code>return_all_views = False</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -356,7 +375,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Let's visualize all the views (one for each model) in the batch:</p>
|
||||
</div>
|
||||
@ -377,14 +397,16 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Render-ShapeNetCore-models-with-PyTorch3D's-differntiable-renderer">3. Render ShapeNetCore models with PyTorch3D's differntiable renderer<a class="anchor-link" href="#3.-Render-ShapeNetCore-models-with-PyTorch3D's-differntiable-renderer">¶</a></h2>
|
||||
<h2 id="3.-Render-ShapeNetCore-models-with-PyTorch3D's-differentiable-renderer">3. Render ShapeNetCore models with PyTorch3D's differentiable renderer<a class="anchor-link" href="#3.-Render-ShapeNetCore-models-with-PyTorch3D's-differentiable-renderer">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Both <code>ShapeNetCore</code> and <code>R2N2</code> dataloaders have customized <code>render</code> functions that support rendering models by specifying their model ids, categories or indices using PyTorch3D's differentiable renderer implementation.</p>
|
||||
</div>
|
||||
@ -406,7 +428,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>First we will try to render three models by their model ids:</p>
|
||||
</div>
|
||||
@ -435,7 +458,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Suppose we would like to render the first three models in the dataset, we can render models by their indices:</p>
|
||||
</div>
|
||||
@ -460,7 +484,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Alternatively, if we are not interested in any particular models but would like see random models from some specific categories, we can do that by specifying <code>categories</code> and <code>sample_nums</code>. For example, if we would like to render 2 models from the category "faucet" and 3 models from the category "chair", we can do the following:</p>
|
||||
</div>
|
||||
@ -486,7 +511,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If we are not interested in any particular categories and just would like to render some random models from the whole dataset, we can set the number of models to be rendered in <code>sample_nums</code> and not specify any <code>categories</code>:</p>
|
||||
</div>
|
||||
@ -511,21 +537,24 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Render-R2N2-models-with-the-same-orientations-as-the-original-renderings-in-the-dataset">4. Render R2N2 models with the same orientations as the original renderings in the dataset<a class="anchor-link" href="#4.-Render-R2N2-models-with-the-same-orientations-as-the-original-renderings-in-the-dataset">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can render R2N2 models the same way as we rendered ShapeNetCore models above. In addition, we can also render R2N2 models with the same orientations as the original renderings in the dataset. For this we will use R2N2's customized <code>render</code> function and a different type of PyTorch3D camera called <code>BlenderCamera</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example, we will render the seventh model with the same orientations as its second and third views. First we will retrieve R2N2's original renderings to compare with the result.</p>
|
||||
</div>
|
||||
@ -544,9 +573,10 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Next, we will visualize PyTorch3d's renderings:</p>
|
||||
<p>Next, we will visualize PyTorch3D's renderings:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -569,21 +599,24 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Visualize-R2N2-models'-voxels">5. Visualize R2N2 models' voxels<a class="anchor-link" href="#5.-Visualize-R2N2-models'-voxels">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>R2N2 dataloader also returns models' voxels. We can visualize them by utilizing R2N2's <code>render_vox_to_mesh</code> function. This will cubify the voxels to a Meshes object, which will then be rendered.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example we will visualize the tenth model in the dataset with the same orientation of its second and third views. First we will retrieve R2N2's original renderings to compare with the result.</p>
|
||||
</div>
|
||||
@ -603,7 +636,8 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Next, we will pass the voxels to <code>render_vox_to_mesh</code>:</p>
|
||||
</div>
|
||||
@ -621,4 +655,4 @@ dataset. The R2N2 dataset also contains its own 24 renderings of each object and
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,14 +82,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Deform-a-source-mesh-to-form-a-target-mesh-using-3D-loss-functions">Deform a source mesh to form a target mesh using 3D loss functions<a class="anchor-link" href="#Deform-a-source-mesh-to-form-a-target-mesh-using-3D-loss-functions">¶</a></h1>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this tutorial, we learn to deform an initial generic shape (e.g. sphere) to fit a target shape.</p>
|
||||
<p>We will cover:</p>
|
||||
@ -115,16 +117,18 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -133,19 +137,25 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -162,20 +172,20 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_obj</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="k">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.ops</span> <span class="k">import</span> <span class="n">sample_points_from_meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_obj</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="kn">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.ops</span> <span class="kn">import</span> <span class="n">sample_points_from_meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">chamfer_distance</span><span class="p">,</span>
|
||||
<span class="n">mesh_edge_loss</span><span class="p">,</span>
|
||||
<span class="n">mesh_laplacian_smoothing</span><span class="p">,</span>
|
||||
<span class="n">mesh_normal_consistency</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
<span class="o">%</span><span class="k">matplotlib</span> notebook
|
||||
<span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="k">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="kn">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib</span> <span class="k">as</span> <span class="nn">mpl</span>
|
||||
<span class="n">mpl</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">'savefig.dpi'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">80</span>
|
||||
@ -193,14 +203,16 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-an-obj-file-and-create-a-Meshes-object">1. Load an obj file and create a Meshes object<a class="anchor-link" href="#1.-Load-an-obj-file-and-create-a-Meshes-object">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Download the target 3D model of a dolphin. It will be saved locally as a file called <code>dolphin.obj</code>.</p>
|
||||
</div>
|
||||
@ -271,7 +283,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Visualize-the-source-and-target-meshes">Visualize the source and target meshes<a class="anchor-link" href="#Visualize-the-source-and-target-meshes">¶</a></h3>
|
||||
</div>
|
||||
@ -314,7 +327,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Optimization-loop">3. Optimization loop<a class="anchor-link" href="#3.-Optimization-loop">¶</a></h2>
|
||||
</div>
|
||||
@ -401,10 +415,10 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s1">'total_loss = </span><span class="si">%.6f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">loss</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Save the losses for plotting</span>
|
||||
<span class="n">chamfer_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_chamfer</span><span class="p">)</span>
|
||||
<span class="n">edge_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_edge</span><span class="p">)</span>
|
||||
<span class="n">normal_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_normal</span><span class="p">)</span>
|
||||
<span class="n">laplacian_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_laplacian</span><span class="p">)</span>
|
||||
<span class="n">chamfer_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_chamfer</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">edge_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_edge</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">normal_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_normal</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">laplacian_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_laplacian</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
<span class="c1"># Plot mesh</span>
|
||||
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="n">plot_period</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
|
||||
@ -419,7 +433,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Visualize-the-loss">4. Visualize the loss<a class="anchor-link" href="#4.-Visualize-the-loss">¶</a></h2>
|
||||
</div>
|
||||
@ -446,7 +461,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Save-the-predicted-mesh">5. Save the predicted mesh<a class="anchor-link" href="#5.-Save-the-predicted-mesh">¶</a></h2>
|
||||
</div>
|
||||
@ -472,10 +488,11 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="6.-Conclusion">6. Conclusion<a class="anchor-link" href="#6.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to load a mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an optimization loop and use four different PyTorch3D mesh loss functions.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,14 +82,16 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Deform-a-source-mesh-to-form-a-target-mesh-using-3D-loss-functions">Deform a source mesh to form a target mesh using 3D loss functions<a class="anchor-link" href="#Deform-a-source-mesh-to-form-a-target-mesh-using-3D-loss-functions">¶</a></h1>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this tutorial, we learn to deform an initial generic shape (e.g. sphere) to fit a target shape.</p>
|
||||
<p>We will cover:</p>
|
||||
@ -115,16 +117,18 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -133,19 +137,25 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -162,20 +172,20 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_obj</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="k">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.ops</span> <span class="k">import</span> <span class="n">sample_points_from_meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_obj</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="kn">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.ops</span> <span class="kn">import</span> <span class="n">sample_points_from_meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">chamfer_distance</span><span class="p">,</span>
|
||||
<span class="n">mesh_edge_loss</span><span class="p">,</span>
|
||||
<span class="n">mesh_laplacian_smoothing</span><span class="p">,</span>
|
||||
<span class="n">mesh_normal_consistency</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
<span class="o">%</span><span class="k">matplotlib</span> notebook
|
||||
<span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="k">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="kn">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib</span> <span class="k">as</span> <span class="nn">mpl</span>
|
||||
<span class="n">mpl</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">'savefig.dpi'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">80</span>
|
||||
@ -193,14 +203,16 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="1.-Load-an-obj-file-and-create-a-Meshes-object">1. Load an obj file and create a Meshes object<a class="anchor-link" href="#1.-Load-an-obj-file-and-create-a-Meshes-object">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Download the target 3D model of a dolphin. It will be saved locally as a file called <code>dolphin.obj</code>.</p>
|
||||
</div>
|
||||
@ -271,7 +283,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Visualize-the-source-and-target-meshes">Visualize the source and target meshes<a class="anchor-link" href="#Visualize-the-source-and-target-meshes">¶</a></h3>
|
||||
</div>
|
||||
@ -314,7 +327,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Optimization-loop">3. Optimization loop<a class="anchor-link" href="#3.-Optimization-loop">¶</a></h2>
|
||||
</div>
|
||||
@ -401,10 +415,10 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s1">'total_loss = </span><span class="si">%.6f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">loss</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Save the losses for plotting</span>
|
||||
<span class="n">chamfer_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_chamfer</span><span class="p">)</span>
|
||||
<span class="n">edge_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_edge</span><span class="p">)</span>
|
||||
<span class="n">normal_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_normal</span><span class="p">)</span>
|
||||
<span class="n">laplacian_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">loss_laplacian</span><span class="p">)</span>
|
||||
<span class="n">chamfer_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_chamfer</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">edge_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_edge</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">normal_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_normal</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
<span class="n">laplacian_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">loss_laplacian</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
<span class="c1"># Plot mesh</span>
|
||||
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="n">plot_period</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
|
||||
@ -419,7 +433,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Visualize-the-loss">4. Visualize the loss<a class="anchor-link" href="#4.-Visualize-the-loss">¶</a></h2>
|
||||
</div>
|
||||
@ -446,7 +461,8 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Save-the-predicted-mesh">5. Save the predicted mesh<a class="anchor-link" href="#5.-Save-the-predicted-mesh">¶</a></h2>
|
||||
</div>
|
||||
@ -472,10 +488,11 @@ the predicted mesh is closer to the target mesh at each optimization step. To ac
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="6.-Conclusion">6. Conclusion<a class="anchor-link" href="#6.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to load a mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an optimization loop and use four different PyTorch3D mesh loss functions.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Fit-a-mesh-via-rendering">Fit a mesh via rendering<a class="anchor-link" href="#Fit-a-mesh-via-rendering">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -95,16 +96,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -113,19 +116,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -143,16 +152,15 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="k">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="kn">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
|
||||
<span class="c1"># Util function for loading meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">chamfer_distance</span><span class="p">,</span>
|
||||
<span class="n">mesh_edge_loss</span><span class="p">,</span>
|
||||
<span class="n">mesh_laplacian_smoothing</span><span class="p">,</span>
|
||||
@ -160,8 +168,8 @@
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">OpenGLPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -186,7 +194,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -198,14 +207,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -223,17 +233,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and it's associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and its associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesVertex</strong> is an auxillary datastructure for storing vertex rgb texture information about meshes.</p>
|
||||
<p><strong>TexturesVertex</strong> is an auxiliary datastructure for storing vertex rgb texture information about meshes.</p>
|
||||
<p><strong>Meshes</strong> has several class methods which are used throughout the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the mesh obj and texture files and save it at the path <code>data/cow_mesh</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -281,7 +293,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="n">verts</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="n">center</span> <span class="o">=</span> <span class="n">verts</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">scale</span> <span class="o">=</span> <span class="nb">max</span><span class="p">((</span><span class="n">verts</span> <span class="o">-</span> <span class="n">center</span><span class="p">)</span><span class="o">.</span><span class="n">abs</span><span class="p">()</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="mi">0</span><span class="p">)[</span><span class="mi">0</span><span class="p">])</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">offset_verts_</span><span class="p">(</span><span class="o">-</span><span class="n">center</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">N</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">offset_verts_</span><span class="p">(</span><span class="o">-</span><span class="n">center</span><span class="p">)</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">scale_verts_</span><span class="p">((</span><span class="mf">1.0</span> <span class="o">/</span> <span class="nb">float</span><span class="p">(</span><span class="n">scale</span><span class="p">)));</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -289,7 +301,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Dataset-Creation">2. Dataset Creation<a class="anchor-link" href="#2.-Dataset-Creation">¶</a></h2><p>We sample different camera positions that encode multiple viewpoints of the cow. We create a renderer with a shader that performs texture map interpolation. We render a synthetic dataset of images of the textured cow mesh from multiple viewpoints.</p>
|
||||
</div>
|
||||
@ -328,7 +341,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="c1"># purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to </span>
|
||||
<span class="c1"># rasterize_meshes.py for explanations of these parameters. We also leave </span>
|
||||
<span class="c1"># bin_size and max_faces_per_bin to their default values of None, which sets </span>
|
||||
<span class="c1"># their values using huristics and ensures that the faster coarse-to-fine </span>
|
||||
<span class="c1"># their values using heuristics and ensures that the faster coarse-to-fine </span>
|
||||
<span class="c1"># rasterization method is used. Refer to docs/notes/renderer.md for an </span>
|
||||
<span class="c1"># explanation of the difference between naive and coarse-to-fine rasterization. </span>
|
||||
<span class="n">raster_settings</span> <span class="o">=</span> <span class="n">RasterizationSettings</span><span class="p">(</span>
|
||||
@ -337,8 +350,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">faces_per_pixel</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured </span>
|
||||
<span class="c1"># phong shader will interpolate the texture uv coordinates for each vertex, </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured </span>
|
||||
<span class="c1"># Phong shader will interpolate the texture uv coordinates for each vertex, </span>
|
||||
<span class="c1"># sample from a texture image and apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
<span class="n">rasterizer</span><span class="o">=</span><span class="n">MeshRasterizer</span><span class="p">(</span>
|
||||
@ -371,7 +384,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Visualize the dataset:</p>
|
||||
</div>
|
||||
@ -391,9 +405,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel.</p>
|
||||
<p>Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -433,7 +448,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Mesh-prediction-via-silhouette-rendering">3. Mesh prediction via silhouette rendering<a class="anchor-link" href="#3.-Mesh-prediction-via-silhouette-rendering">¶</a></h2><p>In the previous section, we created a dataset of images of multiple viewpoints of a cow. In this section, we predict a mesh by observing those target images without any knowledge of the ground truth cow mesh. We assume we know the position of the cameras and lighting.</p>
|
||||
<p>We first define some helper functions to visualize the results of our mesh prediction:</p>
|
||||
@ -451,7 +467,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">target_image</span><span class="o">=</span><span class="n">target_rgb</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">title</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span>
|
||||
<span class="n">silhouette</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
|
||||
<span class="n">inds</span> <span class="o">=</span> <span class="mi">3</span> <span class="k">if</span> <span class="n">silhouette</span> <span class="k">else</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
|
||||
<span class="n">predicted_images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">predicted_mesh</span><span class="p">)</span>
|
||||
<span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span>
|
||||
<span class="n">predicted_images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">predicted_mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">predicted_images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="n">inds</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
@ -459,7 +476,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">target_image</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Plot losses as a function of optimization iteration</span>
|
||||
@ -478,7 +494,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Starting from a sphere mesh, we will learn offsets of each vertex such that the predicted mesh silhouette is more similar to the target silhouette image at each optimization step. We begin by loading our initial sphere mesh:</p>
|
||||
</div>
|
||||
@ -497,7 +514,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We create a new differentiable renderer for rendering the silhouette of our predicted mesh:</p>
|
||||
</div>
|
||||
@ -532,7 +550,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We initialize settings, losses, and the optimizer that will be used to iteratively fit our mesh to the target silhouettes:</p>
|
||||
</div>
|
||||
@ -585,9 +604,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:</p>
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -622,7 +642,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">sum_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">loss</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
|
||||
<span class="n">sum_loss</span> <span class="o">+=</span> <span class="n">l</span> <span class="o">*</span> <span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"weight"</span><span class="p">]</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">l</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
|
||||
<span class="c1"># Print the losses</span>
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s2">"total_loss = </span><span class="si">%.6f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">sum_loss</span><span class="p">)</span>
|
||||
@ -654,7 +675,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Mesh-and-texture-prediction-via-textured-rendering">3. Mesh and texture prediction via textured rendering<a class="anchor-link" href="#3.-Mesh-and-texture-prediction-via-textured-rendering">¶</a></h2><p>We can predict both the mesh and its texture if we add an additional loss based on the comparing a predicted rendered RGB image to the target image. As before, we start with a sphere mesh. We learn both translational offsets and RGB texture colors for each vertex in the sphere mesh. Since our loss is based on rendered RGB pixel values instead of just the silhouette, we use a <strong>SoftPhongShader</strong> instead of a <strong>SoftSilhouetteShader</strong>.</p>
|
||||
</div>
|
||||
@ -691,7 +713,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We initialize settings, losses, and the optimizer that will be used to iteratively fit our mesh to the target RGB images:</p>
|
||||
</div>
|
||||
@ -738,7 +761,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh and its vertex colors from the sphere mesh into a mesh that matches the target images:</p>
|
||||
</div>
|
||||
@ -787,7 +811,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">sum_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">loss</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
|
||||
<span class="n">sum_loss</span> <span class="o">+=</span> <span class="n">l</span> <span class="o">*</span> <span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"weight"</span><span class="p">]</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">l</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
<span class="c1"># Print the losses</span>
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s2">"total_loss = </span><span class="si">%.6f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">sum_loss</span><span class="p">)</span>
|
||||
@ -817,14 +841,16 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Save the final predicted mesh:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Save-the-final-predicted-mesh">4. Save the final predicted mesh<a class="anchor-link" href="#4.-Save-the-final-predicted-mesh">¶</a></h2>
|
||||
</div>
|
||||
@ -850,10 +876,11 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Conclusion">5. Conclusion<a class="anchor-link" href="#5.-Conclusion">¶</a></h2><p>In this tutorial, we learned how to load a textured mesh from an obj file, create a synthetic dataset by rendering the mesh from multiple viewpoints. We showed how to set up an optimization loop to fit a mesh to the observed dataset images based on a rendered silhouette loss. We then augmented this optimization loop with an additional loss based on rendered RGB images, which allowed us to predict both a mesh and its texture.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Fit-a-mesh-via-rendering">Fit a mesh via rendering<a class="anchor-link" href="#Fit-a-mesh-via-rendering">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -95,16 +96,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -113,19 +116,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -143,16 +152,15 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="k">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.utils</span> <span class="kn">import</span> <span class="n">ico_sphere</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="k">import</span> <span class="n">tqdm</span>
|
||||
<span class="kn">from</span> <span class="nn">tqdm.notebook</span> <span class="kn">import</span> <span class="n">tqdm</span>
|
||||
|
||||
<span class="c1"># Util function for loading meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">save_obj</span>
|
||||
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.loss</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">chamfer_distance</span><span class="p">,</span>
|
||||
<span class="n">mesh_edge_loss</span><span class="p">,</span>
|
||||
<span class="n">mesh_laplacian_smoothing</span><span class="p">,</span>
|
||||
@ -160,8 +168,8 @@
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">OpenGLPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -186,7 +194,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -198,14 +207,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -223,17 +233,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and it's associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and its associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesVertex</strong> is an auxillary datastructure for storing vertex rgb texture information about meshes.</p>
|
||||
<p><strong>TexturesVertex</strong> is an auxiliary datastructure for storing vertex rgb texture information about meshes.</p>
|
||||
<p><strong>Meshes</strong> has several class methods which are used throughout the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the mesh obj and texture files and save it at the path <code>data/cow_mesh</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -281,7 +293,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="n">verts</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="n">center</span> <span class="o">=</span> <span class="n">verts</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">scale</span> <span class="o">=</span> <span class="nb">max</span><span class="p">((</span><span class="n">verts</span> <span class="o">-</span> <span class="n">center</span><span class="p">)</span><span class="o">.</span><span class="n">abs</span><span class="p">()</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="mi">0</span><span class="p">)[</span><span class="mi">0</span><span class="p">])</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">offset_verts_</span><span class="p">(</span><span class="o">-</span><span class="n">center</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">N</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">offset_verts_</span><span class="p">(</span><span class="o">-</span><span class="n">center</span><span class="p">)</span>
|
||||
<span class="n">mesh</span><span class="o">.</span><span class="n">scale_verts_</span><span class="p">((</span><span class="mf">1.0</span> <span class="o">/</span> <span class="nb">float</span><span class="p">(</span><span class="n">scale</span><span class="p">)));</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -289,7 +301,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Dataset-Creation">2. Dataset Creation<a class="anchor-link" href="#2.-Dataset-Creation">¶</a></h2><p>We sample different camera positions that encode multiple viewpoints of the cow. We create a renderer with a shader that performs texture map interpolation. We render a synthetic dataset of images of the textured cow mesh from multiple viewpoints.</p>
|
||||
</div>
|
||||
@ -328,7 +341,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="c1"># purposes only we will set faces_per_pixel=1 and blur_radius=0.0. Refer to </span>
|
||||
<span class="c1"># rasterize_meshes.py for explanations of these parameters. We also leave </span>
|
||||
<span class="c1"># bin_size and max_faces_per_bin to their default values of None, which sets </span>
|
||||
<span class="c1"># their values using huristics and ensures that the faster coarse-to-fine </span>
|
||||
<span class="c1"># their values using heuristics and ensures that the faster coarse-to-fine </span>
|
||||
<span class="c1"># rasterization method is used. Refer to docs/notes/renderer.md for an </span>
|
||||
<span class="c1"># explanation of the difference between naive and coarse-to-fine rasterization. </span>
|
||||
<span class="n">raster_settings</span> <span class="o">=</span> <span class="n">RasterizationSettings</span><span class="p">(</span>
|
||||
@ -337,8 +350,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">faces_per_pixel</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
|
||||
<span class="p">)</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured </span>
|
||||
<span class="c1"># phong shader will interpolate the texture uv coordinates for each vertex, </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured </span>
|
||||
<span class="c1"># Phong shader will interpolate the texture uv coordinates for each vertex, </span>
|
||||
<span class="c1"># sample from a texture image and apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
<span class="n">rasterizer</span><span class="o">=</span><span class="n">MeshRasterizer</span><span class="p">(</span>
|
||||
@ -371,7 +384,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Visualize the dataset:</p>
|
||||
</div>
|
||||
@ -391,9 +405,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We contruct a soft silhouette shader to render this alpha channel.</p>
|
||||
<p>Later in this tutorial, we will fit a mesh to the rendered RGB images, as well as to just images of just the cow silhouette. For the latter case, we will render a dataset of silhouette images. Most shaders in PyTorch3D will output an alpha channel along with the RGB image as a 4th channel in an RGBA image. The alpha channel encodes the probability that each pixel belongs to the foreground of the object. We construct a soft silhouette shader to render this alpha channel.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -433,7 +448,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Mesh-prediction-via-silhouette-rendering">3. Mesh prediction via silhouette rendering<a class="anchor-link" href="#3.-Mesh-prediction-via-silhouette-rendering">¶</a></h2><p>In the previous section, we created a dataset of images of multiple viewpoints of a cow. In this section, we predict a mesh by observing those target images without any knowledge of the ground truth cow mesh. We assume we know the position of the cameras and lighting.</p>
|
||||
<p>We first define some helper functions to visualize the results of our mesh prediction:</p>
|
||||
@ -451,7 +467,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">target_image</span><span class="o">=</span><span class="n">target_rgb</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">title</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span>
|
||||
<span class="n">silhouette</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
|
||||
<span class="n">inds</span> <span class="o">=</span> <span class="mi">3</span> <span class="k">if</span> <span class="n">silhouette</span> <span class="k">else</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
|
||||
<span class="n">predicted_images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">predicted_mesh</span><span class="p">)</span>
|
||||
<span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span>
|
||||
<span class="n">predicted_images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">predicted_mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">predicted_images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="n">inds</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
@ -459,7 +476,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">target_image</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Plot losses as a function of optimization iteration</span>
|
||||
@ -478,7 +494,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Starting from a sphere mesh, we will learn offsets of each vertex such that the predicted mesh silhouette is more similar to the target silhouette image at each optimization step. We begin by loading our initial sphere mesh:</p>
|
||||
</div>
|
||||
@ -497,7 +514,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We create a new differentiable renderer for rendering the silhouette of our predicted mesh:</p>
|
||||
</div>
|
||||
@ -532,7 +550,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We initialize settings, losses, and the optimizer that will be used to iteratively fit our mesh to the target silhouettes:</p>
|
||||
</div>
|
||||
@ -585,9 +604,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the sillhouettes of the target images:</p>
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh from the sphere mesh into a mesh that matches the silhouettes of the target images:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -622,7 +642,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">sum_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">loss</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
|
||||
<span class="n">sum_loss</span> <span class="o">+=</span> <span class="n">l</span> <span class="o">*</span> <span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"weight"</span><span class="p">]</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">l</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
|
||||
<span class="c1"># Print the losses</span>
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s2">"total_loss = </span><span class="si">%.6f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">sum_loss</span><span class="p">)</span>
|
||||
@ -654,7 +675,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Mesh-and-texture-prediction-via-textured-rendering">3. Mesh and texture prediction via textured rendering<a class="anchor-link" href="#3.-Mesh-and-texture-prediction-via-textured-rendering">¶</a></h2><p>We can predict both the mesh and its texture if we add an additional loss based on the comparing a predicted rendered RGB image to the target image. As before, we start with a sphere mesh. We learn both translational offsets and RGB texture colors for each vertex in the sphere mesh. Since our loss is based on rendered RGB pixel values instead of just the silhouette, we use a <strong>SoftPhongShader</strong> instead of a <strong>SoftSilhouetteShader</strong>.</p>
|
||||
</div>
|
||||
@ -691,7 +713,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We initialize settings, losses, and the optimizer that will be used to iteratively fit our mesh to the target RGB images:</p>
|
||||
</div>
|
||||
@ -738,7 +761,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We write an optimization loop to iteratively refine our predicted mesh and its vertex colors from the sphere mesh into a mesh that matches the target images:</p>
|
||||
</div>
|
||||
@ -787,7 +811,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">sum_loss</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">loss</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
|
||||
<span class="n">sum_loss</span> <span class="o">+=</span> <span class="n">l</span> <span class="o">*</span> <span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"weight"</span><span class="p">]</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||||
<span class="n">losses</span><span class="p">[</span><span class="n">k</span><span class="p">][</span><span class="s2">"values"</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">l</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()))</span>
|
||||
|
||||
<span class="c1"># Print the losses</span>
|
||||
<span class="n">loop</span><span class="o">.</span><span class="n">set_description</span><span class="p">(</span><span class="s2">"total_loss = </span><span class="si">%.6f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">sum_loss</span><span class="p">)</span>
|
||||
@ -817,14 +841,16 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Save the final predicted mesh:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Save-the-final-predicted-mesh">4. Save the final predicted mesh<a class="anchor-link" href="#4.-Save-the-final-predicted-mesh">¶</a></h2>
|
||||
</div>
|
||||
@ -850,10 +876,11 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Conclusion">5. Conclusion<a class="anchor-link" href="#5.-Conclusion">¶</a></h2><p>In this tutorial, we learned how to load a textured mesh from an obj file, create a synthetic dataset by rendering the mesh from multiple viewpoints. We showed how to set up an optimization loop to fit a mesh to the observed dataset images based on a rendered silhouette loss. We then augmented this optimization loop with an additional loss based on rendered RGB images, which allowed us to predict both a mesh and its texture.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -62,4 +62,4 @@
|
||||
};
|
||||
}
|
||||
});
|
||||
</script></nav></div><div class="container mainContainer documentContainer postContainer"><div class="wrapper"><div class="post"><header class="postHeader"><h1 class="postHeaderTitle">Welcome to the PyTorch3D Tutorials</h1></header><p>Here you can learn about the structure and applications of Pytorch3D from examples which are in the form of ipython notebooks.</p><h3> Run interactively </h3><p>At the top of each example you can find a button named <strong>"Run in Google Colab"</strong> which will open the notebook in <a href="https://colab.research.google.com/notebooks/intro.ipynb"> Google Colaboratory </a> where you can run the code directly in the browser with access to GPU support - it looks like this:</p><div class="tutorialButtonsWrapper"><div class="tutorialButtonWrapper buttonWrapper"><a class="tutorialButton button" target="_blank"><img class="colabButton" align="left" src="/img/colab_icon.png"/>Run in Google Colab</a></div></div><p> You can modify the code and experiment with varying different settings. Remember to install the latest stable version of PyTorch3D and its dependencies. Code to do this with pip is provided in each notebook. </p><h3> Run locally </h3><p> There is also a button to download the notebook and source code to run it locally. </p></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</script></nav></div><div class="container mainContainer documentContainer postContainer"><div class="wrapper"><div class="post"><header class="postHeader"><h1 class="postHeaderTitle">Welcome to the PyTorch3D Tutorials</h1></header><p>Here you can learn about the structure and applications of PyTorch3D from examples which are in the form of ipython notebooks.</p><h3> Run interactively </h3><p>At the top of each example you can find a button named <strong>"Run in Google Colab"</strong> which will open the notebook in <a href="https://colab.research.google.com/notebooks/intro.ipynb"> Google Colaboratory </a> where you can run the code directly in the browser with access to GPU support - it looks like this:</p><div class="tutorialButtonsWrapper"><div class="tutorialButtonWrapper buttonWrapper"><a class="tutorialButton button" target="_blank"><img class="colabButton" align="left" src="/img/colab_icon.png"/>Run in Google Colab</a></div></div><p> You can modify the code and experiment with varying different settings. Remember to install the latest stable version of PyTorch3D and its dependencies. Code to do this with pip is provided in each notebook. </p><h3> Run locally </h3><p> There is also a button to download the notebook and source code to run it locally. </p></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-a-colored-point-cloud">Render a colored point cloud<a class="anchor-link" href="#Render-a-colored-point-cloud">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -94,16 +95,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Import-modules">Import modules<a class="anchor-link" href="#Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -112,19 +115,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -143,15 +152,14 @@
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="c1"># Util function for loading point clouds|</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Pointclouds</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="k">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Pointclouds</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="kn">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVOrthographicCameras</span><span class="p">,</span>
|
||||
<span class="n">PointsRasterizationSettings</span><span class="p">,</span>
|
||||
@ -167,7 +175,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Load-a-point-cloud-and-corresponding-colors">Load a point cloud and corresponding colors<a class="anchor-link" href="#Load-a-point-cloud-and-corresponding-colors">¶</a></h3><p>Load and create a <strong>Point Cloud</strong> object.</p>
|
||||
<p><strong>Pointclouds</strong> is a unique datastructure provided in PyTorch3D for working with batches of point clouds of different sizes.</p>
|
||||
@ -175,7 +184,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the pointcloud data and save it at the path <code>data/PittsburghBridge</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -223,9 +233,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses an <strong>orthographic camera</strong>, and applies <strong>alpha compositing</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
<p>[1] <a href="https://arxiv.org/abs/1912.08804">SynSin: End to end View Synthesis from a Single Image.</a> Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson. CVPR 2020.</p>
|
||||
</div>
|
||||
@ -270,7 +281,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -278,7 +288,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now modify the <strong>renderer</strong> to use <strong>alpha compositing</strong> with a set background color.</p>
|
||||
</div>
|
||||
@ -299,7 +310,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -307,7 +317,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses an <strong>orthographic camera</strong>, and applies <strong>weighted compositing</strong>.</p>
|
||||
</div>
|
||||
@ -351,7 +362,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -359,7 +369,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now modify the <strong>renderer</strong> to use <strong>weighted compositing</strong> with a set background color.</p>
|
||||
</div>
|
||||
@ -379,7 +390,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -387,7 +397,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Using-the-pulsar-backend">Using the pulsar backend<a class="anchor-link" href="#Using-the-pulsar-backend">¶</a></h2><p>Switching to the pulsar backend is easy! The pulsar backend has a compositor built-in, so the <code>compositor</code> argument is not required when creating it (a warning will be displayed if you provide it nevertheless). It pre-allocates memory on the rendering device, that's why it needs the <code>n_channels</code> at construction time.</p>
|
||||
<p>All parameters for the renderer forward function are batch-wise except the background color (in this example, <code>gamma</code>) and you have to provide as many values as you have examples in your batch. The background color is optional and by default set to all zeros. You can find a detailed explanation of how gamma influences the rendering function here in the paper <a href="https://arxiv.org/pdf/2004.07484.pdf">Fast Differentiable Raycasting for Neural Rendering using
|
||||
@ -410,7 +421,6 @@ Sphere-based Representations</a>.</p>
|
||||
<span class="n">bg_col</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -418,7 +428,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="View-pointclouds-in-Plotly-figures">View pointclouds in Plotly figures<a class="anchor-link" href="#View-pointclouds-in-Plotly-figures">¶</a></h3><p>Here we use the PyTorch3D function <code>plot_scene</code> to render the pointcloud in a Plotly figure. <code>plot_scene</code> returns a plotly figure with trace and subplots defined by the input.</p>
|
||||
</div>
|
||||
@ -440,7 +451,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now render a batch of pointclouds. The first pointcloud is the same as above, and the second is all-black and offset by 2 in all dimensions so we can see them on the same plot.</p>
|
||||
</div>
|
||||
@ -483,7 +495,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>For batches, we can also use <code>plot_batch_individually</code> to avoid constructing the scene dictionary ourselves.</p>
|
||||
</div>
|
||||
@ -516,7 +529,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can also modify the axis arguments and axis backgrounds for either function, and title our plots in <code>plot_batch_individually</code>.</p>
|
||||
</div>
|
||||
@ -540,4 +554,4 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-a-colored-point-cloud">Render a colored point cloud<a class="anchor-link" href="#Render-a-colored-point-cloud">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -94,16 +95,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Import-modules">Import modules<a class="anchor-link" href="#Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -112,19 +115,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -143,15 +152,14 @@
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="c1"># Util function for loading point clouds|</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Pointclouds</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="k">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Pointclouds</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="kn">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVOrthographicCameras</span><span class="p">,</span>
|
||||
<span class="n">PointsRasterizationSettings</span><span class="p">,</span>
|
||||
@ -167,7 +175,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Load-a-point-cloud-and-corresponding-colors">Load a point cloud and corresponding colors<a class="anchor-link" href="#Load-a-point-cloud-and-corresponding-colors">¶</a></h3><p>Load and create a <strong>Point Cloud</strong> object.</p>
|
||||
<p><strong>Pointclouds</strong> is a unique datastructure provided in PyTorch3D for working with batches of point clouds of different sizes.</p>
|
||||
@ -175,7 +184,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the pointcloud data and save it at the path <code>data/PittsburghBridge</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -223,9 +233,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthgraphic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses an <strong>orthographic camera</strong>, and applies <strong>alpha compositing</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
<p>[1] <a href="https://arxiv.org/abs/1912.08804">SynSin: End to end View Synthesis from a Single Image.</a> Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson. CVPR 2020.</p>
|
||||
</div>
|
||||
@ -270,7 +281,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -278,7 +288,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now modify the <strong>renderer</strong> to use <strong>alpha compositing</strong> with a set background color.</p>
|
||||
</div>
|
||||
@ -299,7 +310,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -307,7 +317,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses an <strong>orthographic camera</strong>, and applies <strong>weighted compositing</strong>.</p>
|
||||
</div>
|
||||
@ -351,7 +362,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -359,7 +369,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now modify the <strong>renderer</strong> to use <strong>weighted compositing</strong> with a set background color.</p>
|
||||
</div>
|
||||
@ -379,7 +390,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">point_cloud</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -387,7 +397,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Using-the-pulsar-backend">Using the pulsar backend<a class="anchor-link" href="#Using-the-pulsar-backend">¶</a></h2><p>Switching to the pulsar backend is easy! The pulsar backend has a compositor built-in, so the <code>compositor</code> argument is not required when creating it (a warning will be displayed if you provide it nevertheless). It pre-allocates memory on the rendering device, that's why it needs the <code>n_channels</code> at construction time.</p>
|
||||
<p>All parameters for the renderer forward function are batch-wise except the background color (in this example, <code>gamma</code>) and you have to provide as many values as you have examples in your batch. The background color is optional and by default set to all zeros. You can find a detailed explanation of how gamma influences the rendering function here in the paper <a href="https://arxiv.org/pdf/2004.07484.pdf">Fast Differentiable Raycasting for Neural Rendering using
|
||||
@ -410,7 +421,6 @@ Sphere-based Representations</a>.</p>
|
||||
<span class="n">bg_col</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -418,7 +428,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="View-pointclouds-in-Plotly-figures">View pointclouds in Plotly figures<a class="anchor-link" href="#View-pointclouds-in-Plotly-figures">¶</a></h3><p>Here we use the PyTorch3D function <code>plot_scene</code> to render the pointcloud in a Plotly figure. <code>plot_scene</code> returns a plotly figure with trace and subplots defined by the input.</p>
|
||||
</div>
|
||||
@ -440,7 +451,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We will now render a batch of pointclouds. The first pointcloud is the same as above, and the second is all-black and offset by 2 in all dimensions so we can see them on the same plot.</p>
|
||||
</div>
|
||||
@ -483,7 +495,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>For batches, we can also use <code>plot_batch_individually</code> to avoid constructing the scene dictionary ourselves.</p>
|
||||
</div>
|
||||
@ -516,7 +529,8 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can also modify the axis arguments and axis backgrounds for either function, and title our plots in <code>plot_batch_individually</code>.</p>
|
||||
</div>
|
||||
@ -540,4 +554,4 @@ Sphere-based Representations</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-DensePose">Render DensePose<a class="anchor-link" href="#Render-DensePose">¶</a></h1><p>DensePose refers to dense human pose representation: <a href="https://github.com/facebookresearch/DensePose">https://github.com/facebookresearch/DensePose</a>.
|
||||
In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p>
|
||||
@ -97,16 +98,18 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Import-modules">Import modules<a class="anchor-link" href="#Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If torch, torchvision and PyTorch3D are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -115,19 +118,25 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -157,17 +166,16 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
|
||||
<span class="c1"># libraries for reading data from files</span>
|
||||
<span class="kn">from</span> <span class="nn">scipy.io</span> <span class="k">import</span> <span class="n">loadmat</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io.utils</span> <span class="k">import</span> <span class="n">_read_image</span>
|
||||
<span class="kn">from</span> <span class="nn">scipy.io</span> <span class="kn">import</span> <span class="n">loadmat</span>
|
||||
<span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span>
|
||||
<span class="kn">import</span> <span class="nn">pickle</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -190,7 +198,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Load-the-SMPL-model">Load the SMPL model<a class="anchor-link" href="#Load-the-SMPL-model">¶</a></h2><h4 id="Download-the-SMPL-model">Download the SMPL model<a class="anchor-link" href="#Download-the-SMPL-model">¶</a></h4><ul>
|
||||
<li>Go to <a href="http://smpl.is.tue.mpg.de/downloads">http://smpl.is.tue.mpg.de/downloads</a> and sign up.</li>
|
||||
@ -222,7 +231,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Load our texture UV data and our SMPL data, with some processing to correct data values and format.</p>
|
||||
</div>
|
||||
@ -253,13 +263,15 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="n">data</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'latin1'</span><span class="p">)</span>
|
||||
<span class="n">v_template</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">'v_template'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (6890, 3)</span>
|
||||
<span class="n">ALP_UV</span> <span class="o">=</span> <span class="n">loadmat</span><span class="p">(</span><span class="n">data_filename</span><span class="p">)</span>
|
||||
<span class="n">tex</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">_read_image</span><span class="p">(</span><span class="n">file_name</span><span class="o">=</span><span class="n">tex_filename</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">'RGB'</span><span class="p">)</span> <span class="o">/</span> <span class="mf">255.</span> <span class="p">)</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">with</span> <span class="n">Image</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">tex_filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">image</span><span class="p">:</span>
|
||||
<span class="n">np_image</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">image</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s2">"RGB"</span><span class="p">))</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span>
|
||||
<span class="n">tex</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">np_image</span> <span class="o">/</span> <span class="mf">255.</span><span class="p">)[</span><span class="kc">None</span><span class="p">]</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
|
||||
|
||||
<span class="n">verts</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s2">"All_vertices"</span><span class="p">])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">verts</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s2">"All_vertices"</span><span class="p">])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829,)</span>
|
||||
<span class="n">U</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_U_norm'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">V</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_V_norm'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">faces</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_Faces'</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (13774, 3)</span>
|
||||
<span class="n">face_indices</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_FaceIndices'</span><span class="p">])</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span>
|
||||
<span class="n">face_indices</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_FaceIndices'</span><span class="p">])</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span> <span class="c1"># (13774,)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@ -273,7 +285,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># Display the texture image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">tex</span><span class="o">.</span><span class="n">squeeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">cpu</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -281,7 +292,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In DensePose, the body mesh is split into 24 parts. In the texture image, we can see the 24 parts are separated out into individual (200, 200) images per body part. The convention in DensePose is that each face in the mesh is associated with a body part (given by the face_indices tensor above). The vertex UV values (in the range [0, 1]) for each face are specific to the (200, 200) size texture map for the part of the body that the mesh face corresponds to. We cannot use them directly with the entire texture map. We have to offset the vertex UV values depending on what body part the associated face corresponds to.</p>
|
||||
</div>
|
||||
@ -301,6 +313,9 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="n">part</span> <span class="o">=</span> <span class="n">rows</span> <span class="o">*</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span> <span class="o">+</span> <span class="mi">1</span> <span class="c1"># parts are 1-indexed in face_indices</span>
|
||||
<span class="n">offset_per_part</span><span class="p">[</span><span class="n">part</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">u</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span>
|
||||
|
||||
<span class="n">U_norm</span> <span class="o">=</span> <span class="n">U</span><span class="o">.</span><span class="n">clone</span><span class="p">()</span>
|
||||
<span class="n">V_norm</span> <span class="o">=</span> <span class="n">V</span><span class="o">.</span><span class="n">clone</span><span class="p">()</span>
|
||||
|
||||
<span class="c1"># iterate over faces and offset the corresponding vertex u and v values</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">faces</span><span class="p">)):</span>
|
||||
<span class="n">face_vert_idxs</span> <span class="o">=</span> <span class="n">faces</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
|
||||
@ -311,15 +326,15 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># vertices are reused, but we don't want to offset multiple times</span>
|
||||
<span class="k">if</span> <span class="n">vert_idx</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">already_offset</span><span class="p">:</span>
|
||||
<span class="c1"># offset u value</span>
|
||||
<span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">/</span> <span class="n">cols</span> <span class="o">+</span> <span class="n">offset_u</span>
|
||||
<span class="n">U_norm</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">/</span> <span class="n">cols</span> <span class="o">+</span> <span class="n">offset_u</span>
|
||||
<span class="c1"># offset v value</span>
|
||||
<span class="c1"># this also flips each part locally, as each part is upside down</span>
|
||||
<span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">])</span> <span class="o">/</span> <span class="n">rows</span> <span class="o">+</span> <span class="n">offset_v</span>
|
||||
<span class="n">V_norm</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">])</span> <span class="o">/</span> <span class="n">rows</span> <span class="o">+</span> <span class="n">offset_v</span>
|
||||
<span class="c1"># add vertex to our set tracking offsetted vertices</span>
|
||||
<span class="n">already_offset</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">vert_idx</span><span class="o">.</span><span class="n">item</span><span class="p">())</span>
|
||||
|
||||
<span class="c1"># invert V values</span>
|
||||
<span class="n">U_norm</span><span class="p">,</span> <span class="n">V_norm</span> <span class="o">=</span> <span class="n">U</span><span class="p">,</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">V</span>
|
||||
<span class="n">V_norm</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">V_norm</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@ -338,20 +353,18 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># Therefore when initializing the Meshes class,</span>
|
||||
<span class="c1"># we need to map each of the vertices referenced by the DensePose faces (in verts, which is the "All_vertices" field)</span>
|
||||
<span class="c1"># to the correct xyz coordinate in the SMPL template mesh.</span>
|
||||
<span class="n">v_template_extended</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">vert</span><span class="p">:</span> <span class="n">v_template</span><span class="p">[</span><span class="n">vert</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">verts</span><span class="p">)))</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (1, 7829, 3)</span>
|
||||
|
||||
<span class="c1"># add a batch dimension to faces</span>
|
||||
<span class="n">faces</span> <span class="o">=</span> <span class="n">faces</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">v_template_extended</span> <span class="o">=</span> <span class="n">v_template</span><span class="p">[</span><span class="n">verts</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="kc">None</span><span class="p">]</span> <span class="c1"># (1, 7829, 3)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-our-textured-mesh">Create our textured mesh<a class="anchor-link" href="#Create-our-textured-mesh">¶</a></h3><p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxillary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxiliary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -360,15 +373,16 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">texture</span> <span class="o">=</span> <span class="n">TexturesUV</span><span class="p">(</span><span class="n">maps</span><span class="o">=</span><span class="n">tex</span><span class="p">,</span> <span class="n">faces_uvs</span><span class="o">=</span><span class="n">faces</span><span class="p">,</span> <span class="n">verts_uvs</span><span class="o">=</span><span class="n">verts_uv</span><span class="p">)</span>
|
||||
<span class="n">mesh</span> <span class="o">=</span> <span class="n">Meshes</span><span class="p">(</span><span class="n">v_template_extended</span><span class="p">,</span> <span class="n">faces</span><span class="p">,</span> <span class="n">texture</span><span class="p">)</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">texture</span> <span class="o">=</span> <span class="n">TexturesUV</span><span class="p">(</span><span class="n">maps</span><span class="o">=</span><span class="n">tex</span><span class="p">,</span> <span class="n">faces_uvs</span><span class="o">=</span><span class="n">faces</span><span class="p">[</span><span class="kc">None</span><span class="p">],</span> <span class="n">verts_uvs</span><span class="o">=</span><span class="n">verts_uv</span><span class="p">)</span>
|
||||
<span class="n">mesh</span> <span class="o">=</span> <span class="n">Meshes</span><span class="p">(</span><span class="n">v_template_extended</span><span class="p">,</span> <span class="n">faces</span><span class="p">[</span><span class="kc">None</span><span class="p">],</span> <span class="n">texture</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2>
|
||||
</div>
|
||||
@ -396,7 +410,7 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># Place a point light in front of the person. </span>
|
||||
<span class="n">lights</span> <span class="o">=</span> <span class="n">PointLights</span><span class="p">(</span><span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">location</span><span class="o">=</span><span class="p">[[</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">]])</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will </span>
|
||||
<span class="c1"># interpolate the texture uv coordinates for each vertex, sample from a texture image and </span>
|
||||
<span class="c1"># apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
@ -416,7 +430,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Render the textured mesh we created from the SMPL model and texture map.</p>
|
||||
</div>
|
||||
@ -430,7 +445,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -438,7 +452,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Different-view-and-lighting-of-the-body">Different view and lighting of the body<a class="anchor-link" href="#Different-view-and-lighting-of-the-body">¶</a></h3><p>We can also change many other settings in the rendering pipeline. Here we:</p>
|
||||
<ul>
|
||||
@ -474,7 +489,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -482,10 +496,11 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Conclusion">Conclusion<a class="anchor-link" href="#Conclusion">¶</a></h2><p>In this tutorial, we've learned how to construct a <strong>textured mesh</strong> from <strong>DensePose model and uv data</strong>, as well as initialize a <strong>Renderer</strong> and change the viewing angle and lighting of our rendered mesh.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-DensePose">Render DensePose<a class="anchor-link" href="#Render-DensePose">¶</a></h1><p>DensePose refers to dense human pose representation: <a href="https://github.com/facebookresearch/DensePose">https://github.com/facebookresearch/DensePose</a>.
|
||||
In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p>
|
||||
@ -97,16 +98,18 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Import-modules">Import modules<a class="anchor-link" href="#Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If torch, torchvision and PyTorch3D are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -115,19 +118,25 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -157,17 +166,16 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
|
||||
<span class="c1"># libraries for reading data from files</span>
|
||||
<span class="kn">from</span> <span class="nn">scipy.io</span> <span class="k">import</span> <span class="n">loadmat</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io.utils</span> <span class="k">import</span> <span class="n">_read_image</span>
|
||||
<span class="kn">from</span> <span class="nn">scipy.io</span> <span class="kn">import</span> <span class="n">loadmat</span>
|
||||
<span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span>
|
||||
<span class="kn">import</span> <span class="nn">pickle</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -190,7 +198,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Load-the-SMPL-model">Load the SMPL model<a class="anchor-link" href="#Load-the-SMPL-model">¶</a></h2><h4 id="Download-the-SMPL-model">Download the SMPL model<a class="anchor-link" href="#Download-the-SMPL-model">¶</a></h4><ul>
|
||||
<li>Go to <a href="http://smpl.is.tue.mpg.de/downloads">http://smpl.is.tue.mpg.de/downloads</a> and sign up.</li>
|
||||
@ -222,7 +231,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Load our texture UV data and our SMPL data, with some processing to correct data values and format.</p>
|
||||
</div>
|
||||
@ -253,13 +263,15 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="n">data</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'latin1'</span><span class="p">)</span>
|
||||
<span class="n">v_template</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">'v_template'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (6890, 3)</span>
|
||||
<span class="n">ALP_UV</span> <span class="o">=</span> <span class="n">loadmat</span><span class="p">(</span><span class="n">data_filename</span><span class="p">)</span>
|
||||
<span class="n">tex</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">_read_image</span><span class="p">(</span><span class="n">file_name</span><span class="o">=</span><span class="n">tex_filename</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">'RGB'</span><span class="p">)</span> <span class="o">/</span> <span class="mf">255.</span> <span class="p">)</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
|
||||
<span class="k">with</span> <span class="n">Image</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">tex_filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">image</span><span class="p">:</span>
|
||||
<span class="n">np_image</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">image</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s2">"RGB"</span><span class="p">))</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span>
|
||||
<span class="n">tex</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">np_image</span> <span class="o">/</span> <span class="mf">255.</span><span class="p">)[</span><span class="kc">None</span><span class="p">]</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
|
||||
|
||||
<span class="n">verts</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s2">"All_vertices"</span><span class="p">])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">verts</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s2">"All_vertices"</span><span class="p">])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829,)</span>
|
||||
<span class="n">U</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_U_norm'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">V</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_V_norm'</span><span class="p">])</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (7829, 1)</span>
|
||||
<span class="n">faces</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">from_numpy</span><span class="p">((</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_Faces'</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">int</span><span class="p">))</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (13774, 3)</span>
|
||||
<span class="n">face_indices</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_FaceIndices'</span><span class="p">])</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span>
|
||||
<span class="n">face_indices</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">(</span><span class="n">ALP_UV</span><span class="p">[</span><span class="s1">'All_FaceIndices'</span><span class="p">])</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span> <span class="c1"># (13774,)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@ -273,7 +285,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># Display the texture image</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">tex</span><span class="o">.</span><span class="n">squeeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">cpu</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -281,7 +292,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>In DensePose, the body mesh is split into 24 parts. In the texture image, we can see the 24 parts are separated out into individual (200, 200) images per body part. The convention in DensePose is that each face in the mesh is associated with a body part (given by the face_indices tensor above). The vertex UV values (in the range [0, 1]) for each face are specific to the (200, 200) size texture map for the part of the body that the mesh face corresponds to. We cannot use them directly with the entire texture map. We have to offset the vertex UV values depending on what body part the associated face corresponds to.</p>
|
||||
</div>
|
||||
@ -301,6 +313,9 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="n">part</span> <span class="o">=</span> <span class="n">rows</span> <span class="o">*</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span> <span class="o">+</span> <span class="mi">1</span> <span class="c1"># parts are 1-indexed in face_indices</span>
|
||||
<span class="n">offset_per_part</span><span class="p">[</span><span class="n">part</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">u</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span>
|
||||
|
||||
<span class="n">U_norm</span> <span class="o">=</span> <span class="n">U</span><span class="o">.</span><span class="n">clone</span><span class="p">()</span>
|
||||
<span class="n">V_norm</span> <span class="o">=</span> <span class="n">V</span><span class="o">.</span><span class="n">clone</span><span class="p">()</span>
|
||||
|
||||
<span class="c1"># iterate over faces and offset the corresponding vertex u and v values</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">faces</span><span class="p">)):</span>
|
||||
<span class="n">face_vert_idxs</span> <span class="o">=</span> <span class="n">faces</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
|
||||
@ -311,15 +326,15 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># vertices are reused, but we don't want to offset multiple times</span>
|
||||
<span class="k">if</span> <span class="n">vert_idx</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">already_offset</span><span class="p">:</span>
|
||||
<span class="c1"># offset u value</span>
|
||||
<span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">/</span> <span class="n">cols</span> <span class="o">+</span> <span class="n">offset_u</span>
|
||||
<span class="n">U_norm</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">U</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">/</span> <span class="n">cols</span> <span class="o">+</span> <span class="n">offset_u</span>
|
||||
<span class="c1"># offset v value</span>
|
||||
<span class="c1"># this also flips each part locally, as each part is upside down</span>
|
||||
<span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">])</span> <span class="o">/</span> <span class="n">rows</span> <span class="o">+</span> <span class="n">offset_v</span>
|
||||
<span class="n">V_norm</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">V</span><span class="p">[</span><span class="n">vert_idx</span><span class="p">])</span> <span class="o">/</span> <span class="n">rows</span> <span class="o">+</span> <span class="n">offset_v</span>
|
||||
<span class="c1"># add vertex to our set tracking offsetted vertices</span>
|
||||
<span class="n">already_offset</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">vert_idx</span><span class="o">.</span><span class="n">item</span><span class="p">())</span>
|
||||
|
||||
<span class="c1"># invert V values</span>
|
||||
<span class="n">U_norm</span><span class="p">,</span> <span class="n">V_norm</span> <span class="o">=</span> <span class="n">U</span><span class="p">,</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">V</span>
|
||||
<span class="n">V_norm</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">V_norm</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@ -338,20 +353,18 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># Therefore when initializing the Meshes class,</span>
|
||||
<span class="c1"># we need to map each of the vertices referenced by the DensePose faces (in verts, which is the "All_vertices" field)</span>
|
||||
<span class="c1"># to the correct xyz coordinate in the SMPL template mesh.</span>
|
||||
<span class="n">v_template_extended</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">vert</span><span class="p">:</span> <span class="n">v_template</span><span class="p">[</span><span class="n">vert</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">verts</span><span class="p">)))</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span> <span class="c1"># (1, 7829, 3)</span>
|
||||
|
||||
<span class="c1"># add a batch dimension to faces</span>
|
||||
<span class="n">faces</span> <span class="o">=</span> <span class="n">faces</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">v_template_extended</span> <span class="o">=</span> <span class="n">v_template</span><span class="p">[</span><span class="n">verts</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="kc">None</span><span class="p">]</span> <span class="c1"># (1, 7829, 3)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Create-our-textured-mesh">Create our textured mesh<a class="anchor-link" href="#Create-our-textured-mesh">¶</a></h3><p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxillary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxiliary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -360,15 +373,16 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">texture</span> <span class="o">=</span> <span class="n">TexturesUV</span><span class="p">(</span><span class="n">maps</span><span class="o">=</span><span class="n">tex</span><span class="p">,</span> <span class="n">faces_uvs</span><span class="o">=</span><span class="n">faces</span><span class="p">,</span> <span class="n">verts_uvs</span><span class="o">=</span><span class="n">verts_uv</span><span class="p">)</span>
|
||||
<span class="n">mesh</span> <span class="o">=</span> <span class="n">Meshes</span><span class="p">(</span><span class="n">v_template_extended</span><span class="p">,</span> <span class="n">faces</span><span class="p">,</span> <span class="n">texture</span><span class="p">)</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">texture</span> <span class="o">=</span> <span class="n">TexturesUV</span><span class="p">(</span><span class="n">maps</span><span class="o">=</span><span class="n">tex</span><span class="p">,</span> <span class="n">faces_uvs</span><span class="o">=</span><span class="n">faces</span><span class="p">[</span><span class="kc">None</span><span class="p">],</span> <span class="n">verts_uvs</span><span class="o">=</span><span class="n">verts_uv</span><span class="p">)</span>
|
||||
<span class="n">mesh</span> <span class="o">=</span> <span class="n">Meshes</span><span class="p">(</span><span class="n">v_template_extended</span><span class="p">,</span> <span class="n">faces</span><span class="p">[</span><span class="kc">None</span><span class="p">],</span> <span class="n">texture</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Create-a-renderer">Create a renderer<a class="anchor-link" href="#Create-a-renderer">¶</a></h2>
|
||||
</div>
|
||||
@ -396,7 +410,7 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<span class="c1"># Place a point light in front of the person. </span>
|
||||
<span class="n">lights</span> <span class="o">=</span> <span class="n">PointLights</span><span class="p">(</span><span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">location</span><span class="o">=</span><span class="p">[[</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">]])</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will </span>
|
||||
<span class="c1"># interpolate the texture uv coordinates for each vertex, sample from a texture image and </span>
|
||||
<span class="c1"># apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
@ -416,7 +430,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>Render the textured mesh we created from the SMPL model and texture map.</p>
|
||||
</div>
|
||||
@ -430,7 +445,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -438,7 +452,8 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="Different-view-and-lighting-of-the-body">Different view and lighting of the body<a class="anchor-link" href="#Different-view-and-lighting-of-the-body">¶</a></h3><p>We can also change many other settings in the rendering pipeline. Here we:</p>
|
||||
<ul>
|
||||
@ -474,7 +489,6 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -482,10 +496,11 @@ In this tutorial, we provide an example of using DensePose data in PyTorch3D.</p
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Conclusion">Conclusion<a class="anchor-link" href="#Conclusion">¶</a></h2><p>In this tutorial, we've learned how to construct a <strong>textured mesh</strong> from <strong>DensePose model and uv data</strong>, as well as initialize a <strong>Renderer</strong> and change the viewing angle and lighting of our rendered mesh.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-a-textured-mesh">Render a textured mesh<a class="anchor-link" href="#Render-a-textured-mesh">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -96,16 +97,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -114,19 +117,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -144,16 +153,15 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="c1"># Util function for loading meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">load_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">load_obj</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="k">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.texture_vis</span> <span class="k">import</span> <span class="n">texturesuv_image_matplotlib</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="kn">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.texture_vis</span> <span class="kn">import</span> <span class="n">texturesuv_image_matplotlib</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -177,7 +185,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -189,14 +198,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -214,17 +224,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and it's associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and its associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxillary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxiliary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>Meshes</strong> has several class methods which are used throughout the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the mesh obj and texture files and save it at the path <code>data/cow_mesh</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -269,7 +281,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h4 id="Let's-visualize-the-texture-map">Let's visualize the texture map<a class="anchor-link" href="#Let's-visualize-the-texture-map">¶</a></h4>
|
||||
</div>
|
||||
@ -283,7 +296,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span><span class="mi">7</span><span class="p">))</span>
|
||||
<span class="n">texture_image</span><span class="o">=</span><span class="n">mesh</span><span class="o">.</span><span class="n">textures</span><span class="o">.</span><span class="n">maps_padded</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">texture_image</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -291,7 +303,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>PyTorch3D has a built-in way to view the texture map with matplotlib along with the points on the map corresponding to vertices. There is also a method, texturesuv_image_PIL, to get a similar image which can be saved to a file.</p>
|
||||
</div>
|
||||
@ -304,7 +317,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span><span class="mi">7</span><span class="p">))</span>
|
||||
<span class="n">texturesuv_image_matplotlib</span><span class="p">(</span><span class="n">mesh</span><span class="o">.</span><span class="n">textures</span><span class="p">,</span> <span class="n">subsample</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -312,10 +324,11 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Create-a-renderer">2. Create a renderer<a class="anchor-link" href="#2.-Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses a <strong>perspective camera</strong>, a <strong>point light</strong> and applies <strong>phong shading</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses a <strong>perspective camera</strong>, a <strong>point light</strong> and applies <strong>Phong shading</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -346,7 +359,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="c1"># -z direction. </span>
|
||||
<span class="n">lights</span> <span class="o">=</span> <span class="n">PointLights</span><span class="p">(</span><span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">location</span><span class="o">=</span><span class="p">[[</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">3.0</span><span class="p">]])</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will </span>
|
||||
<span class="c1"># interpolate the texture uv coordinates for each vertex, sample from a texture image and </span>
|
||||
<span class="c1"># apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
@ -366,14 +379,16 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Render-the-mesh">3. Render the mesh<a class="anchor-link" href="#3.-Render-the-mesh">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>The light is in front of the object so it is bright and the image has specular highlights.</p>
|
||||
</div>
|
||||
@ -387,7 +402,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -395,9 +409,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Move-the-light-behind-the-object-and-re-render">4. Move the light behind the object and re-render<a class="anchor-link" href="#4.-Move-the-light-behind-the-object-and-re-render">¶</a></h2><p>We can pass arbirary keyword arguments to the <code>rasterizer</code>/<code>shader</code> via the call to the <code>renderer</code> so the renderer does not need to be reinitialized if any of the settings change/</p>
|
||||
<h2 id="4.-Move-the-light-behind-the-object-and-re-render">4. Move the light behind the object and re-render<a class="anchor-link" href="#4.-Move-the-light-behind-the-object-and-re-render">¶</a></h2><p>We can pass arbitrary keyword arguments to the <code>rasterizer</code>/<code>shader</code> via the call to the <code>renderer</code> so the renderer does not need to be reinitialized if any of the settings change/</p>
|
||||
<p>In this case, we can simply update the location of the lights and pass them into the call to the renderer.</p>
|
||||
<p>The image is now dark as there is only ambient lighting, and there are no specular highlights.</p>
|
||||
</div>
|
||||
@ -423,7 +438,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -431,7 +445,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Rotate-the-object,-modify-the-material-properties-or-light-properties">5. Rotate the object, modify the material properties or light properties<a class="anchor-link" href="#5.-Rotate-the-object,-modify-the-material-properties-or-light-properties">¶</a></h2><p>We can also change many other settings in the rendering pipeline. Here we:</p>
|
||||
<ul>
|
||||
@ -475,7 +490,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -483,7 +497,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="6.-Batched-Rendering">6. Batched Rendering<a class="anchor-link" href="#6.-Batched-Rendering">¶</a></h2><p>One of the core design choices of the PyTorch3D API is to support <strong>batched inputs for all components</strong>.
|
||||
The renderer and associated components can take batched inputs and <strong>render a batch of output images in one forward pass</strong>. We will now use this feature to render the mesh from many different viewpoints.</p>
|
||||
@ -525,7 +540,7 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># We can pass arbirary keyword arguments to the rasterizer/shader via the renderer</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer</span>
|
||||
<span class="c1"># so the renderer does not need to be reinitialized if any of the settings change.</span>
|
||||
<span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">meshes</span><span class="p">,</span> <span class="n">cameras</span><span class="o">=</span><span class="n">cameras</span><span class="p">,</span> <span class="n">lights</span><span class="o">=</span><span class="n">lights</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
@ -545,7 +560,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="7.-Plotly-visualization">7. Plotly visualization<a class="anchor-link" href="#7.-Plotly-visualization">¶</a></h2><p>If you only want to visualize a mesh, you don't really need to use a differentiable renderer - instead we support plotting of Meshes with plotly. For these Meshes, we use TexturesVertex to define a texture for the rendering.
|
||||
<code>plot_meshes</code> creates a Plotly figure with a trace for each Meshes object.</p>
|
||||
@ -668,7 +684,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>For batches, we can also use <code>plot_batch_individually</code> to avoid constructing the scene dictionary ourselves.</p>
|
||||
</div>
|
||||
@ -693,7 +710,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can also modify the axis arguments and axis backgrounds in both functions.</p>
|
||||
</div>
|
||||
@ -739,10 +757,11 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="8.-Conclusion">8. Conclusion<a class="anchor-link" href="#8.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to <strong>load</strong> a textured mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an <strong>Renderer</strong> consisting of a <strong>Rasterizer</strong> and a <strong>Shader</strong>, and modify several components of the rendering pipeline. We also learned how to render Meshes in Plotly figures.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -82,7 +82,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h1 id="Render-a-textured-mesh">Render a textured mesh<a class="anchor-link" href="#Render-a-textured-mesh">¶</a></h1><p>This tutorial shows how to:</p>
|
||||
<ul>
|
||||
@ -96,16 +97,18 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="0.-Install-and-Import-modules">0. Install and Import modules<a class="anchor-link" href="#0.-Install-and-Import-modules">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If <code>torch</code>, <code>torchvision</code> and <code>pytorch3d</code> are not installed, run the following cell:</p>
|
||||
<p>Ensure <code>torch</code> and <code>torchvision</code> are installed. If <code>pytorch3d</code> is not installed, install it using the following cell:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -114,19 +117,25 @@
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>pip install torch torchvision
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">==</span><span class="s1">'1.6.0+cu101'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'linux'</span><span class="p">):</span>
|
||||
<span class="o">!</span>pip install pytorch3d
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="n">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">False</span>
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">pytorch3d</span>
|
||||
<span class="k">except</span> <span class="ne">ModuleNotFoundError</span><span class="p">:</span>
|
||||
<span class="n">need_pytorch3d</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="k">if</span> <span class="n">need_pytorch3d</span><span class="p">:</span>
|
||||
<span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"1.9"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"linux"</span><span class="p">):</span>
|
||||
<span class="c1"># We try to install PyTorch3D via a released wheel.</span>
|
||||
<span class="n">version_str</span><span class="o">=</span><span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
|
||||
<span class="sa">f</span><span class="s2">"py3</span><span class="si">{</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="o">.</span><span class="n">minor</span><span class="si">}</span><span class="s2">_cu"</span><span class="p">,</span>
|
||||
<span class="n">torch</span><span class="o">.</span><span class="n">version</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">""</span><span class="p">),</span>
|
||||
<span class="sa">f</span><span class="s2">"_pyt</span><span class="si">{</span><span class="n">torch</span><span class="o">.</span><span class="n">__version__</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">5</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
|
||||
<span class="p">])</span>
|
||||
<span class="o">!</span>pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/<span class="o">{</span>version_str<span class="o">}</span>/download.html
|
||||
<span class="k">else</span><span class="p">:</span>
|
||||
<span class="c1"># We try to install PyTorch3D from source.</span>
|
||||
<span class="o">!</span>curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
|
||||
<span class="o">!</span>tar xzf <span class="m">1</span>.10.0.tar.gz
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"CUB_HOME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getcwd</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"/cub-1.10.0"</span>
|
||||
@ -144,16 +153,15 @@
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">torch</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">skimage.io</span> <span class="k">import</span> <span class="n">imread</span>
|
||||
|
||||
<span class="c1"># Util function for loading meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="k">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">load_obj</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.io</span> <span class="kn">import</span> <span class="n">load_objs_as_meshes</span><span class="p">,</span> <span class="n">load_obj</span>
|
||||
|
||||
<span class="c1"># Data structures and functions for rendering</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="k">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="k">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.texture_vis</span> <span class="k">import</span> <span class="n">texturesuv_image_matplotlib</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="k">import</span> <span class="p">(</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.structures</span> <span class="kn">import</span> <span class="n">Meshes</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.plotly_vis</span> <span class="kn">import</span> <span class="n">AxisArgs</span><span class="p">,</span> <span class="n">plot_batch_individually</span><span class="p">,</span> <span class="n">plot_scene</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.vis.texture_vis</span> <span class="kn">import</span> <span class="n">texturesuv_image_matplotlib</span>
|
||||
<span class="kn">from</span> <span class="nn">pytorch3d.renderer</span> <span class="kn">import</span> <span class="p">(</span>
|
||||
<span class="n">look_at_view_transform</span><span class="p">,</span>
|
||||
<span class="n">FoVPerspectiveCameras</span><span class="p">,</span>
|
||||
<span class="n">PointLights</span><span class="p">,</span>
|
||||
@ -177,7 +185,8 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If using <strong>Google Colab</strong>, fetch the utils file for plotting image grids:</p>
|
||||
</div>
|
||||
@ -189,14 +198,15 @@
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="k">import</span> <span class="n">image_grid</span>
|
||||
<span class="kn">from</span> <span class="nn">plot_image_grid</span> <span class="kn">import</span> <span class="n">image_grid</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>OR if running <strong>locally</strong> uncomment and run the following cell:</p>
|
||||
</div>
|
||||
@ -214,17 +224,19 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and it's associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<h3 id="1.-Load-a-mesh-and-texture-file">1. Load a mesh and texture file<a class="anchor-link" href="#1.-Load-a-mesh-and-texture-file">¶</a></h3><p>Load an <code>.obj</code> file and its associated <code>.mtl</code> file and create a <strong>Textures</strong> and <strong>Meshes</strong> object.</p>
|
||||
<p><strong>Meshes</strong> is a unique datastructure provided in PyTorch3D for working with batches of meshes of different sizes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxillary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>TexturesUV</strong> is an auxiliary datastructure for storing vertex uv and texture maps for meshes.</p>
|
||||
<p><strong>Meshes</strong> has several class methods which are used throughout the rendering pipeline.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>If running this notebook using <strong>Google Colab</strong>, run the following cell to fetch the mesh obj and texture files and save it at the path <code>data/cow_mesh</code>:
|
||||
If running locally, the data is already available at the correct path.</p>
|
||||
@ -269,7 +281,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h4 id="Let's-visualize-the-texture-map">Let's visualize the texture map<a class="anchor-link" href="#Let's-visualize-the-texture-map">¶</a></h4>
|
||||
</div>
|
||||
@ -283,7 +296,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span><span class="mi">7</span><span class="p">))</span>
|
||||
<span class="n">texture_image</span><span class="o">=</span><span class="n">mesh</span><span class="o">.</span><span class="n">textures</span><span class="o">.</span><span class="n">maps_padded</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">texture_image</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -291,7 +303,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>PyTorch3D has a built-in way to view the texture map with matplotlib along with the points on the map corresponding to vertices. There is also a method, texturesuv_image_PIL, to get a similar image which can be saved to a file.</p>
|
||||
</div>
|
||||
@ -304,7 +317,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span><span class="mi">7</span><span class="p">))</span>
|
||||
<span class="n">texturesuv_image_matplotlib</span><span class="p">(</span><span class="n">mesh</span><span class="o">.</span><span class="n">textures</span><span class="p">,</span> <span class="n">subsample</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -312,10 +324,11 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="2.-Create-a-renderer">2. Create a renderer<a class="anchor-link" href="#2.-Create-a-renderer">¶</a></h2><p>A renderer in PyTorch3D is composed of a <strong>rasterizer</strong> and a <strong>shader</strong> which each have a number of subcomponents such as a <strong>camera</strong> (orthographic/perspective). Here we initialize some of these components and use default values for the rest.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses a <strong>perspective camera</strong>, a <strong>point light</strong> and applies <strong>phong shading</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
<p>In this example we will first create a <strong>renderer</strong> which uses a <strong>perspective camera</strong>, a <strong>point light</strong> and applies <strong>Phong shading</strong>. Then we learn how to vary different components using the modular API.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -346,7 +359,7 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<span class="c1"># -z direction. </span>
|
||||
<span class="n">lights</span> <span class="o">=</span> <span class="n">PointLights</span><span class="p">(</span><span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">location</span><span class="o">=</span><span class="p">[[</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">3.0</span><span class="p">]])</span>
|
||||
|
||||
<span class="c1"># Create a phong renderer by composing a rasterizer and a shader. The textured phong shader will </span>
|
||||
<span class="c1"># Create a Phong renderer by composing a rasterizer and a shader. The textured Phong shader will </span>
|
||||
<span class="c1"># interpolate the texture uv coordinates for each vertex, sample from a texture image and </span>
|
||||
<span class="c1"># apply the Phong lighting model</span>
|
||||
<span class="n">renderer</span> <span class="o">=</span> <span class="n">MeshRenderer</span><span class="p">(</span>
|
||||
@ -366,14 +379,16 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="3.-Render-the-mesh">3. Render the mesh<a class="anchor-link" href="#3.-Render-the-mesh">¶</a></h2>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>The light is in front of the object so it is bright and the image has specular highlights.</p>
|
||||
</div>
|
||||
@ -387,7 +402,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">mesh</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -395,9 +409,10 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="4.-Move-the-light-behind-the-object-and-re-render">4. Move the light behind the object and re-render<a class="anchor-link" href="#4.-Move-the-light-behind-the-object-and-re-render">¶</a></h2><p>We can pass arbirary keyword arguments to the <code>rasterizer</code>/<code>shader</code> via the call to the <code>renderer</code> so the renderer does not need to be reinitialized if any of the settings change/</p>
|
||||
<h2 id="4.-Move-the-light-behind-the-object-and-re-render">4. Move the light behind the object and re-render<a class="anchor-link" href="#4.-Move-the-light-behind-the-object-and-re-render">¶</a></h2><p>We can pass arbitrary keyword arguments to the <code>rasterizer</code>/<code>shader</code> via the call to the <code>renderer</code> so the renderer does not need to be reinitialized if any of the settings change/</p>
|
||||
<p>In this case, we can simply update the location of the lights and pass them into the call to the renderer.</p>
|
||||
<p>The image is now dark as there is only ambient lighting, and there are no specular highlights.</p>
|
||||
</div>
|
||||
@ -423,7 +438,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -431,7 +445,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="5.-Rotate-the-object,-modify-the-material-properties-or-light-properties">5. Rotate the object, modify the material properties or light properties<a class="anchor-link" href="#5.-Rotate-the-object,-modify-the-material-properties-or-light-properties">¶</a></h2><p>We can also change many other settings in the rendering pipeline. Here we:</p>
|
||||
<ul>
|
||||
@ -475,7 +490,6 @@ If running locally, the data is already available at the correct path.</p>
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">images</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s2">"off"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
@ -483,7 +497,8 @@ If running locally, the data is already available at the correct path.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="6.-Batched-Rendering">6. Batched Rendering<a class="anchor-link" href="#6.-Batched-Rendering">¶</a></h2><p>One of the core design choices of the PyTorch3D API is to support <strong>batched inputs for all components</strong>.
|
||||
The renderer and associated components can take batched inputs and <strong>render a batch of output images in one forward pass</strong>. We will now use this feature to render the mesh from many different viewpoints.</p>
|
||||
@ -525,7 +540,7 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
<div class="prompt input_prompt">In [ ]:</div>
|
||||
<div class="inner_cell">
|
||||
<div class="input_area">
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># We can pass arbirary keyword arguments to the rasterizer/shader via the renderer</span>
|
||||
<div class="highlight hl-ipython3"><pre><span></span><span class="c1"># We can pass arbitrary keyword arguments to the rasterizer/shader via the renderer</span>
|
||||
<span class="c1"># so the renderer does not need to be reinitialized if any of the settings change.</span>
|
||||
<span class="n">images</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">(</span><span class="n">meshes</span><span class="p">,</span> <span class="n">cameras</span><span class="o">=</span><span class="n">cameras</span><span class="p">,</span> <span class="n">lights</span><span class="o">=</span><span class="n">lights</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
@ -545,7 +560,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="7.-Plotly-visualization">7. Plotly visualization<a class="anchor-link" href="#7.-Plotly-visualization">¶</a></h2><p>If you only want to visualize a mesh, you don't really need to use a differentiable renderer - instead we support plotting of Meshes with plotly. For these Meshes, we use TexturesVertex to define a texture for the rendering.
|
||||
<code>plot_meshes</code> creates a Plotly figure with a trace for each Meshes object.</p>
|
||||
@ -668,7 +684,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>For batches, we can also use <code>plot_batch_individually</code> to avoid constructing the scene dictionary ourselves.</p>
|
||||
</div>
|
||||
@ -693,7 +710,8 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<p>We can also modify the axis arguments and axis backgrounds in both functions.</p>
|
||||
</div>
|
||||
@ -739,10 +757,11 @@ The renderer and associated components can take batched inputs and <strong>rende
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
|
||||
</div><div class="inner_cell">
|
||||
</div>
|
||||
<div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="8.-Conclusion">8. Conclusion<a class="anchor-link" href="#8.-Conclusion">¶</a></h2><p>In this tutorial we learnt how to <strong>load</strong> a textured mesh from an obj file, initialize a PyTorch3D datastructure called <strong>Meshes</strong>, set up an <strong>Renderer</strong> consisting of a <strong>Rasterizer</strong> and a <strong>Shader</strong>, and modify several components of the rendering pipeline. We also learned how to render Meshes in Plotly figures.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</div></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -6,4 +6,4 @@
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
<!DOCTYPE html><html lang=""><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>PyTorch3D · A library for deep learning with 3D data</title><meta name="viewport" content="width=device-width, initial-scale=1.0"/><meta name="generator" content="Docusaurus"/><meta name="description" content="A library for deep learning with 3D data"/><meta property="og:title" content="PyTorch3D · A library for deep learning with 3D data"/><meta property="og:type" content="website"/><meta property="og:url" content="https://pytorch3d.org/"/><meta property="og:description" content="A library for deep learning with 3D data"/><meta property="og:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://pytorch3d.org/img/pytorch3dlogoicon.svg"/><link rel="shortcut icon" href="/img/pytorch3dfavicon.png"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
@ -6,4 +6,4 @@
|
||||
|
||||
ga('create', 'UA-157376881-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2020 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
</script><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/pytorch3dfavicon.png" alt="PyTorch3D"/><h2 class="headerTitleWithLogo">PyTorch3D</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/docs/why_pytorch3d" target="_self">Docs</a></li><li class=""><a href="/tutorials" target="_self">Tutorials</a></li><li class=""><a href="https://pytorch3d.readthedocs.io/" target="_self">API</a></li><li class=""><a href="https://github.com/facebookresearch/pytorch3d" target="_self">GitHub</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="mainContainer"><div class="container paddingBottom paddingTop"><div class="wrapper"><div class="showcaseSection"><div class="prose"><h1>Who is Using This?</h1><p>This project is used by many folks</p></div><div class="logos"><a href="https://www.facebook.com"><img src="/img/undraw_open_source.svg" alt="User1" title="User1"/></a></div><p>Are you using this project?</p><a href="undefined/edit/master/website/siteConfig.js" class="button">Add your company</a></div></div></div></div><footer class="nav-footer" id="footer"><section class="sitemap"><div class="footerSection"><div class="social"><a class="github-button" href="https://github.com/facebookresearch/pytorch3d" data-count-href="https://github.com/facebookresearch/pytorch3d/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star PyTorch3D on GitHub">pytorch3d</a></div></div></section><a href="https://opensource.facebook.com/" target="_blank" rel="noreferrer noopener" class="fbOpenSource"><img src="/img/oss_logo.png" alt="Facebook Open Source" width="170" height="45"/></a><section class="copyright">Copyright © 2021 Facebook Inc<br/>Legal:<a href="https://opensource.facebook.com/legal/privacy/" target="_blank" rel="noreferrer noopener">Privacy</a><a href="https://opensource.facebook.com/legal/terms/" target="_blank" rel="noreferrer noopener">Terms</a></section></footer></div></body></html>
|
||||
Loading…
x
Reference in New Issue
Block a user