(breaking) image_size-agnostic GridRaySampler

Summary:
As suggested in #802. By not persisting the _xy_grid buffer, we can allow (in some cases) a model with one image_size to be loaded from a saved model which was trained at a different resolution.

Also avoid persisting _frequencies in HarmonicEmbedding for similar reasons.

BC-break: This will cause load_state_dict, in strict mode, to complain if you try to load an old model with the new code.

Reviewed By: patricklabatut

Differential Revision: D30349234

fbshipit-source-id: d6061d1e51c9f79a78d61a9f732c9a5dfadbbb47

pytorch3d/renderer/implicit/raysampling.py

@@ -96,7 +96,7 @@ class GridRaysampler(torch.nn.Module):
             ),
             dim=-1,
         )
-        self.register_buffer("_xy_grid", _xy_grid)
+        self.register_buffer("_xy_grid", _xy_grid, persistent=False)
 
     def forward(self, cameras: CamerasBase, **kwargs) -> RayBundle:
         """