Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warping operation formula in MVSplat paper #80

Open
trungphien opened this issue Nov 29, 2024 · 1 comment
Open

Warping operation formula in MVSplat paper #80

trungphien opened this issue Nov 29, 2024 · 1 comment

Comments

@trungphien
Copy link

Hi, could you describe the warping function that you use in the paper in more detail? I noticed that the code you provided and the corresponding formula in the paper "Unifying Flow, Stereo and Depth Estimation" do not match. Thank you very much!

@donydchen
Copy link
Owner

donydchen commented Dec 19, 2024

Hi, @trungphien, sorry for the late reply. They should be mathematically equivalent. The main difference is that the formula is in Homogeneous coordinates, while the implementation is in Cartesian coordinates.

In particular, with reference to UniMatch Eq. (9),

where $H$ denotes the homogeneous coordinates, which represent rotation and translation as unified transformations.

While in the implementation, you can see that we perform separately the rotation (pose[:, :3, :3]) and then the translation (pose[:, :3, -1:]), as in Cartesian coordinates, as shown in

points = torch.bmm(pose[:, :3, :3], points).unsqueeze(2).repeat(
1, 1, d, 1
) * depth.view(
b, 1, d, h * w
) # [B, 3, D, H*W]
points = points + pose[:, :3, -1:].unsqueeze(-1) # [B, 3, D, H*W]

Note that the pose here is a relative pose to the first view, i.e., $E_2E_1^{-1}$, rather than the original extrinsic, i.e., $E_2$, see

pose_ref = extrinsics[:, 0].clone().detach()
pose_tgt = extrinsics[:, 1].clone().detach()
pose = pose_tgt.inverse() @ pose_ref

Let me know if it is unclear. Cheers.

@donydchen donydchen pinned this issue Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants