RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

Di Chang; Aljaž Božič; Tong Zhang; Qingsong Yan; Yingcong Chen; Sabine Süsstrunk; Matthias Nießner

doi:10.1007/978-3-031-19821-2_38

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

Di Chang^*, Aljaž Božič, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, Matthias Nießner

^*Corresponding author for this work

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

53 Citations (Scopus)

Abstract

Finding accurate correspondences among different views is the Achilles’ heel of unsupervised Multi-View Stereo (MVS). Existing methods are built upon the assumption that corresponding pixels share similar photometric features. However, multi-view images in real scenarios observe non-Lambertian surfaces and experience occlusions. In this work, we propose a novel approach with neural rendering (RC-MVSNet) to solve such ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface to alleviate occlusions. Concurrently, we introduce a reference view synthesis loss to generate consistent supervision, even for non-Lambertian surfaces. Extensive experiments on DTU and Tanks &Temples benchmarks demonstrate that our RC-MVSNet approach achieves state-of-the-art performance over unsupervised MVS frameworks and competitive performance to many supervised methods. The code is released at https://github.com/Boese0601/RC-MVSNet.

Original language	English
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	665-680
Number of pages	16
ISBN (Print)	9783031198205
DOIs	https://doi.org/10.1007/978-3-031-19821-2_38
Publication status	Published - 2022
Externally published	Yes
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13691 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

Depth estimation
End-to-end Unsupervised Multi-View Stereo
Neural rendering

Access to Document

10.1007/978-3-031-19821-2_38

Cite this

Chang, D., Božič, A., Zhang, T., Yan, Q., Chen, Y., Süsstrunk, S., & Nießner, M. (2022). RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings (pp. 665-680). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13691 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-19821-2_38

Chang, Di ; Božič, Aljaž ; Zhang, Tong et al. / RC-MVSNet : Unsupervised Multi-View Stereo with Neural Rendering. Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. pp. 665-680 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d814ed79ed424c01a8ddb2d48ca19e47,

title = "RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering",

abstract = "Finding accurate correspondences among different views is the Achilles{\textquoteright} heel of unsupervised Multi-View Stereo (MVS). Existing methods are built upon the assumption that corresponding pixels share similar photometric features. However, multi-view images in real scenarios observe non-Lambertian surfaces and experience occlusions. In this work, we propose a novel approach with neural rendering (RC-MVSNet) to solve such ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface to alleviate occlusions. Concurrently, we introduce a reference view synthesis loss to generate consistent supervision, even for non-Lambertian surfaces. Extensive experiments on DTU and Tanks \&Temples benchmarks demonstrate that our RC-MVSNet approach achieves state-of-the-art performance over unsupervised MVS frameworks and competitive performance to many supervised methods. The code is released at https://github.com/Boese0601/RC-MVSNet.",

keywords = "Depth estimation, End-to-end Unsupervised Multi-View Stereo, Neural rendering",

author = "Di Chang and Alja{\v z} Bo{\v z}i{\v c} and Tong Zhang and Qingsong Yan and Yingcong Chen and Sabine S{\"u}sstrunk and Matthias Nie{\ss}ner",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-19821-2\_38",

language = "English",

isbn = "9783031198205",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "665--680",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, \{Giovanni Maria\} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, Proceedings",

}

Chang, D, Božič, A, Zhang, T, Yan, Q, Chen, Y, Süsstrunk, S & Nießner, M 2022, RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13691 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 665-680, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-19821-2_38

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering. / Chang, Di; Božič, Aljaž; Zhang, Tong et al.
Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. p. 665-680 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13691 LNCS).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - RC-MVSNet

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Chang, Di

AU - Božič, Aljaž

AU - Zhang, Tong

AU - Yan, Qingsong

AU - Chen, Yingcong

AU - Süsstrunk, Sabine

AU - Nießner, Matthias

PY - 2022

Y1 - 2022

N2 - Finding accurate correspondences among different views is the Achilles’ heel of unsupervised Multi-View Stereo (MVS). Existing methods are built upon the assumption that corresponding pixels share similar photometric features. However, multi-view images in real scenarios observe non-Lambertian surfaces and experience occlusions. In this work, we propose a novel approach with neural rendering (RC-MVSNet) to solve such ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface to alleviate occlusions. Concurrently, we introduce a reference view synthesis loss to generate consistent supervision, even for non-Lambertian surfaces. Extensive experiments on DTU and Tanks &Temples benchmarks demonstrate that our RC-MVSNet approach achieves state-of-the-art performance over unsupervised MVS frameworks and competitive performance to many supervised methods. The code is released at https://github.com/Boese0601/RC-MVSNet.

AB - Finding accurate correspondences among different views is the Achilles’ heel of unsupervised Multi-View Stereo (MVS). Existing methods are built upon the assumption that corresponding pixels share similar photometric features. However, multi-view images in real scenarios observe non-Lambertian surfaces and experience occlusions. In this work, we propose a novel approach with neural rendering (RC-MVSNet) to solve such ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface to alleviate occlusions. Concurrently, we introduce a reference view synthesis loss to generate consistent supervision, even for non-Lambertian surfaces. Extensive experiments on DTU and Tanks &Temples benchmarks demonstrate that our RC-MVSNet approach achieves state-of-the-art performance over unsupervised MVS frameworks and competitive performance to many supervised methods. The code is released at https://github.com/Boese0601/RC-MVSNet.

KW - Depth estimation

KW - End-to-end Unsupervised Multi-View Stereo

KW - Neural rendering

UR - https://www.scopus.com/pages/publications/85142691911

U2 - 10.1007/978-3-031-19821-2_38

DO - 10.1007/978-3-031-19821-2_38

M3 - Conference Paper published in a book

AN - SCOPUS:85142691911

SN - 9783031198205

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 665

EP - 680

BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 October 2022 through 27 October 2022

ER -

Chang D, Božič A, Zhang T, Yan Q, Chen Y, Süsstrunk S et al. RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 665-680. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-19821-2_38

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this