Jiazhao Zhang
Kai Xu
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the Particle Filter Optimization (PFO). To surmount the computation-intensive particle sampling and update in standard PFO, we propose to accelerate the randomized search via updating a particle swarm template (PST). PST is a set of particles pre-sampled uniformly within the unit sphere in the 6D space of camera pose. Through moving and rescaling the pre-sampled PST guided by swarm intelligence, our method is able to drive tens of thousands of particles to locate and cover a good local optimum extremely fast and robustly. The particles, representing candidate poses, are evaluated with a fitness function defined based on depth-model conformance. Therefore, our method, being depth-only and correspondence-free, mitigates the motion blur impediment as (ToF-based) depths are often resilient to motion blur. Thanks to the efficient template-based particle set evolution and the effective fitness function, our method attains good quality pose tracking under fast camera motion (up to 4m/s) in a realtime framerate without including loop closure or global pose optimization. Through extensive evaluations on public datasets of RGB-D sequences, especially on a newly proposed benchmark of fast camera motion, we demonstrate the significant advantage of our method over the state of the arts.
Supplemental Material
- Christophe Andrieu and Arnaud Doucet. 2002. Particle filtering for partially observed Gaussian state space models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64, 4 (2002), 827--836.Google Scholar
Cross Ref
- Elise Arnaud and Etienne Mémin. 2005. An efficient Rao-Blackwellized particle filter for object tracking. In IEEE International Conference on Image Processing 2005, Vol. 2. IEEE, II-426.Google ScholarCross Ref
- Robert Bridson. 2007. Fast Poisson disk sampling in arbitrary dimensions. SIGGRAPH sketches 10 (2007), 1.Google Scholar
- Erik Bylow, Jürgen Sturm, Christian Kerl, Fredrik Kahl, and Daniel Cremers. 2013. Real-time camera tracking and 3D reconstruction using signed distance functions. In Robotics: Science and Systems, Vol. 2. 2.Google Scholar
- Jiawen Chen, Dennis Bautembach, and Shahram Izadi. 2013. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics (ToG) 32, 4 (2013), 1--16.Google ScholarDigital Library
- Changhyun Choi and Henrik I Christensen. 2012. Robust 3D visual tracking using particle filtering on the special Euclidean group: A combined approach of keypoint and edge features. The International Journal of Robotics Research 31, 4 (2012), 498--519.Google ScholarDigital Library
- Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. 2015. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5556--5565.Google Scholar
- Zhaopeng Cui, Lionel Heng, Ye Chuan Yeo, Andreas Geiger, Marc Pollefeys, and Torsten Sattler. 2019. Real-time dense mapping for self-driving vehicles using fisheye cameras. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 6087--6093.Google ScholarDigital Library
- Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In Proc. of SIGGRAPH. 303--312.Google ScholarDigital Library
- Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Reintegration. ACM Transactions on Graphics (TOG) 36, 3 (2017), 24.Google ScholarDigital Library
- Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. 2019. Poserbpf: A rao-blackwellized particle filter for 6d object pose tracking. arXiv preprint arXiv:1905.09304 (2019).Google Scholar
- Siyan Dong, Kai Xu, Qiang Zhou, Andrea Tagliasacchi, Shiqing Xin, Matthias Nießner, and Baoquan Chen. 2019. Multi-robot collaborative dense scene reconstruction. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--16.Google ScholarDigital Library
- Felix Endres, Jürgen Hess, Nikolas Engelhard, Jürgen Sturm, Daniel Cremers, and Wolfram Burgard. 2012. An evaluation of the RGB-D SLAM system. In 2012 IEEE International Conference on Robotics and Automation. IEEE, 1691--1696.Google ScholarCross Ref
- Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-scale direct monocular SLAM. In European conference on computer vision. Springer, 834--849.Google ScholarCross Ref
- Jakob Engel, Jörg Stückler, and Daniel Cremers. 2015. Large-scale direct SLAM with stereo cameras. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1935--1942.Google ScholarDigital Library
- Jakob Engel, Jürgen Sturm, and Daniel Cremers. 2013. Semi-dense visual odometry for a monocular camera. In Proceedings of the IEEE international conference on computer vision. 1449--1456.Google ScholarDigital Library
- Anders Eriksson, John Bastian, Tat-Jun Chin, and Mats Isaksson. 2016. A consensus-based framework for distributed bundle adjustment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1754--1762.Google ScholarCross Ref
- Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. 2016. On-Manifold Preintegration for Real-Time Visual-Inertial Odometry. IEEE Transactions on Robotics 33, 1 (2016), 1--21.Google ScholarDigital Library
- Christian Forster, Matia Pizzoli, and Davide Scaramuzza. 2014. SVO: Fast semi-direct monocular visual odometry. In 2014 IEEE international conference on robotics and automation (ICRA). IEEE, 15--22.Google ScholarCross Ref
- Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Daniilidis, et al. 2019. Event-based vision: A survey. arXiv preprint arXiv:1904.08405 (2019).Google Scholar
- Arturo Gil, Óscar Reinoso, Mónica Ballesta, and Miguel Juliá. 2010. Multi-robot visual SLAM using a Rao-Blackwellized particle filter. Robotics and Autonomous Systems 58, 1 (2010), 68--80.Google ScholarDigital Library
- Neil J Gordon, David J Salmond, and Adrian FM Smith. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE proceedings F (radar and signal processing), Vol. 140. IET, 107--113.Google ScholarCross Ref
- Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard. 2007. Improved techniques for grid mapping with rao-blackwellized particle filters. IEEE transactions on Robotics 23, 1 (2007), 34--46.Google ScholarDigital Library
- Giorgio Grisettiyz, Cyrill Stachniss, and Wolfram Burgard. 2005. Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. In Proceedings of the 2005 IEEE international conference on robotics and automation. IEEE, 2432--2437.Google ScholarCross Ref
- Ankur Handa, Thomas Whelan, John McDonald, and Andrew J Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In 2014 IEEE international conference on Robotics and automation (ICRA). IEEE, 1524--1531.Google ScholarCross Ref
- Miles Hansard, Seungkyu Lee, Ouk Choi, and Radu Patrice Horaud. 2012. Time-of-flight cameras: principles, methods and applications. Springer Science & Business Media.Google Scholar
- Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. 2014. RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In Experimental robotics. Springer, 477--491.Google Scholar
- Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu. 2020. DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors. arXiv preprint arXiv:2012.05551 (2020).Google Scholar
- Jin Huang, Xiaohan Shi, Xinguo Liu, Kun Zhou, Li-Yi Wei, Shang-Hua Teng, Hujun Bao, Baining Guo, and Heung-Yeung Shum. 2006. Subspace gradient domain mesh deformation. In ACM SIGGRAPH 2006 Papers. 1126--1134.Google ScholarDigital Library
- Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In UIST. 559--568.Google Scholar
- Chunlin Ji, Yangyang Zhang, Mengmeng Tong, and Shengxiang Yang. 2008. Particle filter with swarm move for optimization. In International Conference on Parallel Problem Solving from Nature. Springer, 909--918.Google ScholarCross Ref
- O. Kahler, V. A. Prisacariu, C. Y. Ren, X. Sun, P. H. S Torr, and D. W. Murray. 2015. Very High Frame Rate Volumetric Integration of Depth Images on Mobile Device. IEEE Trans. Vis. & Computer Graphics (ISMAR) 22, 11 (2015).Google Scholar
- Maik Keller, Damien Lefloch, Martin Lambers, Shahram Izadi, Tim Weyrich, and Andreas Kolb. 2013. Real-time 3d reconstruction in dynamic scenes using point-based fusion. In 2013 International Conference on 3D Vision-3DV 2013. IEEE, 1--8.Google ScholarDigital Library
- Christian Kerl, Jürgen Sturm, and Daniel Cremers. 2013. Dense visual SLAM for RGB-D cameras. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2100-2106.Google ScholarCross Ref
- Georg Klein and David Murray. 2007. Parallel tracking and mapping for small AR workspaces. In 2007 6th IEEE and ACM international symposium on mixed and augmented reality. IEEE, 225--234.Google ScholarDigital Library
- Rainer Kuemmerle, Giorgio Grisetti, Hauke Strasdat, Kurt Konolige, and Wolfram Burgard. 2011. g2o: A General Framework for Graph Optimization. In Proc. ICRA.Google Scholar
- Tristan Laidlow, Michael Bloesch, Wenbin Li, and Stefan Leutenegger. 2017. Dense rgb-d-inertial slam with map deformations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 6741--6748.Google ScholarDigital Library
- Seungkyu Lee. 2014. Time-of-flight depth camera motion blur detection and deblurring. IEEE Signal Processing Letters 21, 6 (2014), 663--666.Google ScholarCross Ref
- Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale. 2015. Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research 34, 3 (2015), 314--334.Google ScholarDigital Library
- Ping Li, Trevor J Hastie, and Kenneth W Church. 2006. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 287--296.Google ScholarDigital Library
- Bin Liu, Shi Cheng, and Yuhui Shi. 2016. Particle filter optimization: A brief introduction. In International Conference on Swarm Intelligence. Springer, 95--104.Google ScholarCross Ref
- Ligang Liu, Xi Xia, Han Sun, Qi Shen, Juzhan Xu, Bin Chen, Hui Huang, and Kai Xu. 2018. Object-Aware Guidance for Autonomous Scene Reconstruction. ACM Trans. on Graph. (SIGGRAPH) 37, 4 (2018).Google ScholarDigital Library
- Anastasios I Mourikis and Stergios I Roumeliotis. 2007. A multi-state constraint Kalman filter for vision-aided inertial navigation. In Proceedings 2007 IEEE International Conference on Robotics and Automation. IEEE, 3565--3572.Google ScholarCross Ref
- Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. 2015. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE transactions on robotics 31, 5 (2015), 1147--1163.Google Scholar
- Raul Mur-Artal and Juan D Tardós. 2017. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics 33, 5 (2017), 1255--1262.Google ScholarDigital Library
- Karthikeyan Natesan Ramamurthy, Chung-Ching Lin, Aleksandr Aravkin, Sharath Pankanti, and Raphael Viguier. 2017. Distributed bundle adjustment. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 2146--2154.Google ScholarCross Ref
- A. Neumaier. 1974. Rounding Error Analysis of Some Methods for Summing Finite Sums. Zeitschrift für Angewandte Mathematik und Mechanik 54, 1 (1974), 39--51.Google ScholarCross Ref
- Richard A Newcombe, Andrew J Davison, Shahram Izadi, Pushmeet Kohli, Otmar Hilliges, Jamie Shotton, David Molyneaux, Steve Hodges, David Kim, and Andrew Fitzgibbon. 2011a. KinectFusion: Real-time dense surface mapping and tracking. In Proc. IEEE Int. Symp. on Mixed and Augmented Reality. 127--136.Google ScholarDigital Library
- Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. 2011b. DTAM: Dense tracking and mapping in real-time. In 2011 international conference on computer vision. IEEE, 2320--2327.Google ScholarDigital Library
- Yinyu Nie, Ji Hou, Xiaoguang Han, and Matthias Nießner. 2020. RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction. arXiv preprint arXiv:2011.14744 (2020).Google Scholar
- Matthias Nießner, Angela Dai, and Matthew Fisher. 2014. Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction.. In Eurographics (Short Papers). Citeseer, 13--16.Google Scholar
- M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. 2013. Real-time 3D Reconstruction at Scale using Voxel Hashing. ACM Trans. on Graph. (SIGGRAPH Asia) 32, 6 (2013), 169.Google ScholarDigital Library
- Marcos Nieto, Andoni Cortés, Oihana Otaegui, Jon Arróspide, and Luis Salgado. 2016. Real-time lane tracking using Rao-Blackwellized particle filter. Journal of Real-Time Image Processing 11, 1 (2016), 179--191.Google ScholarDigital Library
- Marc Pollefeys, David Nistér, J-M Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David Gallup, S-J Kim, Paul Merrell, et al. 2008. Detailed real-time urban 3d reconstruction from video. International Journal of Computer Vision 78, 2 (2008), 143--167.Google ScholarDigital Library
- Vivek Pradeep, Christoph Rhemann, Shahram Izadi, Christopher Zach, Michael Bleyer, and Steven Bathiche. 2013. MonoFusion: Real-time 3D reconstruction of small scenes with a single web camera. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 83--88.Google ScholarCross Ref
- Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016).Google Scholar
- Olivier Saurer, Marc Pollefeys, and Gim Hee Lee. 2016. Sparse to dense 3d reconstruction from rolling shutter images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3337--3345.Google ScholarCross Ref
- Davide Scaramuzza and Zichao Zhang. 2019. Visual-inertial odometry of aerial robots. arXiv preprint arXiv:1906.03289 (2019).Google Scholar
- Jochen Schmidt and Heinrich Niemann. 2001. Using Quaternions for Parametrizing 3-D Rotations in Unconstrained Nonlinear Optimization.. In Vmv, Vol. 1. Citeseer, 399--406.Google Scholar
- Thomas Schops, Torsten Sattler, and Marc Pollefeys. 2019. BAD SLAM: Bundle adjusted direct rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 134--144.Google ScholarCross Ref
- Yuhui Shi and Russell C Eberhart. 1999. Empirical study of particle swarm optimization. In Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), Vol. 3. IEEE, 1945--1950.Google ScholarCross Ref
- Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. 2019. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 (2019).Google Scholar
- Jörg Stückler and Sven Behnke. 2014. Multi-resolution surfel maps for efficient dense 3D modeling and tracking. Journal of Visual Communication and Image Representation 25, 1 (2014), 137--147.Google ScholarDigital Library
- Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. 2012. A benchmark for the evaluation of RGB-D SLAM systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 573--580.Google ScholarCross Ref
- Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. science 290, 5500 (2000), 2319--2323.Google Scholar
- Sebastian Thrun. 2002. Probabilistic robotics. Commun. ACM 45, 3 (2002), 52--57.Google ScholarDigital Library
- Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. 1999. Bundle adjustment: a modern synthesis. In International workshop on vision algorithms. Springer, 298--372.Google Scholar
- Thomas Whelan, Michael Kaess, Maurice Fallon, Hordur Johannsson, John Leonard, and John McDonald. 2012. Kintinuous: Spatially Extended KinectFusion. In RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras.Google Scholar
- Thomas Whelan, Stefan Leutenegger, Renato F Salas-Moreno, Ben Glocker, and Andrew J Davison. 2015. ElasticFusion: Dense SLAM without a pose graph. In Proc. Robotics: Science and Systems.Google ScholarCross Ref
- Kai Xu, Hui Huang, Yifei Shi, Hao Li, Pinxin Long, Jiannong Caichen, Wei Sun, and Baoquan Chen. 2015. Autoscanning for Coupled Scene Reconstruction and Proactive Object Analysis. ACM Trans. on Graph. 34, 6 (2015), 177.Google ScholarDigital Library
- Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Nießner, Oliver Deussen, Daniel Cohen-Or, and Hui Huang. 2017. Autonomous Reconstruction of Unknown Indoor Scenes Guided by Time-varying Tensor Fields. ACM Transactions on Graphics 2017 (TOG) (2017).Google Scholar
- Chi Zhang, Amirhossein Taghvaei, and Prashant G Mehta. 2017. A controlled particle filter for global optimization. arXiv preprint arXiv:1701.02413 (2017).Google Scholar
- Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, and Kai Xu. 2020. Fusion-aware point convolution for online semantic 3d scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4534--4543.Google ScholarCross Ref
Index Terms
- ROSEFusion: random optimization for online dense reconstruction under fast camera motion
Recommendations
Real-time dense 3D reconstruction and camera tracking via embedded planes representation
AbstractThis paper proposes a novel approach for robust plane matching and real-time RGB-D fusion based on the representation of plane parameter space. In contrast to previous planar-based SLAM algorithms estimating correspondences for each plane-pair ...
MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction
We introduce MIPS-Fusion, a robust and scalable online RGB-D reconstruction method based on a novel neural implicit representation - multi-implicit-submap. Different from existing neural RGB-D reconstruction methods lacking either flexibility with a ...
Ant Colony Optimization Algorithm for Solving the Provider - Modified Traveling Salesman Problem
ACIIDS 2014: Proceedings, Part I, of the 6th Asian Conference on Intelligent Information and Database Systems - Volume 8397The paper concerns the introduced and defined problem which was called the Provider. This problem coming from practice and can be treated as a modified version of Travelling Salesman Problem. For solving the problem an algorithm called ACO based on ant ...
Comments