TEAM MEMBERS

LATEST NEWS

RESEARCH PROJECTS

ACADEMIC PUBLICATIONS

VOLDOR-SLAM: For the Times When Feature-Based or Direct Methods Are Not Good Enough
Zhixiang Min and Enrique Dunn
International Conference on Robotics and Automation (ICRA 2021)

×

BibTex

@article{DBLP:journals/corr/abs-2104-06800, author = {Zhixiang Min and Enrique Dunn}, title = {VOLDOR+SLAM: For the Times When Feature-Based or Direct Methods Are Not Good Enough}, journal = {CoRR}, volume = {abs/2104.06800}, year = {2021}, url = {https://arxiv.org/abs/2104.06800}, archivePrefix = {arXiv}, eprint = {2104.06800} }

×

Abstract

We present a dense-indirect SLAM system using external dense optical flows as input. We extend the recent probabilistic visual odometry model VOLDOR [Min et al. CVPR'20], by incorporating the use of geometric priors to 1) robustly bootstrap estimation from monocular capture, while 2) seamlessly supporting stereo and/or RGB-D input imagery. Our customized back-end tightly couples our intermediate geometric estimates with an adaptive priority scheme managing the connectivity of an incremental pose graph. We leverage recent advances in dense optical flow methods to achieve accurate and robust camera pose estimates, while constructing fine-grain globally-consistent dense environmental maps. Our open source implementation [this https URL] operates online at around 15 FPS on a single GTX1080Ti GPU.

VOLDOR: Visual Odometry From Log-Logistic Dense Optical Flow Residuals
Zhixiang Min, Yiding Yang and Enrique Dunn
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)

×

BibTex

@inproceedings{cvprMinYD20, author = {Zhixiang Min and Yiding Yang and Enrique Dunn}, title = {{VOLDOR:} Visual Odometry From Log-Logistic Dense Optical Flow Residuals}, booktitle = {2020 {IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR} 2020, Seattle, WA, USA, June 13-19, 2020}, pages = {4897--4908}, publisher = {{IEEE}}, year = {2020}, url = {https://doi.org/10.1109/CVPR42600.2020.00495}, doi = {10.1109/CVPR42600.2020.00495}, }

×

Abstract

We propose a dense indirect visual odometry method taking as input externally estimated optical flow fields instead of hand-crafted feature correspondences. We define our problem as a probabilistic model and develop a generalized-EM formulation for the joint inference of camera motion, pixel depth, and motion-track confidence. Contrary to traditional methods assuming Gaussian-distributed observation errors, we supervise our inference framework under an (empirically validated) adaptive log-logistic distribution model. Moreover, the log-logistic residual model generalizes well to different state-of-the-art optical flow methods, making our approach modular and agnostic to the choice of optical flow estimators. Our method achieved top-ranking results on both TUM RGB-D and KITTI odometry benchmarks. Our open-sourced implementation is inherently GPU-friendly with only linear computational and storage growth.

Discrete Laplace Operator Estimation for Dynamic 3D Reconstruction
Xiangyu Xu and Enrique Dunn
IEEE International Conference on Computer Vision (ICCV 2019)

×

BibTex

@inproceedings{XuICCV19, author = {Xiangyu Xu and Enrique Dunn},title = {Discrete Laplace Operator Estimation for Dynamic 3D Reconstruction}, booktitle = {2019 {IEEE/CVF} International Conference on Computer Vision, {ICCV} 2019, Seoul, Korea (South), October 27 - November 2, 2019}, pages = {1548--1557}, publisher = {{IEEE}}, year = {2019}, doi = {10.1109/ICCV.2019.00163}}

×

Abstract

We present a general paradigm for dynamic 3D reconstruction from multiple independent and uncontrolled image sources having arbitrary temporal sampling density and distribution. Our graph-theoretic formulation models the spatio-temporal relationships among our observations in terms of the joint estimation of their 3D geometry and its discrete Laplace operator. Towards this end, we define a tri-convex optimization framework that leverages the geometric properties and dependencies found among a Euclidean shape-space and the discrete Laplace operator describing its local and global topology. We present a reconstructability analysis, experiments on motion capture data and multi-view image datasets, as well as explore applications to geometrybased event segmentation and data association.

Self-expressive Dictionary Learning for Dynamic 3D Reconstruction
Enliang Zheng, Dinghuang Ji, Enrique Dunn, Jan-Michael Frahm
IEEE Transactions on Pattern Analysis and Machine Intelligence

×

BibTex

@article{zheng2017self, title={Self-expressive Dictionary Learning for Dynamic 3D Reconstruction}, author={Enliang Zheng and Dinghuang Ji and Enrique Dunn and Jan-Michael Frahm},journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},year={2017},publisher=IEEE}

×

Abstract

We target the problem of sparse 3D reconstruction of dynamic objects observed by multiple unsynchronized video cameras with unknown temporal overlap. To this end, we develop a framework to recover the unknown structure without sequencing information across video sequences. Our proposed compressed sensing framework poses the estimation of 3D structure as the problem of dictionary learning, where the dictionary is defined as an aggregation of the temporally varying 3D structures. Given the smooth motion of dynamic objects, we observe any element in the dictionary can be well approximated by a sparse linear combination of other elements in the same dictionary (i.e. self-expression). Our formulation optimizes a biconvex cost function that leverages a compressed sensing formulation and enforces both structural dependency coherence across video streams, as well as motion smoothness across estimates from common video sources. We further analyze the reconstructability of our approach under different capture scenarios, and its comparison and relation to existing methods. Experimental results on large amounts of synthetic data as well as real imagery demonstrate the effectiveness of our approach.

Learned Contextual Feature Reweighting for Image Geo-Localization
Hyo-Jin Kim, Enrique Dunn, Jan-Michael Frahm
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017)

×

BibTex

@inproceedings{kim2017learned, title={Learned Contextual Feature Reweighting for Image Geo-Localization}, author= {Hyo-Jin Kim and Enrique Dunn and Jan-Michael Frahm}, booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2017} }

×

Abstract

We address the problem of large scale image geo- localization where the location of an image is estimated by identifying geo-tagged reference images depicting the same place. We propose a novel model for learning image rep- resentations that integrates context-aware feature reweight- ing in order to effectively focus on regions that positively contribute to geo-localization. In particular, we introduce a Contextual Reweighting Network (CRN) that predicts the importance of each region in the feature map based on the image context. Our model is learned end-to-end for the im- age geo-localization task, and requires no annotation other than image geo-tags for training. In experimental results, the proposed approach significantly outperforms the previ- ous state-of-the-art on the standard geo-localization bench- mark datasets.We also demonstrate that our CRN discovers task-relevant contexts without any additional supervision.

Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras
Akash Bapat, Enrique Dunn, Jan-Michael Frahm
IEEE Transactions on Visualization and Computer Graphics

×

BibTex

@article{DBLP:journals/tvcg/BapatDF16, author = {Akash Bapat and Enrique Dunn and Jan-Michael Frahm}, title = {Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras}, journal = {IEEE Trans. Vis. Comput. Graph.}, volume = {22}, number = {11}, pages = {2358--2367}, year = {2016} }

×

Abstract

To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate.

Bringing 3D Models Together: Mining Video Liaisons in Crowdsourced Reconstructions
Ke Wang, Enrique Dunn, Mikel Rodriguez, Jan-Michael Frahm
Asian Conference on Computer Vision (ACCV 2016)

×

BibTex

@inproceedings{DBLP:conf/accv/WangDRF16, author = {Ke Wang and Enrique Dunn and Mikel Rodriguez and Jan-Michael Frahm}, title = {Bringing 3D Models Together: Mining Video Liaisons in Crowdsourced Reconstructions}, booktitle = {Computer Vision - {ACCV} 2016 - 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part {IV}}, pages = {408--423}, year = {2016}}

×

Abstract

The recent advances in large-scale scene modeling have enabled the automatic 3D reconstruction of landmark sites from crowdsourced photo collections. Here, we address the challenge of leveraging crowdsourced video collections to identify connecting visual observations that enable the alignment and subsequent aggregation, of disjoint 3D models. We denote these connecting image sequences as video liaisons and develop a data-driven framework for fully unsupervised extraction and exploitation. Towards this end, we represent video contents in terms of a histogram representation of iconic imagery contained within existing 3D models attained from a photo collection. We then use this representation to efficiently identify and prioritize the analysis of individual videos within a large-scale video collection, in an effort to determine camera motion trajectories connecting different landmarks. Results on crowdsourced data illustrate the efficiency and effectiveness of our proposed approach.

Spatio-Temporally Consistent Correspondence for Dense Dynamic Scene Modeling
Dinghuang Ji, Enrique Dunn, Jan-Michael Frahm
European Conference on Computer Vision (ECCV 2016)

×

BibTex

@inproceedings{DBLP:conf/eccv/JiDF16, author = {Dinghuang Ji and Enrique Dunn and Jan-Michael Frahm}, title = {Spatio-Temporally Consistent Correspondence for Dense Dynamic Scene Modeling}, booktitle = {Computer Vision - {ECCV} 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part {VI}}, pages = {3--18}, year = {2016} }

×

Abstract

We address the problem of robust two-view correspondence estimation within the context of dynamic scene modeling. To this end, we investigate the use of local spatio-temporal assumptions to both identify and refine dense low-level data associations in the absence of prior dynamic content models. By developing a strictly data-driven approach to correspondence search, based on bottom-up local 3D motion cues of local rigidity and non-local coherence, we are able to robustly address the higher-order problems of video synchronization and dynamic surface modeling. Our findings suggest an important relationship between these two tasks, in that maximizing spatial coherence of surface points serves as a direct metric for the temporal alignment of local image sequences. The obtained results for these two problems on multiple publicly available dynamic reconstruction datasets illustrate both the effectiveness and generality of our proposed approach.

Efficient joint stereo estimation and land usage classification for multiview satellite data
Ke Wang, Craig Stutts, Enrique Dunn, Jan-Michael Frahm
IEEE Winter Conference on Applications of Computer Vision (WACV 2016)

×

BibTex

@inproceedings{DBLP:conf/wacv/WangSDF16, author = {Ke Wang and Craig Stutts and Enrique Dunn and Jan-Michael Frahm}, title = {Efficient joint stereo estimation and land usage classification for multiview satellite data}, booktitle = {2016 IEEE Winter Conference on Applications of Computer Vision, {WACV} 2016, Lake Placid, NY, USA, March 7-10, 2016}, pages = {1--9}, year = {2016} }

×

Abstract

We propose an efficient algorithm to jointly estimate geometry and semantics for a given geographical region observed by multiple satellite images. Our joint estimation leverages an efficient PatchMatch inference framework de- fined over lattice discretization of the environment. Our cost function relies on the local planarity assumption to model scene geometry and neural network classification to determine semantic (e.g. land use) labels for geometric structures. By utilizing the commonly available direct (i.e. space to image) rational polynomial coefficients (RPC) satellite camera models, our approach effectively circumvents the need for estimating or refining inverse RPC models. Experiments illustrate both the computational efficiency and high quality scene geometry estimates attained by our approach for satellite imagery. To further illustrate the generality of our representation and inference framework, experiments on standard benchmarks for ground-level imagery are also included.

Reconstructing the world* in six days *(As Captured by the Yahoo 100 Million Image Dataset)
Jared Heinly, Johannes L. Schonberger, Enrique Dunn, Jan-Michael Frahm
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015)

×

BibTex

@inproceedings{DBLP:conf/cvpr/HeinlySDF15, author = {Jared Heinly and Johannes L. Schonberger and Enrique Dunn and Jan-Michael Frahm}, title = {Reconstructing the world* in six days *(As Captured by the Yahoo 100 Million Image Dataset)}, booktitle = {IEEE Conference on Computer Vision and Pattern Recognition, {CVPR} 2015, Boston, MA, USA, June 7-12, 2015}, pages = {3287--3295}, year = {2015} }

×

Abstract

We propose a novel, large-scale, structure-from-motion framework that advances the state of the art in data scalability from city-scale modeling (millions of images) to world-scale modeling (several tens of millions of images) using just a single computer. The main enabling technology is the use of a streaming-based framework for connected component discovery. Moreover, our system employs an adaptive, online, iconic image clustering approach based on an augmented bag-of-words representation, in order to balance the goals of registration, comprehensiveness, and data compactness. We demonstrate our proposal by operating on a recent publicly available 100 million image crowdsourced photo collection containing images geographically distributed throughout the entire world. Results illustrate that our streaming-based approach does not compromise model completeness, but achieves unprecedented levels of efficiency and scalability.

Synthesizing Illumination Mosaics from Internet Photo-Collections
Dinghuang Ji, Enrique Dunn, Jan-Michael Frahm
IEEE International Conference on Computer Vision (ICCV 2015)

×

BibTex

@inproceedings{DBLP:conf/iccv/JiDF15, author = {Dinghuang Ji and Enrique Dunn and Jan-Michael Frahm}, title = {Synthesizing Illumination Mosaics from Internet Photo-Collections}, booktitle = {2015 IEEE International Conference on Computer Vision, {ICCV} 2015, Santiago, Chile, December 7-13, 2015}, pages = {3988--3996}, year = {2015} }

×

Abstract

We propose a framework for the automatic creation of time-lapse mosaics of a given scene. We achieve this by leveraging the illumination variations captured in Internet photo-collections. In order to depict and characterize the illumination spectrum of a scene, our method relies on building discrete representations of the image appearance space through connectivity graphs defined over a pairwise image distance function. The smooth appearance transitions are found as the shortest path in the similarity graph among images, and robust image alignment is achieved by leveraging scene semantics, multi-view geometry, and image warping techniques. The attained results present an insightful and compact visualization of the scene illuminations captured in crowd-sourced imagery.

Predicting Good Features for Image Geo-Localization Using Per-Bundle VLAD
Hyo Jin Kim, Enrique Dunn, Jan-Michael Frahm
IEEE International Conference on Computer Vision (ICCV 2015)

×

BibTex

@inproceedings{DBLP:conf/iccv/KimDF15, author = {Hyo Jin Kim and Enrique Dunn and Jan-Michael Frahm}, title = {Predicting Good Features for Image Geo-Localization Using Per-Bundle VLAD}, booktitle = {2015 IEEE International Conference on Computer Vision, {ICCV} 2015, Santiago, Chile, December 7-13, 2015}, pages = {1170--1178}, year = {2015} }

×

Abstract

We address the problem of recognizing a place depicted in a query image by using a large database of geo-tagged images at a city-scale. In particular, we discover features that are useful for recognizing a place in a data-driven manner, and use this knowledge to predict useful features in a query image prior to the geo-localization process. This allows us to achieve better performance while reducing the number of features. Also, for both learning to predict features and retrieving geo-tagged images from the database, we propose per-bundle vector of locally aggregated descriptors (PBVLAD), where each maximally stable region is described by a vector of locally aggregated descriptors (VLAD) on multiple scale-invariant features detected within the region. Experimental results show the proposed approach achieves a significant improvement over other baseline methods.

Sparse Dynamic 3D Reconstruction from Unsynchronized Videos
Enliang Zheng, Dinghuang Ji, Enrique Dunn, Jan-Michael Frahm
IEEE International Conference on Computer Vision (ICCV 2015)

×

BibTex

@inproceedings{DBLP:conf/iccv/ZhengJDF15, author = {Enliang Zheng and Dinghuang Ji and Enrique Dunn and Jan-Michael Frahm}, title = {Sparse Dynamic 3D Reconstruction from Unsynchronized Videos}, booktitle = {2015 IEEE International Conference on Computer Vision, {ICCV} 2015, Santiago, Chile, December 7-13, 2015}, pages = {4435--4443}, year = {2015} }

×

Abstract

We target the sparse 3D reconstruction of dynamic objects observed by multiple unsynchronized video cameras with unknown temporal overlap. To this end, we develop a framework to recover the unknown structure without sequencing information across video sequences. Our proposed compressed sensing framework poses the estimation of 3D structure as the problem of dictionary learning. Moreover, we define our dictionary as the temporally varying 3D structure, while we define local sequencing information in terms of the sparse coefficients describing a locally linear 3D structural interpolation. Our formulation optimizes a biconvex cost function that leverages a compressed sensing formulation and enforces both structural dependency coherence across video streams, as well as motion smoothness across estimates from common video sources. Experimental results demonstrate the effectiveness of our approach in both synthetic data and captured imagery.

Minimal Solvers for 3D Geometry from Satellite Imagery
Enliang Zheng, Ke Wang, Enrique Dunn, Jan-Michael Frahm
IEEE International Conference on Computer Vision (ICCV 2015)

×

BibTex

@inproceedings{DBLP:conf/iccv/ZhengWDF15, author = {Enliang Zheng and Ke Wang and Enrique Dunn and Jan-Michael Frahm}, title = {Minimal Solvers for 3D Geometry from Satellite Imagery}, booktitle = {2015 IEEE International Conference on Computer Vision, {ICCV} 2015, Santiago, Chile, December 7-13, 2015}, pages = {738--746}, year = {2015} }

×

Abstract

We propose two novel minimal solvers which advance the state of the art in satellite imagery processing. Our methods are efficient and do not rely on the prior existence of complex inverse mapping functions to correlate 2D image coordinates and 3D terrain. Our first solver improves on the stereo correspondence problem for satellite imagery, in that we provide an exact image-to-object space mapping (where prior methods were inaccurate). Our second solver provides a novel mechanism for 3D point triangulation, which has improved robustness and accuracy over prior techniques. Given the usefulness and ubiquity of satellite imagery, our proposed methods allow for improved results in a variety of existing and future applications.

Recovering Correct Reconstructions from Indistinguishable Geometry
Jared Heinly, Enrique Dunn, Jan-Michael Frahm
International Conference on 3D Vision (3DV 2014)

×

BibTex

@inproceedings{DBLP:conf/3dim/HeinlyDF14, author = {Jared Heinly and Enrique Dunn and Jan-Michael Frahm}, title = {Recovering Correct Reconstructions from Indistinguishable Geometry}, booktitle = {2nd International Conference on 3D Vision, 3DV 2014, Tokyo, Japan, December 8-11, 2014, Volume 1}, pages = {377--384}, year = {2014} }

×

Abstract

Structure-from-motion (SFM) is widely utilized to generate 3D reconstructions from unordered photo-collections. However, in the presence of non unique, symmetric, or otherwise indistinguishable structure, SFM techniques often incorrectly reconstruct the final model. We propose a method that not only determines if an error is present, but automatically corrects the error in order to produce a correct representation of the scene. We find that by exploiting the co-occurrence information present in the scene’s geometry, we can successfully isolate the 3D points causing the incorrect result. This allows us to split an incorrect reconstruction into error-free sub-models that we then correctly merge back together. Our experimental results show that our technique is efficient, robust to a variety of scenes, and outperforms existing methods.

Stereo under Sequential Optimal Sampling: A Statistical Analysis Framework for Search Space Reduction
Yilin Wang, Ke Wang, Enrique Dunn, Jan-Michael Frahm
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014)

×

BibTex

@inproceedings{DBLP:conf/cvpr/WangWDF14, author = {Yilin Wang and Ke Wang and Enrique Dunn and Jan-Michael Frahm}, title = {Stereo under Sequential Optimal Sampling: A Statistical Analysis Framework for Search Space Reduction}, booktitle = {2014 IEEE Conference on Computer Vision and Pattern Recognition, {CVPR} 2014, Columbus, OH, USA, June 23-28, 2014}, pages = {485--492}, year = {2014} }

×

Abstract

We develop a sequential optimal sampling framework for stereo disparity estimation by adapting the Sequential Probability Ratio Test (SPRT) model. We operate over local image neighborhoods by iteratively estimating single pixel disparity values until sufficient evidence has been gathered to either validate or contradict the current hypothesis regarding local scene structure. The output of our sampling is a set of sampled pixel positions along with a robust and compact estimate of the set of disparities contained within a given region. We further propose an efficient plane propagation mechanism that leverages the pre-computed sampling positions and the local structure model described by the reduced local disparity set. Our sampling framework is a general pre-processing mechanism aimed at reducing computational complexity of disparity search algorithms by ascertaining a reduced set of disparity hypotheses for each pixel. Experiments demonstrate the effectiveness of the proposed approach when compared to state of the art methods.

PatchMatch Based Joint View Selection and Depthmap Estimation
Enliang Zheng, Enrique Dunn, Vladimir Jojic, Jan-Michael Frahm
IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014)

×

BibTex

@inproceedings{DBLP:conf/cvpr/ZhengDJF14, author = {Enliang Zheng and Enrique Dunn and Vladimir Jojic and Jan-Michael Frahm}, title = {PatchMatch Based Joint View Selection and Depthmap Estimation}, booktitle = {2014 IEEE Conference on Computer Vision and Pattern Recognition, {CVPR} 2014, Columbus, OH, USA, June 23-28, 2014}, pages = {1510--1517}, year = {2014} }

×

Abstract

We propose a multi-view depthmap estimation approach aimed at adaptively ascertaining the pixel level data associations between a reference image and all the elements of a source image set. Namely, we address the question, what aggregation subset of the source image set should we use to estimate the depth of a particular pixel in the reference image? We pose the problem within a probabilistic framework that jointly models pixel-level view selection and depthmap estimation given the local pairwise image photoconsistency. The corresponding graphical model is solved by EM-based view selection probability inference and PatchMatch-like depth sampling and propagation. Experimental results on standard multi-view benchmarks convey the state-of-the art estimation accuracy afforded by mitigating spurious pixellevel data associations. Additionally, experiments on large Internet crowd sourced data demonstrate the robustness of our approach against unstructured and heterogeneous image capture characteristics. Moreover, the linear computational and storage requirements of our formulation, as well as its inherent parallelism, enables an efficient and scalable GPU-based implementation.

Correcting for Duplicate Scene Structure in Sparse 3D Reconstruction
Jared Heinly, Enrique Dunn, Jan-Michael Frahm
European Conference on Computer Vision (ECCV 2014)

×

BibTex

@inproceedings{DBLP:conf/eccv/HeinlyDF14, author = {Jared Heinly and Enrique Dunn and Jan-Michael Frahm}, title = {Correcting for Duplicate Scene Structure in Sparse 3D Reconstruction}, booktitle = {Computer Vision - {ECCV} 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part {IV}}, pages = {780--795}, year = {2014} }

×

Abstract

Structure from motion (SfM) is a common technique to recover 3D geometry and camera poses from sets of images of a common scene. In many urban environments, however, there are symmetric, repetitive, or duplicate structures that pose challenges for SfM pipelines. The result of these ambiguous structures is incorrectly placed cameras and points within the reconstruction. In this paper, we present a postprocessing method that can not only detect these errors, but successfully resolve them. Our novel approach proposes the strong and informative measure of conflicting observations, and we demonstrate that it is robust to a large variety of scenes.

3D Reconstruction of Dynamic Textures in Crowd Sourced Data
Dinghuang Ji, Enrique Dunn, Jan-Michael Frahm
European Conference on Computer Vision (ECCV 2014)

×

BibTex

@inproceedings{DBLP:conf/eccv/JiDF14, author = {Dinghuang Ji and Enrique Dunn and Jan-Michael Frahm}, title = {3D Reconstruction of Dynamic Textures in Crowd Sourced Data}, booktitle = {Computer Vision - {ECCV} 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part {I}}, pages = {143--158}, year = {2014} }

×

Abstract

We propose a framework to automatically build 3D models for scenes containing structures not amenable for photo-consistency based reconstruction due to having dynamic appearance. We analyze the dynamic appearance elements of a given scene by leveraging the imagery contained in Internet image photo-collections and online video sharing websites. Our approach combines large scale crowd sourced SfM techniques with image content segmentation and shape from silhouette techniques to build an iterative framework for 3D shape estimation. The developed system not only enables more complete and robust 3D modeling, but it also enables more realistic visualizations through the identifi- cation of dynamic scene elements amenable to dynamic texture mapping. Experiments on crowd sourced image and video datasets illustrate the effectiveness of our automated data-driven approach.

Joint Object Class Sequencing and Trajectory Triangulation (JOST)
Enliang Zheng, Ke Wang, Enrique Dunn, Jan-Michael Frahm
European Conference on Computer Vision (ECCV 2014)

×

BibTex

@inproceedings{DBLP:conf/eccv/ZhengWDF14, author = {Enliang Zheng and Ke Wang and Enrique Dunn and Jan-Michael Frahm}, title = {Joint Object Class Sequencing and Trajectory Triangulation (JOST)}, booktitle = {Computer Vision - {ECCV} 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part {VII}}, pages = {599--614}, year = {2014} }

×

Abstract

We introduce the problem of joint object class sequencing and trajectory triangulation (JOST), which is defined as the reconstruction of the motion path of a class of dynamic objects through a scene from an unordered set of images. We leverage standard object detection techinques to identify object instances within a set of registered images. Each of these object detections defines a single 2D point with a corresponding viewing ray. The set of viewing rays attained from the aggregation of all detections belonging to a common object class is then used to estimate a motion path denoted as the object class trajectory. Our method jointly determines the topology of the objects motion path and reconstructs the 3D object points corresponding to our object detections. We pose the problem as an optimization over both the unknown 3D points and the topology of the path, which is approximated by a Generalized Minimum Spanning Tree (GMST) on a multipartite graph and then refined through a continuous optimization over the 3D object points. Experiments on synthetic and real datasets demonstrate the effectiveness of our method and the feasibility to solve a previously intractable problem.

P-HRTF: Efficient personalized HRTF computation for high-fidelity spatial sound
Alok Meshram, Ravish Mehra, Hongsheng Yang, Enrique Dunn, Jan-Michael Frahm, Dinesh Manocha
IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2014)

×

BibTex

@inproceedings{DBLP:conf/ismar/MeshramMYDFM14, author = {Alok Meshram and Ravish Mehra and Hongsheng Yang and Enrique Dunn and Jan-Michael Frahm and Dinesh Manocha}, title = {P-HRTF: Efficient personalized HRTF computation for high-fidelity spatial sound}, booktitle = {IEEE International Symposium on Mixed and Augmented Reality, {ISMAR} 2014, Munich, Germany, September 10-12, 2014}, pages = {53--61}, year = {2014} }

×

Abstract

Accurate rendering of 3D spatial audio for interactive virtual auditory displays requires the use of personalized head-related transfer functions (HRTFs). We present a new approach to compute personalized HRTFs for any individual using a method that combines state-of-the-art image-based 3D modeling with an efficient numerical simulation pipeline. Our 3D modeling framework enables capture of the listener’s head and torso using consumer-grade digital cameras to estimate a high-resolution non-parametric surface representation of the head, including the extended vicinity of the listener’s ear. We leverage sparse structure from motion and dense surface reconstruction techniques to generate a 3D mesh. This mesh is used as input to a numeric sound propagation solver, which uses acoustic reciprocity and Kirchhoff surface integral representation to efficiently compute an individual’s personalized HRTF. The overall computation takes tens of minutes on multi-core desktop machine. We have used our approach to compute the personalized HRTFs of few individuals, and we present our preliminary evaluation here. To the best of our knowledge, this is the first commodity technique that can be used to compute personalized HRTFs in a lab or home setting

Rotation estimation from cloud tracking
Sangwoo Cho, Enrique Dunn, Jan-Michael Frahm
IEEE Winter Conference on Applications of Computer Vision (WACV 14)

×

BibTex

@inproceedings{DBLP:conf/wacv/ChoDF14, author = {Sangwoo Cho and Enrique Dunn and Jan-Michael Frahm}, title = {Rotation estimation from cloud tracking}, booktitle = {IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, CO, USA, March 24-26, 2014}, pages = {917--924}, year = {2014} }

×

Abstract

We address the problem of online relative orientation estimation from streaming video captured by a sky-facing camera on a mobile device. Namely, we rely on the detection and tracking of visual features attained from cloud structures. Our proposed method achieves robust and efficient operation by combining realtime visual odometry modules, learning based feature classification, and Kalman filtering within a robustness-driven data management framework, while achieving framerate processing on a mobile device. The relatively large 3D distance between the camera and the observed cloud features is leveraged to simplify our processing pipeline. First, as an efficiency driven optimization, we adopt a homography based motion model and focus on estimating relative rotations across adjacent keyframes. To this end, we rely on efficient feature extraction, KLT tracking, and RANSAC based model fitting. Second, to ensure the validity of our simplified motion model, we segregate detected cloud features from scene features through SVM classification. Finally, to make tracking more robust, we employ predictive Kalman filtering to enable feature persistence through temporary occlusions and manage feature spatial distribution to foster tracking robustness. Results exemplify the accuracy and robustness of the proposed approach and highlight its potential as a passive orientation sensor.

Combining semantic scene priors and haze removal for single image depth estimation
Ke Wang, Enrique Dunn, Joseph Tighe, Jan-Michael Frahm
IEEE Winter Conference on Applications of Computer Vision (WACV 2014)

×

BibTex

@inproceedings{DBLP:conf/wacv/WangDTF14, author = {Ke Wang and Enrique Dunn and Joseph Tighe and Jan-Michael Frahm}, title = {Combining semantic scene priors and haze removal for single image depth estimation}, booktitle = {IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, CO, USA, March 24-26, 2014}, pages = {800--807}, year = {2014} }

×

Abstract

We consider the problem of estimating the relative depth of a scene from a monocular image. The dark channel prior, used as a statistical observation of haze free images, has been previously leveraged for haze removal and relative depth estimation tasks. However, as a local measure, it fails to account for higher order semantic relationship among scene elements. We propose a dual channel prior used for identifying pixels that are unlikely to comply with the dark channel assumption, leading to erroneous depth estimates. We further leverage semantic segmentation information and patch match label propagation to enforce semantically consistent geometric priors. Experiments illustrate the quantitative and qualitative advantages of our approach when compared to state of the art methods.

Increasing the Efficiency of Local Stereo by Leveraging Smoothness Constraints
Yilin Wang, Enrique Dunn, Jan-Michael Frahm
International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission (3DIMPVT 2012)

×

BibTex

@inproceedings{DBLP:conf/3dim/WangDF12, author = {Yilin Wang and Enrique Dunn and Jan-Michael Frahm}, title = {Increasing the Efficiency of Local Stereo by Leveraging Smoothness Constraints}, booktitle = {2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, Zurich, Switzerland, October 13-15, 2012}, pages = {246--253}, year = {2012} }

×

Abstract

We introduce a novel framework for efficient stereo disparity estimation leveraging the spatial smoothness typically assumed in stereo and formalized by the various smoothness constraints. The smoothness constraint presumes that a neighboring set of pixels shares the same disparity or the disparity varies smoothly. Our key insight is that it hence suffices to evaluate any single one of those pixels at the correct disparity to identify a valid estimate for the entire set. We leverage this insight into the formulation of a complexity reducing mechanism. We distribute the exploration of the disparity search space among neighboring pixels, effectively reducing the set of disparity hypothesis evaluated at each individual pixel. Moreover, we integrate a recently proposed concept to deploy sparsity within this neighborhood of distributed disparities into our novel mechanism, in order to further reduce the computational burden. Our experiments clearly demonstrate the effectiveness of our approach by achieving comparable results to the baseline of exhaustive disparity search. The analysis of the computational complexity of our proposed mechanisms illustrates how, by making moderate assumptions on the smoothness of the observed scene, we can reduce the computational complexity of local stereo disparity search by upwards of two orders of magnitude while maintaining the comparable result quality.

Efficient and Scalable Depthmap Fusion
Enliang Zheng, Enrique Dunn, Rahul Raguram, Jan-Michael Frahm
British Machine Vision Conference (BMVC 2012)

×

BibTex

@inproceedings{DBLP:conf/bmvc/ZhengDRF12, author = {Enliang Zheng and Enrique Dunn and Rahul Raguram and Jan-Michael Frahm}, title = {Efficient and Scalable Depthmap Fusion}, booktitle = {British Machine Vision Conference, {BMVC} 2012, Surrey, UK, September 3-7, 2012}, pages = {1--12}, year = {2012} }

×

Abstract

The estimation of a complete 3D model from a set of depthmaps is a data intensive task aimed at mitigating measurement noise in the input data by leveraging the inherent redundancy in overlapping multi-view observations. In this paper we propose an effi- cient depthmap fusion approach that reduces the memory complexity associated with volumetric scene representations. By virtue of reducing the memory footprint we are able to process an increased reconstruction volume with greater spatial resolution. Our approach also improves upon state of the art fusion techniques by approaching the problem in an incremental online setting instead of batch mode processing. In this way, are able to handle an arbitrary number of input images at high pixel resolution and facilitate a streaming 3D processing pipeline. Experiments demonstrate the effectiveness of our proposal both at 3D modeling from internet-scale crowd source data as well as close-range 3D modeling from high resolution video streams

Comparative Evaluation of Binary Features
Jared Heinly, Enrique Dunn, Jan-Michael Frahm
European Conference on Computer Vision (ECCV 2012)

×

BibTex

@inproceedings{DBLP:conf/eccv/HeinlyDF12, author = {Jared Heinly and Enrique Dunn and Jan-Michael Frahm}, title = {Comparative Evaluation of Binary Features}, booktitle = {Computer Vision - {ECCV} 2012 - 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part {II}}, pages = {759--773}, year = {2012} }

×

Abstract

Performance evaluation of salient features has a long-standing tradition in computer vision. In this paper, we fill the gap of evaluation for the recent wave of binary feature descriptors, which aim to provide robustness while achieving high computational efficiency. We use established metrics to embed our assessment into the body of existing evaluations, allowing us to provide a novel taxonomy unifying both traditional and novel binary features. Moreover, we analyze the performance of different detector and descriptor pairings, which are often used in practice but have been infrequently analyzed. Additionally, we complement existing datasets with novel data testing for illumination change, pure camera rotation, pure scale change, and the variety present in photo-collections. Our performance analysis clearly demonstrates the power of the new class of features. To benefit the community, we also provide a website for the automatic testing of new description methods using our provided metrics and datasets (www.cs.unc.edu/feature-evaluation).

Disparity Map Estimation by Combining Cost Volume Measures Using Genetic Programming
Enrique Naredo, Enrique Dunn, Leonardo Trujillo
EVOLVE - A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation (EVOLVE 2012)

×

BibTex

@inproceedings{DBLP:conf/evolve/NaredoDT12, author = {Enrique Naredo and Enrique Dunn and Leonardo Trujillo}, title = {Disparity Map Estimation by Combining Cost Volume Measures Using Genetic Programming}, booktitle = {{EVOLVE} - {A} Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation II, {EVOLVE} 2012, Mexico City, Mexico, August 7-9, 2012, Proceedings}, pages = {71--86}, year = {2012} }

×

Abstract

Stereo vision is one of the most active research areas in modern computer vision. The objective is to recover 3-D depth information from a pair of 2-D images that capture the same scene. This paper addresses the problem of dense stereo correspondence, where the goal is to determine which image pixels in both images are projections of the same 3-D point from the observed scene. The proposal in this work is to build a non-linear operator that combines three well known methods to derive a correspondence measure that allows us to retrieve a better approximation of the ground truth disparity of stereo image pair. To achieve this, the problem is posed as a search and optimization task and solved with genetic programming (GP), an evolutionary paradigm for automatic program induction. Experimental results on well known benchmark problems show that the combined correspondence measure produced by GP outperforms each standard method, based on the mean error and the percentage of bad pixels. In conclusion, this paper shows that GP can be used to build composite correspondence algorithms that exhibit a strong performance on standard tests.

An Approach for Shape from Surface Normals with Local Discontinuity Detection
Yilin Wang, Enrique Dunn, Jan-Michael Frahm
International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT 2011)

×

BibTex

@inproceedings{DBLP:conf/3dim/WangDF11, author = {Yilin Wang and Enrique Dunn and Jan-Michael Frahm}, title = {An Approach for Shape from Surface Normals with Local Discontinuity Detection}, booktitle = {International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, 3DIMPVT 2011, Hangzhou, China, 16-19 May 2011}, pages = {188--195}, year = {2011} }

×

Abstract

We present a multi-modal surface reconstruction approach, which utilizes direct surface orientation measurements along with luminance information to obtain high quality 3D reconstructions. The proposed approach models local surface geometry as a set of intersecting natural cubic splines estimated through least squares fitting of our input pixel-wise surface normal measurements. We use this representation to detect discontinuities and segment our scene into disjoint continuous surfaces, which are constructed by an aggregation of connected local surface geometry elements. In order to obtain absolute depth estimates, we introduce the concept of multi-view patch sweeping, where we search for the most photo-consistent patch displacement along a viewing ray. Our approach improves on existing shape from normals methods by enabling absolute depth estimates for scenes with multiple objects. Furthermore, in contrast to existing multi-view stereo methods, we are able to reconstruct textureless regions through the propagation of relative surface orientation measurements. Experiments on synthetic and real data are presented to validate our proposal.

Adaptive Scale Selection for Hierarchical Stereo
Yi-Hung Jen, Enrique Dunn, Pierre Fite Georgel, Jan-Michael Frahm
British Machine Vision Conference (BMVC 2011)

×

BibTex

@inproceedings{DBLP:conf/bmvc/JenDGF11, author = {Yi-Hung Jen and Enrique Dunn and Pierre Fite Georgel and Jan-Michael Frahm}, title = {Adaptive Scale Selection for Hierarchical Stereo}, booktitle = {British Machine Vision Conference, {BMVC} 2011, Dundee, UK, August 29 - September 2, 2011. Proceedings}, pages = {1--10}, year = {2011} }

×

Abstract

Hierarchical stereo provides an efficient coarse-to-fine mechanism for disparity map estimation. However, common drawbacks of such an approach include the loss of high frequency structures not observable at coarse scale levels, as well as the unrecoverable propagation of erroneous disparity estimates through the scale space. This paper presents an adaptive scale selection mechanism to determine a suitable resolution level from which to begin the hierarchical depth estimation process for each pixel. The proposed scale selection mechanism allows us to robustly implement variable cost aggregation in order to reduce the variability of the photo-consistency measure across scale space. We also incorporate a weighted shiftable window mechanism to enable error correction during coarse-to-fine depth refinement. Experiments illustrate the effectiveness of our approach in terms of disparity accuracy, while attaining a computational efficiency compromise between full resolution and hierarchical disparity map estimation.

A geometric solver for calibrated stereo egomotion
Enrique Dunn, Brian Clipp, Jan-Michael Frahm
IEEE International Conference on Computer Vision (ICCV 2011)

×

BibTex

@inproceedings{DBLP:conf/iccv/DunnCF11, author = {Enrique Dunn and Brian Clipp and Jan-Michael Frahm}, title = {A geometric solver for calibrated stereo egomotion}, booktitle = {IEEE International Conference on Computer Vision, {ICCV} 2011, Barcelona, Spain, November 6-13, 2011}, pages = {1187--1194}, year = {2011} }

×

Abstract

This paper introduces a novel geometrical solution for the pose estimation of a stereo camera system as commonly used in robotics, where the camera system balances between coverage and overlap. The proposed approach considers a set of features observed, respectively, in four, three and two views. In contrast to most algebraic solutions our constraints are geometrically meaningful. Initially, we use a four view feature to restrict our translation vector to lie on the surface of a sphere while setting orientation as a function of translation up to a single rotational degree of freedom. Next, we use a three view feature to restrict the translation vector to lie on a circle on the sphere, while completely defining orientation as a function of translation. Finally, we use a two view feature to determine the translation vector lying on the intersection of the circle and one of the generator lines of a doubly ruled quadric. We show how for this final step, the problem can be reduced to the intersection of two coplanar circles. We also analyze the degenerate configurations of the proposed solver and perform an experimental evaluation.

Building Rome on a Cloudless Day
Jan-Michael Frahm, Pierre Fite Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik
European Conference on Computer Vision (ECCV 2010)

×

BibTex

@inproceedings{DBLP:conf/eccv/FrahmGGJRWJDCL10, author = {Jan-Michael Frahm and Pierre Fite Georgel and David Gallup and Tim Johnson and Rahul Raguram and Changchang Wu and Yi-Hung Jen and Enrique Dunn and Brian Clipp and Svetlana Lazebnik}, title = {Building Rome on a Cloudless Day}, booktitle = {Computer Vision - {ECCV} 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part {IV}}, pages = {368--381}, year = {2010} }

×

Abstract

This paper introduces an approach for dense 3D reconstruc- 7 8 tion from unregistered Internet-scale photo collections with about 3 mil- 8 9 lion of images within the span of a day on a single PC (“cloudless”). Our 9 10 method advances image clustering, stereo, stereo fusion and structure 10 11 from motion to achieve high computational performance. We leverage 11 12 geometric and appearance constraints to obtain a highly parallel imple- 12 13 mentation on modern graphics processors and multi-core architectures. 13 14 This leads to two orders of magnitude higher performance on an order 14 15 of magnitude larger dataset than competing state-of-the-art approaches.

Next Best View Planning for Active Model Improvement
Enrique Dunn, Jan-Michael Frahm
British Machine Vision Conference, {BMVC} 2009

×

BibTex

@inproceedings{DBLP:conf/bmvc/DunnF09, author = {Enrique Dunn and Jan-Michael Frahm}, title = {Next Best View Planning for Active Model Improvement}, booktitle = {British Machine Vision Conference, {BMVC} 2009, London, UK, September 7-10, 2009. Proceedings}, pages = {1--11}, year = {2009} }

×

Abstract

We propose a novel approach to determining the Next Best View (NBV) for the task of efficiently building highly accurate 3D models from images. Our proposed method deploys a hierarchical uncertainty driven model refinement process designed to select vantage viewpoints based on the model’s covariance structure and appearance, as well as the camera characteristics. The developed NBV planning system incrementally builds a sensing strategy by sequentially finding the single camera placement, which best reduces an existing model’s 3D uncertainty. The generic nature of our system’s design and internal data representation makes it well suited to be applied to a wide variety of 3D modeling algorithms. It can be used within active computer vision systems as well as for optimized view selection from the set of available views. Experimental results are presented to illustrate the effectiveness and versatility of our approach.

Developing visual sensing strategies through next best view planning
Enrique Dunn, Jur P. van den Berg, Jan-Michael Frahm
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2009)

×

BibTex

@inproceedings{DBLP:conf/iros/DunnBF09, author = {Enrique Dunn and Jur P. van den Berg and Jan-Michael Frahm}, title = {Developing visual sensing strategies through next best view planning}, booktitle = {2009 {IEEE/RSJ} International Conference on Intelligent Robots and Systems, October 11-15, 2009, St. Louis, MO, {USA}}, pages = {4001--4008}, year = {2009} }

×

Abstract

We propose an approach for acquiring geometric 3D models using cameras mounted on autonomous vehicles and robots. Our method uses structure from motion techniques from computer vision to obtain the geometric structure of the scene. To achieve an efficient goal-driven resource deployment, we develop an incremental approach, which alternates between an accuracy-driven next best view determination and recursive path planning. The next best view is determined by a novel cost function that quantifies the expected contribution of future viewing configurations. A sensing path for robot motion towards the next best view is then achieved by a cost-driven recursive search of intermediate viewing configurations. We discuss some of the properties of our view cost function in the context of an iterative view planning process and present experimental results on a synthetic environment.

Individual Evolution as an Adaptive Strategy for Photogrammetric Network Design
Gustavo Olague, Enrique Dunn, Evelyne Lutton
Adaptive and Multilevel Metaheuristics

×

BibTex

@incollection{DBLP:series/sci/OlagueDL08, author = {Gustavo Olague and Enrique Dunn and Evelyne Lutton}, title = {Individual Evolution as an Adaptive Strategy for Photogrammetric Network Design}, booktitle = {Adaptive and Multilevel Metaheuristics}, pages = {157--176}, year = {2008} }

×

Abstract

This chapter introduces individual evolution as a strategy for problem solving. This strategy proposes to partition the original problem into a set of homogeneous elements, whose individual contribution to the problem solution can be evaluated separately. A population comprised of these homogeneous elements is evolved with the goal of creating a single solution by a process of aggregation. The goal of individual evolution is to locally build better individuals that jointly form better global solutions. The implementation of the proposed approach requires addressing aspects such as problem decomposition and representation, local and global fitness integration, as well as diversity preservation mechanisms. The benefit of applying the individual evolution approach for problem solving is a substantial reduction in computational effort expended in the evolutionary optimization process. This chapter shows an example from vision metrology where experimental results coincide with previous state of the art photogrammetric network design methodologies, while incurring in only a fraction of the computational cost.

Parisian camera placement for vision metrology
Enrique Dunn, Gustavo Olague, Evelyne Lutton
Pattern Recognition Letters

×

BibTex

@article{DBLP:journals/prl/DunnOL06, author = {Enrique Dunn and Gustavo Olague and Evelyne Lutton}, title = {Parisian camera placement for vision metrology}, journal = {Pattern Recognition Letters}, volume = {27}, number = {11}, pages = {1209--1219}, year = {2006} }

×

Abstract

This paper presents a novel camera network design methodology based on the Parisian evolutionary computation approach. This methodology proposes to partition the original problem into a set of homogeneous elements, whose individual contribution to the problem solution can be evaluated separately. A population comprised of these homogeneous elements is evolved with the goal of creating a single solution by a process of aggregation. The goal of the Parisian evolutionary process is to locally build better individuals that jointly form better global solutions. The implementation of the proposed approach requires addressing aspects such as problem decomposition and representation, local and global fitness integration, as well as diversity preservation mechanisms. The benefit of applying the Parisian approach to our camera placement problem is a substantial reduction in computational effort expended in the evolutionary optimization process. Moreover, experimental results coincide with previous state of the art photogrammetric network design methodologies, while incurring in only a fraction of the computational cost.

CURRICULUM VITAE