1st Workshop and Challenge on
Computer Vision in the Built Environment
for the Design, Construction, and Operation of Buildings

Held in conjunction with the
IEEE Conference on Computer Vision and Pattern Recognition 2021.


The workshop will connect the domains of Architecture, Engineering and Construction (AEC) with that of Computer Vision by establishing a common ground of interaction and identify shared research interests. These topics will be presented from the dual lens of Computer Vision and AEC, highlighting the limitations and bottlenecks related to developing applications for this specific domain. The objective is for the two communities to come together and solve problems that are relevant in the AEC community, such as understanding the construction sites as it is changing, as well as automatically generating building models from scans of real data.

This workshop will also host the first Scan-to-BIM challenge focused on key problems when converting 3D point cloud data obtained using lidar, photogrammetry, or depth map cameras to Building Information Models (BIMs). BIMs represent different elements of a building in a semantically rich manner, identifying objects, associated attributes, and geometrical and topological relationships. AEC firms and public agencies create BIMs to design new buildings and other structures; however, BIMs are rarely available for existing buildings or facilities. For maintenance, retrofitting, or renovation AEC entities could greatly benefit from access to BIMs of their facilities, because interpreting 2D as-built drawings is very time consuming and leads to many errors and omissions. Furthermore, asset owners could use BIMs for operations and maintenance as BIMs enable storing and having access to attributes such as maintenance history, material type, manufacturer specifications among others.

However, current computer vision evaluations are mainly done on familiar evaluation metrics such as Intersection-over-Union (IoU) or classification accuracy, rather than metrics that are more relevant in practice in the AEC community, such as geometrical and topological connections between different spaces and openings. This challenge aims to bring the computer vision, civil engineering, geomatics, and construction communities together by creating new tasks and evaluation metrics that can be tackled by computer vision researchers while being of interest to and fulfilling the objectives of the AEC community.


Times are in Central Daylight Time zone

Attend from The Zoom Webinar Link here:




01.00-01.30 PM :   

Introduction to the Workshop and Challenge [recording]

Dr. Fuxin Li, EECS, Oregon State University

01.30-02.00 PM :   

Vision for Construction in Practice: Lessons from Startup Experience, Keynote Talk - [abstract, recording]

02.00-02.10 PM :   

2nd-Place Presentation, Challenge I: Floorplan Reconstruction [recording]

Dr. Jing Dao Chen, CEE, Georgia Institute of Technology

02.10-02.30 PM :   

Winner Presentation, Challenge I: Floorplan Reconstruction [recording]

Jiali Han, IA, Chinese Academy of Sciences

02.30-03.00 PM :   

Digitalisation beyond BIM: Towards a data-driven future of the Architecture, Engineering and Construction (AEC), Keynote Talk - [abstract, recording]

03.00-03.15 PM :   


03.15-03.25 PM :   

2nd-Place Presentation, Challenge II: 3D Building Model Reconstruction, [recording]

Miguel Vega, CGEE, Technical University of Munich

03.25-03.45 PM :   

Winner Presentation, Challenge II: 3D Building Model Reconstruction, [recording]

Liu Yang and Yi-Chun Lin, CE, Purdue University

03.45-04.15 PM :   

HouseGAN: Teaching Computers to Design Architecture, Keynote Talk - [abstract, recording]

04.15-04.45 PM :   

Scan-to-BIM for Energy Efficiency Renovation of Buildings, Keynote Talk - [abstract, recording]

04.45-05.15 PM :   

Panel Discussion recording]

05.15-05.30 PM :   

Closing Remarks & topics for next workshop recording]

Dr. Martin Fischer, CEE, Stanford University



Marzia Bolpagni

Head of BIM International at Mace, Honorary Lecturer at UCL

Dr. Marzia Bolpagni works as Head of BIM International at Mace where she develops and implements digital construction solutions for public and private international clients in five international hubs. She holds a PhD in ICT and Smart Construction and she is passionate in filling the gap between industry and academia. She is glad to be a member of the BIMExcellence Initiative, Assistant Editor of the BIM Dictionary where she coordinates more than 120 volunteers worldwide, Ambassador of the UK BIMAlliance and Expert at the European Committee for Standardisation (CEN) TC 442 where she chairs a Task Group on information requirements standardisation (Level of Information Need). She is lead author of the Level of Information Need standard EN 17412-1, Chair of EC3 Modelling and Standards Committee and Honorary Lecturer at UCL The Bartlett School of Sustainable Construction. She is also founder of Italians in Digital Transformation UK, she loves sharing her knowledge with students and she is often invited as keynote speaker at academic and industrial events. She received several awards for her activities including ‘Woman Ingenious’ in 2017 and Star Award for Innovation and Service Excellence by Mace in 2019.


Frédéric Bosché

Professor, CEE, University of Edinburgh

Dr. Frédéric Bosché is a Senior Lecturer in Construction Informatics at the University of Edinburgh. Within the University, he is a member of the School of Engineering, the Centre for Future Infrastructure (CFI) and the Edinburgh Centre for Robotics. Within that network, Frédéric leads the CyberBuild Lab whose research covers the areas of data processing, information management and information visualisation, all within the context of digital construction, project management and asset monitoring. Frédéric is the Programme Director of the MSc Leading Major Programmes, the current President of the International Association for Automation and Robotics in Construction (IAARC), and Associate Editor of the international journal of Automation in Construction. He has received several awards, including the IAARC Tucker-Hasegawa Award in 2018 for “distinguished contributions to the field of automation and robotics in construction”. Two of his main on-going research projects are the H2020-funded BIMERR and COGITO projects.


Yasutaka Furukawa

Professor, CS, Simon Fraser University

Dr. Yasutaka Furukawa is an Associate Professor in the School of Computing Science at Simon Fraser University (SFU). Prior to joining SFU in 2017, he was an assistant professor at Washington University in St. Louis USA, and a software engineer at Google. He completed his Ph.D. at Computer Science Department of the University of Illinois at Urbana-Champaign in 2008. Dr. Furukawa received the best student paper award at ECCV 2012, the NSF CAREER Award in 2015, CS-CAN Outstanding Young CS Researcher Award 2018, Google Faculty Research Awards in 2016, 2017, and 2018, and PAMI Longuet-Higgins prize in 2020.


Derek Hoiem

Professor, CS, University of Illinois at Urbana-Champaign

Dr. Derek Hoiem is an Associate Professor in Computer Science at the University of Illinois at Urbana-Champaign, since January 2009. Derek earned his PhD in Robotics from Carnegie Mellon University in 2007 and completed a postdoctoral fellowship at the Beckman Institute in 2008. Awards include ACM Doctoral Dissertation Award honorable mention, CVPR best paper award, Intel Early Career Faculty award, Sloan Fellowship, and PAMI Significant Young Researcher award. Derek Hoiem is also co-founder and leads the vision team at Reconstruct, a construction technology company that integrates image 3D processing and analysis with plans and schedule to compare as-built to as-planned.


Eleni Papadonikolaki

Professor, The Bartlett School of Sustainable Construction, UCL

Dr. Eleni Papadonikolaki ARB, MAPM, SFHEA, is an Associate Professor in Digital Innovation and Management at University College London (UCL) Bartlett School of Sustainable Construction (BSSC) and a management consultant with Digital Outlook. She has a PhD in Design and Construction Management from TU Delft, Netherlands. Bringing practical experience of working as an architect engineer and design manager on a number of complex and international projects in Europe and the Middle East, she is researching and helping teams manage the interfaces between digital technology and their work. Eleni is the author of over 60 peer reviewed publications, for instance in the International Journal of Project Management, Construction Management and Economics and others. She has attracted and delivered collaborative research projects of total worth circa £10M as Principal and Co-Investigator funded by European and UK research councils. She is the Director of the MSc Digital Engineering Management at UCL where she develops the new generation of leaders in digital transformation.

Keynote Abstracts

Vision for Construction in Practice: Lessons from Startup Experience

Prof. Derek Hoiem, CS, University of Illinois at Urbana-Champaign

As computer vision becomes increasingly useful, researchers should think beyond the abstract technical problems to contexts and workflows of application. Doing so increases potential for research impact and can reveal new technical challenges. I will show examples of evolution from technical solutions to workflow solutions for the built environment from my work at Reconstruct and highlight important research problems that remain.

Digitalisation beyond BIM: Towards a data-driven future of the Architecture, Engineering and Construction (AEC)

Prof. Eleni Papadonikolaki, The Bartlett School of Sustainable Construction, UCL

The Architecture, Engineering and Construction (AEC) undergoes massive transformations through the influence of its socio-political environment and digital economy. Digital economy becomes increasingly topical in the AEC since skills shortage on construction sites and estates offices is growing and recent restrictions in the movement of workers (see Brexit) continue the traditional overreliance on people’s judgement and experience, without leveraging technology and data. In our technology-laden world, various opportunities arise from novel ways to capture, clean, analyse and visualise a wealth of data, previously not tamed. Digital technologies and the pervasiveness of data create new opportunities and challenges in AEC projects, especially in information sharing, decision-making and knowledge management. From these, many proprietary solutions promise to provide quick, ‘hands-off’ and off-the-shelf answers to persistent problems such as low efficiency, productivity and poor quality. However, the complexity of data and AEC projects have shown that quick and off-the-shelf solutions, such as the promise of Building Information Modelling (BIM) for the AEC rarely work on their own. Instead, as digital technologies continuously evolve and become more integrated, there is a need to push the boundaries of BIM especially during project execution and hand-over phases – the less digitised phases of AEC – ensuring a meaningful digital thread, supported by automation and machine learning. Problematising around the interfaces among people, technology and data, how can we lead project teams and develop our people to leverage data and excel in digital economy?

HouseGAN: Teaching Computers to Design Architecture

Prof. Yasutaka Furukawa, CS, Simon Fraser University

We have made a breakthrough in the domain of automated floorplan generation with the research in the past few years. Our latest system takes a form of a "relational refinement GAN with convolutional message passing", which is a result of our 3 recent papers (CVPR 2020, ECCV 2020, CVPR 2021). I will explain what the three key research insights are in the three papers, forming the latest system. An interactive demo is available only at http://www.houseganpp.com. This research is a joint work with Greg Mori and Autodesk Research.

Scan-to-BIM for Energy Efficiency Renovation of Buildings

Prof. Frédéric Bosché, CEE, University of Edinburgh

The mission of the EU-funded BIMERR project is to design and develop an ICT-enabled Renovation 4.0 toolkit comprising tools for Architecture, Engineering & Construction (AEC) actors in the process of delivering the energy efficiency renovation of existing buildings. This talk will first briefly present the BIMERR project, but then focus on the Scan-to-BIM solution developed to output models that can be directly and effectively used by the Renovation Decision Support System (RenoDSS) tool. The process involves several tools: the Scan-to-BIM tool, the BIM Management Platform, and ultimately the RenoDSS tool. The talk will present that process, but will naturally focus on the Scan-to-BIM tool and the different functionalities that it includes for `structural’ modelling (i.e. modelling of walls, floors and openings), MEP modelling and product property editing. The talk will conclude with some reflection on the contributions made and necessary future developments.

Workshop 2021 Recordings

Introduction to Workshop and Challenge

Keynote Talk by Prof. Derek Hoiem

Floorplan Reconstruction Challenge Winners

Keynote Talk by Prof. Eleni Papadonikolaki

3D Building Model Reconstruction Challenge Winners

Keynote Talk by Prof. Yasutaka Furukawa

Keynote Talk by Prof. Frédéric Bosché

Panel Discussion

Closing remarks

Important Dates

  • Challenge submission deadline: June 1st, 2021 June 11th, 2021 11:59PM PT (Extended!)
  • Notification to authors: June 10th, 2021 June 15th, 2021
  • Workshop day: June 20, 2021, Sunday. Day 2 of CVPR 2021.


The workshop will host the 1st International Scan-to-BIM challenge. The challenge will include the following tasks:

I. Floorplan Reconstruction
II. 3D Building Model Reconstruction


The dataset is available HERE. You will need a registered account to download the dataset. Once you submit a request for registration, it will be inspected and a registration link will be sent to you within 24 hours. The challenge contains a total of 31 buildings with multiple floors each and dozens of rooms on each floor. Of which, 20 buildings are designated as the training set, with a total of 49 point clouds. The validation and testing sets contain 5.5 buildings with 21 point clouds each. For each model, there is a point cloud in LAZ format. For the training and validation sets, a corresponding floorplan aligned with the coordinate system of the point cloud is also provided.
NOTE: If you have downloaded the data prior to May 3rd noon PT, please note that we have an updated version of the data, with the CAD model coordinates aligned with the input point cloud, which has replaced the original version.

Floorplan Reconstruction Task: For the floorplan reconstruction task, the training set consists of the point clouds of 20 buildings with multiple floors each, with a total of 49 point clouds, as well as corresponding 2D building models for each floor with multiple extracted layers. These layers include: walls, curved walls, doors, windows, stairs, and columns. The buildings have complicated floor structures which could include dozens of rooms, curved walls and many doors on each floor. We have released 6 new buildings as the validation set and another 6 as the testing set. The validation set contains point clouds and their corresponding 2D building models. For the testing set, only the point clouds will be made available, while the 2D building models will not be made available and we will host a server to evaluate the submissions.

NEW (May 12th) The validation and testing sets have been released!

3D Building Model Reconstruction Task: For the 3D building model reconstruction task, the training set consists of point clouds of 4 buildings with multiple floors, as well as the corresponding 3D building model in AutoCAD DXF format. We aim to release 2 more buildings for the validation set and 2 more as the testing set. The evaluation on the testing set will also be hosted on the evaluation server in a similar manner to the floorplan reconstruction task.


(Note that you will get an email within 24 hours after requesting to register for an account. After you receive confirmation, please use the emailed link in the confirmation email to register your account. Following this, you can download the Scan-To-BIM training set for the two tasks. The building models for the 3D reconstruction task are also available in separate files.)

File Specifications: A description of the file formats and sample code to read/write these files can be found HERE.


We will include metrics to evaluate the reconstruction of the walls, doors, floor area and windows.

2D Evaluation Metrics

Geometric Metrics:

  • IoU of the each room (a room is defined as a completely separated area with walls and doors).
  • Accuracy of endpoints. Precision/Recall and F-measure will be evaluated in the coordinate system of the point cloud. The provided endpoints will be matched with the Hungarian algorithm to the point cloud, and every point that is within a certain threshold will be determined as a match.
  • Orientation. For each matched line between the ground truth, we will compute the cosine similarity metric between them as the normalized dot product. If a line is not matched with ground truth, the cosine metric will be zero. Finally, the metric will be averaged over all the ground truth lines.

Topological Metrics:
We will also evaluate topological metrics that measure whether the connectivity of the rooms match the ground truth.

  • Warping error(Jain et al. 2010).The warping error will first warp the predicted floorplan to the ground truth with a homotopic deformation, and then compute the pixels that cannot match after the deformation.
  • Betti number error.The Betti number error will compare the Betti numbers between the prediction and the ground truth and output the absolute value of the difference.

3D Evaluation Metrics

Figures and evaluation metrics (accuracy, correctness, completeness) are from:
Tran, H., Khoshelham, K., & Kealy, A. (2019). Geometric comparison and quality evaluation of 3D models of indoor environments.ISPRS journal of photogrammetry and remote sensing,149, 29-39.

Evaluation and Submission

The evaluation code for the 2D floorplan task is available from This GitHub Repository. For 3D, because of the difficulty to come up with correct ground truth and have a consistent file format that is acceptable to both the computer vision and AEC communities, we have decided that we will use an expert panel to evaluate the submissions manually. We will aim to have automatic evaluations for the challenges in next years.

Submission of results on the testing set and evaluation on the ground truth data is now available on the evaluation server. For 2D, the submission will be in the same JSON format as in the ground truth provided to you. For 3D, the submission will be in DXF or Wavefront OBJ formats.

2D Challenge Results

Method Name Team Members Affiliation Precision (2cm) Precision (5cm) Precision (10cm) Recall (2cm) Recall (5cm) Recall (10cm) Intersection-Over-Union Warping Error Betti Error
VecISR-2D Jiali Han, Shuhan Shen Institute of Automation, Chinese Academy of Sciences 5.73% 23.39% 38.54% 2.08% 8.48% 13.59% 56.88% 0.256 1.186
Jiali Han, Mengqi Rong, Hanqing Jiang, Hongmin Liu, Shuhan Shen. Vectorized indoor surface reconstruction from 3D point cloud with multistep 2D optimization, ISPRS Journal of Photogrammetry and Remote Sensing,Volume 177,2021,Pages 57-74
GTS2B Jingdao Chen, Jisoo Park, Yosuke Yajima, Seongyong Kim Georgia Institute of Technology 2.18% 9.69% 19.12% 0.88% 0.41% 8.18% 34.75% 0.232 1.132
FloorPP-Net Yijie Wu, Fan Xue Hong Kong University 1.14% 4.23% 6.52% 7.09% 25.57% 38.59% 11.98% 0.268 1.204
Yijie Wu, Fan Xue. FloorPP-Net: Reconstructing Floor Plans using Point Pillars for Scan-to-BIM.
PointWeaver Fiona Collins, Saeed Mafipour, Florian Noichl, Yuandong Pan, Miguel Vega Technical University of Munich 0.18% 0.89% 2.07% 0.60% 3.05% 6.96% 11.48% 0.336 1.915
Yusuf_Sahin Yusuf Sahin Istanbul Technical University 0.03% 0.10% 0.17% 3.96% 9.57% 15.58% 8.43% 0.268 1.764

3D Challenge Results

Method Name Team Members Affiliation Accuracy Completeness Correctness Bonus Total Score
Point cloud semantic segmentation with Superpoint Graphs and augmented synthetic-real point cloud Liu Yang, Yi-Chun Lin, Ayman Habib, Hubo Cai Purdue University 688 643 695 330 2356
Ma JW, Czerniawski T, Leite F. Semantic segmentation of point clouds of building interiors with deep learning: Augmenting training datasets with synthetic BIM-based point clouds, Automation in Construction, 113:103144.
PointWeaver Fiona Collins, Saeed Mafipour, Florian Noichl, Yuandong Pan, Miguel Vega Technical University of Munich 688 643 695 330 2356
VecISR-3D Jiali Han, Shuhan Shen Institute of Automation, Chinese Academy of Sciences 700 629 680 213 2222
Jiali Han, Mengqi Rong, Hanqing Jiang, Hongmin Liu, Shuhan Shen. Vectorized indoor surface reconstruction from 3D point cloud with multistep 2D optimization, ISPRS Journal of Photogrammetry and Remote Sensing,Volume 177,2021,Pages 57-74
GTS2B Jingdao Chen, Jisoo Park, Yosuke Yajima, Seongyong Kim Georgia Institute of Technology 624 593 565 331 2112


Contact the organizers at: cv4aec.3d@gmail.com



Iro Armeni

Postdorctoral Researcher, ETHZ


Erzhuo Che

Professor, CEE, Oregon State


Yong Cho

Professor, CEE, Georgia Tech


Martin Fischer

Professor, CEE, Stanford


Daniel Hall

Professor, CEE, ETHZ


Jaehoon Jung

Professor, CEE, Oregon State


Fuxin Li

Professor, CS, Oregon State


Michael Olsen

Professor, CEE, Oregon State


Marc Pollefeys

Professor, CS, ETHZ


Silvio Savarese

Professor, CS, Stanford


Yelda Turkan

Professor, CEE, Oregon State