The National BIM Standard Capability Maturity Model


Early in 2007, the National Institute of Building Sciences released the draft National Building Information Modeling Standard (NBIMS), “Version 1.0 – Part 1:  Overview, Principles, and Methodologies,” with the intent that this standard would serve as a guide for information generation and management between all phases of a facility’s life cycle.  It includes useful information about the scope of NBIMS, information exchange concepts and content, in addition to the NBIMS development process.  In Chapter 4.1, there exists a definition for the minimum standard providing a baseline for measuring information exchange capabilities within a given BIM.  This baseline standard establishes what constitutes a minimum BIM. 

                 

The Interactive Capability Maturity Model and 2007 AIA TAP BIM Award Winners


The next chapter, Chapter 4.2, takes the baseline established for a minimum BIM, expands on this standard, and develops a Capability Maturity Model (CMM).  The CMM is most successfully used as a tool for BIM users to evaluate their practices and processes.  Furthermore, it can be used for portfolio-wide analysis to establish an organization’s current strategic or operational BIM implementation.  In addition, it can be used to set goals for achieving greater information maturity on future BIM projects.   The intended use of the CMM is only as an internal tool and when utilized properly, the CMM provides valuable information about leveraging BIM information management, along with opportunities for developing more mature information flow for future BIM projects.  It would be a mistake to think of this as a tool to compare BIMs or BIM implementations.  The CMM is really a tool to determine the level of maturity of an individual BIM as measured against a set of weighted criteria agreed to be desirable in a Building Information Model. 


The CMM is a matrix with 11 area of interest on the x-axis and 10 levels of maturity on the y-axis.  Areas of interest include; data richness, life-cycle views, change management (formerly ITIL maturity assessment), roles or disciplines, business processes, timeliness/response, delivery method, graphical information, spatial capability, information accuracy, and interoperability/IFC support.  Ratings for maturity levels are on a scale of 1 to 10, with 10 being the most mature.  Two versions of the CMM exist.  The first version is the tabular CMM, which is a static Microsoft Excel workbook consisting of three worksheets.  The second version, based on the tabular CMM, is the Interactive CMM (I-CMM).  The I-CMM is a multi-tab Microsoft Excel workbook, which includes several interdependent worksheets of functionality.  The worksheets are interactive and actively update the BIM’s maturity level as the user enters information.


Testing Team Interactive Capability Maturity Model (I-CMM) Project


Within the NBIMS Project Committee are task committees or teams, one of which is the NBIMS Testing Team.  Of the myriad of responsibilities, one of the NBIMS Testing Team tasks is to evaluate and integrate existing industry best business practices.  Based on these responsibilities, the NBIMS Executive Committee felt it appropriate for the Testing Team to be the group testing and validating the Capability Maturity Model established in NBIMS, Version 1-Part 1.                 


After their announcement of the 2007 AIA TAP (Technology in Architectural Practice) BIM award winners, AIA and NBIMS agreed to partner on a project in which the NBIMS Testing Team would apply the I-CMM to each of the TAP BIM award winners.  It is important to note that the purpose of the test project was to focus on the I-CMM tool and not on conclusions about the BIM projects themselves.  The Testing Team’s focus was to apply the scientific method provided by the I-CMM to “real” BIM projects.  In doing so, the project’s outcome would provide adequate evidence and analysis in support of the validity of the Capability Maturity Model.  Instructions given to Testing Team members who volunteered to participate in the project were as follows:    

  •     Access the winning AIA TAP BIM packages.
  •     Primary evaluator rates an assigned project independent of secondary evaluator.
  •     Follow-up evaluators score submission as a “blind” secondary evaluator.
  •     Compare the two independent scores.
  •     Confer until consensus on score if variance is ±10%.
  •     If variance exceeds ±10%, then Testing Team Leader will be notified and new evaluators will be selected. (NOTE:  this was not necessary since no BIM score was more than 5% different, and usually only 1 or 2% different.)
  •    Primary evaluator to call each BIM award winner and discuss the scoring and methodology.
  •    Finalize score after conversation with BIM award winner.  Post score.


A group of NBIMS Testing Team members, led by Major Patrick Suermann, P.E., University of Florida, applied the I-CMM to nine winners in four categories.  Testing Team members participating as evaluators were:


  •     Chris Hubbard, Quarry Group, Inc.
  •     Mat Krogulecki, MACTEC, Inc.
  •     Tammy McCuen, University of Oklahoma
  •     David Scheer, University of Utah
  •     Craig Thomas, Leo A Daly


Evaluators scored each project within the 11 areas of interest and 10 levels of maturity provided by the I-CMM.  In addition to the measurements provided by the I-CMM, evaluators also obtained a list of software used in each of the BIM projects and identified the strengths of each submission through the lens of the I-CMM.  Evaluators also noted any challenges or opportunities for improvement they experienced in their use of the I-CMM for which future clarification and modification to the CMM may be necessary in NBIMS.     


Following are the I-CMM results from each evaluator’s analysis of the individual BIMs organized by their AIA TAP BIM award category.   The list includes each project along with the outcomes from the Testing Team evaluation using the I-CMM as a tool to rate information contained within the BIM.  As a reminder, the scoring levels within the I-CMM reflect the maturity level of an individual BIM as measured against a set of weighted criteria agreed to be desirable in a BIM.  The I-CMM should not be used to compare BIMs or BIM implementations, but only to measure an individual BIM’s maturity level. 


Category: Creating Stellar Architecture Using BIM


Project:  The Loblolly House

Award Type: Citation


Project:  The Loblolly House


Strengths of the BIM: BIM aids beautiful design

Challenges for evaluators using the I-CMM: How does one evaluate modularization on the CMM?



Category: Design/Delivery Process Innovation Using BIM


Project:  Benjamin D. Hall Research Building at the University of Washington

Award Type: Citation



Strengths of the BIM: Design-Build-Operate-Maintain project delivery method clearly aided by integrated practice upfront in BIM

Challenges for evaluators using the I-CMM: 5D cost data information



Category: Design/Delivery Process Innovation Using BIM


Project:  Food and Drug Administration Headquarters

Award Type: Honorable Mention



Strengths of the BIM: Campus of multiple BIMs

Challenges for evaluators using the I-CMM: Grading the “portfolio” or the buildings



Category: Design/Delivery Process Innovation Using BIM


Project:  U.S. Coast Guard Web Enabled Services

Award Type: Honorable Mention



Strengths of the BIM:  Extremely advanced and early design pre-construction BIM

Challenges for evaluators using the I-CMM:  Do pre-design BIMs not used in CDs qualify for nD in graphical information?



Category: Support for Human Use and Innovative Program Requirements Using BIM


Project:  The Royal London Hospital

Award Type: Citation



Strengths of the BIM: Commitment to BIM through entire building lifecycle

Challenges for evaluators using the I-CMM:  ITIL – relative unknown to AECO Industry, Clarification of written material helpful



Category: Support for Human Use and Innovative Program Requirements Using BIM


Project:  Open Geospatial Consortium Open Web Services

Award Type: Honorable Mention



Strengths of the BIM: Maximized interoperability, real-time data sharing

Challenges for evaluators using the I-CMM: “Roles or disciplines” spectrum skewed towards traditional design on project-by-project basis



Category: Jury's Choice


Project:  GSA Pilot Project Successes

Award Type: Citation



Strengths of the BIM: Pre-qualified software vendors to meet requirements

Challenges for evaluators using the I-CMM: Receiving multiple projects in one submission. 

Is 3-D laser scanning better/different than as-built data?

 


Category: Jury's Choice


Project:  Opera Theater, Sydney Opera House, and Western Colonnade

Award Type: Honorable Mention

    


         

Strengths of the BIM: Integrated into organizational transformation plan and used as an analysis tool

Challenges for evaluators using the I-CMM:  How facilities management fits into current CMM ITIL concerns



Testing Team I-CMM Project Outcomes


It is important to re-emphasize that the primary objective for the Testing Team was to evaluate and score the award winning BIMs based solely on the measurements within the I-CMM, which is to leverage information management, rather than to perform evaluations based on architectural, engineering, construction, or management metrics.  Accordingly, the BIMs scored received a wide range of scores commensurate with their project requirements.  Logically, the highest scoring BIM submission was a test bed BIM pushing the edge of current interoperability, while the lowest scoring BIM (which received a “Minimum BIM” rating) was for a custom-designed residential home. 


With this in mind, it is important to note that while the I-CMM is very effective at measuring BIM information management, it should not be used as a benchmark for any other metrics.  In other words, just as owners’ needs do not require that every building be built to LEED-Platinum standards, neither should every BIM be perceived as any less successful if they do not achieve an I-CMM Platinum score. Ultimately, the I-CMM facilitated quantitative evaluation of qualitative traits. 


The process above has resulted in three primary changes:


- The “ITIL Maturity Assessment” portion of the CMM underwent a name change and is now titled “Change Management,” a more general and all encompassing term.  In addition, the Minimum BIM Chapter, Chapter 4.1, was changed to note that currently, Change Management did not need to be considered to score a minimum BIM.


- Due to some difficulty understanding a few of the definitions of the level of maturity, the author of the original CMM, Dana K. “Deke” Smith, AIA, Executive Director of buildingSMARTalliance™, created a tab in the I-CMM that elaborates on the existing definitions which allows people to better understand the terminology and more easily agree on scores for their BIMs.


- Finally, changes made to the “Area of Interest Chart” tab that automatically updates with the Interactive Maturity Model changes the original bar chart to a “radar” or “spider web” style chart, as shown below, which better represents the areas of maturity.  This was the recommendation of Kimon Onuma and adopted by the Task Team. 



Conclusion


The Testing Team’s project this year provided information not only to NBIMS about Building Information Models, but also to the AIA TAP BIM award winners by providing them with a rational measure of current maturity on each of these projects.  NBIMS has set the minimum BIM score for 2007 at 20.1 points.  NBIMS currently proposes that the score for a ‘Minimum BIM’ be raised to 40 points.  This proposal recognizes that in time, owners will raise the standard for expectations from models delivered on projects in the future.  The Capability Maturity Model is available and users should try to utilize it to gauge their current maturity level, in addition to using it as a tool to set goals for future maturity levels.  As with any new tool, users can expect the CMM to evolve.  According to the National BIM Standard Version 1 – Part 1, “As industry evolves and more rapidly adopts greater levels of maturity, this model will change to accurately reflect best industry practices.” We hope that this tool will serve to bridge BIM implementation from theory into practice in a way that provides goals for the best way to manage information in a pragmatic approach.