Tools for BIM maturity and benefits not fit for purpose, says major study

There is a lack of effective tools for assessing BIM maturity, or for evaluating its benefits, says a report commissioned from Northumbria University by the Centre for Digital Built Britain (CDBB) in partnership with the UK BIM Alliance.

The report’s authors examined 25 different ways of evaluating BIM maturity and benefits available to industry. This included 15 maturity tools, four “maturity” methods for organisations and projects, and six benefits tools.

None of the maturity assessment tools examined were aligned with ISO 19650, the international standard for managing information over the whole lifecycle of an asset, as they were all developed prior to its release in 2018. The report recommended that a framework be developed for maturity assessment tools, aligned with the ISO 19650 series, to ensure a common approach.

The study found that BIM benefits evaluation tools have value in helping to promote BIM and encourage collaborative working but recommended that these tools be extended to look at benefits throughout the whole supply chain.

With the launch of the ISO 19650 series in 2018, CDBB decided it was a sensible time to review the various tools used for assessing BIM maturity and measuring BIM benefits, to understand what the gaps were.

The research project team included UNN, BIM Academy, the Royal Institute of British Architects (RIBA) and HKA. The project was led by associate professor Mohamad Kassem (pictured above), who also led the authoring of the report with Jennifer Li of UNN. Contributing authors were Professor Bimal Kumar and Richard Watson of UNN, Adrian Malleson of RIBA, Dr Graham Kelly of BIM Academy and Dr David-John Gibbs of HKA.

The study found that BIM maturity assessment tools are not widely used. Only 28% of the 184 survey respondents used a tool to measure BIM maturity. Another 18% measured BIM maturity without a tool.

The research showed gaps between the characteristics of existing BIM maturity assessment tools and the industry requirements. Criticisms include:

• The tools are typically rigid, with binary (yes/no) inputs from users and are largely focused on readiness and capability, rather than maturity;
• The tools tend to lack granularity, which means the depth of the assessment is low or moderate. Where the granularity is higher, assessments take significantly longer to conduct;
• A lack of precision limits the usefulness of some tools; there is insufficient distinction made between readiness, capability and maturity.
• Maturity assessments do not reach far enough down into the supply chain, instead focusing on tier one contractors and lead designers;
• The gaps and limitations in existing tools are driving organisations to develop their own internal maturity assessments. Some 45% of respondents had already done this, according to a survey conducted by the report’s authors;
• The depth of assessment enabled by most tools is low, offering limited understanding of BIM maturity of organisations or projects;
• Maturity assessment is currently geared towards complying with clients’ BIM requirements, but these differ from client to client.

The survey identified several positive reasons for measuring BIM maturity, including that it helps organisations identify their BIM implementation challenges and develop improvement strategies.

As with maturity assessment tools, the report found that BIM benefits evaluation tools are not commonly used. Just 16% of survey respondents used a tool to measure BIM benefits, with 35% saying they measure benefits without a tool, and 49% saying they do not measure benefits at all.

The task of measuring BIM benefits is complex, says the report: it is hard to eliminate alternative explanations for the relationship between a BIM capability and an end benefit; there is a lack of benchmarking data and there is variation in the knowledge of those inputting data. There is a further challenge in communicating benefits.  

If organisations are to adopt BIM benefits evaluation methodologies, they need to ensure that the benefits sought are material and relevant over long project lifespans.

Some 92% of survey respondents agreed that measuring BIM benefits encourages a collaborative way of working, while 77% agreed there is a need for better measurement tools.

Gaps identified by the report in the assessment of both BIM maturity and benefits evaluation tools need to be addressed, to fulfil the requirements and expectations of the construction and asset management industry.

The report’s key recommendations are:

• All BIM maturity assessment tools should be aligned with the ISO 19650 series.
• At an organisational level, a multi-level framework for effective BIM maturity assessment should be developed to provide a common approach within the industry. This should identify a range of BIM competencies required and propose metrics for their assessment. The framework should have one common level, relevant to all disciplines within construction and asset management, which is also adaptable to suit specific organisations. Additional levels should be specific to different disciplines.
• At a project level, a BIM maturity assessment method should be developed based on the UK BIM Framework, including the ISO 19650 series.
• BIM competencies, including maturity, should play a greater role at the “invitation to tender”, “tender response”, “appointment”, and “mobilisation” stages of a project. The competencies should be extended beyond readiness and capability to include maturity. This process can be assisted by adopting the ISO 19650 series which sets out enabling requirements throughout the tender process for each appointment.
• BIM benefits evaluation tools should be extended to include the broader benefits of embracing a digital culture and driving digital maturity in the supply chain.
• At a project level, the tools should establish metrics at the outset of a project and consistently measure against targets. These metrics need to address challenges that are commonly experienced with construction projects, including: lengthy project lifetime; time lag between an activity and its corresponding benefit; variation in a project’s scope; and the continuing evolution of technology and processes.

Image: Associate professor Mohamad Kassem (Northumbria University)