Open source software assessment methodologies

Chamindra de Silva - Technology Strategist, Global Technology Office, Virtusa Corp.and

Madu Ratnayake - Global Head of Process, Quality and Knowledge Management, Virtusa Corp.

During difficult times, like the current economic downturn, open source systems are increasingly being adopted due to their significant cost advantages. Additionally, open source products are now moving and maturing the value chain in areas such as Customer Relationship Management, Business Intelligence, Business Process Management and Enterprise Content Management to name a few.

You now find that there are over a hundred thousand registered open source software products available for free download on the internet.

All Free and Open Source projects provide the end user with the freedom to access, modify and distribute the software. However, while some of these have the maturity and stability to support mission critical production environments, others are considerably less mature.

This is because open source ultimately is a liberal license and a development methodology and provides no inherent assurances on the quality of the end result of the software.

Free and Open Source projects range from the lone teen developer to the multi-organisation supported communities such as Linux and Firefox, and there is a wide disparity in terms of the product quality, support and licensing models.

So a smart CIO needs to be able to pick the right open source project from the many out there, and if this is done well, many benefits can result including a significant benefit in operating costs and time to market. So how do you pick the right ones for your enterprise? This is where an assessment framework can be of great assistance.

What are Capability Maturity Models?

Maturity models are not new in the software engineering industry. The Capability Maturity Model Integration (CMMI) is one such example. CMMI covers best-practices for planning, engineering and managing product development and maintenance.

The key element we need to focus on is that the maturity model provides clear quantifiable measures and an evaluation framework of engineering maturity. Such principles can be taken out of the typical commercial software engineering environment that they are applied in and modified to assess open source project maturity.

Open Source Maturity Models

Since the source code is openly available for public consumption and contribution, the way open source software is built differs from how proprietary software is built. A cornerstone of the success of an open source project is the strength of the people and community that surrounds it.

A strong community can provide a wealth of diverse mindshare from the best minds around the world. In comparison, a proprietary product can only benefit from the mindshare of those within the company. The second item that needs extra consideration is the licensing terms and intellectual property management policies and controls in the project.

There are popular established open source licenses that have proven themselves in court cases, but there are others that have not been tested in this regard. And finally, open source software comes free, but with no warranty.

Thus most companies need to fill a gap in accountability by getting into a support contract with a provider that is able to provide certain service levels for the software.

Apart from these three areas that need special consideration in open source, all other assessments areas are typically what you would or should do for proprietary products such as quality, security, scalability, adoption and functionality.

The open source software assessment methodologies we evaluated here are:

* Open Source Maturity Model (OSMM) from Capgemini

* Open Source Maturity Model (OSMM) from Navica

* Qualification and Selection of Open Source Software (QSOS) from Atos Origin

* Open Business Readiness Rating (OpenBRR) sponsored by Carnegie Mellon West Centre for Open Source Investigation, O'Reilly CodeZoo, SpikeSource and Intel

All the assessment methodologies provide three basic aspects: evaluation criteria, requirements weightings (which allow the evaluator to assign weights according to the importance of each criterion) and final scores. Each one has its strengths and the following table provides a high level assessment of the coverage of each of maturity model.

A comprehensive assessment tended to be on the criteria that had the most detailed measures on each of those respective areas. The following table provides a high level assessment based on research and our own internal assessment from our process and open source teams:

In addition to the coverage, we also looked at the assessment methodologies themselves.

popularity and conditions placed on the utilisation of these assessment methodologies also can have an impact on what is the most appropriate for your organisation.

For example, going with a community run assessment methodology better ensures that the model is peer reviewed and verified.

Conclusion

Looking at the comparison, the Capgemini model is the only one that is proprietary and restricts access. The Navica assessment does not fare too well on coverage.

OpenBRR and QSOS seem to fare the best on coverage, but ultimately it depends on the priorities of the target organisation for which you are assessing the project.

For example, the priorities and resultant weightage for an Independent Software Vendor (ISV) would be different from that of an enterprise.

Unfortunately, none of the maturity models have been able to grow a community of peer reviewers, as was initially hoped. Still they have provided us with a valuable baseline for evaluating open source assets.