Software architecture and engineering - From dog houses to skyscrapers

Chandika Mendis, Director Technology Virtusa Corporation Sri Lanka

During my interviews of young newcomers into the IT field, I notice that many of them have trouble differentiating "coding" from the discipline of "software engineering."

This is unfortunate, for there is a hugely important difference between the two that young software engineers must appreciate. This article discusses some of these software engineering aspects, with learning's from Virtusa's Technology Office.

A statement by Booch is very relevant to the topic at hand: "Dog houses have been built. You can't build a skyscraper the way you build a dog house." The "engineering" part of software engineering becomes really important-neigh, critical-for large-scale complex projects-the software equivalent of skyscrapers. So what are the important engineering aspects of building the software equivalent of skyscrapers?

First, it's important to understand what attributes are expected from the skyscrapers of the software world. Besides accurately and fully supporting immediate business requirements, the software also needs to support future expansion and scale.

Future expansion translates to both increasing business volume and changing business requirements. The software also needs to adhere to industry best practices that helps future maintainability of the code.

The use of the proper software engineering practices can result in software that adheres to the above attributes, while at the same time facilitating the productive use of a large team to improve time-to-result and enhance productivity of the software development process itself.

Requirements

Requirements gathering is usually the starting point of the software engineering cycle. Verbal and informal methods of gathering requirements, while quick and useful in small projects, are insufficient for large-scale projects.

At the same time, the traditional waterfall method of first gathering all the requirements before proceeding to design and development tends to also fail in a world where time-to-result is becoming a prime differentiator.

Instead, modern agile development techniques tend to engage customers earlier and more frequently and also provide for shorter iterations, enabling customers to see the product earlier and decide implementation priorities. In our experience, agile methods are growing in acceptance and momentum, and are even becoming the approach of choice in offshore development projects.

Metrics, such as the Requirements Clarity Index and Requirements Stability Index, are useful measures of the requirements gathering and management processes. One interesting tool I've come across is an Automated Requirements Measurement tool from NASA.

The tool can scan a requirements document and capture many requirements-related metrics; for example, the usage of weak English phrases that lead to ambiguities, structural inconsistencies, etc. The output of this tool can be very useful to optimize manual requirements reviews.

Requirements gathering includes both business requirements and technical requirements.

The project architect is responsible for visualising the technical solution, and must therefore ensure that any technical requirements and constraints are fully captured during the requirements process.

Architecture and Design

For large projects to be delivered on-time, it is important to effectively engage a large team, and parallelise as much work as possible.

Software architecture provides the means to decompose a large project into smaller sub-systems or sub-projects. Decomposition involves identifying the functional modules, architectural layers and cross-cutting concerns.

Architectural layers focus on specific functions, such as the user interface, business logic or access to database/external systems.

By organising these layers such that, for example, the user interface layer can only access the business logic layer, and the business logic layer can only access the persistence layer, layered architecture reduces the interconnections between components and reduces the complexity of the resulting software. Cross-cutting concerns deal with common technical requirements that go across layers, such as security, error handling, transactions, etc.

A technical requirements checklist of the right questions to ask in these areas can help avoid expensive requirements gaps in new development engagements.

Defining the architecture requires broad and deep knowledge of technology choices and best practices. The team should leverage industry standards and best practices associated with its technology selections.

These standards should not be limited to the main programming language, but should also address user interface (HTML, accessibility, XHTML standards, etc.), database (database object naming conventions, SQL standards, etc.), and any other specific technologies in use. Virtusa's Technology Office provides a portal with all these specific standards and best practices, from which teams can select the subset they require for each project.

In many industries, existing blueprints are used to create new applications. So rather than architecting from scratch, it is often better to use established architecture blueprints as a starting point. Creating an architecture blueprint in code during the early stages of the design process is a highly-recommended best practice that results in better quality code aligned with the architectural vision.

This approach has the advantage of mitigating technical risks early in the project and describing the architecture in developers' language. This thin slice of code representing all the architectural layers can also be run through profiling tools to capture potential performance and scalability bottlenecks with the architecture.

Architecture blueprint generators exist that can generate the early architecture blueprint for new applications.

The architecture blueprints generated by these tools already adhere to best practices and can be a huge boost to both quality and productivity. Virtusa's Technology Office provides a custom blueprint generator for the standard Java/Spring technology stack, and also provides a list of recommendations and guidance for different sets of technologies.

Architecture blueprint generators can go a long way to ensuring that the technologies and standards used in an enterprise are consistent.

Virtusa has also created custom architecture blueprint generators for customers, based on their unique technology decisions and nuances.

Automated Unit Testing

As teams start doing parallel development work, a good team infrastructure and rigorous engineering practices are needed to ensure that conflicts are avoided, and any defects are caught early.

Defects that are caught later in the lifecycle cost much more to fix than those that are caught early in the lifecycle. Also, as the code base starts to grow in the thick of the construction cycle, there is a growing risk of developers not considering dependencies sufficiently and breaking functionality that worked earlier. Such defects can be extremely destabilizing and can easily reduce the productivity of both development and quality assurance (QA) teams to the breakpoint.

Dependency-based defects can be easily avoided by creating a suite of automated unit tests. In fact, developers must be informed that development is not considered complete until their code is sufficiently covered by automated unit tests.

Frameworks, such as JUnit for Java and NUnit or MSTest for Microsoft .NET, can be used to create and run automated unit tests. Code coverage tools, such as Emma for Java and NCover or MS Code Coverage for .NET, provide valuable information about exactly how much of the code is covered by the unit tests.

They point out which lines of code are covered and which lines are not covered by the unit tests-valuable information, indeed, for technical leaders assessing test coverage and thus stability of the code base.

Having a suite of automated unit tests can also be a huge benefit for the maintenance phase of the lifecycle, as it ensures that defect fixes or enhancements don't end up causing instability in the system.

An interesting side note is the relationship between code complexity and unit testing. Code complexity can be measured using McCabe's Cyclomatic complexity metric, which effectively counts the number of conditional branches through the system.

A rule of thumb is that the number of unit tests should at least be equal to the Cyclomatic complexity. Another interesting metric, called the CRAP metric (Change Risk Analyzer and Predictor), provides a measure of the maintainability of software. The CRAP metric increases as the complexity increases, and decreases as the code coverage from unit tests increases.

A high CRAP count can be lowered either by refactoring code to reduce complexity, or by increasing test coverage using more unit tests.

Team Infrastructure

Standard team infrastructure includes source control and defect/change tracking systems, team collaboration portals and continuous integration (CI) build servers. CI servers promote frequent code integration, by "listening" to the source control system, building the code automatically at frequent intervals, and, if automated unit tests exist, executing these unit tests.

CI servers are usually configured to send an email to the team on each build or if a build or unit test failure occurs. Having a CI server ensures that any integration failures are caught early and are therefore much easier to fix.

A defect tracking system can provide useful information on the trending of defects and the readiness to release software to the customer-typically, the number and density of defects decrease and the time to find a defect increases as the system moves closer to release.

Code Analysis

The percentage of defects caught before delivery to the customer as a percentage of the total number of defects (also called the defect removal efficiency) can provide a very useful indicator of the overall efficiency of the QA processes.

Configuration and release management is a well defined activity that can reduce conflicts and ensure a smooth transition of code and other artifacts between the Development, QA and production support teams.

Automation is key to ensuring software quality in a large project. Code analysis tools, such as Checkstyle and Findbugs for Java or FxCop for NET, automatically check code against coding conventions and best practices.

Virtusa's Technology Office provides an integrated development environment (IDE) based on Eclipse that bundles the Virtusa coding standards and conventions, as well as a standard set of code-analysis plug-ins.

Developers are expected to run these tools and correct any errors reported before code is checked in. Another custom tool provided by the Technology Office is installed on the build servers, analyses the code base and publishes metrics to an enterprise-level metrics dashboard.

These metrics are available for technical leadership to analyze and act upon, providing an additional safety net for code quality. Such infrastructure can reduce the subjectivity of code assessments and can focus manual code reviews on higher-order activities, such as finding logical and design-level issues with code.

Software development is often seen as an industry with low barrier to entry. While this may be true for small projects, complex projects that leverage large teams require a very high degree of professionalism and engineering discipline to be successful.

This is especially true as customers also become more sophisticated, demanding high quality solutions to complex business problems with significantly reduced time-to-results. In an increasingly competitive industry, it's the "architecture and engineering" aspect of software engineering that separates the men from the boys.