What makes for a successful software development project? I was mulling over a number of projects I had been involved with in the past, some very successful, some not so, and thinking about the key differences between the two. There are any number of aspects that can be considered - the quality of project management, clearness of requirements, thoroughness of the analysis, quality of execution and technical leadership to name but a few. Here I am going to focus on two key technical aspects, not being a career project manager, and explain why they are both important and also both frequently neglected.
From my experience there are two main aspects to a technically successful software development project, the Software Architecture, and the Software Engineering Process (I am taking it as read that your teams have the basic capabilities to develop software using your chosen technical platform, which in my case is Java).
Software Architecture
When projects fail from a technical perspective it is generally down to a failure to satisfy the headline non-functional requirements of the system - performance and scalability are usually the biggest culprits here, often closely followed by transactionality and/or reliability. It should be said that the blame here is not necessarily on the heads of the engineering team alone; such requirements are all too easily swept under the carpet because the customer often doesn't know what they need in this area, or with bigger projects, isn't willing to invest the effort to find out what the requirements should be.
Enter the Software Architecture Document. The purpose of this document is to nail down the outline structure of the software, in the context of the hardware that it will run on, to satisfy the key non-functional requirements. These are the architectural goals and constraints - what do we want to achieve, what targets are we setting for performance, scalability etc - and how do we best structure our application to make optimum use of the environment in which it will run.
Now the process of preparing a Software Architecture Document is a subject all by itself, and one I will cover another time, here I want to get over the importance of having such a document, the benefits it provides, and the pitfalls it can help you avoid.
The Software Architecture Document is in effect the launching point for your development project as well as providing the bridge between the non-functional requirements and the framework within which the functional requirements of the system will be implemented. It starts by framing the key non-functional requirements (also referred to as the architectural goals) and the architectural constraints. These can be such disparate items as "the system shall support 200 concurrent users", "the system shall respond to 80% of web server requests inside 1s", "the search function will use a Google search appliance", and "the system shall integrate with PayPal's Instant Payment Notification and Google Checkout". These are high level and should come straight from the customer's overall requirements for the system.
The non-functional requirements will have a fundamental influence over the design and implementation of the system. To not consider any of these requirements, or to ignore architectural constraints ("we have an Oracle license so you will be deploying to Oracle 10gAS", for example) can doom a project before a single line of code has been written, and it is vital that as many of these requirements and constraints are flushed out as early as possible, and documented. This will then allow the project team to develop a software architecture that allows the functional requirements to be implemented, but also ensures that the non-functional requirements are considered from the earliest possible stage of the project.
A failure to address the non-functional requirements at the start of a project is often fatal, as the cost and/or complexity of re-engineering a finished solution (and the delays associated with doing so) to meet 'new' requirements can be prohibitive, not to say unpopular with the customer. This is where strong project management is useful - both in ensuring that the customer provides some metrics (customer volumes, response times, infrastructure limitations etc), and in ensuring that the project team fold them into their design decisions.
Fundamentally, the Software Architecture Document is a vehicle for stating those metrics, and then outlining the software architecture that will support them and provide documentation of the framework for the developers to work against. The document also allows the team to trace from non-functional requirements and technical constraints through to the architectural design decisions, demonstrating how requirements are satisfied and, where appropriate, why a particular approach was taken.
If the Software Architecture is lacking key metrics, or worse still does not exist at all, then the project is at severe risk of failing to meet its non-functional requirements. The customer may not even realise that they have non-functional requirements, and in the event of this it is the development team's responsibility to push their management team hard to agree a set of requirements with the customer. The larger the system, the more important and deeper the non-functional requirements need to be. Conversely it is only fair to point out that a small project with a small target user base may be able to get away with no such requirements specified - simply because the functional requirements can never tax the deployed system sufficiently for any non-functional issues to arise.
In summary then, the software architecture is important because it ensures that non-functional requirements have been explicitly drawn up and considered in the design and development of the system. The Software Architecture is frequently neglected however, for this very reason - it all to often serves to highlight what hasn't been thought of. Also it cannot help that writing such a document is sometimes viewed as a paper exercise with no immediate value to the project. Hopefully I have explained why this is an incorrect viewpoint, that the Software Architecture is a vital component of the development project, and that neglecting to undertake this work in some form increases the risk of a technical project failure.
Having ensured that both functional and (technical) non-functional requirements will be addressed during the architecture and design phase, your team can move on and start producing code. And this is where the other key technical aspect comes in.
Software Engineering Process
If we accept that developers are, by definition, good at producing code that satisfies the requirements laid down (both functional and non-functional) then I would also posit that developers each, by and large, have their own style of working. Left to their own devices, a project team of six people that produced a working system with no formal engineering process in place would produce six sets of differently styled code, six different approaches to unit testing, and so on. Of course, you may be fortunate and get less than six sets, but you will definitely have two or more unless you set out from the beginning to avoid such a situation. And the bigger the project, the bigger the problem.
This is where the Software Engineering Process comes in. It lays out a modus operandi for the development of code at a high level (so that everybody works in the same way, produces the same sets of deliverables etc), lays down standards for the deliverables being produced (coding and testing standards, User Interface design standards, documentation standards where applicable, etc) and finally and most importantly (and frequently the most neglected part) provides a technical environment that supports and promotes the process and the standards, and makes it easy to work with/to them.
Most companies have a culture that takes care of the first of these three parts of the process. New people join an existing team and are inducted, almost by default, into the way the team works to produce code, and the team work as a unit going through the same development lifecycle and producing the same deliverables. It happens so naturally that it is taken for granted.
The second part of the Software Engineering Process is that of working to standards. Fewer companies have pre-prepared standards and guidelines documentation to hand for a project team. With large companies this is not because they don't exist (exercises like ISO9001 certification ensure that they do), but more generally because the groups that develop such documentation are too far removed from the people expected to use them. Therefore adoption is low for all sorts of reasons - people are not aware of them, can't work out which to apply and how, or there is insufficient enforcement of their use by the company as a whole. With smaller companies, if these documents don't exist it is frequently because the emphasis is more on doing the 'real' work than documenting 'how to make the code look nice'.
The final part of the process is the concept of providing technical resources to simplify the implementation of standards within the overall development cycle. This is, in my experience, the part most frequently missing from a project team, despite the fact that this is now supported better than ever. Coding to another person's standards always takes more time and effort than coding to one's own "internal style guide", so to ensure a high level of compliance there are all sorts of things that can be done to make life, and conformance to standards, easier. For example in Eclipse you can set up templates, formatting rules, on-save actions and checkstyle rules which all help to produce code that is by default closely aligned to coding standards, with the IDE providing warnings to the developer when they diverge from those standards.
Other tools such as CruiseControl allow the project to run regular repeatable builds and to re-run all automated tests, thereby providing a high level of visibility as to the state of the code ("are the unit tests all passing?"). Further tools can attest to the level of quality present - Emma can highlight where testing is insufficiently thorough, FindBugs can highlight programming antipatterns in the code, and so on. All this can be scripted to happen, for example, every evening, so that any issues introduced into the code can be caught quickly and resolved.
All these tools support the development team in producing better quality and more maintainable code than would otherwise be the case (and for little additional cost too), aspects of the development process that become more vital as the project progresses through test and moves into live deployment and maintenance. A good engineering process, together with the tools and standards to match, can help ensure that your project delivers code to a known level of quality that can more easily be maintained, even if the vast majority of your original team move on.
In Summary
Firm foundations (in the Software Architecture Document) and a well supported, repeatable process with quality built in along the way (in the Software Engineering Process) are key to a successful development project. They help to ensure that key non-functional requirements are addressed from day one, that the development team produce code of consistent quality, and that testing happens early and often. It is easy to fall into the error of writing standards for the sake of standards (and causing many a developer to become disgruntled by their arbitrary nature), but with modern tools it is now possible to focus on promoting practices that are a more natural fit with the tools and therefore easier and less demanding to adhere to.
