© 1995 David Harvey. All rights reserved
Acknowledgements to Object Designers Limited for permission to make this paper available here.

In spite of all the promises, it appears that project pressure is still a fact of life. Deadlines come and go, and you and your teams end up working nights and weekends to produce systems which need fixing as soon as they are delivered. Only now you're designing using an OO design method, and coding in Smalltalk or C++. If you are lucky enough to have time to reflect on things, you might remember reading somewhere that object orientation was supposed to cure all this.

On the whole, of course, it hasn't. The generous claims made for object technology are frequently justified on a small scale by appeals to flexibility, adaptability, the power of tools which are still largely single-user, and productivity of individual developers. Yet in spite of the visions of large-scale use and re-use of objects throughout organisations, unfortunately it is still the case that projects fail more frequently than they succeed. Even those that eventually deliver have a rough time of it.

Like all advances in software, object-orientation has been driven by experiment and vision at the implementation level (pioneers generally prove their points by doing something). An implementation technology by itself clearly cannot solve the organisational problems inherent in building systems of significant size and complexity. Organisations have spent twenty-five years attempting to deliver 'traditional' software systems using structured programming languages and associated methodologies. Many are now embarking on the same journey with object orientation, trying to build systems with distributed architectures and highly interactive user interfaces. These new systems are more complicated by orders of magnitude than those which they are to replace. Crucially, this is being done without stopping to learn the lessons of earlier failures. The use of object-oriented techniques to analyse, design and implement business systems does not in itself address any of the issues of development process. In fact, adopting this or any new technology will stretch to breaking point and beyond a process culture which is already inadequate or ill-prepared. There are already rumblings of a backlash, a reaction against object technology. This is unfortunate: one might as well blame a new spade for a hole dug in the wrong part of the garden.

The problem, simply stated, is that the waterfall approach to the project lifecycle does not work for most of the systems people want to build today. Methodologies which embody this approach, such as SSADM, reduce the craft of analysis and design to a clerical exercise. They accommodate change either by legislating against it or making it expensive, and so fail to recognise the nature of risk in software development. Dinosaurs such as these belong in museums. In so far as a consensus has been achieved (object methodologists spend most of their pages proposing notations and interpretations rather than processes) there is general agreement that some sort of iterative or evolutionary model is a more appropriate vehicle for object-oriented systems development. However, there has been relatively little consideration of what it means to build a system in this way, and still less concerning the planning and management of such processes.

Evolutionary Delivery

As long ago as 1988, Tom Gilb was propounding Evolutionary Delivery (ED) as a solution to the problems of system development. His classic book Principles of Software Engineering Management (Addison-Wesley, 1988: ISBN 0 201 19246 2) says nothing specific about object-oriented approaches, but it is easy to recognise that, applied appropriately, object technology provides an environment in which ED can flourish. Some clarifications are necessary, as the purity of Gilb's vision of ED has become compromised over the years by a general failure to recognise its essential points. Firstly,

ED is not Exploratory Development

Some projects pretend they are pursuing ED when what they are really doing is exploratory development. At its most blatant, this involves nothing more than modifying and extending the code for a system until you have something that works. It is an approach that is as successful with objects as it ever was before: not at all. This does not mean there is no place for exploratory development. For example, if you don't know what the requirements are, you may need to build a prototype to help elicit them. You may plan to exploit an unfamiliar feature of your implementation hardware or software or need to test interfaces and explore performance characteristics. All these are opportunities for genuine exploratory development. Furthermore, any large organisation with an interest in software should support pure research and development. Exploration of tools and technologies, keeping abreast of developments, all this is important (you never know where the next competitive edge might come from) and developers need to be able to play a part in this. But this should not be part of the effort put into delivering systems.

This makes us realise that

ED is about delivery

Projects often pay more attention to evolution than to delivery: this is wrong, and it is why I prefer Gilb's term, stressing delivery, to incremental or evolutionary development. 'Evolutionary' and 'development' are both misused here to signify change, growth: 'evolutionary development' is, therefore, a tautology. Projects should embark upon ED with a strong view about what is going to be delivered, when it will be delivered, how these deliveries will be achieved, and who is going to take delivery. All this implies a framework of activities to be pursued during the course of a project, a view of what project milestones and deliverables mean, and some distinct roles and responsibilities.

Setting up an ED project

Work in projects has traditionally been organised in terms of the interim deliverables of a process. There is a requirements phase: you know this is finished when the requirements specification is agreed. Analysis, design at different levels, all these are tracked by the completion of documents, yet only implementation delivers a system. We know from experience that in most cases this approach doesn't work, so it is time to consider the alternatives. A user is primarily interested in a system, so it is an important principle of ED that we start delivering working software early, not just pieces of paper. However, a certain amount of preliminary work is still necessary. Some call this analysis, others initial design or project preparation. Whatever it is called, the purpose of this work is to establish an architecture and develop an initial plan identifying the timing and contents of deliveries.

Assuming, then, that a system is to be built and delivered in stages, how do we decide what to deliver when? Again, to take the users' perspective, what is required is software that offers testable functionality. Prototypes which effectively do nothing, for example of a user interface, are not useful as a measure of progress, particularly as these are often all an external user sees of the system for many months. Instead, the contents of the several deliveries of evolutions should proceed according to the users' view of the functionality of the system. This way of proceeding is distinct from top-down and phased approaches, both of which have more to do with the developer's view of a system's structure than a users view of its operation.

A delivery should completely implement one or more user interactions with the system. These interactions are becoming an important part of requirements capture in OO methodologies, where they are familiar as use cases or scenarios. As you will see, a system requirement expressed in these terms is a necessary starting point: if you start from a more traditional functional specification, it is a straightforward task to re-cast descriptions of required functionality into a set of use cases, an exercise which frequently clarifies the original requirement.

A set of scenarios by itself will help us neither to judge the complexity of a system, nor to come to any estimate of how long we can expect to take building the components of the system. They need to be considered in the light of an initial object model. This identifies the major abstractions to be used in the system, supports the system's information requirements through properties and associations, and proposes an initial grouping of these into domains or subsystems. The boundaries of these domains frequently arise from the characteristics of the requirements themselves, and are often suggested by 'clustering' of closely associated types. This partitioned model forms an important component of the system architecture: by taking care in the identification of domain boundaries we can make it feasible to develop and test each domain more or less independently. In general, each domain becomes the responsibility of one project sub-team: depending on the size of the project these could be real teams or simply individuals. A degree of flexibility is important: developers may need to move between domains as work is required in particular areas (domain teams can be 'virtual teams') or double up on domain responsibilities where necessary. The runtime locations and characteristics of object instances form a second view of the architecture, which is particularly important for systems where objects are distributed, are saved and shared in databases, or where concurrency is an issue.

With a set of scenarios and a system architecture document, we can start making decisions about the implementation of functionality. One way to kick off the process is to explore system behaviour at the level of domains through design workshops. Each tackles a number of scenarios or use cases. Through role-playing the collaborations between domains, we establish the high-level domain interfaces required to support the functionality. The results of these sessions are captured using interaction diagrams such as OMT event traces or Syntropy's mechanism diagrams.

Having done this, we can provisionally scope the contents of evolutionary deliveries. It may be that before we can implement the first interaction a certain amount of infrastructure is required. If so, this should be part of the first evolution, but it is important that it is not the sole contents: it must always be the case that testable functionality is delivered. There are several approaches we can take in deciding which scenarios to implement first. There is often some degree of 'natural' order, which it is as well to follow: you don't build a house by putting the roof on first. Sometimes it is appropriate to tackle first those scenarios whose domain interactions seem to be straightforward. This helps particularly with an inexperienced development team, and is also a useful strategy when an early first delivery is required: the principle of best gain for least effort applies. Conversely (and all other things being equal) we might implement a complicated scenario first, if it is particularly important that the functionality it requires should be visible sooner rather than later.

What's in an Evolution?

Deliveries should be time-boxed: the full duration of the project is split into periods or 'time-boxes' with a commitment to delivering an evolution of the system at the end of each. The lengths of these time-boxes will clearly influence the character of the project, in addition to the amount of functionality which can be implemented in each evolutionary cycle. If you are talking about 6-monthly cycles, then you are not talking about ED. A time-box longer than three months is likely to be interpreted as an old-fashioned project phase and you will lose momentum and focus. Conversely, I have seen projects working on aggressive schedules delivering an evolution every two weeks: this is an extreme case, and can put an intolerable burden on those receiving the deliveries. Typically, three to six weeks should be enough to deliver a significant increment in usable functionality. Work is planned around building domain classes to implement the functionality required to support the scenarios. The interactions between a domain's classes are explored in design workshops in much the same way as the high-level interactions between domains: interaction diagrams identify the operations required on classes to support the scenarios, along with any additional domain-specific information. These are captured in a domain design document, the classes are coded and tested by the individual domain teams, and finally integrated for delivery. As a cycle delivers a system (not a prototype) all releases must be of production quality. This is central to the notion of ED as delivery. The design is always a working document rather than a finished product: it describes the classes implemented up to and including the current evolution, and is complete (in the sense of describing the design for delivery of all system functionality) only at the end of the process.

Once delivery dates for each evolution are established, they must not change. What can change is the contents of the deliveries: if it becomes apparent that a scenario cannot be implemented in the course of a current evolution, it is moved to a succeeding evolution. The knock-on effect of this may be that an extra delivery needs to be planned, or that the final evolution delivers most but not all of the required functionality: one advantage of ED is that this becomes apparent early rather than late in the project. It appears to be the case that delivering less is forgiven more readily than delivering late (of course, it has often been the case that projects end up delivering less and late). As in all project planning, there are only four things we can change when problems occur: resources (not usually an option, and in any case the dangers of adding staff to a project which is failing are well known), timescales (we will deliver, but late), quality (we will deliver on time, but not fully tested) and functionality (we will deliver, but not everything). In general, ED mandates only the last of these.

The contents for a delivery are confirmed at the start of each cycle, but only finally established at the end. As work progresses it may be that scenarios are implemented earlier than originally planned, or that functionality can be swapped between evolutions. This is to be expected: it is important that a degree of flexibility is maintained, to offset the constraining effect of the promise that evolutionary deliveries will occur on the planned dates.

Project roles

A project organised along ED lines needs a number of key roles filled effectively to make it work. A core team at the outset should consist of the project manager, chief architect, and project administrator. While these may be combined in the same individuals for small project, it is likely that for any project of significant size that at least the project manager and chief architect will be different people: both are pretty much full-time roles. In the initial stage of a project, the core team will be supplemented by additional designers, who take technical responsibility for subsystem design and implementation through the system's subsequent evolutions. Experts in the area the system is to address, either on secondment from the commissioning users or more closely connected with the development team, will assist in preparing requirements. Developers working on implementing the system undertake design of their subsystems, detailed class design and coding.

The users of a system may not wish to take responsibility for accepting every single evolutionary delivery, particularly if the evolutions are short. Part of the planning process involves deciding which of the evolutions are to be released to external users: other deliveries must be made to an internal user with responsibility for validating the delivered functionality against the scenarios. This role is readily filled during the implementation cycles by a domain expert who has contributed to formulating system requirements and the initial model (traditionally an 'analyst' role). It is important that these internal deliveries are taken just as seriously by the development team as external releases.

Making it work

ED requires a sustained commitment on the part of all concerned. It is easier to stay focused on a project if deadlines are frequent and deliveries can be achieved. The most important principles are clear: ED is not experimental development, and ED means delivery. We've also seen that people in central roles need to be capable and motivated.

In addition, a development infrastructure needs to be in place which will allow evolutionary cycles to proceed without hold-up. The process must make it easy to capture and validate increments to design and code. This requires tools and a common understanding of their use, and a commitment to timely review and approval. The tools must be integrated: it must not be necessary to enter information more than once when working on design, for example, so one essential is a repository-based diagramming tool which can output text and graphical descriptions of classes to a word-processor for formatting and publication. Source control software for the documentation and code developed during increments must be capable of working with the structure of domains defined in the architecture. It will also need to support regular baselining of the entire system. Just as important are policies for tool use. I would not expect to see all the features of a powerful case tool used on a project just because they are there: conversely it is possible to make do with imperfect tools as long as their limitations are known and respected.

The nature of ED means that a project may never be finished in the traditional sense, although by delivering production-quality components from the start there is a sense in which each evolution is indeed completely finished. So there needs to be a notion of enough design for the task in hand, which is the delivery of the next chunk of functionality. All this must be supported by a lightweight design process, in which design and code review is both constant and non-intrusive. A week spent by a developer waiting for a design review is a week wasted. We can distinguish between the products of the initial project phase (the architecture document and scenarios) and the domain designs produced in the evolutionary cycles. The former, as the documents which establish and define the project, should be thoroughly and widely reviewed by developers and users, preferably using formal techniques such as inspection. The latter, on the contrary, are 'work in progress': they are most effectively published and reviewed by email. These domain designs need rapid reaction from a minimal team of reviewers consisting of at least the chief architect and clients of the services provided by the domain in question. It must be possible to identify and review only what has changed in an evolution. Formal review meetings should be available if requested, but are generally necessary only in cases of disagreement between designer and reviewer. In all cases the architect has final authority.

ED - the payoff

Managing software projects is managing risk. ED provides a means to control risk by providing feedback at an early stage on a testable system delivered in the first evolutions. By implementing a system around chunks of usable functionality, we ensure that there is always a system which is at least demonstrable, and after the first few cycles genuinely useful. Flexibility in the planning of the contents of evolutions makes it possible to use the experience of the early cycles to better estimate the effort required to produce later deliveries, and in any case most of us are happier estimating for shorter than for longer periods. This becomes an important consideration in an organisation without previous experience of object-oriented development or with no project metrics.

There are less tangible but equally important effects of ED. By repeatedly meeting the frequent achievable dates for delivery, the development team gets used to meeting deadlines. A culture is established in which all involved expect the project as a whole to succeed. There is no gruelling final integration and fixing stage: this effort is spread throughout the implementation.

Project standards and procedures get established and understood early on. All the arguments about content, format and style in design documents are disposed of in the first rounds of reviews, as are debates about coding layout and style. Each cycle adds to the team's experience of the process, so in later cycles there is less dissipation of effort. Everyone knows what happens, and when.

Most importantly, the deliverables of the first few evolutions exercise the system architecture. In many cases, these can be used to subject that architecture to volume and stress testing. Any weaknesses and omissions are identified early on and can be rectified before they become expensive to correct. By building to an architecture, as cycles progress the system becomes more coherent rather than less. The phenomenon of software critical mass, where integrating new functionality causes errors to manifest themselves faster than they can be fixed, does not arise. Because objects localise the effects of change, it is feasible to build a system by programming changes to classes. The encapsulation provided by object-oriented languages makes it possible to hide an incomplete or substitute implementation behind a fixed interface. This enables us to add functionality piece-by-piece as required to fulfil the requirements of individual scenarios without serious effects on the rest of the system.

Working on a project which is making a success of object technology by doing ED and doing it right is a rewarding experience. A palpable sense of common purpose is generated in producing a real, working system evolution at intervals of a month or less. The short cycles bring home the relevance of design: on a system of any significant complexity this is the only effective way of ensuring consistency with the architectural vision which drives the implementation. In turn, this requires effective tool support and a streamlined process in which the emphasis is on delivery.

ED is a powerful idea in its own right, but its conjunction with object-orientation generates a powerful synergy. The potential of this combination to revitalise systems development stands in sharp contrast to the cumbersome development methodologies which have arisen in the last quarter century. Dinosaurs, stand aside: go find a lost continent in which to finish the job of becoming extinct. The process models we need to deliver the systems of the future will be small, fast, adaptable and warm-blooded: the age of OO-ED has arrived.