The InfoQ article included a really important question: “But how does it all tie up in the real world?” In other words, what is this ‘backbone’? How does it work? And how do we define it, build it, maintain it, use and reuse it to support enterprise agility? So from my admittedly architecture-oriented side of the fence, here are some suggestions towards a practical answer.

Step 1: It’s all about the enterprise. In effect, the backbone represents the core of the organisation’s relationship with the enterprise. Agility only makes sense if it directly supports the enterprise, and that relationship with the extended-enterprise. So to make any sense of this, we need to identify what that extended-enterprise is – in other words the deep-’Why’ for the organisation and its relationship with others, as expressed in its business-model, system-interfaces and so on. There are quite a few ways to do this: I typically use a workshop based on the sequence Vision, Role, Mission, Goal (there’s more detail on this in various of my books, such as Doing EA and Mapping the Enterprise), but there are plenty of other options.

What we need to end up with at the end of this step is some kind of vision-descriptor (about the extended-enterprise, not solely the organisation!), and an overview of the various players and key assets, activities and resources in use throughout that extended-enterprise. That becomes our starting-point for the next step.

Step 2: What part do we play in the enterprise? For what assets, activities and resources do others come to us and our organisations, or which we need from others? In the IT part of the organisation, this should tell us the types of information of which we need to keep track. We need to take a note at this point to try to distinguish between information whose structure remains stable and is used by many different areas of the organisation and its partners – because those items are likely candidates for the ‘backbone’.

Step 3: What enterprise items are unique to this organisation, and/or to its immediate partners or competitors? For example, the national mail-provider (Australia Post, Royal Mail, USPS etc) will usually have the responsibility ‘on behalf of the nation’ to maintain an up-to-date list of addresses, locations, post-codes and even probable addressees; Google maintains huge physical data-storage capacity and unique search-algorithms; Amazon builds and maintains a vast index and repository of opinions about products. Any such ‘unique items’ are almost certain candidates for the ‘backbone’. Note that these items can be much more than data alone: for example, the armed forces in principle represent unique capabilities for the nation, not just weaponry, but communications, logistics, field-hospitals and much more besides. We may also be able to identify some of these items by reviewing business-processes and other activities: for example, a national mail-provider will also maintain street post-boxes, which represent a ‘unique enterprise asset’.

In some ways what we’re looking for here are similar to ‘core competencies‘ and the like – yet it’s also sufficiently different that we need to take care not to get too sidetracked down that route. For example, equivalents for some data-items, such as industry-wide product- or service-identifiers, may also be held by competitors, by suppliers, or even by customers. The point here is that they’re unique within the shared-enterprise – which again makes them key candidates for inclusion in the ‘backbone’.

Step 4: What enterprise items are essential to our work for and contribution to this enterprise, and will need to be shared across our organisation? For example, if we are a commercial organisation, we’re very likely to have customer-records and service- or inventory records; almost every organisation will have HR records, organisational-structure records, account-codes, and so on. The point is that these are items that will need to be shared by everyone, in a consistent way – because if it’s not described and handled consistently, it’s almost certain to cause problems. If they’re essential to the business of the organisation, and shared across the organisation, they need to be in the ‘backbone’.

Step 5: Using all of the items identified in the previous steps, define the ‘backbone’. We would typically do this through the tools and techniques of the various domain-architectures: data-architecture, applications-architecture, process-architecture, organisational-architecture, brand-architecture and so on. (Notice, by the way, that not everything needs to be in the backbone! We don’t try to record everything in ‘excruciating detail’, in classic Zachman style – instead, this applies only to the items that really are core to the organisation and enterprise as a whole.) We may well need to do a ‘clean-up’ at this point to improve consistency and reduce duplication, such as described in TOGAF and the like. Once any item is included in the backbone, it should be placed under strict governance and change-control, with an explicit item-owner. Each backbone item should also be included in an explicit glossary and thesaurus, which itself should be subject to strict governance and change-control. The scope of governance depends on the context: in some cases it may be constrained to a single business-function or business-discipline, but in others the scope may need to be organisation-wide, or even broader.

Step 6: Define the interfaces to the ‘backbone’. This is where service-oriented architectures come into their own – IT-SOA, service-based process interfaces, and so on – with explicit service-catalogues and the like. Anyone who uses the information or other items (process-functions, assets etc) will need clear rules and, in many cases, development-support, on how to use the item, and the conditions under which it may be changed. There’s nothing particularly new in this: it’s fairly straightforward service-architectures, except that it should not necessarily be constrained only to information or IT.

Step 7: Define the spectrum of governance from waterfall-control to free-form agile, and the conditions that apply to change-projects and experiments at each point along that spectrum. In other words, define a flexible form of governance across the change-space, in which, for example, ‘Shadow IT’ would be explicitly encouraged as long as it’s outside of clearly-defined bounds. Provide support to enable agile projects to identify non-negotiable constraints such as regulation or mandatory standards.

Step 8: Define governance needed to migrate new items into the backbone. This is an important part, because the core will slowly change – some things will be dropped, but new processes and information-stores and the like that have proven both useful and reusable out in the agile space will be valuable in the core. As they move more towards the backbone – i.e. become shared by more people, and those crosslinks become more business-critical – they would also move ‘upward’ in terms of strictness of governance, from free-form agile towards waterfall-control.

The end-point of all of this is that we end up with a layered structure: a core that has a set of defined content with defined interfaces that can be be accessed anywhere required, and all tightly controlled; then a mid-layer in which interfaces tend to be shared across a more limited range; and finally agile hacks and mashups that may well be used only for one business-unit or even for one short- to medium-term purpose, but which still accesses the backbone for anything that is and needs to be common across the whole organisation. Some people might see this as somewhat hierarchical, but in fact it isn’t: mashups can connect to other mashups as required, perhaps bypassing the backbone completely. The point is that the backbone interfaces are guaranteed not to break (or, if they do need to change, then anyone who might be affected by the change would know about it in advance); the mid-range interfaces are not guaranteed; whilst the agile-layer interfaces – if they exist at all – can be all but guaranteed that they will break at some point. And it doesn’t matter that they might break – that’s the whole point. In other words, whilst the backbone must usually be as ‘fail-safe’ as we can make it, and the mid-layer must ‘fail gracefully’, the agile layer is intentionally designed to be ‘safe-fail’ – we often want it to fail, so as to learn from the ‘failure’, and adapt and retry accordingly.

This also resolves some of the classic fights between ‘the IT department’ – the conventional big-IT systems – and the ‘shadow-IT’ of little Excel quick-and-dirty kludges and local databases and HTML5/JSON/NoSQL hacks. In this scheme, each has their explicit place, with their own governance, their own role: each needs the other, and hence needs to respect the other, too. And it also clarifies the role and place of upcoming technologies: cloud, for example, has a really strong place in the agile context – but is often likely to be a very bad idea for the backbone.

So that’s my understanding of all of this: a way to unify architecture and agile, linking process-management, asset-management, information-management and everything else via a systematic enterprise-scope architecture, to support agility wherever it’s needed, and in whatever form is needed.

Something to play with, anyway: comments/feedback/suggestions, anyone?

No matter what it is, it seems – whether in IT-architectures, business-models, the organisation as a whole – agility of any kind is promoted as ‘the answer’. With the increasing hype around cloud-based computing, there’s been plenty of talk amongst the less-realistic IT-centric ‘enterprise’-architects that EA itself is redundant. And perhaps that large organisations don’t even need an IT department any more.

Which, to be blunt, is just plain stupid.

It’s not an ‘either/or’. It’s a ‘both/and’. Sure, over-rigid structures can be a real problem. But agility often needs a backbone to give it some direction – something to push against so it that can do more than merely flop about at random. As usual, it’s a question of balance – getting the right balance between the solid bone and the agile muscle.

Which is where enterprise-architecture comes into the picture, to help create a proper sense of the whole as whole.

To extend that simple metaphor, yes, it’s true that there are a few creatures, or parts of creatures, that consist only of agile muscle, pushing against itself:  the octopus, the cuttlefish, the elephant’s trunk. But most muscle needs something to provide an anchor around which or against which to push and pull. And if we want both agility and speed – as so many companies do – then we’re even more likely to need a backbone: think of the cat, the cheetah, the snake. If you want high efficiency, you’re going to need the right kind of balance between rigid and flexible: think of the kangaroo, that uses less energy the faster it goes. If you want agility in the air – a swallow, a falcon – you’re going to need another kind of balance, another subtle trade-off between rigidity, flexibility, lightness and strength. Many other examples there, of course.

And for large organisations, the much-hated ‘IT Department’ provides a key part of that backbone. You may not like it; you may not like its seemingly slow, lumbering rigidity, its need to get everything right, everything certain. But without it, that organisation is going nowhere. Even if everything in IT could somehow shift to the cloud and to all employee-provided technology – which would involve some massive shifts in itself, for HR and pay and expenses and tax-issues and the rest – there’ll still need to be someone who sets up and maintains the wi-fi networks, the security-policies, the software licenses and all the other myriad of ‘keeping the lights on’ issues that are so often forgotten by the cloud crew. (Yes, folks, even with everything-cloud there’ll still be some software-licenses to worry about: not everything can go onto the cloud, ‘cos you still have to connect to the cloud somehow…)

A real enterprise-architecture would show us that the backbone consists of a variety of different things that can link together in different ways to create different kinds of flexibility:

• the vision and values of the extended-enterprise, to which the organisation aligns itself – its core ‘Why’
• the core ‘things’, information,  services, events and places that are most relevant to that enterprise – its core ‘What’, ‘How’, ‘When’ and ‘Where’
• the structures of relationships implemented through the organisation and its partnerships with others – its core ‘Who’

That backbone gives us something around which we can build agility. We can try all kinds of different business-models – as long as they align to that core ‘Why’. We can work on many different kinds of ‘things’, physical, virtual or otherwise – as long they link back to the core ‘things’ of the extended-enterprise. We can work with all kinds of information – but we must be able to link it back to the core-information that defines the scope of the enterprise.

Which, once we think about that for more than a few minutes, makes it plain that no business larger than the smallest start-up should ever even consider storing all of its business-critical data on someone else’s cloud. Not without some really solid questions on escrow, reverse-backup, long-term migration, jurisdictions, business-continuity, disaster-recovery and the rest, at any rate.

And no-one - not even the smallest start-up – should ever consider outsourcing any part of its core-strategy to anyone else. Ever. Whether to the cloud, to some mob of external consultants, to some government department, or to anything else. Outsourcing your strategy is a really quick way to commit commercial suicide…

Agility takes place out at the edge: things happen fast there. But in so many, many cases they can only happen fast out there because the core takes care to move slowly, cautiously, providing the solid, certain backbone for the agile edge to push against. And as in living bodies, getting the right balance between them can be a literal make-or-break. A point that it’s probably wise not to forget?

Update, twenty minutes later:

Typical. I’ve been working on the ideas behind this post for months, and been brewing how to describe it for at least a week. And literally five minutes after I post it, along comes a really nice summary of some other key issues around agility versus ‘the backbone’: Mark Ferraro’s ’10 Dynamics of Effective Agile Organizations‘. Well, at least the link is here now, anyway: hope it’s useful?

My focus is on extending TOGAF to include SOA.

My approach was to add SOA as a cross-cutting architecture (cutting across BA,ISA,TA & also the governance of TOGAF (since SOA also needs SOA governance).

Roland [Ettema] has also suggested a nice approach above of having SOA in place of IA and making IA a horizontal.

How do you guys compare these 2 approaches and is there any other way around for customizing TOGAF for SOA ?

As many people have discovered the hard way, trying to use the standard TOGAF methodology for SOA seems to work well at first, but will suddenly hit intractable problems that just don’t make sense if we come from an IT-centric perspective. (In fact if we come from that perspective alone it’s almost impossible to see why they don’t make sense – which is why SOA so often seems so damn’ hard…) The actual reason is three-fold:

1. an iterative approach – e.g. an Agile methodology – suits SOA developments best, but TOGAF’s methodology (despite its claims to be iterative) is still essentially a ‘big bang’ approach to architecture development
2. the TOGAF methodology purports to describe the whole of ‘enterprise architecture’, but in fact describes a very narrow IT-centric subset – too narrow even for an IT-centric SOA, let alone a full ‘service-oriented enterprise’
3. the TOGAF framework still draws heavily on Zachman, which itself draws on technologies that are at least twenty years old – long before the days of interactive user-interfaces and layered servers, let alone current SOA web-services and service-buses

To resolve items #1 and #2, we need to rework the first half the TOGAF methodology – Phases A to D – with a few lesser amendments in the Preliminary Phase and Phases E to H:

• for item #1 – agile – we split the existing Phase A in half, moving the ‘architecture vision’, documentation, framework skeleton and overall authorisation into the Preliminary Phase, with Phase A amended to focus only on the needs of the specific iteration

• for item #2 – methodology – we expand the scope of TOGAF’s layering of ‘business architecture’, ‘information systems architecture’ and ‘technical architecture’:
  • ‘business architecture’ describes the overlighting business concerns – i.e. Zachman’s ‘contextual’ and ‘conceptual’ layers, with some touch into the ‘logical’ layer, and not the generalised ‘everything not-IT’ grab-bag as described in the present TOGAF 8.1 spec
  • ‘information-systems architecture’ becomes the common interface layer – i.e. Zachman’s ‘logical’ and ‘physical’ layers, but across every aspect of the enterprise, not just the IT [Roland Ettema's placing of SOA here is just one possible subset of this]
  • ‘technology architecture’ becomes the detail-layers across the whole enterprise, not just the IT – e.g. skills, logistics, vehicles, and much else besides, right down to the real-time operations layers

We could use these amended layers as the content and focus for Phases B to D respectively. The catch is that, once we’re tackling a true whole of enterprise scope, we end up with a requirement for an almost endless stream of stakeholder reviews within each phase. So for purely practical reasons I’d recommend using an alternate split for Phases B to D, as ‘As-Is’, ‘To-Be’ and ‘Gap Analysis’ respectively: with this split, the stakeholder reviews become the phase boundaries.

To resolve item #3, we need to rework Zachman itself – a major rework, since a bit of careful analysis shows not only that he’s missing a layer at the top, but an entire dimension of depth (which I’ve described as ‘segments’), and also the original content and descriptions for several of his columns are, frankly, a scrambled mess. (His latest revision in effect quadruples the complexity of his framework without addressing any of the core underlying problems – and he’s still stuck in the metaphor of ‘engineering the enterprise’, which does not and cannot work with the complexity and dynamics of real-world systems. But that’s another story for another time…)

But in that rework, what we don’t do is add more columns. The basic principle of What, How, Where, Who, When, Why is sound – the catch is that they don’t necessarily correlate directly with anything in the real world. Instead, we focus on Zachman’s valid principle that ‘architecture is primitives; solutions come from composites’. An interface, for example is a composite – it doesn’t sit in a single Zachman cell. (Depending on the context, it’s a composite of How and Who, plus probably When and What as well.) EA-toolset vendors have made this worse by trying to shoehorn all their models into single columns or single cells. If you add ‘interface’ as a column – which I’ve seen several people do – you end up with a Zachman-like muddled mixture of primitives and composites, which is not a good idea. It may look fine at first, and may seem to make solution design easier; but it really does cause utter chaos later down the track, when the technology changes on you, or when you try to model alternate implementations such as for disaster recovery. In short, don’t shoehorn: primitives are primitives, composites are composites – don’t mix them up.

So for SOA, what we need first is an exercise in meta-models and meta-architecture: we start from primitives of the Zachman frame – single cells – and build a layered set of root-composites (meta-patterns) that describe our key entities. (Meta-architecture is a swine to get your head around, but if you want your solutions to be ‘future-proofed’ you can’t afford to skip over this stage.) Services are a good place to start: they’re composites of How (functions) and Who (capabilities), whilst the messages and trigger-events are composites of What and When. We then build cross-linking composites (functional patterns) that are closer to our implementation layer: a web-service, for example, links a service definition with a message definition with an interface definition. That gives us our logical layer, which we can still specify in a true implementation independent manner – which is what we need for, say, disaster recovery planning. Then we can describe entities (implementation patterns) for the the implementation specific physical layer. Then we can do our SOA designs – which may well traverse IT-based and non-IT-based processes. But because they’re all anchored all the way back to the root-primitives, we have a straightforward trail of derivation and dependency to guide redesign when (not ‘if‘…) we get hit by a change of technology, or context or whatever.

To see how this works, have a look at my presentation for the last TOGAF conference: ‘Enterprise architecture and the service oriented enterprise‘ (PDF, 850kb) – it summarises all of this, together with a broader view of ‘service oriented architecture’.

I’m also working on a book on this – Bridging the Silos: enterprise architecture for IT architects – which tackles this general problem of extending TOGAF and Zachman to whole of enterprise architecture. It’s taking a little longer than I’d hoped but there’s a PDF of a partial draft already available up on the Tetradian Books website, at tetradianbooks.com/2008/04/silos-ebook/ – any comments much appreciated!

Hope this helps, anyway.