Tom Graves / Tetradian

Just how much are organisations themselves ‘their own worst enemy’ for the enterprise?

Have been thinking about this one for quite a while, following up on some great conversations with Kevin Smith (of PEAF fame) and Nigel Green (of VPEC-T fame) about Kevin’s concept of Enterprise Debt – an expansion into the whole-enterprise scope of Ward Cunningham’s concept of Technical Debt:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation.

In classic IT-oriented ‘enterprise’-architecture, Enterprise Debt occurs with any ‘quick-fix’ IT solution, or any solution that breaches the architecture guidelines. A lot of ‘shadow-IT’ fits into this category, of course. But exactly as with a credit-card, for example, the ‘debt’ doesn’t matter much if it’s cleared quickly: short-term debt is rarely much of a problem or risk, and if managed well does help us to achieve short-term goals. Likewise for longer-term debt, such as a mortgage or a car-loan: as long as we know that the debt is there, and that we’ve allowed for it and managing it, it’s not a problem. Yes, it’s a risk, but it’s a managed risk, and one that allows us to reach towards our longer-term goals. So, to take the metaphor back to IT-architectures, a large-scale solution that breaches the architecture-principles may indeed be a risk – but it’s probably not too much of a risk if we’re aware of it, are managing it and do have specific plans in place to ‘write down the debt’.

For enterprise-architectures and any of the other non-IT-oriented architectures, we take the same principles, and adapt them to the respective scope to identify and address the overall Enterprise Debt and concomitant risks. It’s really no different to Technical Debt, other than in its scope: the tasks to identify and address it are certainly much the same.

So far, so good. But it’s the hidden debt – the debt that we don’t notice or don’t know about – that can be the real killer here. Because we don’t know or notice that it exists, it keeps on building up and building up in the background, until suddenly the metaphoric bill is presented, and we have little or no way to pay it off. A classic example was Y2K – otherwise known as the ‘short-term thinking bug’ which comes back to haunt us again and again. In that context, the use of two-digit dates was a design decision that was sensible enough given the technical constraints of the time, but whose short-term expediency was never redressed, and came back to bite us big-time… a good example of a kurtosis risk.

In enterprise-architectures and the like, there are many sources for hidden risk of this kind, often quietly building up quite terrifying amounts of Enterprise Debt. Some examples to watch for:

• small ‘knock it up over a weekend’ applications and databases becoming used in business-critical processes [no documentation, lack of links to enterprise standards]
• a database for a single team, developed using office software tools such as MS Access or cloud apps, becoming used across a wider spread [security concerns, database stability in a true multi-user context]
• business-processes continuing indefinitely without review [loss of 'fitness for purpose', high efficiency but low effectiveness]
• anything that depends on or assumes specific silo boundaries or organisational structures [because those boundaries or structures may change at any time]
• anything that depends on a single individual or small team [serious maintainability-risks, risk of inability to review design-decisions]

We can tackle some of this with proper governance that provides an appropriate balance between the competing needs for agility and stability, such as in a ‘backbone’ governance-model. We also need a long-term view – often reaching out into decades, for many types of business-critical information and processes – and strategies and tactics for systematic ‘entropy-reduction‘ in the overall architecture.

To me, this is one of the most important tasks of an enterprise-architecture unit: addressing the issues of the past so as to support the needs of the future.

Yet there’s a really nasty booby-trap here, a huge source of hidden Enterprise Debt, of which we definitely need to be aware at every level, especially at an enterprise-wide scope. It’s a variant of Parkinson’s Law, often nicknamed the Shirky Principle after Clay Shirky’s aphorism: “Institutions will try to preserve the problem to which they are the solution“. This is a classic wicked-problem whose symptoms include:

• people whose incomes and identities depend on ‘the problem’ not being resolved [very common in social-work contexts]
• ‘hero’-managers and other ‘fire-fighters’ whose prestige depends on having metaphoric fires to fight [hence have vested-interest against preventive strategies or tactics]

To say this is ‘political’ is an understatement… And unfortunately, by its business-anarchist nature, enterprise-architecture tends to highlight the inherent absurdities of these kinds of situations – which means that, when accidentally exposing this, architects themselves often tend to be attacked, often with extreme ‘irrational’ anger, in a classic shoot-the-messenger ‘defence’.

This does mean, though, that many organisations – especially in the government or non-profit sectors – are actually ‘their own worst enemy’. The people, the processes and even the structures will often in effect act to perpetuate the problem they purport to resolve. Tricky… especially for the enterprise-architects who work in those organisations…

There are some practical options, though, for enterprise-architects unfortunate enough to be faced with that kind of ‘mess’. These include:

• be careful never to describe any of this as anyone’s ‘fault’ – it’s a natural tendency at the systemic level
• use narrative or story-based methods such as SEMPER or Causal Layered Analysis to ‘surface’ the structural issues
• always reframe motivations and requirements in terms of being ‘for‘ something – never ‘against’ something

As enterprise-architects, we have to deal with the painful realities of Enterprise Debt as best we can. But first we have to learn to become aware of it: and warnings such as the Shirky Principle can be a lot of help in this.

Hope this helps, anyway – comments/suggestions, as usual?

I’ve just finished editing yet another book on enterprise-architecture, and set up its production via my Tetradian Books publishing setup. But you won’t see it on the site, and in fact it may never be published as such in its present form – though you will see it coming out quite soon under someone else’s imprint.

As you can imagine, there’s a story behind this. (A happy story, I hasten to add!) But first, here’s the book-cover that you probably won’t see much elsewhere:

The Pragmatic Enterprise Architecture Framework, or PEAF for short, is the brainchild of my neighbour Kevin Smith. (‘Neighbour’ here is a relative term – he actually lives about fifteen miles away from where I’ve been staying in England for the past few years. But given that the EA community is pretty small, and so scattered around the globe, fifteen miles is pretty close. ) We have irregular ‘HEC meetups’ – where ‘HEC’, of course, is the ubiquitous English pub-lunch of ‘Ham Egg & Chips’.

For me these meetings have been great, because we tackle such different aspects of the enterprise-architecture domain. I work mostly on ‘big-picture’ theory, these days with a strong emphasis on the ‘people-stuff’ of the enterprise, whilst Kevin concentrates on the day-to-day practicalities of doing enterprise-architecture. And that’s the whole point of his “Pragmatic EA’ – it’s all about the pragmatics. It’s all about “cutting EA to the bone”, as Kevin puts it, so that people can get started and get moving on EA, within the very real constraints and tortuous politics of large organisations. We’ve talked a lot about this EA ‘world’, using our mutual misunderstandings of each other’s approach to get a better sense of the EA whole.

That’s the backstory. Onto the story itself…

Two weeks ago as I write this, we were having another HEC-meetup. We talked about the upcoming AE Rio 2011 conference in Rio de Janeiro, where Kevin had been booked as one of the lead presenters, and for which I’d harboured some ambitions to go. (That is now happening for me, courtesy of much help from AOGEA-Brazil‘s Roberto Severo and Alan Rodrigues, but that’s another story for another time.) Somewhen during that discussion, I’d talked about how I use my books as a key part of my business-marketing: as many people will know, I often turn up at EA conferences with a great stack of books to give away. Which, of course, led to the idea that Kevin needed a book too. Preferably in time for AE Rio. Which, at that point, was effectively just over three weeks away.

Well, you can pretty much guess what happened next.

We’d previously talked about setting up a publishing-operation for Kevin’s own business, but it hadn’t really got very far. I’d looked at publishing the existing PEAF materials, but I’m not comfortable about publishing others’ work, with all the nightmare complications of record-keeping and royalties that any ‘real’ publisher would have to deal with. And the PEAF material includes a lot of large-format graphics, almost all of it in colour – which would be insanely expensive to produce in print as-is. All of that material is readable online for free, anyway. So the only practicable option for a book that would also add some real value would be a simplified Introduction, summarising all the PEAF material in an immediately-usable way.

Yet although it ought to be published under Kevin’s own imprint, setting that up would take more time than we had – hence, for now, the only viable option would be to use my Tetradian Books imprint, my production-templates, and so on. It would need to be at least eighty pages, and preferably more than a hundred, in order for the book to have a conventional paperback-style spine. And given the known production lead-times, we would have less than two weeks at best to get it to press.

From start to finish, I did it in just twelve days. Yikes…

If POD-printers Lightning Source live up to their usual excellent performance, we should have the first hundred copies late next week, in time to take to Rio. Once we get back, we’ll go through all the usual rigmarole – ISBNs, trade-accounts, British Library registration and the rest – to set up Kevin’s new imprint to publish PEAF and the other upcoming ‘Pragmatic’ materials. And at that point I’ll be able to hand over all the master-files and set everything up for Kevin to (re-)publish it himself. And once that’s done, this first version will quietly disappear from view – as it should, because although, yes, I did edit it, and write all of the summaries from scratch, too, it really isn’t mine to publish. The real work on PEAF is Kevin’s – and that’s where it belongs.

It’s been an interesting challenge, not just the sheer physical effort of getting everything done in time, but also getting my head around what is still a somewhat alien style of thinking about a very different part of the business ‘world’ that I work in. It’s been a very worthwhile challenge, too. Yet not, I think, something I’d want to repeat for anyone else any time soon?

A couple of weeks back, Gabriel Morgan of Microsoft’s internal Enterprise Strategic Planning unit posted an article on what he described as a ‘breakthrough’ in enterprise-architecture, “A Breakthrough: Maturing EA to be a Catalyst to Transform the Company“:

It’s time to rethink enterprise architecture people. Well, at least here in Microsoft IT’s Enterprise Architecture Team it is.

… For the past year or so, I’ve led a crack team of experts focused on aligning IT and the Business, and from this journey I wanted to share with you my current thinking. My team is called Enterprise Strategic Planning and it is a service team dedicated to enterprise-wide strategy and planning. Our initial goal is to become critical to the planning process with the intention of providing data to qualify an optimal set of IT Programs to invest in. … In this article I wanted to share with you something that has occurred to us during our journey and describe some changes we are making that I think is ground-breaking in the Enterprise Architecture domain.

Assuming a primary goal of EA is to align IT to the Business, the problem is that most, if not all, EA Frameworks are not equipped to actually deliver on this goal. They are limited to drawing associations from IT things … to Business things … . Some of the more mature EA teams have partnered with their finance department to apply financial modeling … to these associations to help describe business value in monetary terms and possibly start a chargeback model. These are all great accomplishments but at best they only capture how IT ‘relates’ to the Business. That is, these EA Frameworks and methods are more about IT transparency, not alignment.

I would have to admit that my first response to this would best be described as ‘unprintable’…  (The polite version of ‘unprintable’ would look at certain key phrases in the above, such as “It’s time to rethink enterprise architecture, people”, “a crack team of experts” and “Assuming a primary goal of EA is to align IT to the Business”, and thence to make extremely disparaging remarks about Microsoft, about apparent arrogance, about a supposedly innovative company being literally years behind the leading edge of enterprise-architecture, about breakthroughs that aren’t, and about the vapidity and shallowness of IT-centric assumptions… Oh well…)

My second response, after I’d had a chance to calm down a bit, was perhaps a bit more charitable, if still more than a little sarcastic: something more along the lines of “Welcome to the club, glad to see you’ve woken up at last, any chance we can actually talk about enterprise architecture?” – because to most of us who have been working at that leading edge of EA-development, this supposed ‘breakthrough’ is very old news indeed. It’s actually the understanding that existed decades ago, before IT-architects came in and hijacked the entire industry; several of my IT-oriented clients had each reached that same point independently half a decade or more ago; and even the Open Group, after we’d nagged and bullied and cajoled them for so long, finally ‘got it’ late last year, with “evolving EA from IT to business” now becoming an explicit key theme of current Open Group (TOGAF) conferences. So in terms of the overall EA industry, there’s not a single thing that’s actually new in that Microsoft ‘breakthrough’: and hearing someone call it a ‘breakthrough’ is frustrating in the extreme.

But that second response still misses the point: this actually is a breakthrough – for Microsoft. Which, because of who Microsoft are, means that it’s also a real breakthrough in another sense as well.

Yes, it’s frustrating to note that, from what appears in the article, Morgan and his group seem not to know that much about any ‘prior art’ – not even from other Microsoft EAs such as Nick Malik, who writes a very good ‘Inside Architecture‘ column at the same MSDN weblog-host, and is even listed in the links on Morgan’s weblog. Yet there is real change here: important change. Take a look, for example, at the business-architecture references that Morgan lists later in the article:

• Porter Five Forces
• Balanced Scorecard
• Business Model Canvas
• Geoffrey Moore ‘Core vs Context’

There’s no IT directly involved in any of those models (unlike, say, Ross, Weill & Robertson’s much-lauded ‘Enterprise Architecture as Strategy‘, which still takes a strongly IT-centric view of business-strategy). For someone like Microsoft, whose whole business-focus is and revolves around IT, that absence of IT-centrism is huge… definitely a real breakthrough.

I also need to remember that my own role in enterprise-architecture is very different from theirs. Morgan and his team are practitioners, dealing with the day-to-day realities of real-world architecture in a very large organisation. I’m a practitioner, too, yes, but my own work these days is mostly about setting-up architecture practices, troubleshooting, and doing practice-refresh for existing architecture groups – it’s consultancy, not mainstream production-level architecture. So although I’m a practitioner, my practice is mostly about theory, futures, creating new tools and techniques: which means that if my work isn’t well ahead of the mainstream, I wouldn’t be doing my job properly. Yet that view gives me a rather jaundiced perspective of the industry: too easy to forget that it does take years – several years at least – for the ideas and themes and concepts that we’re working on now to filter down into everyday EA practice. In short, too easy for me to become arrogant about what I see as other people’s ‘arrogance’… Oops…

Coincidentally, Gartner published their current ‘EA Hype Cycle’ this week, described in a press-release and a one-page graphic. They state that there are now two distinct generations of EA: the ‘traditional’ IT-centric one, which has now reached what Gartner term ‘the Plateau of Productivity’; and an upcoming “more business-focused” version that will help to prevent EA from falling permanently into ‘the Trough of Disillusionment’.

To me, though, these two ‘generations’ respectively represent maturity-levels 2 (“clean up the mess”) and 3 (“top-down strategy”), out of five distinct maturity-levels, as described in my book ‘Doing Enterprise Architecture‘. There are at least two more ‘generations’ to go before we reach a fully usable architecture for the enterprise; and, worryingly, far too few architecture-teams seem to have properly implemented maturity-level 1 – “what business are we in?” – which in the longer term places the entire architecture at risk. (It’s not adequately addressed in either TOGAF or FEAF: TOGAF 9 does sort-of include part of it in a kind of half-baked form in the muddled mess that it describes as ‘Business Architecture’, but that’s about it. To me, the only major framework that does cover it properly is Kevin Smith’s Pragmatic EA, which is still not as well-known as it deserves to be.) So there’s a long way still to go – and a lot of repair-and-refresh to do, too, to clean up the problems caused by the IT-centrism of the existing ‘enterprise’-architecture frameworks.

But I’m wondering just how much this article of Morgan’s should change the view that Gartner shows us. Gartner shows ‘Business-Driven Architecture’ as having only just reached ‘the Peak of Inflated Expectations’: yet the TOGAF conferences, and now Morgan’s article, seem to show it much further on, more like the start of ‘the Slope of Enlightenment’. Fact is that Microsoft is just about as mainstream as they get: so if Microsoft’s EA has now turned beyond IT-centrism to a more explicit whole-of-business focus, what it really tells us is that business-driven architecture has just gone mainstream.

It’s still not a ‘real’ enterprise-architecture as I would understand the term: but it’s a heck of a lot better than than the old IT-centrism. That is a real breakthrough – and very good news indeed.

Watch This Space, at last?