Tom Graves / Tetradian

At least he was open about it, I guess. “Tell you what I’ll do”, he says to my colleague here in Guatemala, “I’ll find you a client, then I’ll sit in, learn everything you do, and then I’ll apply it in my own business. How does that sound to you?”

Uh, no. Not a good idea. Not just because it’s a really bad deal from our perspective, but much more that Reality Department really doesn’t work that way: there’s no short-cut to experience.

Yes, it all looks simple enough – in fact that’s the whole point. A lot of simple visual summaries, and surprisingly simple-seeming methods, too. Yet it only looks simple because we’ve been through a heck of lot of hard work to make it that way. Hard-won experience, won the hard way through years and years of practice in many, many different contexts, getting it ‘wrong’ time and time again, in many, many different ways in order to get it right.

The real trap is that these simple-seeming ideas and methods aren’t simple rules, prepackaged sense-making and decision-making that will always work in every context. These are simple principles, simple guidelines, the kind of easy-to-memorise information that helps decision-making in real-time, in circumstances that are subtly different every time. (See my SCAN posts for more on these distinctions.) If you try to use them as ‘rules’ for inherently-uncertain contexts, without understanding why those principles apply, or how they need to be tweaked every time to match each different context, you’re going to be in real trouble – along with everyone else around you. Not a good idea…

The same often applies even to things that really are ’rules’. Take that example of perhaps the greatest simplification ever made: e=mc². All the core information you need to build a nuclear power-station is right there in that equation: but there’s a heck of a long way – a heck of a lot of engineering-experience – to go from that one equation to building a nuclear-power station that actually works.

Same with everything else, really: simplification is essential, but can also be deceptive – especially when people mistake ‘simple’ for ‘easy’…

Which is also why I get a bit hot-under-the-collar about the current proliferation of ‘certification-schemes’ in enterprise-architecture and elsewhere. Some of them are genuinely valuable, but others – to be blunt – are little better than money-spinning scams, in terms of the actual value that they (don’t) deliver. And the crucial distinction revolves around the role and recognition of experience.

For example, the TOGAF Foundation and Archimate Foundation certifications have real value. They verify that the respective person has a credible command of the terminology and language – a requirement that matters a lot for communication across a dispersed and disparate team.

Likewise the ATAC certifications should have real value, because each explicitly tests practical experience in the respective area.

But unless they’ve changed it in the past year or so, the full TOGAF certification is delivered through the absurdly-inappropriate mechanism of a multiple-choice test. And to my mind, that’s not merely useless, it’s actually worse than useless, because it’s exactly how not to test for the kind of experience that that type of competence requires. (When I did the TOGAF 8 exam some years back, I almost failed because I answered several key questions correctly in terms of real-world experience, rather than the theory-based wrong-assumptions that the test thought were ‘right’.) The result of that kind of pseudo-test is a bevy of people who can wave a certificate around, but have no idea how to do the work in any real-world context.

A good training-course can make all the difference, and the better training-providers do take up some of the slack here. (I’ll wave a flag at this point for John Polgreen at Architecting The Enterprise, who’s been doing sterling work for years on adapting TOGAF for the US-government context.) Yet none of that competence carries through anywhere into the actual test: so unless we know each of the training-providers, we have no way to tell whether a candidate does actually know what they’re doing, or merely that they have a piece of paper to prove that they know just enough to get into real trouble, but not enough to get out of it again.

In effect, right now, the full TOGAF certification is of less real-world value than the Foundation certification – which is both bizarre and sadly pointless. And I’ll hasten to add that I’m using TOGAF here merely as one example amongst many: it may well be that most of the so-called ‘certifications’ in this field are even more meaningless than that. And the results can be seen everywhere in the trade…

In short, it’s a mess.

What we need to be testing for is genuine understanding of a context, and the ability to adapt for uniqueness. And that calls for much, much more than can be covered in a crude multiple-choice test delivered through a mindless machine. Sure, that kind of test is cheap, and relatively easy to administer: but it’s also all but meaningless for anything than foundation-level rote-knowledge. It really does take years of painful practice to develop the experience needed to do this work well: and if this trade is to gain the credibility that it needs, we need to stop pretending that we don’t need to test for that experience.

Time to re-think how we do this, and how we respect this, too: there’s no short-cut to experience.

Yikes! Talk about misinterpreted in a sound-bite…

(Before I go any further, a note to all in the TOGAF training/education community: from what you’ve read elsewhere, you may at present believe that I’ve been attacking you personally. As you’ll see below, this is not the case – so please accept my apologies for others’ interpretations of what I wrote. Do read on – and thank you.)

There are a few Tweets going round that suggests I’m attacking TOGAF (again), this time by suggesting that TOGAF training is worse than useless:

harsh deduction by @tetradian: TOGAF certification almost an indication that one is NOT capable of doing #EA

“… close to … a TOGAF certification is an indication that someone is not capable of doing enterprise architecture.” http://bit.ly/uZDZd

I’ll admit my original post summarising the London TOGAF conference last month does indeed include that latter quoted text. But it’s quoted way out of context: so please, do read the whole post before you jump to that conclusion! – because it isn’t what I’m saying at all.

First, it’s not my own ‘deduction’: it’s a near-verbatim report from a broader discussion at the TOGAF conference. From the certification perspective, four key themes came up from the conference, all from leading members of the TOGAF community:

• the reference-architectures (Part VI of the TOGAF spec: ‘Technical Reference Model’ and ‘Integrated Information Infrastructure Reference Model’) are way out of date, and at the least need a complete overhaul, if not dumped altogether [that was from the Open Group's lead Allen Brown, in one of the plenary sessions]
• “almost no-one” uses the ADM in the form described in the TOGAF specification [in my last post I said I thought that was one of the guys from Deloitte, but my notes indicate it was Mike Lambert from Architecting the Enterprise, one of the lead TOGAF training groups]
• enterprise architecture is much broader than IT, and must now encompass the whole of the enterprise [that theme came up at least a dozen times, in plenary sessions and elsewhere]
• enterprise architecture needs to be understood as a professional discipline, comparable to other professional disciplines such as medicine and building-architecture [again, many people, but particularly Len Fehskens, Open Group VP on Skills and Capabilities]

These are all points that, yes, I have personally pushed hard over the past few years: but you can see from the above that it’s not just me that’s saying it – it’s being echoed now right from the centre of the TOGAF community itself. (Just this once, I’m not ‘the Outsider’ here! <wrygrin>)

So, to the problem of certification. The key point is that certification alone is not an indication of professional competence. Back in my aero-engineering days, it was common knowledge that newly-graduated engineers were a potentially lethal danger to everyone in the place: they knew just enough to think that they knew what they were doing, with that arrogant certainty of the newly-qualified, but had no idea of how to work with the subtle complexities and constraints of the real world of engineering. For example, they would specify components that couldn’t actually be made, or assemblies that could be made but couldn’t physically be assembled. Even for the best, it usually took a year or two at least “to learn how to make my mathematics sufficiently imprecise to be usable”, as one of them put it. Crucially, there were a few who never learnt that lesson, and instead clung on to their certification as ‘proof’ that they were competent – which in practice more proved that they were not competent to be let loose on a real aircraft. Or, in this context, a real enterprise.

On its own – and again I’ll emphasise, on its own – an enterprise architecture certification does not and cannot indicate competence: it needs to be balanced by real-world practice. For which, again, crucially, this profession at present has no means to monitor or measure.

Next, look at what’s actually covered in the existing TOGAF certification: it’s primarily about the ‘standard’ ADM and the reference-models – which are no longer used in that form in practice. And – as also indicated in those themes from the conference – real enterprise architecture is much, much broader than IT: yet everything in the existing certification is centred on IT. So anyone who does slavishly follow the ‘standard’ will be almost guaranteed to create an architecture that might at first seem ‘efficient’, but will be so outdated, so IT-centric and so far off the real mark that it will at best be useless, and possibly much worse than that.

What the old TOGAF 8 certification exam did not cover was how to adapt the ADM to the enterprise, or how to create reference models and use them for compliance-monitoring and risk-management – which is what is actually most needed in those stages of architecture that the TOGAF spec aims to cover. And there’s no way that any of that kind of context-dependent knowledge could be assessed in a simplistic multiple-choice exam such as is still used for TOGAF certification. As I mentioned in the previous post, I nearly failed my TOGAF exam because in many parts of the test, none of the options shown on the screen actually matched what I knew from experience works in practice, and the nearest available guess turned out to be ‘wrong’ according to the specification in the book. Conversations at the conference made it clear that I was far from alone in that experience: in effect, anyone who presents a high score in their TOGAF certification may have the book-knowledge but know nothing about the practice, whereas many will score low because they are competent in practice. So as it stands, the TOGAF certification not only tells us nothing about professional competence, but can be actively misleading: a high score may well indicate that someone is not competent to do the work, whereas a low score indicates either high competence or complete failure, with no apparent means to distinguish between those two extremes.

All of which adds up to a serious problem.

It does not, however, mean that TOGAF training is wrong. Quite the opposite: many of the trainers I talked with at the conference made it clear that their training-courses emphasise the importance of adaptation of the ADM, development and use of reference-models, and all the other skills needed to assess and adapt to the enterprise context, and how to extend EA beyond the IT domain itself. To develop those professional skills, we’re likely to need more training, not less; and much of that training needs to be context-specific, too. The catch is that almost none of this material is in the current TOGAF specification, and none at all is assessed in the current TOGAF certification. So yes, whilst to my mind the TOGAF specification is still annoyingly limited and limiting, that’s not the real problem in this case. The point here is that, as it stands, the TOGAF certification is not only meaningless but actively misleading: and right now that is a real, genuine, active, in-your-face, fundamental problem for the profession.

This is a problem that’s being addressed: as I said in the previous post, Len Fehskens is specifically tasked with this on behalf of the Open Group, and others are tackling it in other ways with other groups. But we must first acknowledge that it is a real problem, and one that won’t go away simply by ignoring it, which is all that had been happening to date. So, yes, whilst it’s an uncomfortable fact to face, one of the key signs that the EA profession is maturing as a profession is that it is now willing to face up to such uncomfortable facts.

‘Bad’ news that’s good news all round, in fact.