IT Certification and Training

November 14, 2006

Cognitive Dissonance: The art of overcoming skepticism to improve certification

Filed under: background, certification — Jim Henderson @ 21:11

The Random House Unabridged Dictionary defines Cognitive Dissonance as:

Anxiety that results from simultaneously holding contradictory or otherwise incompatible attitudes, beliefs, or the like, as when one likes a person but disapproves strongly of one of his or her habits.

In my last posting, I talk about being skeptical of certification and describe the reasons why I am. So how do I resolve the inherent conflict between being skeptical of technical certification and the fact that a large part of my job is to help develop and manage certification programs?
The answer to this question may not be as surprising as you might think.

Throughout my career, I’ve never underestimated the benefits of being informed on a subject. Indeed, when I give technical interviews, interviewees are often surprised by the questions I ask. No, I don’t ask questions like “if you could be any superhero, which one and why” or nonsense like that. I do a technical interview asking about their skills and knowledge. As a subject matter expert on eDirectory and directory technologies myself, I can ask questions that I am certain the candidate isn’t going to be able to answer.

For example, I once asked a job candidate how he’d resolve an intermittant -649 error reported by NWAdmin when modifying a user object.

(As a side note, this question was a question I was actually asked after my interview at the same company – not because they wanted to stump me, but because it was a problem they actually were having and they thought I’d be able to help resolve the problem. I didn’t know, said so, but offered to research the issue and they said “No problem, we just wondered if you knew of a quick fix”. A week later they extended an offer to me.)

My motiviation in asking this question (pretty much as written above) was not to elicit a technical answer from the candidate, but rather to put him in the position of not knowing the answer. I knew, based on my experience with that specific problem and the ensuing 2-year long support call (it wasn’t critical, just annoying) that the odds of him being able to answer the question were very small. I wanted to see how he reacted to a situation where he didn’t know the answer to see what he’d do. Predictably, he responded “I don’t know”, but with a little prodding that “I don’t know” was an acceptable answer, he outlined where he’d go to get help.

This little scenario is similar to a written test. You are asked a question and given an opportunity to provide a freeform answer. Your “grade” is based on how well you answer.

If I could have, I would have loved to have walked him into the lab and presented him with that exact scenario. By that time, we knew the cause and I could’ve easily reproduced it for him. Give him 10-15 minutes to try to troubleshoot the problem, and observe. How much better an evaluation can you get beyond actually having to perform the task with all the resources avaialble that would be available in the actual job? Discuss the scenario afterwards and see if the troubleshooting style is consistent with someone who has the experience we were looking for. Brilliant.

(Footnote: The candidate did get the job, in part due to my recommendation, and to my knowledge is still employed at the company in question)

How does this answer the question about the inconsistency between my working on certification programs and being skeptical about them? Simple: Practical evaluation of skills is what’s necessary for a certification to have value to the holder.

It also cuts down on the feasibility of “braindump” type “cheating” tools. Rather than answer the question “how do you create a home directory at the time of user creation using useradd?”, actually require the candidate to perform the task. The end result is what’s important, the “how they got there” isn’t so important because different people work in different ways.

I (and possibly others in the industry) refer to this as “competency-based skills assessment”, and see that as being pivotal in the development of a valued certification program. The idea is that in order to achieve certification, you have to actually perform tasks, rather than describe how to perform tasks.

Why hello, Novell Practicum Exam.

If I look at the marketingspeak about the practicum, Novell pioneered the use of practical examinations with the Certified Directory Engineer certification. That certification required you pass a couple of written exams (Directory Technolgies and NDS Troubleshooting as I recall) and then sit down in a real envrionment and solve real problems in a scenario-based environment. This concept is currently used for the Novell CLP and Novell CLE certifications (both version 9 and version 10).

Practical testing takes the “Paper” out of “Certification”.

That’s the first part of the answer to the question.

The second part to the answer has to do with reversing the trend of commoditization of certification. Commoditization happens when certification exams become too easy to pass. The overt goal is for the vendor to aim for hundreds of thousands or even millions of individuals holding the certification for the purposes of product marketing (and consequently higher market penetration) at the expense of the individual candidate’s sense of self-worth. If you’re only one in a million certification candidates, the certification holds little real value. If you’re one of 900 who hold the certification, there’s a sense of accomplishment for the candidate.

When you commotiditze certification, you commoditize certification holders. When you have a large pool of holders, the average skill of those workers drops and the infrastructure suffers. Think of it like acquiring a drivers’ license. There are millions of people who are able to drive because the tests are (relatively) simple to pass. In Utah, the “written” test requires that you press a button next to a picture in order to answer the question posed (which is usually done in video form as I recall). There is a practical test, but practical testing alone doesn’t solve the problem of competency. If a driving “practical” test includes turning the key to turn on the engine, that doesn’t prove your ability to drive: The test needs to be valid for the skill presented, and the grading has to accurately reflect what is considered “competency” in demonstrating that skill. Set the bar too low, and you end up with the same problem. Now how many drivers think they’re “above average”? (The joke goes that 85% of drivers think they’re “above average”).

The same holds true for IT certification – the bar has to be set high enough. The only way to do that is to not focus on commoditization, but rather to focus on building competency in systems management. That doesn’t mean that commoditization can’t happen, but rather that commoditization shouldn’t be a goal of a certification program. If it’s the goal, the testing suffers, competency suffers, and you end up with a million certification candidates who can’t do the job competently or efficiently.

With this in mind, I see certification and training (at least at Novell) moving in that direction. It won’t happen overnight – no, scratch that. It can’t happen overnight. Change takes time, and making the right changes requires a lot of work and careful planning.

I moved into my current role in November of 2005 – and I can tell you that it has been a very busy year for me and the team I’m on. The results of what we’ve been working on will start to become visible in November of 2006, and I have confidence that the steps we’re taking are in the right direction. What’s more, I see that when the changes are finally announced, they will be seen to be in the right direction by the industry.

As the announcements come out, let me ask you to remember one thing: This is only the beginning.


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

%d bloggers like this: