Category Archives for How to

Assess ITSM with CAIR

Are you planning to assess and improve ITSM processes in 2018?

4 things you must do to maximise ROI in ITSM process assessment.

IT leaders assess and improve ITSM processes for a variety of reasons. All of them are valid, but it's worth thinking carefully about the primary motivation for an assessment before you start.

Why? Because when handled well, every assessment can and should be a springboard for:

  • Long-term process improvement
  • Efficiency and effectiveness gains
  • Less waste and rework
  • Higher staff morale and engagement
  • The foundation of a culture of continual improvement

For each reason to assess an organisation’s processes, there is a danger of closing down our thinking and vision so that we find only what we are looking for and miss some of the benefits listed above. As the saying goes, ‘for a person with a hammer, every problem looks like a nail’.

Some Reasons to Assess and Improve ITSM

  • Assess to benchmark (establish how well we do things today, in comparison to internal or external measures or maturity models)
  • Assess as a first step in undertaking a general process improvement project
  • Assess to address perceived issues with one or more specific processes
  • Assess to address perceived issues with IT services as a whole, as a result of customer comments or customer satisfaction metrics

Potential Drawbacks

  • Sometimes, benchmarking efforts fail to take the logical next step of addressing the issues they find. Also, benchmarking is sometimes set-up with too narrow a scope and an over-dependence on metrics and objective data – effectively ignoring the knowledge and observations of IT staff and their willingness to engage with improvement efforts.
  • Assessment should almost always be a first step in undertaking a general process improvement project. But the amounts of data brought to light need to be categorised, prioritised and reported so that the improvement process delivers the biggest possible return on investment. Ideally, categorisation takes place automatically as a part of the assessment process (a Zeno feature, by the way).
  • When processes are assessed to rectify specific issues it’s possible to miss the real underlying causes of process problems. If you go in thinking you already know the answer, you will most probably find the answer you’re looking for – but it may not be the right one. A better approach is to form hypotheses, analyse assessment data, then test the hypotheses based on evidence.
  • A similar problem arises when we assess and improve ITSM processes in response to customer comments or customer satisfaction metrics. Listening to the voice of the customer to understand issues is an excellent way to do things, but we need to exercise caution when customers propose solutions. They pay us to be the experts, so we should use our knowledge of the big picture to discover and remove root causes, rather than just reacting to the loudest complaints.
Assess and Improve ITSM

Zeno: Powerful process improvement made simple

Avoiding the Pitfalls

So how can you avoid these pitfalls and maximise ROI in projects that assess and improve ITSM? One way is to use the CAIR framework – which is a common-sense approach to the complete cycle of assessment and improvement, methodically (but very quickly) moving through stages of Contextualise – Assess – Improve – Re-calibrate.

Using the CAIR model to assess and improve ITSM processes and organisations

Assess and Improve ITSM - CAIR model

'C' is for Context

The Context phase makes sure that stakeholders agree on the scope and vision for the assessment, and then does some planning. It includes activities such as

  • understand business objectives for the project
  • paint the big picture of possible underlying causes by asking initial open questions such as ‘what works well around here? What doesn’t work well?’
  • decide which IT elements will be assessed
  • establish the methods for an ITSM maturity assessment and who will carry them out
  • understand organization-specific customizations needed in the assessment process
  • define the details of the project

'A' is for Assessment

This phase sets out to find evidence, interview people and observe work in progress. Technical skills are useful but insufficient. Interview and listening techniques, establishing empathy, and the ability to formulate and test hypotheses are all put to use.

Tools are important for this assessment stage and the following stages – which is why we developed Zeno assessment software to make assessment faster, more effective, and more intuitive. Minimally, people need to know which questions to ask, how to rate and categorise the answers, how to capture comments and use them to generate reports, and how to apply basic statistical analysis. Zeno covers all of these bases and more.

Sorry to interrupt, but if you would like to hear more about Zeno’s launch and be among the first to use its features, you can sign-up here (no credit card details or any financial or other commitment is required to join the wait list):

'I' is for Improvement

Communication is an important part of the communication phase. When reports and recommendations are accessible, attractive and meet the needs of various audiences, it’s easier to gain wide support for improvement activities.

More technical aspects come into play once communication is clear. This includes CSF and KPI design, managing change, and building roadmaps.

It’s vitally important that momentum isn’t lost during this stage. Zeno includes a Process Tracker which is used by a process owner or other CSI professional to track improvements, estimate new maturity and capability scores, and report on progress.

'R' is for Re-calibration

Re-calibration is about managing the new environment created by the improvements that have been made. One critical feature is to ensure that no negative unintended consequences are taking place.

15 Elements of an ITSM Maturity Assessment

Much more detail and a splendid infographic can be found here: A Visual Guide to ITSM Maturity Assessment. The infographic identifies 15 elements of a successful assessment.

If you would like to hear more about Zeno’s launch and be among the first to use its features, you can sign-up here. No credit card details or any financial or other commitment is required to join the wait list:

Zeno: Powerful process improvement made simple

ITSM Assessment: How many interviewers?

How many interviewers needed for an ITSM Assessment?

This article discusses the question of how many interviewers are needed to carry out an ITSM Maturity Assessment. It includes pointers on the benefits and drawbacks of three different approaches, and shows how to get a successful outcome whether you have one, two or no interviewers involved. 

It's important to understand the different interview options that are available when planning an assessment. There are two dimensions in play: quality and cost. However, as we’ll see, how many interviewers are involved isn’t the only factor governing these two variables: with good organization and preparation the different approaches can all be made to work.

Let’s look at all three options in turn.

The 2-Interviewer Solution

The set-up

In a two-interviewer set up, the first interviewer usually focuses on engaging with the interviewee by putting the questions or statements to them. The interviewer will have the questions and / or statements relating to the process, function or other IT element being assessed in front of them – usually on a laptop and usually in a spreadsheet.

This first interviewer may go ‘off-piste’ and delve deeper with follow-up questions. He or she will then capture the main response of the person being interviewed. By ‘main response’ we mean either

  • The ‘yes’ or ‘no’ response… or
  • A score on a Likert scale or similar number-based scale (separate article coming on using the Likert scale)

The first interviewer, then, does most of the talking, and only needs to capture a ‘yes’, a ‘no’ or a score (1 -5 in the Zeno software, see screenshots below) relating to each question or statement.

The second interviewer often joins in the conversation to follow up on a specific point, but their main task is to capture details of the response being given, plus their own version of the yes, no or number score.

Zeno: Powerful process improvement made simple

Advantages of using two interviewers

  • More detail is captured (because the second interviewer can focus on typing)
  • Each interview can take less time and flow more freely when the main interviewer doesn’t have to regularly stop to take notes
  • Interviewers can compare scores and investigate any differences in their results. In effect, there are two pairs of eyes on any problems
  • The chances of having in-depth practical knowledge of the subject in hand are doubled
  • The chances of spotting any discrepancies or important nuances are doubled
  • Follow-up tasks such as finding corroborative evidence are more easily shared

Disadvantages of using two interviewers

  • Although the interview may be shorter, this doesn’t fully compensate for the extra effort and cost of using two interviewers
  • If not handled correctly, the interviewee may feel intimidated by facing two interviewers

The 1 Interviewer Solution

The set-up

In a one-interviewer set up, the interviewer focuses on engaging with the interviewee by putting the questions or statements to them and will then capture their main response (yes, no or score). The interviewer also needs to capture notes on the responses given so that they can be followed up or reconsidered later. The notes are also used to give context for an answer when it is found later that two or more interviewees have given very different responses.

Advantages of using one interviewer

  • Less costly in terms of time and money
  • Can be easier to establish trust and empathy when only two people are involved

Disadvantages of using one interviewer

  • Less detail is captured (and / or the interview takes longer because of the need to take notes). Note taking in spreadsheets tends to be awkward and adds to the time taken
  • Interviewers don’t have a second opinion at hand
  • There is less chance of having in-depth practical knowledge of the subject in hand
  • The chances of spotting any discrepancies or important nuances are halved
  • Follow-up tasks such as finding corroborative evidence may require more effort

How to mitigate the disadvantages of a 1 interviewer set-up

Mitigation of the disadvantages of using one interviewer boils down to two things –

  • Preparation in terms of deepening understanding of the IT element being assessed (covered in this post). This will help to minimise any awkward pauses, apart from being a basic requirement of an interviewer!
  • Preparation in terms of capturing responses and taking notes

Here’s how we do this in Zeno, but you can use the same kind of approach using a spreadsheet.

1 Capture the 'main response'. In this case the user clicks on 'Agree' and either adds a comment or moves on to the next statement. The following sequence of actions would be used irrespective of how many interviewers are involved.

Zeno - how many interviewers - 1

Click for larger image

2. Prepare to add additional comments. In this case the user clicks the 'down' arrow to add a comment.

Zeno - how many Interviewers - 2

Click for larger image

3. Add additional comment. The comment is added and saved. Also note that lots of space has been set aside to capture detailed notes, without having to scroll right. In the case of Zeno, these notes can be directly exported into a report (after some editing if necessary)

Zeno - how many interviewers - 3

Click for larger image

The No Interviewer Solution

The set-up

In a no-interviewer set up, the assessment questions and statements (or a link to them) are sent to each respondent. The participants respond to each statement and question, add their own explanatory notes, and complete the questionnaire in their own time (subject to any project constraints). The responses are then collated.  

Traditionally, self-assessment has only been used with a cut-down set of statements (for example, in the case of ITIL) but as we shall see, self-assessment can now be extended much further.

Advantages of using no interviewers

  • Much less costly in terms of time and money
  • Scalable - the administrative overhead of questioning twenty (or any number) of respondents is hardly any more than questioning one
  • No scheduling headaches
  • Potentially larger (and more statistically significant) data sets
  • Easier to capture different perspectives (for example, the customer perspective) because any number of respondents are available
  • Respondents can go into detail about what they perceive as problem areas
  • Discrepancies in scoring or important nuances that appear in respondent’s notes can be analysed offline

Disadvantages of using no interviewers

  • No guidance from an interviewer if the meaning of a question is unclear

How to mitigate the disadvantages of a no-interviewer set-up

Mitigation of the disadvantages of using no interviewers can mostly be resolved by using in-context help. In a spreadsheet this would be done by adding comments to cells. In Zeno, comprehensive in-context help is built-in.

So: how many interviewers?

Although we've used Zeno to demonstrate some of the principles, the same effects can be achieved with spreadsheets. The quality and usefulness of an ITSM assessment no longer has to be so strongly influenced by how many interviewers are used. With good guidance and some practice, effective assessments can now be carried out more frequently and more cheaply.

Zeno: Powerful process improvement made simple

Want more?

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

If you have any questions, please let me know in the comments or contact me via LinkedIn.

If you need assessment software, you might want to see what Zeno has to offer.

Was this article useful? Please share with your network...

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
start IT assessment interview

How to start an ITSM Maturity Assessment Interview

This article describes how to start an ITSM Assessment interview. It includes pointers on what to say to the interviewee and the importance of personality and mood.

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

Let’s get into it!

The ITSM Assessment Interview: Opening Impressions

It's important to gauge the mood of the interviewee as soon as possible. In general, an ITSM assessment interview will go more smoothly when there is some rapport between the two (and sometimes three) parties.

When we talk about 'mood' here, we're referring mainly to the interviewee's attitude to the interview and to the assessment process in general. Most people will be interested by the process and glad to help, but occasionally you'll meet someone who feels unsure at best, or threatened at worst. Establishing rapport is especially important in those circumstances and we'll come to how to do that in a moment.

One of the reasons we're interested in the mood or attitude of the interviewee (besides the fact that we want it to be a pleasant experience for them) is that strong attitudes may introduce bias. Even with a positive attitude towards the assessment, some people will tend to have a 'glass half full' or half-empty bias to their answers. It's a good idea to corroborate and find supporting evidence whenever you feel that some bias has entered into the responses you're getting.

Opening Remarks: establishing rapport

Establishing rapport with an interviewee is 80% art, and 20% science. It isn't possible to supply absolute rules (although there are many books on the subject of rapport) because there are so many different kinds of people and circumstances.

For that reason, although I am going to state clearly what I think you MUST say, I think the order in which you say these things should be allowed to emerge semi-naturally from the conversation. 'Semi-naturally' because it is your responsibility as the interviewer to steer the opening conversation and the rest of the interview. Time is usually short, so you only have a few minutes to set the scene and get the ITSM assessment interview off to a good start.

So, before you start the interview proper, here are the areas you MUST cover, with a suggested order which may vary slightly with the circumstances.

You MUST cover these 7 areas

1. Welcome the interviewee into the room. Briefly introduce yourself ( if you're an external assessor, this is NOT a chance to advertise yourself or your company!)

2. Confirm that you're speaking to the right person about the right subject (for example the Change Management process).

3. Ask if the interviewee was briefed by their manager (they should have been!). If they were not, brief them now. For example, you could explain that a senior manager has asked you to look at the change management process to see what's working well and what isn't so that we can make some improvements, and that the interviewee's name was put forward as someone with a good understanding of the subject.

4. Ask for brief details of the interviewee's role, and how long they've been in that role. This will help you to put into context the answers you receive. This is really important. For example, a lack of understanding or knowledge from a process owner or process manager indicates a different kind of underlying issue than lack of understanding from someone with a more limited role within the process.

5. Explain that the interviewee is anonymous for the purposes of the assessment process. In other words, no remarks made during the interview will ever be directly attributed to the interviewee. It's all 'in confidence'. You have to mean this, and you have to stick to it. To digress slightly, this is one of the reasons why you should always interview at least two people on any single subject. If only one person is interviewed, it will be clear who the source of comments is.

6. Equally, if the assessment process was designed so that interviewees are NOT anonymous (not always something I'd recommend), then make that clear.

7. Finally, explain how the ITSM assessment interview will proceed. Give an example of a question or statement and the kind of response you're looking for. For example, you could explain that you are looking for a yes / no answer, or a rating from 1-5 - or whatever method you are using. When I use my preferred 1-5 rating system (Likert scale) I print a brief explanation of each rating and place it in front of the interviewee so that they can consider where they are on the scale. They will quickly get used to that and won't need to refer to the written explanation after the first few questions.


Zeno: Powerful process improvement made simple

Summary

We are really doing just a few simple things to get the interview off on the right foot:

- gauge the mood and potential bias of the interviewee

- establish rapport

- give the interviewee the information they need

If you have any questions, please let me know in the comments or contact me via LinkedIn.

If you need assessment software, you might want to see what Zeno has to offer.

Was this ITSM Assessment interview article useful? If so, please share with your network...

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
The Interview Environment for ITSM Maturity Assessments

The Interview Environment for ITSM Maturity Assessments

This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's necessary to make the interview as successful as possible. 

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

Let’s get into it!

The Interview Room

I hope you like the picture at the top of this page. Believe it or not, it isn't absolutely necessary to have an inspiring view of distant mountains when conducting an ITSM maturity assessment interview. Which is handy, if you happen to be in Lincolnshire or Florida.

What is vital, though, is that the interview environment is reasonably private so that the interviewee can speak openly (we have covered the subject of respondent anonymity in other posts). Once the environment is right, everything else goes more smoothly.

Interview Environment Practicalities

This is what you need to do to get the interview environment just right:

- Don't carry out the assessment interview in an open office. Make the interviewee confident that privacy is assured. 'Job shadowing' and informal conversations related to the assessment can of course take place in open offices, but don't discuss specific contentious issues outside of the formal interview

- The interview room should be soundproof if possible, again so that conversation can flow freely. Other common sense environmental factors (light, temperature) should be considered

- Ensure that power sockets and a network connection are available for your lap top and that of your co-interviewer if there is one

- Interviews can last up to two hours, depending on what is being assessed. Ensure your interviewee is comfortable, allow for quick and disciplined breaks if necessary

- Ensure that you have to hand hard or soft copies of any documentation you want or need to discuss (such as reports or a formal process description)

- Make sure that your access to software (such as an ITSM tool) is arranged and tested before the interview

- You may not know before an interview that you'll need to access ITSM software. Sometimes an interesting and relevant point will come up that can be dealt with right away as long as there is access to the right tool

Zeno: Powerful process improvement made simple

Common Sense

Although these suggestions are mostly basic common sense, I have seen interviews go less smoothly than they should for simple, avoidable reasons. The most common problem is that no room has been booked, or an unsuitable room has been booked. It's possible to conduct interviews in break-out spaces or staff rest areas, but they don't provide the ideal interview environment.

That kind of unpreparedness worries me when it happens in client organisations: it sends a signal that the assessment and improvement efforts aren't really important enough to make professional arrangements.

Please let me know in the comments or on LinkedIn if I've forgotten anything or you have suggestions.

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
IT Assessment Interview Preparation

A Guide to IT Assessment Interview Preparation

How to Prepare for an IT Assessment Interview

This post describes IT Assessment interview preparation from the perspective of the interviewer. You will learn some simple guidelines on how to prepare to conduct an interview as part of an ITSM process maturity assessment.

Maturity assessment is of course much more than just a series of interviews, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover these subjects:

Without more ado, let’s get into it.

Assessment Interview Preparation

First, the bad news: if you’re going to ask questions about an IT subject (often a process) you need a sound grasp of that subject.

If that sounds daunting given the complexity of IT, let me explain that there are various levels of expertise and with good preparation you can approach any assessment confidently. I am not an expert on every single IT process in the sense that I would be able to step into a job in – for example - Release and Deployment Management, and immediately be effective. That process is a discipline with its own body of knowledge, tools, technical practices, technical shorthand (jargon), etc.

So what level of expertise is needed? The answers fall into two main categories.

1. Understand as a management discipline, not a technical one

Let’s suppose that you will be assessing the ITIL process called Release and Deployment Management. If this is not a main area of expertise for you, go to the source material (ITIL Service Transition 2011 CH4.4) and review it. You can start by grasping the main objectives of the process, and then drill down into the actual practices. I like to use Mind Maps for this.

The process has its own jargon and if you come across something you don’t understand – DML, perhaps, or V-model – make sure you learn a clear definition or at least get the concepts clear in your mind. Any difficult terms or acronyms can be colour-coded in the mind map or other note-taking software. The source material should include the right definitions, but if not, apparently there’s something called the internet that may be able to help…

You will come away from this exercise with a refreshed understanding of Release and Deployment as a management discipline, but not as a technical practice. So what happens if the interviewee starts to ‘talk tech’ in response to a question?

How to deal with ‘tech talk’

Don’t try to shut the interviewee down, because the devil is often in the technical detail. There are a few things that you can do though:

  • Be open and admit that you are not a technical expert in this area. Be sure to strongly imply that the interviewee is a technical expert. Flattery will get you everywhere
  • Ask your interviewee to explain the issue in lay terms. Then guide the conversation to the substantive effects of the technical issue, not the technical details themselves
  • Make sure to take good notes that you can follow up later
  • If appropriate, arrange to follow up with the interviewee so that you can see the issue at close quarters

2. Make sure you understand the meaning of the question you're asking!

If you are using a standard set of statements such as those available through Axelos (or better still, through Zeno) as a basis for ITIL process assessment, make sure you look at each statement long before you start the interview.

If not, you will come a cropper.

It’s not that the statements and questions are technically difficult to understand, it’s that some of them are semantically difficult. And if you, the assessor, can’t understand the question, what chance has your interviewee got?

Resource: ITIL Maturity Model: High Level Self-Assessment Service *

* Alternatively, you could wait for the feature rich but ITIL Maturity Model compliant Zeno SaaS version, due in Q1 2018. Just sign up here for early access and more info:

Zeno: Powerful process improvement made simple

My approach to this is to practice putting a question or statement in my own words, retaining the essence but making the question as simple as possible. 

It may seem that I’m exaggerating the problem here. But imagine you’re an hour into an IT Service Continuity assessment where the conversation has been non-stop, and the interviewee is beginning to flag. The statements are starting to sound a bit similar to the interviewee's ears because they are each focused on a slightly different detail of the process. Then you put this corker to your unsuspecting victim:

“IT service continuity management works closely with projects and changes and other design activities for new services, to ensure that the designs for the service continuity of new services match the needs of the business and are consistent with continuity design criteria and architectures and business continuity requirements.”

Yes, that’s a real ITIL process assessment statement and by no means the worst example of its kind!

Bear in mind that you will be looking for an answer from the interviewee that indicates whether or not they agree with the statement. But in the unlikely event that they understood the statement on first hearing it, exactly which part of it will they respond to? There are three approaches to preparing for the problem of over-complex statements and questions.    

How to Simplify Statements and Questions

        Approach 1. Break the statement down into its constituent parts and find the essence of what you want to know. It will often be the case that you’ve asked about several parts of the question before. For instance, in our example you will already know how well the process a) matches business needs and b) is consistent with design criteria and architectures and c) is also consistent with business continuity requirements.

Therefore, we can simply ask about the crux of the matter: "does ITSC work well with project management?". This is a more human way of putting it and gets at what you need to know in a straightforward way.

        Approach 2. With more experience you’ll realise that the statements are sometimes grouped into themes. For example, there are a series of ITSC statements around ‘how well does the ITSC process work with X?” (so, the general theme here is ‘interfaces with other processes’). 

Then we just repeat the technique of 'Approach 1' with the other processes that are referenced in the assessment statements, as follows:

- does ITSC work well with financial management to ensure that investments required by continuity plans are consistent with budgets and financial plans?

- does ITSC work well with capacity management to… etc.

        Approach 3. Finally, you could choose to move away from the ‘official’ ITIL statements altogether. This has the advantage of enabling the design of more user-friendly questions; however, it has the disadvantage of moving away from the objective ITIL standard and therefore makes it harder to benchmark or compare against other maturity scores. In Zeno we provide both: a drastically shortened set of easily understandable questions, plus the ‘standard’ statements with some built-in contextual help. The circumstances will dictate which is best to use.

IT Assessment Interview Preparation: not rocket science

Assessment Interview Preparation Summary

Assessment Interview Preparation Summary. CLICK FOR FULL SIZE IMAGE

Hopefully by now you’re convinced that one key to successful IT assessment interviewing is that you need to study the statements and questions you’ll be asking!

All of the above may seem a little complex, but all will become clear when you have the assessment questions in front of you, and have picked out and understood the themes and repetitions

It’s not rocket science, but like any other area of endeavour, hard work and practice pay off. The hard work comes in when you make sure you are up to speed with the subject you’re assessing (and by reading about techniques in articles like this one). You can do practice runs in your head or in reality, or you can conduct an actual assessment interview. A great way to learn is to sit in with an experienced assessor, watch and listen.

If you have any questions on IT Assessment interview preparation , feel free to ask in the comments below.

Zeno: Powerful process improvement made simple

ITSM Maturity Assessment chat

ITSM Maturity Assessment: Infographics and In-depth Guide

How do I carry out the best possible ITSM Maturity Assessment?

For years, ITSM maturity assessment and IT process assessment have rightfully been a cornerstone of process improvement projects. Given that, it's strange that so little has been written about the strategy, tactics and techniques of assessment, especially from a non-vendor perspective.

This article lifts the lid on how professionals plan and implement ITSM maturity and IT process assessments. In the course of a wider series of posts, we will go into great detail to create a complete guide to the subject. This post is an introduction to what an IT assessment is, the techniques employed, the skills required, and whatever else we meet along the way.

My intention is that this article will become the go-to resource for anyone needing to learn about assessing ITSM maturity, and all that entails. Over the coming months many resources will be added such as sample assessment statements / questions, sample reports, links to more in-depth articles on a specific subject, etc. So please check back and let me know in the comments exactly what information you need!

But first, a quick definition of what is meant by an ITSM Maturity Assessment.

Definitions

An ITSM maturity assessment rates the performance of IT elements against an industry standard 'maturity scale'. It does this by interviewing stakeholders, observing work in progress, and by identifying material evidence. The intention is to improve the effectiveness and efficiency of the process, service or other element being assessed.

"IT elements" include IT processes, services, value streams, supporting tools, functions and teams, and specific activities carried out by the IT organisation.

Because ITIL is the dominant IT Service Management framework, ITSM assessments are usually based on ITIL questions and statements and the accompanying ITIL maturity model.

An 'IT process assessment' refers to an assessment of one or more IT processes. A maturity scale is also used, but both the scale and the framework which it relates to may be non-ITIL. For instance, an IT process assessment could be carried out using the COBIT framework. This uses the COBIT Process Assessment Model.

Another common maturity model originally designed for software development is CMMI for Services.

Basic Overview of an ITSM Maturity Assessment

Let's start with a graphical overview of an assessment project:

ITSM Maturity Assessment basic overview


CAIR Overview - an ITSM Maturity Assessment approach

The CAIR graphic shows a basic map of an assessment divided into four main phases. The mnemonic I use to remember this is 'Create An Improvement Revolution'. Let's take a slightly closer look before we go into detail.

Zeno: Powerful process improvement made simple

CAIR: "Create An Improvement Revolution"

'C' is for Context

The Context phase is about getting agreement from the relevant stakeholders on how the assessment will be carried out, and then doing some planning. It covers areas such as 

  • understanding the business objectives
  • deciding which IT elements will be assessed
  • establishing the methods for an ITSM maturity assessment and who will carry them out
  • an understanding of organisation-specific customizations needed in the assessment process
  • defining the details of the project

'A' is for Assessment

The skillful work of finding evidence, interviewing people and observing work in progress happens in the Assessment phase. This is what we call the 'solid foundation' of an assessment project. The skills needed are not just technical - they include interview and listening techniques, empathy, and the ability to formulate and test hypotheses.

Related Article: A Guide to Interview Preparation

Related Article: The Interview Envi​ronment

Related Article: How to start the interview

Related Article: One Interviewer or Two? How to Interview Efficiently

Related Article: How to use the ITIL maturity model and Likert scale to assess IT processes

'I' is for Improvement

The improvement phase is partly about communication. Reports and recommendations should be accessible, attractive and mindful of the needs of various audiences. Without those qualities it's difficult to gain broad support for improvement activities.

Once communication is clear more technical aspects come into play, such as designing CSFs and KPIs, managing change and building roadmaps.

'R' is for Re-calibration

Re-calibration is about managing the new environment created by the improvements that have been made. One critical feature is to ensure that no negative unintended consequences are taking place.

The ITSM Maturity Assessment Big Picture

CAIR is the skeleton of an assessment and improvement effort. Let's start to put meat on the bones with this graphic:

ITSM Maturity Assessment Infographic

Zeno: Powerful process improvement made simple

The 15 Elements of ITSM Maturity Assessments

Let's dig a little more deeply into each of the elements of a great IT Service Management assessment. Many of the points here will be expanded upon in separate posts.

Context 1. Customized and uniquely valuable

There is no such thing as a 'cookie cutter' assessment. Each organisation will have different requirements at different times. They will be responding to unique internal and external (market) pressures. The job of the assessor is to understand the environment and ensure that the assessment approach is fit for purpose (but see 'Ethical, Professional and Open', below). There are several aspects to this...

- What is the business context for the assessment? What are the objectives? 

Some organisations have a straightforward desire to improve processes, functions or services. Others might have a more sophisticated approach and be looking to improve an IT value stream. 

More broadly, at this point it's essential to understand:

  • problems the organisation faces and the objective of the assessment
  • behaviours which need to change. An ITSM maturity assessment should unearth the reasons for such behaviours and recommend how to change them
  • perceived competency gaps
- What are the business drivers and expected business value?

But there could be other reasons to assess. An organisation may want to measure the capability of its IT organisation to compare it to other organisations or outsourcers. Some will want to ensure that they are obtaining value for money from their current IT provider. Or an IT provider might want to prove that its services are meeting and excelling industry standards.

Each of these scenarios can introduce constraints into the assessment. Some also involve professional and ethical issues: just who are we assessing for, and should it matter? We'll come back to those questions later.

- Ask open questions

Never underestimate the value of big, open questions. Sometimes senior stakeholders will have a strong opinion about what kind of IT issues exist.Their views will strongly influence the scope of the ITSM maturity assessment. But ask other stakeholders their opinions before the scope is agreed: "what is working well around here?" "What's not working well around here?"

- Be aware of internal politics

You won't be surprised to hear that internal politics sometimes colour the objectives or scope of an IT assessment. If this seems to be the case in the organisation being assessed, try to understand the big 'political' issues that may be in play without getting drawn into them.

- Agree the scope of the assessment

Once these factors are understood, it's time to agree the scope of the assessment. There is more context to consider though, because there are now decisions to make about exactly what will be assessed. Which is coming up next...

Context 2. Context specific

Once the scope has been agreed, it will be clear what kind of assessment strategy is required. For example the assessment of a specific process will involve different techniques to an assessment of an IT Value stream.

Context 3. Perfectly prepared

Timely and effective project management is essential for a successful ITSM maturity assessment. Part of the skill of this is to make sure that a good sampling of stakeholders is selected to be interviewed. Depending on the objectives, for example, a mix of senior managers, associates and customers might be appropriate. All of these stakeholders will have limited time, and this needs to be managed with some sensitivity.

Assessment 4. Built on a solid foundation

This is where four of the five fundamental skills and activities happen ('evidence based' is covered separately). Assessment interviewing, job shadowing, tools reviews and documentation reviews are usually all essential in understanding how the work gets done.

Assessment 5. Aligned with specific frameworks

Because ITIL has become the dominant framework for ITSM, most assessments will relate to ITIL processes. ITIL has an accompanying model with defined maturity levels, and of course a set of specific questions / statements which are used as the fundamental assessment tool. How to use those questions is a big subject and one which we'll go into in a separate post.

Of course, other service management-related frameworks and approaches are available. It is possible to carry out COBIT, Devops and other types of assessment. Each will have its own set of questions and statements and will use an appropriate maturity model. Again, this is a big subject and one we'll go into elsewhere.

Assessment 6. Methodologically sound and statistically valid

This element applies to the business of how we conduct interviews, which questions are asked, and how we record and 'score' the answers. It's important that there is a consistency to the way in which we rate responses to statements. 

The standard for ITSM maturity assessment is to read a statement and record a 'yes' or 'no' response as to whether or not the statement is true. There are numerous reasons why this is NOT a good idea. We will go into this at great depth in another post.

Assessment 7. Open to non-statistical information

Responses to formal statements and questions are important and are the basis of maturity scores. The customer (i.e. the party that commissioned the project) is often focused on these scores, but it is the job of the assessor to listen carefully to information that comes from other sources. 

By listening to anecdotal stories and comparing with other more formal information sources, we start to form a picture of what is really happening in the organisation. In fact, we begin to explain the evidence and its effects, and we note that new evidence supports or contradicts these explanations.

The bottom line is: if the ITSM maturity assessment sticks to formal questions, you may be missing vital facts and perspectives that will help the organisation improve. 

Assessment 8. Psychologically in-tune

Assessments are about people. It's vital that our 'respondents' are treated with respect.

Some people are positively happy about being asked to contribute. Someone remarked to me recently that he had worked in the organisation for fifteen years, and this was the first time anyone had ever asked for his opinion and valued his experience in this way.

Other people will be fearful or negative. Some will have political agendas or try to control the interview. Some will be introverts and others extroverts. The job of the assessor is to make sure that each of these kinds of people is able to make as significant a contribution as possible. 

Assessment 9. Evidence based

Some of the testimony gathered by assessors is opinion. Some of it is fact. The assessor's job is to substantiate the most important findings. This can play out at the level of overall process performance, or in the details of an activity, and everywhere in between. 

Assessment 10. Ethical, professional and open

There are two main aspects to this. The first is to respect the anonymity of respondents. People should feel free to speak their minds without fear of recrimination. The principle of anonymity should be established and agreed in the Context phase.

The second aspect is to ensure that what we find is what we report. An IT service provider might well want the best possible maturity results; but a new CIO has good reason to hope for the worst possible maturity results. We should already have discovered this sort of bias in the Context phase.

Neither of those desires or expectations must be allowed to affect an ITSM maturity  assessment in any way. They are absolutely not part of the scope, and should not change in any way the questions that are asked or the way they are reported. Of course, whoever commissioned the assessment report is in a position to change it or refuse to publish it. That situation is usually beyond the control of the assessor.

In the short term, this stance may seem suicidal for anyone specialising in assessment. In the longer term, the attempt to establish an objective, neutral position is fundamental to success and the trust of the organisations who want to assess and improve themselves.

Improvement 11. Specific and clear recommendations

One of the delights of assessment reporting is that very often the number of actionable recommendations can be huge. I say 'delights' because the recommendations are an indicator that our knowledge of whatever is being assessed has increased, and that there is obvious room for improvement.

It's important that these recommendations don't become overwhelming. They should be categorised and prioritised at a minimum, and arranged into a logical sequence in a roadmap (see below).

Improvement 12. Visually engaging, built for sharing

There is a strong case for presenting the plain facts in a plain way. But the reality is that the large amounts of information produced by an assessment need to be ordered and presented for different audiences.

Visual engagement is vital for some audiences, though we have to sound a word of warning here: make sure that any visuals, and particularly graphs, don't distort the data. At Zeno, we happen to have a tamed statistician with a brain the size of Pluto, and he keeps us on the straight and narrow. More about this in another article.

Typically, different audiences want to consume information differently. The obvious example is that for some people an executive summary will be appropriate; but for a process owner, an understanding of the detail is required.

Improvement 13. Clear on the roadmap and objectives

Producing a roadmap is not usually something an assessor would do on their own. It's important that when prioritising actions based on the recommendations, we go back to the business drivers that were established in the Context phase. Roadmaps are especially important when different 'streams' of improvement effort will be carried out simultaneously. The roadmap should literally provide the 'vision'; a project plan should provide the detail.

Re-calibration 14. Monitored

Once a process improvement plan is agreed and up and running, it should be monitored. Those familiar with Lean IT and the Theory of Constraints will understand that process improvements (or function, or service improvements, etc.) can have unintended consequences.

Put at its simplest, if activity A gets much more efficient and increases throughput, what's the effect on activities B and C? The overall effect could be negative.

This should be initially considered in the Context phase when CSFs and KPIs are designed: the measures should address desired outcomes at the level of business value, not at the level of localised process improvements.

Re-calibration 15. Focused on action and accountability

The best possible reason to assess ITSM maturity is that it provides fantastic insights into how to improve the way that services and IT in general are managed.

But if there is no clear accountability for making sure that appropriate recommendations are carried out, the assessment will have been a waste of time.

Equally, if the organisation's senior management does not strongly and visibly support the improvement efforts, again - the assessment will have been a waste of time.

Finally!

You may want to consider bookmarking this page. I intend to update it with links to other detailed posts and resources.

If there are particular assessment and improvement-related resources you would like access to in the future, please let me know in the comments below or contact me via LinkedIn.

For more information on Zeno, just fill out the form below. No obligation, no payment details required.

Zeno: Powerful process improvement made simple

Did you like this article? We'd really appreciate a testimonial!

Was this Zeno content valuable for you?
Will this Zeno content help you to improve at your job?
Will this Zeno content help your organisation?
How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
>