Category Archives for Interviewing

ITSM Assessment: How many interviewers?

How many interviewers needed for an ITSM Assessment?

This article discusses the question of how many interviewers are needed to carry out an ITSM Maturity Assessment. It includes pointers on the benefits and drawbacks of three different approaches, and shows how to get a successful outcome whether you have one, two or no interviewers involved. 

It's important to understand the different interview options that are available when planning an assessment. There are two dimensions in play: quality and cost. However, as we’ll see, how many interviewers are involved isn’t the only factor governing these two variables: with good organization and preparation the different approaches can all be made to work.

Let’s look at all three options in turn.

The 2-Interviewer Solution

The set-up

In a two-interviewer set up, the first interviewer usually focuses on engaging with the interviewee by putting the questions or statements to them. The interviewer will have the questions and / or statements relating to the process, function or other IT element being assessed in front of them – usually on a laptop and usually in a spreadsheet.

This first interviewer may go ‘off-piste’ and delve deeper with follow-up questions. He or she will then capture the main response of the person being interviewed. By ‘main response’ we mean either

  • The ‘yes’ or ‘no’ response… or
  • A score on a Likert scale or similar number-based scale (separate article coming on using the Likert scale)

The first interviewer, then, does most of the talking, and only needs to capture a ‘yes’, a ‘no’ or a score (1 -5 in the Zeno software, see screenshots below) relating to each question or statement.

The second interviewer often joins in the conversation to follow up on a specific point, but their main task is to capture details of the response being given, plus their own version of the yes, no or number score.

Zeno: Powerful process improvement made simple

Advantages of using two interviewers

  • More detail is captured (because the second interviewer can focus on typing)
  • Each interview can take less time and flow more freely when the main interviewer doesn’t have to regularly stop to take notes
  • Interviewers can compare scores and investigate any differences in their results. In effect, there are two pairs of eyes on any problems
  • The chances of having in-depth practical knowledge of the subject in hand are doubled
  • The chances of spotting any discrepancies or important nuances are doubled
  • Follow-up tasks such as finding corroborative evidence are more easily shared

Disadvantages of using two interviewers

  • Although the interview may be shorter, this doesn’t fully compensate for the extra effort and cost of using two interviewers
  • If not handled correctly, the interviewee may feel intimidated by facing two interviewers

The 1 Interviewer Solution

The set-up

In a one-interviewer set up, the interviewer focuses on engaging with the interviewee by putting the questions or statements to them and will then capture their main response (yes, no or score). The interviewer also needs to capture notes on the responses given so that they can be followed up or reconsidered later. The notes are also used to give context for an answer when it is found later that two or more interviewees have given very different responses.

Advantages of using one interviewer

  • Less costly in terms of time and money
  • Can be easier to establish trust and empathy when only two people are involved

Disadvantages of using one interviewer

  • Less detail is captured (and / or the interview takes longer because of the need to take notes). Note taking in spreadsheets tends to be awkward and adds to the time taken
  • Interviewers don’t have a second opinion at hand
  • There is less chance of having in-depth practical knowledge of the subject in hand
  • The chances of spotting any discrepancies or important nuances are halved
  • Follow-up tasks such as finding corroborative evidence may require more effort

How to mitigate the disadvantages of a 1 interviewer set-up

Mitigation of the disadvantages of using one interviewer boils down to two things –

  • Preparation in terms of deepening understanding of the IT element being assessed (covered in this post). This will help to minimise any awkward pauses, apart from being a basic requirement of an interviewer!
  • Preparation in terms of capturing responses and taking notes

Here’s how we do this in Zeno, but you can use the same kind of approach using a spreadsheet.

1 Capture the 'main response'. In this case the user clicks on 'Agree' and either adds a comment or moves on to the next statement. The following sequence of actions would be used irrespective of how many interviewers are involved.

Zeno - how many interviewers - 1

Click for larger image

2. Prepare to add additional comments. In this case the user clicks the 'down' arrow to add a comment.

Zeno - how many Interviewers - 2

Click for larger image

3. Add additional comment. The comment is added and saved. Also note that lots of space has been set aside to capture detailed notes, without having to scroll right. In the case of Zeno, these notes can be directly exported into a report (after some editing if necessary)

Zeno - how many interviewers - 3

Click for larger image

The No Interviewer Solution

The set-up

In a no-interviewer set up, the assessment questions and statements (or a link to them) are sent to each respondent. The participants respond to each statement and question, add their own explanatory notes, and complete the questionnaire in their own time (subject to any project constraints). The responses are then collated.  

Traditionally, self-assessment has only been used with a cut-down set of statements (for example, in the case of ITIL) but as we shall see, self-assessment can now be extended much further.

Advantages of using no interviewers

  • Much less costly in terms of time and money
  • Scalable - the administrative overhead of questioning twenty (or any number) of respondents is hardly any more than questioning one
  • No scheduling headaches
  • Potentially larger (and more statistically significant) data sets
  • Easier to capture different perspectives (for example, the customer perspective) because any number of respondents are available
  • Respondents can go into detail about what they perceive as problem areas
  • Discrepancies in scoring or important nuances that appear in respondent’s notes can be analysed offline

Disadvantages of using no interviewers

  • No guidance from an interviewer if the meaning of a question is unclear

How to mitigate the disadvantages of a no-interviewer set-up

Mitigation of the disadvantages of using no interviewers can mostly be resolved by using in-context help. In a spreadsheet this would be done by adding comments to cells. In Zeno, comprehensive in-context help is built-in.

So: how many interviewers?

Although we've used Zeno to demonstrate some of the principles, the same effects can be achieved with spreadsheets. The quality and usefulness of an ITSM assessment no longer has to be so strongly influenced by how many interviewers are used. With good guidance and some practice, effective assessments can now be carried out more frequently and more cheaply.

Zeno: Powerful process improvement made simple

Want more?

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

If you have any questions, please let me know in the comments or contact me via LinkedIn.

If you need assessment software, you might want to see what Zeno has to offer.

Was this article useful? Please share with your network...

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
start IT assessment interview

How to start an ITSM Maturity Assessment Interview

This article describes how to start an ITSM Assessment interview. It includes pointers on what to say to the interviewee and the importance of personality and mood.

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

Let’s get into it!

The ITSM Assessment Interview: Opening Impressions

It's important to gauge the mood of the interviewee as soon as possible. In general, an ITSM assessment interview will go more smoothly when there is some rapport between the two (and sometimes three) parties.

When we talk about 'mood' here, we're referring mainly to the interviewee's attitude to the interview and to the assessment process in general. Most people will be interested by the process and glad to help, but occasionally you'll meet someone who feels unsure at best, or threatened at worst. Establishing rapport is especially important in those circumstances and we'll come to how to do that in a moment.

One of the reasons we're interested in the mood or attitude of the interviewee (besides the fact that we want it to be a pleasant experience for them) is that strong attitudes may introduce bias. Even with a positive attitude towards the assessment, some people will tend to have a 'glass half full' or half-empty bias to their answers. It's a good idea to corroborate and find supporting evidence whenever you feel that some bias has entered into the responses you're getting.

Opening Remarks: establishing rapport

Establishing rapport with an interviewee is 80% art, and 20% science. It isn't possible to supply absolute rules (although there are many books on the subject of rapport) because there are so many different kinds of people and circumstances.

For that reason, although I am going to state clearly what I think you MUST say, I think the order in which you say these things should be allowed to emerge semi-naturally from the conversation. 'Semi-naturally' because it is your responsibility as the interviewer to steer the opening conversation and the rest of the interview. Time is usually short, so you only have a few minutes to set the scene and get the ITSM assessment interview off to a good start.

So, before you start the interview proper, here are the areas you MUST cover, with a suggested order which may vary slightly with the circumstances.

You MUST cover these 7 areas

1. Welcome the interviewee into the room. Briefly introduce yourself ( if you're an external assessor, this is NOT a chance to advertise yourself or your company!)

2. Confirm that you're speaking to the right person about the right subject (for example the Change Management process).

3. Ask if the interviewee was briefed by their manager (they should have been!). If they were not, brief them now. For example, you could explain that a senior manager has asked you to look at the change management process to see what's working well and what isn't so that we can make some improvements, and that the interviewee's name was put forward as someone with a good understanding of the subject.

4. Ask for brief details of the interviewee's role, and how long they've been in that role. This will help you to put into context the answers you receive. This is really important. For example, a lack of understanding or knowledge from a process owner or process manager indicates a different kind of underlying issue than lack of understanding from someone with a more limited role within the process.

5. Explain that the interviewee is anonymous for the purposes of the assessment process. In other words, no remarks made during the interview will ever be directly attributed to the interviewee. It's all 'in confidence'. You have to mean this, and you have to stick to it. To digress slightly, this is one of the reasons why you should always interview at least two people on any single subject. If only one person is interviewed, it will be clear who the source of comments is.

6. Equally, if the assessment process was designed so that interviewees are NOT anonymous (not always something I'd recommend), then make that clear.

7. Finally, explain how the ITSM assessment interview will proceed. Give an example of a question or statement and the kind of response you're looking for. For example, you could explain that you are looking for a yes / no answer, or a rating from 1-5 - or whatever method you are using. When I use my preferred 1-5 rating system (Likert scale) I print a brief explanation of each rating and place it in front of the interviewee so that they can consider where they are on the scale. They will quickly get used to that and won't need to refer to the written explanation after the first few questions.


Zeno: Powerful process improvement made simple

Summary

We are really doing just a few simple things to get the interview off on the right foot:

- gauge the mood and potential bias of the interviewee

- establish rapport

- give the interviewee the information they need

If you have any questions, please let me know in the comments or contact me via LinkedIn.

If you need assessment software, you might want to see what Zeno has to offer.

Was this ITSM Assessment interview article useful? If so, please share with your network...

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
The Interview Environment for ITSM Maturity Assessments

The Interview Environment for ITSM Maturity Assessments

This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's necessary to make the interview as successful as possible. 

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

Let’s get into it!

The Interview Room

I hope you like the picture at the top of this page. Believe it or not, it isn't absolutely necessary to have an inspiring view of distant mountains when conducting an ITSM maturity assessment interview. Which is handy, if you happen to be in Lincolnshire or Florida.

What is vital, though, is that the interview environment is reasonably private so that the interviewee can speak openly (we have covered the subject of respondent anonymity in other posts). Once the environment is right, everything else goes more smoothly.

Interview Environment Practicalities

This is what you need to do to get the interview environment just right:

- Don't carry out the assessment interview in an open office. Make the interviewee confident that privacy is assured. 'Job shadowing' and informal conversations related to the assessment can of course take place in open offices, but don't discuss specific contentious issues outside of the formal interview

- The interview room should be soundproof if possible, again so that conversation can flow freely. Other common sense environmental factors (light, temperature) should be considered

- Ensure that power sockets and a network connection are available for your lap top and that of your co-interviewer if there is one

- Interviews can last up to two hours, depending on what is being assessed. Ensure your interviewee is comfortable, allow for quick and disciplined breaks if necessary

- Ensure that you have to hand hard or soft copies of any documentation you want or need to discuss (such as reports or a formal process description)

- Make sure that your access to software (such as an ITSM tool) is arranged and tested before the interview

- You may not know before an interview that you'll need to access ITSM software. Sometimes an interesting and relevant point will come up that can be dealt with right away as long as there is access to the right tool

Zeno: Powerful process improvement made simple

Common Sense

Although these suggestions are mostly basic common sense, I have seen interviews go less smoothly than they should for simple, avoidable reasons. The most common problem is that no room has been booked, or an unsuitable room has been booked. It's possible to conduct interviews in break-out spaces or staff rest areas, but they don't provide the ideal interview environment.

That kind of unpreparedness worries me when it happens in client organisations: it sends a signal that the assessment and improvement efforts aren't really important enough to make professional arrangements.

Please let me know in the comments or on LinkedIn if I've forgotten anything or you have suggestions.

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]
IT Assessment Interview Preparation

A Guide to IT Assessment Interview Preparation

How to Prepare for an IT Assessment Interview

This post describes IT Assessment interview preparation from the perspective of the interviewer. You will learn some simple guidelines on how to prepare to conduct an interview as part of an ITSM process maturity assessment.

Maturity assessment is of course much more than just a series of interviews, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover these subjects:

Without more ado, let’s get into it.

Assessment Interview Preparation

First, the bad news: if you’re going to ask questions about an IT subject (often a process) you need a sound grasp of that subject.

If that sounds daunting given the complexity of IT, let me explain that there are various levels of expertise and with good preparation you can approach any assessment confidently. I am not an expert on every single IT process in the sense that I would be able to step into a job in – for example - Release and Deployment Management, and immediately be effective. That process is a discipline with its own body of knowledge, tools, technical practices, technical shorthand (jargon), etc.

So what level of expertise is needed? The answers fall into two main categories.

1. Understand as a management discipline, not a technical one

Let’s suppose that you will be assessing the ITIL process called Release and Deployment Management. If this is not a main area of expertise for you, go to the source material (ITIL Service Transition 2011 CH4.4) and review it. You can start by grasping the main objectives of the process, and then drill down into the actual practices. I like to use Mind Maps for this.

The process has its own jargon and if you come across something you don’t understand – DML, perhaps, or V-model – make sure you learn a clear definition or at least get the concepts clear in your mind. Any difficult terms or acronyms can be colour-coded in the mind map or other note-taking software. The source material should include the right definitions, but if not, apparently there’s something called the internet that may be able to help…

You will come away from this exercise with a refreshed understanding of Release and Deployment as a management discipline, but not as a technical practice. So what happens if the interviewee starts to ‘talk tech’ in response to a question?

How to deal with ‘tech talk’

Don’t try to shut the interviewee down, because the devil is often in the technical detail. There are a few things that you can do though:

  • Be open and admit that you are not a technical expert in this area. Be sure to strongly imply that the interviewee is a technical expert. Flattery will get you everywhere
  • Ask your interviewee to explain the issue in lay terms. Then guide the conversation to the substantive effects of the technical issue, not the technical details themselves
  • Make sure to take good notes that you can follow up later
  • If appropriate, arrange to follow up with the interviewee so that you can see the issue at close quarters

2. Make sure you understand the meaning of the question you're asking!

If you are using a standard set of statements such as those available through Axelos (or better still, through Zeno) as a basis for ITIL process assessment, make sure you look at each statement long before you start the interview.

If not, you will come a cropper.

It’s not that the statements and questions are technically difficult to understand, it’s that some of them are semantically difficult. And if you, the assessor, can’t understand the question, what chance has your interviewee got?

Resource: ITIL Maturity Model: High Level Self-Assessment Service *

* Alternatively, you could wait for the feature rich but ITIL Maturity Model compliant Zeno SaaS version, due in Q1 2018. Just sign up here for early access and more info:

Zeno: Powerful process improvement made simple

My approach to this is to practice putting a question or statement in my own words, retaining the essence but making the question as simple as possible. 

It may seem that I’m exaggerating the problem here. But imagine you’re an hour into an IT Service Continuity assessment where the conversation has been non-stop, and the interviewee is beginning to flag. The statements are starting to sound a bit similar to the interviewee's ears because they are each focused on a slightly different detail of the process. Then you put this corker to your unsuspecting victim:

“IT service continuity management works closely with projects and changes and other design activities for new services, to ensure that the designs for the service continuity of new services match the needs of the business and are consistent with continuity design criteria and architectures and business continuity requirements.”

Yes, that’s a real ITIL process assessment statement and by no means the worst example of its kind!

Bear in mind that you will be looking for an answer from the interviewee that indicates whether or not they agree with the statement. But in the unlikely event that they understood the statement on first hearing it, exactly which part of it will they respond to? There are three approaches to preparing for the problem of over-complex statements and questions.    

How to Simplify Statements and Questions

        Approach 1. Break the statement down into its constituent parts and find the essence of what you want to know. It will often be the case that you’ve asked about several parts of the question before. For instance, in our example you will already know how well the process a) matches business needs and b) is consistent with design criteria and architectures and c) is also consistent with business continuity requirements.

Therefore, we can simply ask about the crux of the matter: "does ITSC work well with project management?". This is a more human way of putting it and gets at what you need to know in a straightforward way.

        Approach 2. With more experience you’ll realise that the statements are sometimes grouped into themes. For example, there are a series of ITSC statements around ‘how well does the ITSC process work with X?” (so, the general theme here is ‘interfaces with other processes’). 

Then we just repeat the technique of 'Approach 1' with the other processes that are referenced in the assessment statements, as follows:

- does ITSC work well with financial management to ensure that investments required by continuity plans are consistent with budgets and financial plans?

- does ITSC work well with capacity management to… etc.

        Approach 3. Finally, you could choose to move away from the ‘official’ ITIL statements altogether. This has the advantage of enabling the design of more user-friendly questions; however, it has the disadvantage of moving away from the objective ITIL standard and therefore makes it harder to benchmark or compare against other maturity scores. In Zeno we provide both: a drastically shortened set of easily understandable questions, plus the ‘standard’ statements with some built-in contextual help. The circumstances will dictate which is best to use.

IT Assessment Interview Preparation: not rocket science

Assessment Interview Preparation Summary

Assessment Interview Preparation Summary. CLICK FOR FULL SIZE IMAGE

Hopefully by now you’re convinced that one key to successful IT assessment interviewing is that you need to study the statements and questions you’ll be asking!

All of the above may seem a little complex, but all will become clear when you have the assessment questions in front of you, and have picked out and understood the themes and repetitions

It’s not rocket science, but like any other area of endeavour, hard work and practice pay off. The hard work comes in when you make sure you are up to speed with the subject you’re assessing (and by reading about techniques in articles like this one). You can do practice runs in your head or in reality, or you can conduct an actual assessment interview. A great way to learn is to sit in with an experienced assessor, watch and listen.

If you have any questions on IT Assessment interview preparation , feel free to ask in the comments below.

Zeno: Powerful process improvement made simple

>