ITSM Assessment: How many interviewers?

How many interviewers needed for an ITSM Assessment?

This article discusses the question of how many interviewers are needed to carry out an ITSM Maturity Assessment. It includes pointers on the benefits and drawbacks of three different approaches, and shows how to get a successful outcome whether you have one, two or no interviewers involved. 

It's important to understand the different interview options that are available when planning an assessment. There are two dimensions in play: quality and cost. However, as we’ll see, how many interviewers are involved isn’t the only factor governing these two variables: with good organization and preparation the different approaches can all be made to work.

Let’s look at all three options in turn.

The 2-Interviewer Solution

The set-up

In a two-interviewer set up, the first interviewer usually focuses on engaging with the interviewee by putting the questions or statements to them. The interviewer will have the questions and / or statements relating to the process, function or other IT element being assessed in front of them – usually on a laptop and usually in a spreadsheet.

This first interviewer may go ‘off-piste’ and delve deeper with follow-up questions. He or she will then capture the main response of the person being interviewed. By ‘main response’ we mean either

  • The ‘yes’ or ‘no’ response… or
  • A score on a Likert scale or similar number-based scale (separate article coming on using the Likert scale)

The first interviewer, then, does most of the talking, and only needs to capture a ‘yes’, a ‘no’ or a score (1 -5 in the Zeno software, see screenshots below) relating to each question or statement.

The second interviewer often joins in the conversation to follow up on a specific point, but their main task is to capture details of the response being given, plus their own version of the yes, no or number score.

Zeno: Powerful process improvement made simple

Advantages of using two interviewers

  • More detail is captured (because the second interviewer can focus on typing)
  • Each interview can take less time and flow more freely when the main interviewer doesn’t have to regularly stop to take notes
  • Interviewers can compare scores and investigate any differences in their results. In effect, there are two pairs of eyes on any problems
  • The chances of having in-depth practical knowledge of the subject in hand are doubled
  • The chances of spotting any discrepancies or important nuances are doubled
  • Follow-up tasks such as finding corroborative evidence are more easily shared

Disadvantages of using two interviewers

  • Although the interview may be shorter, this doesn’t fully compensate for the extra effort and cost of using two interviewers
  • If not handled correctly, the interviewee may feel intimidated by facing two interviewers

The 1 Interviewer Solution

The set-up

In a one-interviewer set up, the interviewer focuses on engaging with the interviewee by putting the questions or statements to them and will then capture their main response (yes, no or score). The interviewer also needs to capture notes on the responses given so that they can be followed up or reconsidered later. The notes are also used to give context for an answer when it is found later that two or more interviewees have given very different responses.

Advantages of using one interviewer

  • Less costly in terms of time and money
  • Can be easier to establish trust and empathy when only two people are involved

Disadvantages of using one interviewer

  • Less detail is captured (and / or the interview takes longer because of the need to take notes). Note taking in spreadsheets tends to be awkward and adds to the time taken
  • Interviewers don’t have a second opinion at hand
  • There is less chance of having in-depth practical knowledge of the subject in hand
  • The chances of spotting any discrepancies or important nuances are halved
  • Follow-up tasks such as finding corroborative evidence may require more effort

How to mitigate the disadvantages of a 1 interviewer set-up

Mitigation of the disadvantages of using one interviewer boils down to two things –

  • Preparation in terms of deepening understanding of the IT element being assessed (covered in this post). This will help to minimise any awkward pauses, apart from being a basic requirement of an interviewer!
  • Preparation in terms of capturing responses and taking notes

Here’s how we do this in Zeno, but you can use the same kind of approach using a spreadsheet.

1 Capture the 'main response'. In this case the user clicks on 'Agree' and either adds a comment or moves on to the next statement. The following sequence of actions would be used irrespective of how many interviewers are involved.

Zeno - how many interviewers - 1

Click for larger image

2. Prepare to add additional comments. In this case the user clicks the 'down' arrow to add a comment.

Zeno - how many Interviewers - 2

Click for larger image

3. Add additional comment. The comment is added and saved. Also note that lots of space has been set aside to capture detailed notes, without having to scroll right. In the case of Zeno, these notes can be directly exported into a report (after some editing if necessary)

Zeno - how many interviewers - 3

Click for larger image

The No Interviewer Solution

The set-up

In a no-interviewer set up, the assessment questions and statements (or a link to them) are sent to each respondent. The participants respond to each statement and question, add their own explanatory notes, and complete the questionnaire in their own time (subject to any project constraints). The responses are then collated.  

Traditionally, self-assessment has only been used with a cut-down set of statements (for example, in the case of ITIL) but as we shall see, self-assessment can now be extended much further.

Advantages of using no interviewers

  • Much less costly in terms of time and money
  • Scalable - the administrative overhead of questioning twenty (or any number) of respondents is hardly any more than questioning one
  • No scheduling headaches
  • Potentially larger (and more statistically significant) data sets
  • Easier to capture different perspectives (for example, the customer perspective) because any number of respondents are available
  • Respondents can go into detail about what they perceive as problem areas
  • Discrepancies in scoring or important nuances that appear in respondent’s notes can be analysed offline

Disadvantages of using no interviewers

  • No guidance from an interviewer if the meaning of a question is unclear

How to mitigate the disadvantages of a no-interviewer set-up

Mitigation of the disadvantages of using no interviewers can mostly be resolved by using in-context help. In a spreadsheet this would be done by adding comments to cells. In Zeno, comprehensive in-context help is built-in.

So: how many interviewers?

Although we've used Zeno to demonstrate some of the principles, the same effects can be achieved with spreadsheets. The quality and usefulness of an ITSM assessment no longer has to be so strongly influenced by how many interviewers are used. With good guidance and some practice, effective assessments can now be carried out more frequently and more cheaply.

Zeno: Powerful process improvement made simple

Want more?

Maturity assessment is more than just a series of interviews, of course, and you can read about the wider context in the Visual Guide to ITSM Maturity Assessment.

In this series on interviewing we’ll cover:

If you have any questions, please let me know in the comments or contact me via LinkedIn.

If you need assessment software, you might want to see what Zeno has to offer.

Was this article useful? Please share with your network...

How to Use the ITIL Maturity Model and Likert Scale to Assess IT Processes
IT process assessments have for many years used maturity models to rate the capability and maturity of processes and functions.[...]
Are you planning to assess and improve ITSM processes in 2018?
4 things you must do to maximise ROI in ITSM process assessment.IT leaders assess and improve ITSM processes for a[...]
ITSM Assessment: How many interviewers?
How many interviewers needed for an ITSM Assessment? This article discusses the question of how many interviewers are needed to[...]
How to start an ITSM Maturity Assessment Interview
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
The Interview Environment for ITSM Maturity Assessments
This post describes how to prepare the ideal interview environment for an ITSM Maturity Assessment. It includes reminders on what's[...]
A Guide to IT Assessment Interview Preparation
How to Prepare for an IT Assessment InterviewThis post describes IT Assessment interview preparation from the perspective of the interviewer.[...]

About the Author Dan McCarthy

Dan is the co-founder of Zeno. He likes finding new ways to improve processes and working life in general.

follow me on:
>