Designing Questionnaires to help Establish Learning needs

PDFPrintEmail

Sending a questionnaire to gather data to help identify learning and development needs is probably the most used method in the trainer or HR professional's repertoire. But what is the best way to design a questionnaire? What are the pitfalls to avoid?  In this article Sarah Cook describes how to go about developing this important, fact finding tool.

Questionnaires are a useful way of establishing learning and development needs. They can be used across an organisation or at a departmental / functional level to help identify current and desired training needs. However there are some common mistakes made in research. These are:

1. Asking too many questions
2. Asking questions that are not relevant to the respondent
3. Lack of responsibility within the organisation for acting on the results

Designing a questionnaire

When designing a questionnaire, many people make the mistake of making it too long, uninteresting and not relevant.  The results of the survey are therefore not helpful to the organisation: as they may be giving the wrong message or are not easy to interpret. The resulting actions may be difficult to prioritise.

Best practice when designing a questionnaire is to first of all decide what you want to do with the results e.g.

- Do you want a picture of the current perceptions of the training and development supplied?
- Do you want to know what improvements could be made to the current offer?
- Would you like to know the priority training and development needs for the next 6 to 12 months? Or longer?
- Do you need to know the needs by department / unit or grade of personnel?

Once you know what you want to achieve from the questionnaire you can begin the design process.

Use of exploratory research

My advice, based on personal experience, is to use qualitative techniques to identify what is important to your potential audience and to design your questionnaire around these attributes. Many people make the mistake of designing questionnaires themselves without checking via qualitative research that the questions are meaningful to respondents. One useful way of ensuring that you are intending to include the right questions is to hold a focus group with a cross-section of respondents to elicit their opinions. The results of the focus group should determine what questions you should ask in the questionnaire.  I will provide more information about focus groups in a forthcoming article.

Golden rules of questionnaires

Once you have determined what you want to establish in using a questionnaire and what types of questions to ask via the exploratory research, you will then be in a position to design the questionnaire.

Here are some useful tips when designing a questionnaire to be sent via post or email:

• Ensure that the questionnaire is easy to complete

• Make the layout of the questionnaire attractive and appealing to the eye. Use plenty of white space and type size that is 
  at least 12 points type so that it is easy to see and follow

• Use plain English so that the wording is clear to all e.g. say 'Learning Needs Analysis ' not 'LNA' and 'Key Performance
   Indicators', not 'KPIs'

• Explain the benefits of completing the questionnaire in the introduction and position why you wants to hear respondents'
   views

• Reassure the respondent that the responses will be treated in confidence

• Make it clear what will happen to the results

• Make the instructions for completion short and to the point

• Include an addressed envelope or return email address

• Use questions generated from exploratory research by typical respondents in their language - avoid jargon

• Layout the questions in a logical sequence so that it makes sense to the respondent 

• Start the questionnaire with easy questions which the respondents can answer quickly

• Make sure the questions and answers match

• If you need to ask more personal details of the respondent e.g. which department they work in or length of time with the
   company, put these questions at the end of the survey

• Do not make the survey too long - response rates drop the longer the questionnaire is

• Decide what type of questions you will ask the respondent

• Fixed answers: e.g. 'Is it a good idea'? Yes or No

• Multiple choice: e.g. 'Which word best describes this'? followed by a range of options

• Free choice: e.g. 'Tick as many words as appropriate to describe..'

• Open-ended questions: e.g. 'please list your reasons'

• Ranking: e.g. 'Rank your preferred choice in order of preference'

• Rating: e.g. 'Spread the 10 points over the words you prefer'

• Don't use double-barrelled questions e.g. 'Did the training meet your needs and was it held at the right time'?  These
  should be two questions not one

• Don't be biased in the questions you use e.g. 'Do we offer average or good training and development currently'? This
  pre-supposes that the training is not poor

• Don't use leading questions e.g. 'Would you agree it is more convenient to be able to book directly on to the course'?
   Rather than 'Is it more convenient to be transferred automatically to the advisor'? as the former style of questioning
   leads the customer to answer 'Yes'

• Avoid ambiguous words e.g. frequent, often, many - you may have a perception of what these words mean which is
   different to that of the customer e.g. how often is 'often'?

• Randomise the order of your questions so that the respondent has to think of the response. When completing a
  questionnaire subconsciously the respondent follows a pattern. It's best to change the order of responses especially if the
  questionnaire is long so that the respondent does not follow a pattern

• Remember to allow space for additional qualitative comments

• Be conscious of customers with special needs and make arrangements so that they are not excluded from responding

• Select an appropriate time to send out the questionnaire. Avoid busy periods such as summer holidays or Christmas as
  you will receive a lower response rate

Response scales

One of the factors that you will have to decide in preparing a questionnaire is the number of open and closed questions that you are going to include. 'Open' questions allow the respondent to give their views and opinions. This can be useful but if the whole questionnaire is based on open questions, it will take a long time to analyse and it can be difficult to draw conclusive results. Best practice is to include an 70/30 split in favour of 'closed' questions.  This means that 80% of the responses are pre-determined in order to allow data to be collated and analysed in a systematic fashion.  For this reason, it is very important to do some exploratory research, otherwise the response options for closed questions may not be appropriate to the audience.

There are a number of different options when it comes to response scales. Below are a number of different possible approaches with my experiences of each.

The key options are

1. Verbal scales: where respondents are offered a choice of verbal responses e.g.

Completely dissatisfied
Quite dissatisfied
Neither dissatisfied or satisfied
Quite satisfied
Completely satisfied

This type of attitudinal response option is called the Likert scale. Each respondent is asked to rate each item on some response scale. For instance, they could rate each item on a 1-to-5 response scale where:

1. = strongly disagree
2. = disagree
3. = undecided
4. = agree
5. = strongly agree

There are a variety possible response scales (e.g. 1-to-7, 1-to-9, and 0-to-5). All of these odd-numbered scales have a middle value is often labelled 'Neutral' or 'Undecided'.

If you decide to use this type of scale, be sure to have an equal balance of positive and negative responses e.g.

Very unhappy
Quite unhappy
Neither happy or unhappy
Quite happy
Very happy

As in the rest of the questionnaire design, be sure to use descriptors that the respondent would also use.  Be wary of biasing the results by offering only one negative option e.g.

Poor
Neither poor nor good
Good
Excellent

Using an unbalanced scale such as this, organisations can claim to have 95% satisfaction ratings e.g. 95% of respondents scored the current training offer neither poor nor good, good and excellent.  This may not be representative of the respondents' true feelings.

The other problem to watch out for is the 'bunching effect'.  If you have only 4 or 5 options, most respondents are likely to select the option 'neither good nor bad' or the option 'quite good'. It is not typical to see many responses at either end of the scale.  There is quite a big difference for example in the response 'good' and 'excellent'.

If you do wish to use a verbal scale, we recommend using one that has 7 options e.g.

Very dissatisfied
Dissatisfied
Somewhat dissatisfied
Neither satisfied nor dissatisfied
Somewhat satisfied
Satisfied
Very satisfied

One organisation I worked with used a 9 point verbal scale:

Dismayed
Very dissatisfied
Dissatisfied
Somewhat dissatisfied
Neither satisfied nor dissatisfied
Somewhat satisfied
Satisfied
Very satisfied
Delighted

This allows the respondent more options and helps spread the range of scores.

It is also possible to use a forced-choice response scale with an even number of responses and no middle neutral or undecided choice. In this situation, the respondent is forced to decide whether they lean more towards the agree or disagree end of the scale for each item.

2. Numerical scales

Using numerical scales respondents are asked to rate attributes within a given range of figures e.g. on a scale of 1 to 5.

Give careful consideration to the choice of scale; if you ask people to rate on a scale of 1 to 5 they go invariably for the middle box.   Again, using this method, results usually fall between scores of 3 and 4.  As a typical example, organisations average scores will be say 3.75.  Some people dismiss the 5 point scale therefore because effectively you end up using a 2 point scale (scores of 3 and 4).

The same caveats apply when you convert verbal scales e.g. Likert scale to numerical scales e.g.

Very unhappy =1
Quite unhappy = 2
Neither happy or unhappy = 3
Quite happy = 4
Very happy = 5

The resulting average scores are likely to fall between 3 and 4.

In my experience it is better to use a 10 point scale (where 1 =low and 10 = high) to describe levels of satisfaction and importance than any other scale. Respondents (probably because of their experiences at school) are likely to give a score of 6 or 7 as average, a 9 for very good and anything 5 or below will be considered as poor.  The resulting data is spread over a wide range (usually scores range between 5 and 9, given that people rarely score 10 and are less likely to score 1 to 4), the data is also more easy to manage and interpret.

An alternative to a scale of 1 to 10 is a scale of 1 to 6. This avoids the problems with the middle box - if you ask people to rate on a scale of 1 to 5 they go invariably for the middle box.

Finally, do remember to include the option - Can't remember / Have insufficient information / Not applicable- especially if you are asking people to recall something that happened some time ago. Alternatively, put a note in the instructions to ask people who do not know the response not to complete the relevant questions.

Examples of LNA questionnaire

Here is part of a questionnaire intended to establish what respondents think of the current training offer and what the key areas of learning and development are needed for the next 12 months. This questionnaire was aimed at managers in each unit.

1. What training and development have you attended or undertaken in the last 12 months?  Please tick the following boxes as appropriate:

- Management development (there followed a list of workshops)
- Induction
- IT training (there followed a list of workshops)
- Personal professional development (there followed a list of options)
- Other (please state)
- Can't remember / Have insufficient information / Not applicable

2. Thinking about the areas where you have undertaken development in the past 12 months, how satisfied were you with the content of the training?

Very dissatisfied
Dissatisfied 
Somewhat dissatisfied  
Neither satisfied nor dissatisfied
Somewhat satisfied
Satisfied
Very satisfied 

- If you were somewhat dissatisfied, dissatisfied or very dissatisfied, please state your reasons why

3. Thinking about the areas where you have undertaken development in the past 12 months, how satisfied were you with the approach of the training? (i.e. how the training was run)

Very dissatisfied 
Dissatisfied
Somewhat dissatisfied
Neither satisfied nor dissatisfied
Somewhat satisfied 
Satisfied
Very satisfied 

- If you were somewhat dissatisfied, dissatisfied or very dissatisfied, please state your reasons why

4. Thinking of your unit as a whole, what internal or external changes are or could affect your business in the next 12 months?

 

- What are your business priorities for the next 12 months?

 

5. Given the anticipated changes and your business priorities for the next 12 months, what training and development do you anticipate that that the organisation needs to offer more of? Please tick the following boxes as appropriate:

- Management development (there followed a list of possible new courses)
- IT training (there followed a list of possible new workshops)
- Personal professional development (there followed a list of options)
- Other (please state)

Response rates

When using postal or email questionnaires you can anticipate a response rate anywhere in the region of 20 - 50%.  The response rate will be better:

a) If the topic of the survey is of particular interest to the respondent
b) If the letter accompanying the survey is addressed in person and sent by a named representative of the organisation
c) If you use an incentive to encourage response (e.g. entry to a free prize draw or first 100 questionnaires returned receive a free prize)
d) If you publicise the survey to remind people about the survey
e) If the survey is not too long - we recommend no more than 4 pages of questions
f) If you follow up with a reminder letter two to three weeks after the initial survey has been sent asking people who have not done so to complete the survey
g)  If you tell respondents that you will publish the findings of the research

Pilot

Once you have prepared your questionnaire, I strongly recommend that you pilot this in one small area. Explain that this area is a pilot and ask for their feedback on the questionnaire after they have completed it. This will allow you to see the likely responses and to make any adjustments that are needed to help you get a full picture of future needs.

In next months article I will describe how to analyse the results of questionnaires.

Key Learning Points

• Questionnaires are a useful way of gaining information about learning and development needs from a wide number of
   people in a short space of time.

• Best practice when designing a questionnaire is to first of all decide what you want to do with the results

• Always conduct exploratory research via such methods as focus groups amongst the target audience prior to sending
   out a questionnaire

• Best practice is to include an 70/30 split in favour of 'closed' questions

• Careful selection is required of the types of responses

• Pilot the questionnaire before it is sent out to ensure it is fit for purpose

Sarah is Director of customer care specialists, The Stairway Consultancy Ltd.  Sarah can be contacted on 01628 526535. 

You may publish this article in whole or in part. The only requirement is that if in print the article must state - article by Sarah Cook, Stairway Consultancy Ltd, www.thestairway.co.uk

If you use this article in whole or in part electronically, the requirement is that you inlcude the following code on the page:

Article by Sarah Cook of the Stairway Consultancy. Specialists in <a href="http://www.thestairway.co.uk/services/leadership-development.html">Leadership</a>, <a href="http://www.thestairway.co.uk/services/customer-service.html">Customer Services</a>, <a href="http://www.thestairway.co.uk/services/training-courses-in-team-building.html">Team Building</a> and <a href="http://www.thestairway.co.uk/services/personal-effectiveness.html">Personal Effectiveness</a>.

The code should look like this:

Article by Sarah Cook of the Stairway Consultancy. Specialists in Leadership, Customer Services, Team Building and Personal Effectiveness.

If you do not do this you will infringe our copyright

image description

"Over the years The Stairway Consultancy has become a key partner to our business. Their services have proved an invaluable tool in taking individuals forward in their careers and thus developing the organisation. Their approach to working in partnership has certainly helped us to achieve our objectives, and we value the contribution Stairway makes to the business. I always look forward to working with their professional, open and warm consultants, and would highly recommend The Stairway Consultancy to any business that is seeking to grow, inspire and develop their teams."

Learning and Development Business Partner
- Southern Housing Group
Looking for high impact training material?

We have a wide range of Training Games, Activities, Toolkits, Books and DVDs.

Training Shop