Distributing LNA Questionnaires, Analysing & Communicating the Results

One of the most frequently used methods of Learning Needs Analysis (LNA) is questionnaires. In last month's article I outlined bst practice in designing a questionnaire to identify learning needs. In this article I discuss how best to distribute questionnaires and analyse and communicate the results.

Distributing LNA questionnaires

When distributing questionnaires to identify learning needs, focus on the 'WIFM' factor - 'What's in it for me?' If people do not understand the reasons for the questionnaire, what the benefits are of completing it and how the results will be used, it is hardly surprising that you may achieve a low response rate.

The more advance warning people have that they are being asked to complete a questionnaire and the reasons why, the higher the response. 

Consider how you might react yourself to an unsolicited questionnaire.

Prior to distribution, it is also helpful to hold a focus group or groups with representatives of the people who are likely to complete the questionnaire. The focus group will help you identify the likely issues that may potentially cause barriers to completion of the questionnaire. For example, one focus group I held recently highlighted the need to include out-reach and part-time workers in the LNA survey and came up with solutions for doing this.

Best practice is to involve people's line manager in engaging employees in undertaking the questionnaire. The topic can be raised by the manager at team briefings as well as one to ones.

A full explanation of how to complete the questionnaire, when and where it needs to be sent in reply and what will happen to the results needs to accompany the survey. It is also helpful to put a reference number or email address if there are any queries.  I suggest that you allow two weeks for completion. Any longer and it is likely that people will forget the questionnaire. Any shorter and they may not have time to complete it.

Some organisations that have a Learning Management System (LMS) make effective use of this to log who the questionnaires have been sent to. In addition LMS can allow organisations to follow-up the questionnaires with a reminder email to people who have not returned their survey. This is providing that questionnaire is not confidential, in which case a reminder is not appropriate. LMS can also be used to log the results of the LNA.

Analysing results

As we have seen in earlier articles, the results of the LNA questionnaire analysis are only representative if the questionnaire itself is robust and sound.  If you have used a biased rating scale for example as part of the questionnaire, the results will in turn be biased.  Statistically, you need at least 100 responses for the results to be valid. This may not always be possible with smaller sample sizes, so do ensure that either before or after the questionnaire has taken place you gather qualitative data via one to one discussions or focus groups to support and substantiate the findings.

Throughout the data gathering phase of the LNA, do keep in mind how you will analyse the data as well as to whom you will be presenting your results.  One organisation we know of sent out a questionnaire about learning needs to all its 1900 employees. The questionnaire contained many open questions to gather views and opinions. The authors had not considered how they would present the resulting data before they undertook the research.  It took one person many weeks to sift through the results and draw worthwhile conclusions. 

Here are some tips before you begin to analyse the information you have collected from customers:

• Keep the objectives for the survey always in mind.

• Keep the analysis as simple as possible, don't over-complicate things

• Ensure that the data you have is fit for purpose

• Keep an open mind - unless the purpose of your research is to prove a particular theory, do not start the analysis thinking you already know the conclusions.  This will cause bias in your analysis as you will probably subconsciously manipulate the findings to your prior conclusions

• Ask someone else who has not been involved in the project to review your analysis before you present it to check that it is logical and that the findings make sense

• Do not be afraid where data is inconclusive, to instigate further research to clarify areas or probe further particularly if important organisational decisions ride on the research

Quantitative data

Questionnaires and other forms of survey provide data that can be analysed. The easiest form of analysis is to turn the data into percentages or mean scores. This allows comparisons to be made.  When analysing:

• Use a spread sheet to enter data.

• Check that you are adding the correct data to arrive at percentages or means

• To arrive at a mean score, multiple the number of respondents for each score, add the total and divide this by the total number of respondents.

Watch out when using mean scores alone however, as they can hide the range of responses  - e.g. the range of scores in the above example is from 1 to 10 showing that there are a wide range of opinions.

• When presenting figures decide whether you wish to round scores between 0.51 to 0.99 up to the nearest whole number and round them down if they are lower than this or to present the figures to the nearest two points e.g. 3.59

• Make sure the questions and answers match

• Note sample size - any sample under 100 lacks statistical validity, the greater the sample size the more valid the results. Make sure you put the sample size (n) on all the figures you present

• Turn data into charts and diagrams. Information is easier to understand when it is visual e.g.

• Where surveys have included open ended questions, analyse these too and present these as statistics e.g.

Areas suggested for improvement (n = 120)
Improved speed of response to course booking requests    46%
More product based training     31%
More visits to customers     13%
Web-site more easy to navigate     8%
Other         2%

• Structure your findings so that they are logical and easy to follow. Here is an example of a report structure:

Executive summary
Background
Objectives of Survey
Methodology
Sample size
Detailed Findings
Conclusions
Recommendations

Presenting results

Whenever you undertake a LNA questionnaire, the chances are that you need to communicate the results and engage people in plans for improvement in the future.

When presenting the results of a LNA questionnanire some advice is, think:

• Who

• What

• How

• When

Typically when preparing a presentation, we focus on what we need to say, rather than thinking who is my audience? Thinking about your audience and their needs helps you decide what information they need to hear. This in turn dictates how you present and when.

The style of presentation needs to fit the audience. For example, I recently presented LNA survey results to a Board of Directors. This was an information giving session. The presentation was a summary using three Powerpoint slides.  Prior to this we had facilitated a meeting with the sponsor and the project team which took a day. Here we presented the feedback and facilitated discussions on the conclusions, recommendations and action points the team wanted to come up with. This was very much an interactive, two-way process.  The objective was to ensure the project team members took ownership of the results and made a commitment to act on them.

Whether a short presentation or a facilitated workshop, here are some tips when presenting results:

• Use diagrams and charts where possible - people interpret results more readily when they are presented visually. Here for example is a histogram representing the length of time in which people have been studying for a professional qualification. Study for professional qualification in years.

• Use lots of white space for impact - make the presentation and report clear and easy to read. It is better to have a few images or paragraphs on a page, rather than clutter the report and make it difficult to read

• Use colour to bring the data to life - it is more memorable and adds interest

• If using Powerpoint, allow one slide to take 60 - 90 seconds on average to explain. Don't overload your audience with information. It is better to have fewer slides and greater discussion about the research, rather than 'Death by Powerpoint'

• Prepare a separate document summarising the results that can be given out at the end of the presentation. This stops people in the audience reading the presentation document whilst you are speaking

• If your audience is very much into detail, provide the tabulations related to the research as a separate document so that they can look at the data after the presentation

• Ensure where possible that you present results face to face so that there is an opportunity to ask questions and discuss outcomes. If appropriate take one session to present the results and another to discuss reactions so that people have time to reflect. Develop a plan of action as a result of your presentation.  This should include a plan on how other people will be informed about the results of the research and subsequent actions.

Possible reactions to the results

When presenting results, be aware that whether positive or negative, you will see a range of reactions to feedback at the presentation and probably afterwards:

DENIAL -   Not ME! That's not true. That doesn't happen here.

EMOTION -    How dare they say that? How can they think that after all I've done for them?

RATIONALISATION -  Ah, may be, BUT ...    Well we've been so busy.

ACCEPTANCE -   Well, yes, it's true. Yes, we could do better

CHANGE -    I know what I am going to do. We can put this right by ...

Denial is a very usual reaction to feedback. You may see people ignoring the results or assuming that the findings do not relate to them. In denial, people need evidence, facts to help them see that the findings are relevant.

You will probably see a range of emotions in people when they hear LNA feedback - if it is positive feedback people can become embarrassed, if it is areas for improvement, people can become angry, tearful or withdrawn depending on their characters. In this phase it is helpful to give people time to reflect and see the feedback in perspective, before asking them to put together an improvement plan.

In rationalisation people tend to justify away the feedback. Typically, people will pick holes in the research methodology or make excuses for their behaviour. To help people move through this stage, be future orientated, 'so given that we are so busy, what do we need to do differently next time'?

Once people accept the feedback they need support and encouragement to bring about change.

You are likely to have a wide variety of reactions therefore to the results of Learning Needs Analysis survey results. It is useful to be aware of possible reactions and how to overcome them, prior to presenting the results. One set of survey results that I presented recently was highly critical of the courses on offer and how people could book onto training.  I knew that the people I was presenting to had been responsible for the current training and development offer and booking system. The key here was not to 'massage' the results of the feedback however to minimise possible reactions. This might give you peace of mind in the short-run but will mean that the voice of the employee is not heard.  Position the feedback as an opportunity for improvement, to better the learning and development offer for the whole organisation and ultimately improve business performance.

Key learning points

Provide a full explanation of how to complete the questionnaire, when and where it needs to be sent in reply and what will happen to the results

Use visuals and graphs to present and order your data

When presenting the results of the survey, tailor your presentation to the audience. Think:

• Who

• What

• How

• When

Be prepared for the likely reactions to the results of the survey

Present the findings as an opportunity for improvement

Sarah Cook is Managing Director of the leadership and service excellence organisation, The Stairway Consultancy.  She can be contacted on: 01628 526535 sarah@thestairway.co.uk.

You may publish this article in whole or in part. The only requirement is that if in print the article must state - article by Sarah Cook, Stairway Consultancy Ltd, www.thestairway.co.uk

If you use this article in whole or in part electronically, the requirement is that you inlcude the following code on the page:

The code should look like this:

Article by Sarah Cook of the Stairway Consultancy. Specialists in Leadership, Customer Services, Team Building and Personal Effectiveness.

If you do not do this you will infringe our copyright