Selecting the Best Techniques to Identify Learning and Development Needs

What is the best method to gain information to identify learning and development needs?

What are the advantages and disadvantages of the different methods available?

In this article Sarah Cook provides an overview of the range of methodologies available to the training professional to identify training and development needs as part of the Learning Needs Analysis (LNA).

Once you have agreed who you should include in your sample, you are in a position to select the best measurement techniques and determine the sample size to use to gain feedback.

Defining the target audience

As we have seen in earlier articles in this series, one of the first steps in the LNA process is defining the target audience for the learning needs analysis. This involves establishing whether you wish the LNA to focus on a particular strata or population of the organisation or whether the LNA needs to be more generic.

Once you know who you are targeting, you can then select the measurement methodology that best suits your target population. Questions to ask are:

• What are the objectives of the LNA?
• Who are the target population?
• In order to satisfy these objectives with this target population, do I need to obtain information that I can quantify or do I need views and opinions?
• Given the target audience, what methodology/ies is / are most suitable?

There are two main forms of measurement techniques:

• Quantitative
• Qualitative

Quantitative methods allow you to quantify opinions. They use fact based methods to help identify the skills, knowledge and behaviour gaps of the target population. This involves using hard data as an objective means of masking a judgement. The results of quantitative research methods can often be presented in percentage terms.

Qualitative methods allow you to understand employees (and their customers') perceptions of the experience needed to perform effectively in the role in terms of skills, knowledge and behaviours. Qualitative techniques involve expressive methods to help identify employees' views and opinions and the reasons behind them. The resultant data allows organisations to better understand the whys, hows and whats of the desired performance.

Here are the main methods that fall under each category:

QUANTITATIVE METHODS

Analysis of performance reviews
Written and email questionnaires
Telephone surveys
Mystery shopping
Structured interviews
Assessment centres
Observation
Analysis of customer surveys, customer complaints and compliments

QUALITATIVE METHODS

Employee focus groups
One to one interviews
On-line discussion forums / web-based focus groups
Conversations
Exploratory interviews

There are advantages and disadvantages to each of these methods:

QUANTITATIVE TECHNIQUES

ADVANTAGES

ANALYSIS OF PERFORMANCE REVIEWS

• Allows you to review existing data on learning and development needs
• Allows trainer to measure and monitor overall trends
• Useful at beginning of research phase to help decide focus
 
DISADVANTAGES

• If not current, may not be representative of current employee or customer views
• Requires more follow up to verify exact need

WRITTEN QUESTIONNAIRES

ADVANTAGES

• Can reach a large amount of people
• Provides quantitative data
• Useful way of quantifying opinions expressed in employee interviews / focus groups

DISADVANTAGES

• Response rate can be low - therefore can be costly
• Results obtained only mirror the questions asked
• Little opportunity for the person completing to express opinions in full

EMAIL SURVEYS

ADVANTAGES

• Can reach a large amount of people
• Provide instant response
• Can provide instant results
• Provide quantitative data
• Useful way of contacting employees at a distance

DISADVANTAGES

• Can be considered as Spam and a nuisance, therefore response rate can be low
• Difficult to obtain information in depth on employee opinions
• The longer the length of the survey, the lower the response rate

TELEPHONE SURVEYS

ADVANTAGES

• Provide instant response
• Provide quantitative data
• Useful way of contacting employees at a distance

DISADVANTAGES

• Many employees dislike intrusion
• Not easy to gain trust
• Difficult to elicit information on employee opinions in a short space of time if employee not prepared

STRUCTURED INTERVIEWS

ADVANTAGES

• Gain employee responses face to face
• Interviewer can explain intentions behind the questions
• Allows sample to be taken of target population
 
DISADVANTAGES

• Requires skill in interviewing techniques
• Time consuming
• Open to bias from interviewer
• Structured nature of questions does not allow for in-depth discussion

MYSTERY SHOPPING

ADVANTAGES

• Provides objective view of the service
• Allows service providers to be assessed using measures which are customer-centric
• Highlights lack of consistency 

DISADVANTAGES

• Questions often formulated by the trainer not the customer, therefore can be biased
• Can be seen as 'spying' by staff
• Low acceptance by staff of results - the mystery shop is only one occasion therefore not representative of total customer experience

ASSESSMENT CENTRES

ADVANTAGES

• Allows organisation to assess using objective methods, employees' skills, knowledge and behaviours
• Uses a variety of assessment methodologies
 DISADVANTAGES
• Timely and costly to run
• Involves a minimum number of employees
• Not statistically valid

OBSERVATION

ADVANTAGES

• Allows trainer to observe employees on the job
• Helps identify required skills, knowledge and behaviours
 
DISADVANTAGES

• Open to observer bias
• Requires skilled observation
• Data sometimes difficult to analyse

ANALYSIS OF CUSTOMER SURVEYS, COMPLAINTS & COMPLIMENTS

ADVANTAGES

• Provides data on customer opinion of service provider
• Can identify trends

DISADVANTAGES

• May not be representative of total customer opinion as  customers who have had 'magical' or 'miserable' experiences are more likely to complete surveys, complain or compliment the service provider
• May not include internal customer opinion

QUALITATIVE METHODS

EMPLOYEE FOCUS GROUPS

ADVANTAGES

• Allows organisation to explore customer opinions
• Allows in-depth discussion and frank feedback
• Useful in exploring sensitive issues
• Consultative / inclusive
• Useful for gathering sensitive data
• Helpful in exploring a range of views prior to using structured, quantitative methods

DISADVANTAGES

• Needs good facilitation to ensure all employees attending can give their opinions
• Not statistically valid
• Does not allow opinions to be quantified. Data sometimes difficult to analyse

ONE TO ONE IN-DEPTH INTERVIEWS

ADVANTAGES

• Allows trainer to explore employees' opinions
• Allows in-depth discussion
• Consultative / inclusive
• Useful for understanding opinions and feelings
• Helpful in exploring a range of views prior to using structured, quantitative method

DISADVANTAGES

• Time consuming
• Not representative of total opinion
• Not statistically valid
• Can be influenced by interviewer

CONVERSATIONS WITH EMPLOYEES

ADVANTAGES

• Provide informal information
• Quick 'dip stick' approach

DISADVANTAGES

• Anecdotal, not representative
• Open to bias and misinterpretation

EXPLORATORY INTERVIEWS

ADVANTAGES

• Useful to discover employee opinions prior to a full-scale research study
• Open-ended questions allow interviewer to explore employees' feelings as well as opinions
• Interactive

DISADVANTAGES

• Time consuming to set up and run
• Not representative of total employee group
• Open to bias from interviewer

How to ensure that you are using the right measurement techniques

As demonstrated above, each of the methods has both advantages and disadvantages - so how do you ensure that you are using the method that will get the best results?

Firstly consider which method is most suitable to your target employee group. Is it feasible for example to hold an employee focus group?  Given that the number of participants is eight to ten, when and how could this number of employees be made available for an hour's discussion?  How many focus groups would need to be run?

Secondly consider a combination of both qualitative and quantitative techniques. This ensures that you have a good mix of methodologies. So for example, you could hold a series of employee focus groups to establish learning needs. Based on these findings you could then develop a questionnaire to be sent to a wider population in order to quantify the opinions.

Determining sample size

Once you have established your LNA methodology, you need to establish your sample size.  How can you ensure that your research is clear, credible and impartial and properly reflects your employee population?

If you have an employee base of 10,000 people, do you need to speak to them all for your survey results to be correct?  The answer is no, with a 5% margin of accuracy you would only need to speak to 370 of your employees to discover their learning and development needs.

The main point to take away is that sample sizes are worked out using statistical formulae and do not work in proportion to each other. So for example if you had a population of 5,000 employees (half the size of the above) the sample size with a 5% margin of error needed to be statistically valid is 357 people.

Population size  Number of respondents to be statistically accurate 
(to 5% margin of error and confidence level of 95%)

10000    370
5000    357
2500    333
1000    278
500    217
250    151

There are some useful web-sites that help you calculate the sample sizes you need to be statistically accurate. For example: www.ncspearson.com/research-notes/sample-calc.htm and www.surveysystem.com/sscalc.htm

Sampling methods

The accuracy of learning needs analysis methodology is also affected by how 'random' your sampling was. So, for example if you conducted a telephone survey to sample the employees of a 24/7 call centre operation and only called employees during the day, your sample would be biased and not random.

There are three main types of sampling that you can use:

1. Random sampling - here you select customers to be contacted as part of the research on a random basis. So if you wanted to random sample an employee base of 1,000 people, you would choose a series of numbers at random to contact e.g. the 9th, 19th, 29th, 39th etc until you have a sufficient number.

2. Stratified random sampling - involves dividing your population into homogeneous subgroups and then taking a simple random sample in each subgroup. This assures that you will be able to represent not only the overall population, but also key subgroups of the population, especially small minority groups.

3. Non-random sampling

Non-random sampling because it is non-random, almost certainly introduces bias. It can be difficult to draw conclusions about the population based on information derived from a non-random sample, as samples are often unrepresentative of the population. So for example, asking the first ten employees who walk through the door to take part in a learning needs analysis will not be representative of the total employee base.

Generally speaking for quantitative methods it is best to work from sample sizes of at least 100 respondents, (less than this and your data is not statistically valid).  When sample sizes fall below 100, as is often the case with small groups of employees, best practice is to use a mixture of qualitative (opinion) as well as quantitative (statistical) methodologies to ensure you capture views and opinions as well as hard facts.

In conclusion

There are a wide range of methods available to assess learning needs. Each has both advantages and disadvantages so the best approach is to combine different methods to ensure an accurate picture of needs emerges.

Assess your current approach to learning needs analysis methodologies

Look at the following statements and assess your organisation's approach to leaning needs analysis methodologies:

Measurement Criteria:  In place / Not applicable / Needs to be actioned / addressed

We have identified the target population for the research  
We have selected measurement techniques that are relevant to our target employee  
We use a combination of quantitative and qualitative research techniques  
We have determined the correct sample size  
We have determined the employees who will take part in the learning needs analysis via random sampling  

You may publish this article in whole or in part. The only requirement is that if in print the article must state - article by Sarah Cook, Stairway Consultancy Ltd, www.thestairway.co.uk

If you use this article in whole or in part electronically, the requirement is that you inlcude the following code on the page:

The code should look like this:

Article by Sarah Cook of the Stairway Consultancy. Specialists in Leadership, Customer Services, Team Building and Personal Effectiveness.

If you do not do this you will infringe our copyright