As companies look to survey their employees more frequently than ever before, they have become increasingly interested in the idea of statistical sampling. With sampling, you only send your survey to a small subset of the overall population. Sampling is made possible by the laws of probability, so in order for it to produce valid reports, two key requirements must be met. First, employees must be selected into the sample randomly, and second, the sample must be of a sufficient size.
While inviting just a sample of employees to a survey has many advantages highly relevant to an overall listening strategy, it also has some drawbacks that should be understood before deploying this technique.
1. The benefits of survey sampling
Because you are only sending your survey to a subset of your population, this means fewer employees set aside their regular work to complete a survey, and less effort may be required in the administration of the survey as compared to trying to reach the full population. This reduced administrative burden can mean surveys are done more quickly, more often, and even somewhat “under the radar,” if desired. These advantages are substantial, as sending a survey to the full population on a frequent basis may not be practical.
2. When it comes to sampling, size matters
It makes sense that larger populations require larger samples. Yet, the increase in size is modest and eventually reaches a maximum of just a few hundred employees. What surprises people is learning that proportionally larger samples are needed for smaller populations. For example, a 5% sample is plenty big for a population of 10,000, but not nearly large enough for a population of 100.
In practice, these statistical rules mean that sampling is not viable for small groups. To continue the example above, a group of 100 requires a sample of 79 to provide meaningful data, which is essentially the entire group itself. Even for a group of 500, you need a sample of almost half (217), and once you factor in a response rate, this means inviting well over half to participate, essentially negating the advantages of sampling in the first place. In fact, sampling really only makes sense for groups of at least 3,000 where you only need a sample 341, which could be doubled to 782 (or roughly 1 in 4 of your employee population) to account for even a weak return of 50%. If reporting results for groups of this size is aligned with your listening strategy, great. If not, sampling is probably not right for your organization.
3. Confirm your data needs in advance
A very important consequence of sampling is the lack of statistically valid data breakdowns. The issue is essentially this – if you create a sample for a group of 3,000 employees, the data you collect will accurately represent the aggregated views of those employees. But if you attempt to slice the collected data to see differences by job level, or functional area, you will soon realize that your sample is not large enough to accurately represent the views of such sub-groups. This drawback is often over-looked until the need for more granular data emerges, usually when results indicate a problem and one wants to identify its source. The challenge, of course, is that these breakdown groups are populations in and of themselves and need to be sampled for properly. It’s very important to account for these groups early on, but bear in mind that doing so can increase sample size requirements substantially and end up undermining the value of sampling in the first place.
4. Complex sampling requires careful planning
Multiple samples combined into a rollup group introduce additional complexity and require special handling. Without getting overly technical, the challenge is that groups sampled at different rates can’t be simply combined. For example, if you have a 10,000-person business unit that you sample at 5% and a 3,000-person business unit that you sample at 10%, you can’t combine these two to get an overall company-wide view without an elaborate weighting scheme.
5. Surveys as a driver for change
Sample surveys also tend to have weaker impact on organizational change, as compared to census surveys. This may be fine if you consider your employee listening program to be more of a monitoring activity, but to the extent you wish to drive change this can be a significant weakness. Because a census survey reaches everyone, they tend to create an organizational momentum or “buzz”, provoke feelings of employee involvement, and typically generate a far higher return rate. These are extremely valuable features of a successful company improvement program not typically found in a sample survey.
In the end, sample surveys can offer a lot of value, but like most good things also come with important trade-offs. If your primary interest is in monitoring high-level groups to provide direction for more focused follow-up activity, sampling can make sense. If you’re looking for a survey to provide detailed insights to drive change, even for smaller groups, it’s likely not.
For more details on sampling, including how to create a valid sample, sign in to your Willis Towers Watson Employee Engagement Software account, or contact us for a demo.
Adam Zuckerman, Product Leader
Willis Towers Watson Employee Engagement Software
Adam is responsible for the overall development and direction of Willis Towers Watson Employee Engagement Software. His goal is to create the world’s greatest software for delivering insight and enabling actions that enhance employee experience, company culture, and business performance. Outside of work, Adam enjoys off-roading in his Jeep and spending time with his family. Follow Adam on Twitter.