It is a sign that the company is more concerned with image than with reality. That’s not a good thing for either employees or the company. Among other things, it means the company will sacrifice real and meaningful improvements to the working environment, in favour of a positive press release.  

Most employees see through the public relations spin. The disconnect between employees experience and the stellar results required of being labelled one of the Best Companies to Work For, does not cause employees to suddenly change their experiences. Though it does cause them to question  the credibility of the Executive team and Human Resources.

As much as employees may be aware of the disconnect between public relations and reality, they may not be aware of how and why it occurs. Here are three big reasons:

1. Improper statistical analysis and interpretation.

When employee survey data is analyzed, statistical significance testing is used to identify important areas where your company (and all other companies participating in the survey) do well or do poorly. This significance testing is used again to identify important differences between your company and those used for comparisons. The problem is that statistical significance is not a measure or test of importance – ask any statistician. Most of what is identified as important in statistical significance testing is trivial – of no practical importance. Much of what is critical to your success is passed over as not statistically significant.

This means that all the strength and weakness areas and comparisons to other companies is nothing more than an extensive statistical fairy tale. Statistical tools and computations are used to give the appearance of being scientific, but it’s junk science, used to sell surveys rather than inform the business. Certainly the fairy tale is entertaining, perhaps even a little mesmerizing, but it is still just a fairy tale with the same correlation to reality. If survey results and subsequent HR strategies seem disconnected from what is going on in the busines, it is probably because the employee research has gone down the rabbit hole and enterred Wonderland – the only place where statistical significance can be used to determine what is important.

2.  The belief you can lead by following.

Comparing company performance to others is rooted in the belief that the best way to be a leader is to follow others. This is the foundational improvement strategy of those that engage in these comparisons.

For example, consider a company identified as being low (as compared to the Best) in an area said to contribute to employee engagement. Forget for the moment that the difference is probably no more than a fairy tale (see point 1 above). Because of the lower scores in this area, the recommended improvement strategy is to raise performance in this area to bring it up to the level of those identified as Best. In other words, become a leader by following the leader.

Meanwhile, organizations displaying real leadership have moved on, pursuing initiatives that actually make a difference to their employees (as oppossed to someone else’s). Meanwhile, our example company is left destined to chasing ghosts and sacrificing real engagement in the process.

3. Publicity is more important than employees.

Being labelled one of The Best Companies to Work For is great publicity. The company can put Best Company logo’s on its web site and recruitment advertising. A trip to accept the award is usually involved  for the executives responsible. Some stories about the company and its commitment to employees will be featured in the media.

Publicity is certainly a good thing, but not when the cost is sacrificing the prospect of obtaining reliable information concerning the strengths and weaknesses of the employer/employee relationship. Yet this is precisely what is happening. Employees are asked to take a back seat to a positive press release. Any company that has its priorities so misplaced is not  likely to be a good company to work for let alone one of the best, it is likely one of the worst.

The best companies to work for? These are companies that:

Live in the real world. As uncomfortable as it may be from time to time, the best companies believe that real issues and concerns need to be raised and are prepared to deal with them head on. They have left the world of fairy tales and make believe behind and are actually engaging employees in the work.

Are prepared to lead. The best companies follow their your own path. Greatness doesn’t come from following the crowd. It comes from understanding what needs to be done and doing it.

Put employees before public relations. Real leaders don’t need endless amounts of positive stroking to feel good about themselves. Nor do they measure success by the level of press coverage. They measure it by results: improvement in levels of employee engagement, loyalty, productivity and satisfaction.

Almost all employee survey research, probably the research you are using now, assumes that what is important to people can be determined by the formula for standard error – significance testing.

If your research draws attention to what is statistically significant, then you are being sold fairy tales and it is costing you. It is costing you in deteriorating employee satisfaction and engagement.

The formula for standard error works well enough for polling, where research problemsfocus on how best to spin political messages. There is a big difference between running for political office and running a business.

For those of us in the real world, concerned less with spin and more with bottom-line results, confusing statistical significance with practical importance is a disastrous error. It means most of what you think your research is telling you is dead wrong. Unimportant factors and differences are labelled ’statistically significant’ and garner organizational attention. Critically important factors are labelled ‘not statistically significant’ and left to fester.

How can statistical significance determine what is important to people?

It can’t.

When you are ready to leave The Cult of Statistical Significance, and begin identifying the real issues and problems in the employer–employee relationship, give us a call or drop us an e-mail. Leave the fairy tales behind. A better organization is just around the corner.

VoE replaces meaningless statistical significance tests with tests of real world or practical significance. Rather than produce an endless list of statistically significant issues, VoE identifies a clear and concrete set of priorities that can be used to drive sustainable change aligned with business priorities.

There is no statistical gobbledygook. Only hard science applied to the problem of identifying issues of real and practical importance to your employees.

That’s the new world of VoE. Developing sustainable, practical improvement solutions for business. Your business.

The Old World of Employee Research

Typical employee research conducted using standard survey research techniques can determine how many people hold certain attitudes or if the differences in attitudes between departments is statistically significant, but standard survey research can’t determine whether any of this actually matters or is of any practical importance. This is because typical survey research techniques are enumerative.

Enumerative research is a formal term within the statistical sciences. Polling is a form of enumerative research with which most of us are familiar. Enumerative research is only capable of describing conditions. Specifically, enumerative research:

• Determines, within a level of statistical confidence, the number of objects (people) in a population,
• Determines, again within a level of statistical confidence, the number or proportion of objects that have one or more characteristics (such as stated opinions or attitudes).

That’s it. In many applications, this may be enough. In advertising research, when seeking to identify how many people saw a particular advertisement, enumerative research is the right method. In an employee survey, you may want to know how many people are aware of the new benefits plan as well as how many people favour it over the old plan. Here again, enumerative research works well. But enumerative research can do no more. It can’t determine the effectiveness of an ad, or why or to what degree employees favour the new benefits plan, because there is nothing in enumerative research that can separate the wheat from the chaff, that can identify and focus on what is actually important to people.

The New World of Analytic Research

Of course, identifying what is really important to people is what most organizations are interested in. Determining counts with a certain statistical precision is usually less of an interest than determining why an ad was effective or not and to what degree in terms of generating sales. They want to know how important the change in the benefit plan is to people and how this may affect employee turnover.

Answering these sorts of questions is the purpose of analytic research. Voice of the Employee (VoE) and Voice of the Customer (VoC) are examples of analytic research as are most of the research techniques used in performance improvement methodologies such as Lean and Lean Six Sigma. Analytic research is the formal term for research designed, not to describe, but to explain and predict. Enumerative research describes conditions, analytic research explains why these conditions came about and what will likely happen if changes are not made. You are probably thinking that it is analytic research that answers the questions that business typically need answered.

VoE Difference #1: Providing Answers

You’re right. This is the first big difference between enumerative and analytic research – analytic research provides answers that businesses need answered. It makes a world of difference.

Demonstrating a statistically significant difference in an employee engagement score between two departments (outcome of an enumerative study) doesn’t in any way answer the question of whether the observed difference between the two departments is actually important (outcome of an analytic study).

This is why so much traditional employee survey research leaves executives the empty feeling that their questions haven’t really been answered. They wanted know why things are the way are, but received only a detailed statistical description of conditions. Senior management are apt to respond to an enumerative study with “So what?“.

With an analytic study, the typical response from executives  is “I see!“. That’s because VoE provides answers to the questions executives want answered. Why is staff turnover high? Why is engagement low? What issues are of greatest concern to employees? What changes would make the greatest difference to employee satisfaction and engagement? All questions that only analytic research such as VoE can answer.

VoE Difference #2: Providing Focus

Here is another world of difference between traditional employee survey research and VoE – VoE provides focus.

Traditional employee enumerative research is characterized by multitudes of statistically significant results. Issues and problems are apparently everywhere. This is because statistically significant differences are so easy to generate. Statistical significance is a function of the research design and most employee research is designed to yield plenty of results (albeit meaningless). This is great for appearances sake, as it implies those purchasing the research received value for the money expended (plenty of results for the dollar). Unfortunately, as almost all of these statistically significant results are meaningless, and the business ends up wasting time and money chasing ghosts. After a few years of wasted efforts, executives are bound to notice and over time, the annual employee survey begins to suffer from declining credibility.

In analytic research such as VoE, the number of findings or significant results are fewer in number. That’s because VoE identifies what is really important people – statistical significance is replaced with real world, practical or economic significance. Obviously those things that are really important to people are fewer in number than all the possibilities. With VoE the number of findings is reduced, but those findings represent what the business really needs to pay attention to if it is to improve performance.

As a result, improvement efforts focus on the important stuff. People begin to notice a difference. Things get better – attitudes toward work and levels of employee engagement rise. When you think about it, what other way could employee engagement improve other than by addressing the issues and concerns of importance to staff?

VoE Difference #3: Driving Business Improvement

Analytic research differs from enumerative research in another way. As Dr. Deming, who first formally defined the distinction between the two research approaches pointed out, enumerative research is useful only when taking action on a population whereas analytic research is useful when taking action on the cause and effect system that produced the population. As confusing as this sounds, in employee research it means:

• traditional enumerative employee research is most useful when you want to sell an existing business program, policy or process to employees,
• analytic research is useful when you want to change existing business programs, polices or processes that affect employees.

Both approaches seek improvement but the strategies are polar opposites. Traditional employee research supports push strategies that attempt to change employee attitudes by spinning positive messages. By definition, push strategies only address symptoms. While effective, such strategies yield only short term improvements, because, no amount of promotion or spinning of messages can sustain positive attitudes in a bad system. Sooner or later, reality catches up with the messaging and a crisis ensues.

This, by the way, is why enumerative research, such as polling, is used so frequently in political contests. The research is used to craft political messages that sell a candidate and only long enough to carry through until election day.

VoE is different world altogether. VoE research supports pull strategies that attempt to change employee attitudes by changing the system in which people work. By identifying what is really important to people, VoE identifies the root causes affecting employee performance, attitude and engagement. Thus VoE drives change in the business by identifying the needs and requirements of the people who work in it. The result is long term sustainable change that makes a difference to the business and its employees.

The New World of Voice of the Employee

VoE represents a whole new world for the HR professional. One that for the first time, allows HR to  fill the promise of effective Organizational Development (OD). OD, after all, was about changing organizations, not changing people. VoE  is the only research methodology designed to gather employee information specifically for this purpose. It drives long-term sustainable change by identifying what factors are impacting employees and making clear focused recommendations for change.

The new world of VoE, therefore, is the real world of business. Research that identifies what is really important to people as opposed to statistically significant. Research that drives change to support meaningful and sustainable improvement. Research that contributes to a more effective, efficient and flexible organization. Isn’t that what employee research is supposed to do?

Traditional research, based on polling methods, support strategies that push or force feed solutions on to people. That’s why polling is used so frequently in political contests and advertising research, where the data is used to spin messages that are pushed onto the public.

An organization responding to the needs of employees needs something very different – an honest assessment of the strength and weaknesses of your relationship with employees. These needs can only be met through Voice of the Employee.

VOE  provides information that drives improvement to process or service design. Improvements that yield changes that employees want and demand – that are pulled by the employee rather than pushed by the company.

Research for the Real World

Traditional customer research (of which polling is an example) uses statistical significance to highlight issues and concerns. As any manager knows, what is statistically significant is not nearly so important as what is significant to the customer or to the business — real world, economic significance. This is the biggest difference between traditional employee research and Voice of the Employee. Traditional research identifies what is statistically significant; VoE identifies what is important!

Doesn’t statistical significance measure importance? Not at all. Equating statistical significance with importance or real world, economic significance is and example of:

(i) scientific/statistical incompetence or

(ii) simple fraud.

Statistical significance is the probability of detecting a result using an assumed theoretical probability distribution as a benchmark of expectations.  Now if that is what you are looking for – great. For those less interested in detectability estimates based on assumed theoretical probability distributions and more interested in finding what is really important to people, then analytic methodologies such as Voice of the Employee are the only scientific approaches capable of delivering the results you are looking for.

You can’t be customer driven if you can’t tell the difference between the wheat and the chaff – between what is really important to customers and what isn’t.

Information for Action

Traditional employee research techniques (such as polling) can tell you how many employees feel this way or that about some organizational characteristic. But what traditional research cannot identify is why people like or dislike the characteristic or what can be done about it. You have probably noticed your employee research has a so what quality to it, leaving management wondering: What am I supposed to do with this?

Voice of the Employee, however, is designed specifically to identify issues and concerns that of real and practical importance to both you and your customers. This is information that is specific, concrete and of real world importance - information that can be used to drive improvements in your products, services and programs. Information for action.

VoE not only produces information intended to support management action but also supports an accurate set of priorities. If you are using traditional customer research, you probably have an overwhelming numbers of statistically significant results. So many, it is hard to know where to start. Of course, most if not all of these results are of no practical importance. This is why statistical significance is used, because it generates so many ‘significant’, albeit meaningless, results.

In contrast, VoE offers  fewer significant results but these few results will be of real world significance to you and the business. This gives you a  clear and concise set of improvement priorities that will actually make a difference.

Employee Driven Organization Development

Some organizations believe they engage in employee driven change simply because they conduct employee research and respond to it. Organizations using traditional research are more driven by meaningless statistical significance tests than by the real needs of employees. The only voice that is being heard is that of an assumed theoretical probability distribution. This results in an increasing disconnect between the business and its customers. Organizational development initiatives must be pushed onto employees. Why? If these initiatives are really what people wanted, why the hard sell? Shouldn’t evidenced-based initiatives be pulled by employees? Yes, they should.

Employee driven organization development responds to the voice of the customer. This voice is embedded into the design and delivery of OD initiatives. The result are initiatives that people demand – without all the pushing. Reputation grows as the business is perceived as effectively responding to employee needs.

All this is predicated on having a research method capable of identifying what is important to employees and what isn’t. Traditional customer research can’t do it.

Voice of the Employee can .

This work is now recognized as the global best practice in customer research for product, service and program design. Japanese success in the automobile, electronics and heavy industries cannot be understood without understanding what Dr. Akao did for building products and services that actually produced customer demand or pull.

Different Business Questions, Different Research Methods

Dr. Akao was Professor of Industrial Engineering at Tamagawa University in Tokyo, served on numerous committees of the Union of Japanese Scientists and Engineers (JUSE), the Japaneses Standards Association (JSA) and the former President of the  Japanese Society for Quality Control (JSQC). He knew that identifying what is really important to customers required a whole new approach to capturing and analyzing customer information.

A student of Edwards Deming during Japan’s quality revolution, Dr. Akao understood the difference between enumerative and analytic research methods. Traditional customer research, such as polling, is enumerative; it can determine how many but not how come or why. For example, enumerative research counts how many many people said this or that in response to a survey question. In contrast, analytic research is less concerned with what people say and more concerned with what they do – identifying those things sufficiently important to influence behavior. Different research methods to answer different research questions.

Dr. Akao wanted to know how important various product and service characteristics were to customers. Enumerative research can’t answer these types of questions. But analytic research can because it is designed to separate the wheat from the chaff – to identify those things that have real value for customers. To accomplish these ends, Dr. Akao adapted analytic research methods to the task of customer research and Voice of the Customer was born.

This leading edge work has now been adopted for employee research in Voice if the Employee (VoE).

Voice of the Employee: An Analytic Research Methodology

Voice of the Customer then,  is analytic research applied to the task of identifying real employee needs and preferences.  Enumerative research, such as polling, identifies what is statistically significant; analytic research, such as VoC,  identifies what is economically significant.

Different Research for Different Purposes and Different Questions

Clearly there is a role for both enumerative and analytic research methodologies in business. Enumerative methods are very good for answering enumerative questions, describing how many people can recall seeing a particular ad for example, is an important question in advertising research. In the same way, determining how many people read the company newsletter is a good application of typical employee research/surveys.

Analytic research in contrast, is good at explaining why sales are down or why people didn’t respond to an ad – or why they are or are not reading the company newsletter. It is not surprising then, that as important as some enumerative questions are, the vast majority of questions that are important to business are analytic. You can’t run a business by describing conditions – you must be able to explain and predict conditions so as to make improvements.

This is one reason that VoE has grown in scope and sophistication since Dr. Akao first gathered and analyzed customer verbatim statements and organized them into design requirements. Voice of the Employee is now the  best practice in employee research because it is the only research methodology capable of answering the questions businesses need answered.

VoE s has proven itself invaluable with companies like Bridgestone and Toyota. Let us show you how invaluable it can be for you.

Part of the move is driven by criticism of traditional employee research from the scientific community, that points out that most employee research misuses scientific methods to lend an air of credibility to what is pure pseudo-science. Another reason for the move is the recognition by organizations that traditional employee research methods simply don’t seem to work all that well and that over time, creating  a gap between HR understanding of organizational issues and organizational realities. 

1. Criticism from the Scientific/Statistical Community

A big reason for the move to VoE is attributable to recent, well founded, attacks by the scientific/statistical community on the widespread misuse and overselling of traditional customer survey research. Traditional research based on enumerative methods (such as polling) simply can’t deliver what is promised. The enumerative emperor has no clothes.

Misuse of statistical significance as an indicator or measure of practical  importance.

Traditional customer research typically uses tests of statistical significance to highlight differences among departments, branches,  products, customer segments, time periods or geographical regions. Differences between customer segments, for example, are said to be important if the differences are statistically significant to some confidence  level such as 95% (the infamous 19 times out of 20 statistic). Management is expected to pay attention and to respond to these statistically significant results.

Of course, statistical significance has nothing to to do with importance and is of little economic or scientific relevance. Unfortunately, that hasn’t stopped some  from selling the fiction that it does. Recently, a detailed and brilliant attack from the scientific community has come from authors Stephen T. Ziliak and Deirdre N. McCloskey  in their highly recommended: The Cult of Statistical Significance: How the Standard Error Cost Us Jobs, Justice and Lives. The title says it all. Organizations that use statistical significance as a measure of importance are literally flaying at windmills and destroying their relationship with customers in the process.

Fortunately, the efforts in the scientific/statistical community are having an impact. In realizing the shortcomings of enumerative approaches, businesses are turning to analytic research methodologies such as Voice of the Customer with its sound scientific basis and focus on practical, real world or economic significance, to properly identify what is, and is not, important to customers.

Flagrant overreaching.

The scientific/statistical community is also taking direct aim at many of the claims made by those selling traditional customer research. Among the two most outrageous:

• that  comparisons can be made among companies to produce a ranking of performance (i.e.; 50 best or top 100),
• that statistical models can predict employee loyalty, satisfaction or engagement (i.e.; 12 things that predict loyalty).

Unfortunately, the simple truth is that both claims are well beyond the ability of the statistical sciences. What is being sold here is pure snake-oil.

Corrupt Comaprisons (i.e. 50 Best)

Creating corrupt comparisons, occurs when the results of one company are compared with the results of a group of companies (i.e; 50 best). For example, comparing your employee engagement score with the employee engagement scores for a comparison group. Conclusions are drawn that usually take the form of  a ranking, where one company is said to be higher or lower in the ranking than others. (We are in the highest quartile!). Such conclusions are examples of overreaching and are complete nonsense, but they sell.

Some of the snake oil involved is in having you believe that a single number, your employee engagement score in the current year for example, can be taken as an indicator for organizational performance generally. Comparing your measures to similar measures of comparative organizations compounds the error while promoting the delusion that scientific comparisons are being made. (Such comparisons also make use of statistical significance testing which adds a whole new layer of nonsense to the process.)  This is the scientific equivalent of deducing a trend from a single data point – a mathematical miracle of immaculate conception.

Basing employee and organizational development decisions on such fanciful conclusions is dangerous. It leads the organization to pursuing directions that have no potential for improving the employee relationship while ignoring others that do. The net result being wildly misdirected efforts. Small wonder research by the GAO in the United States has indicated that the least effective customer strategies are built on a foundation of organizational comparisons (see Benchmarking Customer Satisfaction Doesn’t Work).

Meaingless Models (i.e.; twelve factors that predict customer loyalty)

Meaningless models are those that use statistical correlations to predict employee engagement, loyalty or satisfaction. Examples are identifying the ten factors determining employee loyalty or the twelve drivers of employee engagement. If a statistics course was part of your education, there is a good chance you can recall one phrase your instructor pleaded with you to remember: correlation is not causation! It was good advice.

Customer research claiming to determine or predict levels of customer loyalty or satisfaction confuse statistical models of correlation (enumerative models) with models of causation (analytic models). Like shell game artists, they take advantage of the confusion to pretend one thing is another.  Such models of employee loyalty, engagement or satisfaction are as scientific as models linking declines in polar bear populations with decreases in the number of accordion players in Missouri. The two are correlated, but the relationship is a fantasy and the model, meaningless.

Edward Tufte, former Professor of Public Affairs at Princeton and now Professor Emeritus at Yale University, recently coined a word to describe such flagrant overreaching - economisting. (See Beautiful Evidence, Graphic Press LLC, 2006.) Derived from a German root, the word equates to the deliberate act or process of converting limited evidence into grand claims … see also the German meaning of mist. Such claims he notes, are evidence of mediocrity. We couldn’t agree more.

2. The Growth of Analytic Methods in Business

Another driver behind the move to Voice of the Employee research is the rise and acceptance of other analytic research methods in business, especially Lean, Six Sigma and Continuous Improvement. As companies gain experience with these improvement methodologies, their understanding of the differences between enumerative and analytic  methods has grown. With this comes the recognition that traditional employee research, which is enumerative, is simply incapable of answering the questions that are typically put to it.

For example, Lean Six Sigma improvement models call for improvement teams to gather the voice of the customer at some point in the improvement process – usually early on in the improvement model. Some companies equated this call with the traditional customer research being used. As experience with Lean Six Sigma grew, however, these companies began to realize that voice of the customer meant just that –  analytic research methods that properly capture customer requirements embedding them into product and process improvement activity.

At the same time, understanding of the distinction between enumerative and analytic methods has also grown. Organizations are simply doing a better job of appreciating the appropriate application and limitations of each research approach and doing a better job at aligning the research question with the research methodology used to answer it. (For more on the enumerative-analytic distinction in the research science see; VoE: Answering the Important Questions).

The Move to Voice of the Employee

The move to Voice of the Employee then has been prompted by three factors:

1. Strong criticism from the scientific/statistical community of the widespread misrepresentation and overreaching in traditional enumerative employee research. This is evidenced by the misuse of statistical significance and the making of corrupt and meaningless comparisons and models.

2. Growth and acceptance of other analytic methods through Lean, Six Sigma and Continuous Improvement. The use of these techniques has increased awareness of the distinction of enumerative and analytic research methods generally along with an increased appreciation for  the importance of aligning the research methodology with the research question at hand.

3. The continuing trend in moving away from traditional customer research to Voice of the Customer. With this has come the realization that the flaws in traditional customer research mirror those in traditional employee research. Both are enumerative research methodologies and cannot answer the really important questions the business needs answered.  You can’t separate the wheat from the chaff with enumerative methods or measures of statistical significance. It requires the use of real world economic significance and Voice of the Employee.