A couple of weeks ago, Brand Strategy attended a Net Promoter conference in London, which left us with some food for thought. We covered it in the magazine’s leader, which subscribers can read here. We have also reprinted it below for non-subscribers:
**********
Brand Strategy leader, July: The full measure
Fred Reichheld, management guru at Bain and author of the seminal business book The Loyalty Effect, claims that he realised customer loyalty was in trouble when he went to an airport. As he was fast-tracked through immigration, he chatted with the man taking down his details. The man commented that Reichheld was listed as an ‘author’, asking what he wrote. “I write about customer loyalty,” he replied. “Oh,” the security guard quipped, “So, you write fiction….”
Reichheld’s response to this (conveniently) amusing tale was to develop the Net Promoter Score (NPS), the basis of his book The Ultimate Question. The idea is that you can determine the growth of a company based on the likelihood of your customers to recommend you. The theory has really taken off, with brands as diverse as LEGO and T-Mobile falling for its charms.
It is not just Reichheld who extols the virtues of what many call ‘the ultimate metric’. Bernhard Klein Wassink, senior vice president of global marketing for GE Real Estate, claimed at the European Net Promoter Conference last month that he was “extremely excited” by the effect on his business of heeding this particular strategy.
Reichheld, speaking at the same conference, added that he had been converted from a customer loyalty expert into someone who believed in the ‘golden rule’ – do unto customers as you would have done to yourself. He joked: “I have been repositioned. I am no longer the high priest of loyalty. I am now the godfather of growth.”
But not everyone is convinced by Reichheld’s ideas. Tim Keiningham, senior vice president at IPSOS Mori, along with a team of academics, has produced a paper in this month’s Journal of Marketing that claims: “We find no support for the claim that Net Promoter is the single most reliable indicator of a company’s ability to grow.” The article argues that the system is no better than any other measure already on the market.
But does this matter? The answer is probably not. While Keiningham’s statistical analysis might be correct – and it’s important that we should take no proprietary management tool as gospel – it perhaps misses the point. Brands everywhere need a simple method to point them in the right direction; if anything encourages them to cut down on long, pointless customer surveys and look more closely at what people really want, surely that is a good thing?
Another person who can offer insight and real-life practical case studies is Concetta Lanciaux, senior adviser at LVMH. She is running our first exclusive Brand Masters seminar on 27 September 2007. Don’t miss out. You can find out more at Brand-Masters.com.
Ruth Mortimer, Editor
**********
But Keiningham has more to say on the topic and wrote Brand Strategy a long letter explaining his views and research in more detail. His letter follows. Are any other readers keen to have their say? Do let us know – please comment below!
**********
From Tim Keiningham:
I read with interest your take on research that I conducted with marketing and loyalty experts, Lerzan Aksoy (Koç University), Bruce Cooil (Vanderbilt University), and Tor Wallin Andreassen (Norwegian School of Management), that pertains to the work of best selling author and loyalty guru, Fred Reichheld that is now available at the Journal of Marketing.
I would like to amend your interpretation of our findings. You state, “The article argues that the system is no better than any other measure already on the market.” The more correct assessment of our findings, however, is that we did not find Net Promoter to be a good predictor of growth at all. Using a methodology designed to replicate that of Mr. Reichheld’s research, it is quite strange that we could not uncover similar findings to those reported by Reichheld given the overwhelming superiority he reports with regard to Net Promoter’s linkage to growth.
The most troubling finding from our research, however, it that we uncovered very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, we were able to replicate a subset of Reichheld’s reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter was not superior to the ACSI. It is virtually impossible to imagine a scenario other than research bias as the cause of this finding.
This is a very serious problem. We expect published research to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science. It is ironic that this issue was ignored in your article despite this being in a field with professional researchers and codified industry rules of conduct.
Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.
Mr. Reichheld made extraordinary claims regarding Net Promoter. In the process, he stridently attacked the work of his competitors; his most withering critiques were aimed at the ACSI, the primary source of competitive advantage for a competitor to Satmetrix and Bain & Company’s loyalty practice, CFI Group (note that neither I nor my co-authors have any affiliation with the ACSI or CFI). Given our findings with regard to Net Promoter vis-à-vis the ACSI, this adds yet another dimension to this issue.
While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased. Research bias is totally unacceptable and inexcusable.
Sincerely,
Tim Keiningham
July 12, 2007 at 8:04 pm |
I agree with Tim, my research does not support the Net Promoter approach.
July 19, 2007 at 7:37 pm |
Ruth,
Thank you for your practical and thoughtful posting. I attended the Net Promoter conference in NYC earlier this year and was impressed with the success stories. The debate regarding the Journal of Marketing research is interesting and caught my attention last week. After ready it I came to the same conclusions as you did in your original posting. My thoughts are noted here:
1. It appears that the Journal of Marketing research referred to shows that Net Promoter is, at a minimum, equivalent to ACSI in correlating to growth. (Figure 1). That says that a simple metric driven by a single question is at least as accurate at predicting growth than a more complex algorithm driven by multiple questions.
2. After discussing the claims with Satmetrix, they stand behind the research. Claiming a research bias is a pretty serious allegation. Both Fred and Satmetrix have spent over 10 years studying the correlation between loyalty and growth and helping companies implement programs that improve loyalty and drive growth. The result of this effort is apparent in reviewing the case studies presented at the Net Promoter conferences: http://netpromoter.typepad.com/npc_london_2007/
3. This conversation is missing the point that is attracting business leaders. The value of Net Promoter is its simplicity. Unlike complex satisfaction indexes, Net Promoter is easy to understand and take action on. Simplified surveys drive higher response rates, a better reflection of the customers that matter, rather than random sampling. Using real-time reporting, leaders can get information in the hands of employees that can address detractors, move the passives and nurture the promoters.
At the end of the day, it’s what companies do to improve loyalty that drives growth. Net Promoter offers an approach that is understandable by everyone in the business, not just the statisticians. This gives an organization a rallying cry for building customer centric organizations.
Full Disclosure: During this conversation I was contemplating joining the Satmetrix team. Having experience deploying Net Promoter programs, and researching the market further, I decided to join Satmetrix as CMO: http://www.satmetrix.com/news/pressrelease_2007-07-17.htm
Let’s continue the conversation.
July 23, 2007 at 2:52 pm |
During last few months I have observed that the debate on NPS is becoming a kind of stereotype. Those who have doubts about its validity try to put forward their research work and there is a sizeable lot of it now. And in response the converts put forth only arguements about its simplicity, appeal to business leaders, years spent in research and few examples of success. Often most of the converts are someway part of or related to the same group which is commercilising the concept. This is not very conducive to healthy debate and proponents of NPS are advised to counter the research with research and not with rhetorics
July 25, 2007 at 10:54 pm |
Ms. Eastman’s statement that research bias is a serious allegation is absolutely correct. So we are waiting for a good answer. The argument that Net Promoter is “at least” as good as the ASCI is irrelevant. Reichheld claims his research reveals no relationship between the ACSI and growth.
In the Harvard Business Review article that introduced Net Promoter, in the book, The Ultimate Question, and in presentations regarding Net Promoter, the American Customer Satisfaction Index has been specifically mentioned as not linking to firm growth by Reichheld. The book, The Ultimate Question, argues that the ACSI does not yield much insight into loyalty or growth, noting that “investors rarely waste money on standard satisfaction surveys” as a result (The Ultimate Question, p. 86).
Similarly, an article in the Harvard Business Review states (p. 49): “Our research indicates that satisfaction lacks a consistently demonstrable connection to actual customer behavior and growth. This finding is borne out by the short shrift that investors give to such reports as the American Customer Satisfaction Index. The ACSI, published quarterly in the Wall Street Journal, reflects customer satisfaction ratings of some 200 U.S. companies. In general, it is difficult to discern a strong correlation between high customer satisfaction scores and outstanding sales growth.”
Furthermore, in a web-based presentation, Mr. Reichheld states that a “Bain team looked at the correlation between growth and customer satisfaction, and found there is none.” A scatter diagram was shown with the X-axis labeled “American consumer satisfaction index annual growth” and the Y-axis labeled “Sales annual growth.” The R-square reported was 0.00, indicating no correlation whatsoever.
Given that our findings show that Net Promoter was not superior to the ACSI when using Reichheld’s best-case scenarios, it is virtually impossible to imagine a scenario other than research bias as the cause. This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.
It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld’s reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]
Sincerely,
Tim Keiningham