Analyst ethics?

27Sep10

Quis custodiet ipsos custodes?

Who guards the guards? This is a question that my peers and I have been discussing for some time. We are in a quandary. Analysts and analyst relations live in a symbiotic relationship where we need each other to thrive – you could argue that we are each others PR team in that the success of one group inevitably helps the other.

Conversely, the effect of a ‘bad apple’ impinges on us all. And so the question remains – who polices the analyst industry?

I have often felt that it is self-policing and that the success of firms is bred on trust, integrity, respect and knowledge. Therefore what should we do if we feel that a firm has lost the respect of the AR field? This is a serious question and one that shouldn’t be taken lightly – if we make a post such as this, are we damning a business out in the open when this should be done behind closed doors?

In private, my peers and I regularly do not recommend working with a specific analyst firm (who for the purpose of this blog shall remain nameless, although those in the industry may be able to guess who I am speaking about). They are seen as guns for hire (note this past link refers to Aberdeen but the firm I talking about here is another one completely). In the past when this has happened with other firms, people simply stopped spending money with them. But now, I believe we are at a tipping point. Money for analysts and AR is scarce and justifying their worth and independence is something that I feel I should not have to do.

Let me give you a couple of scenarios and you can judge whether this firm is unethical or simply doing business.

Scenario 1 – the award

A client of mine was contacted out of the blue to inform them that they have won an award for being the best at their chosen field. Naturally they were excited, and despite the feeling that they felt they deserved it, were delighted as not many people realised in the mass market they were as good as they were. Call it ‘the worlds best secret’. However, they were then told that in order to receive their award they would have to pay a significant sum of money. When they refused to do this, the award was presented to somebody else who now appeared first.

The question I asked was if a company deserved to win the award, then surely they should still be first despite the fact that they have not paid “to pick it up”?

Scenario 2 – the review

In this scenario, the analyst firm was commissioned to survey two phone handsets.  In an independent evaluation, the handset that came out top was the one produced by the commissioning firm. Now I have no problem with the firm coming out higher, the issue I have is that even though the analyst firm did mention the report had been commissioned, it failed to say by whom. The sceptics amongst us may presume that it was worried that, given the result, everyone would think the report was biased.

We have two situations here that call ethics into question. Can a firm write positively about vendors only if they have been paid to do so and secondly, can a firm be commissioned to write independently about a vendor if they have been given money in advance?

The simple answer is that no firm should provide positive commentary because they have been paid. Advocacy is determined by value not by money. It is the very basics of integrity that is at the heart of this matter and unless this process is stopped, and vendors refuse to spend money on such obvious gimmicks, as the market perceived value of these reports is so low, then I fear this practice will continue.

The specific firm in question has been deliberately omitted. Indeed you can argue that this practice isn’t limited to them. Which analyst firms disclose the clients mentioned in a report? None of the large analyst firms do that. Can we really say that white papers and commissioned research are ethically so different?

Of course, this last statement was deliberately provocative. Any firm can produce independent commissioned research. Take Freeform Dynamics for example who have a business model comprising of patronage. Sure IBM, Microsoft and others have paid them money to write research – but they have never told FD what the conclusions should be, what methodology they should practice, what criteria they should take note of, what weightings to follow or what assumptions to make.

My thoughts aren’t alone on this. Merv Adrian and Curt Monash recently discussed ‘White Paper Sponsorship and Labelling’. It was clear that much of what is commissioned by analysts was used as marketing material which should by default be taken with a pinch of salt. However, the need for authenticity still remained. Indeed one of the telling comments in the post by Carl Olofson is good practice for us all:

My recommendation for analysts is that any sponsored documents should clearly show their sponsorship right up front with the title and author, and also to consider that you do have some influence over how it is displayed. At IDC, we have forced vendors to make changes to their references to our work when the references improperly imply that the documents in question explicitly endorse them or their product (IDC has a firm policy prohibiting the endorsement of vendors or products; we may endorse or affirm technologies, methodologies, or trends, but in a vendor-neutral way).

To prospective buyers, I would say this: assume that the analyst reports page, like the rest of the website, is designed to impress you. If you just skim the titles of documents on this page without looking at them, you are allowing the vendor to create cumulative impression that could be misleading, just like looking at the very impressive customer logos that all vendors show in their presentations as if they are all strategic customers, when really, many are not. Caveat emptor, my friends.

I strongly believe that any time a vendor gives money to an analyst firm for paid-for work it needs to be referenced and authentic. Our industry needs firms it can trust to help them make informed decisions – without them, and if it’s reputation is tarnished by the acts of a few who practice methods I don’t agree with then the whole market fails.

Who analyses the analysts?

We all do.

Vendors need to stop looking for the easy marketing win by using these firms and the analyst houses themselves need to be more transparent. As to the specific firm in question – sort it out!

Post originally published on the IIAR



3 Responses to “Analyst ethics?”

  1. 1 Marc Duke

    Jonny

    Great post, my only thoughts on this one is that people are people and while in ideal world we all should play nicely that is never the case. I remember once having a very heating ‘discussion’ with a very senior analyst (no names etc..) where I contended that every analyst firm has its price and when money changes hands it will have some impact on independance. He fiercly disagreed.

    At the end of the day its a personal thing, can you/one/be part of a firm that behaves in an ethically questionable way?

    I’ll finish with an addage from Giles Fraser founder of Brands2Life who gave my first ever consultancy role ; ‘be nice to people when you are on the way up, as you will see them all on the way down!’

    May sound trite but I am a firm believer that what goes around does come around.

  2. I’ve written about forty white papers, all commissioned by vendors. Each one clearly displays the sponsoring company’s name and logo next to a caption that reads: “Sponsored by” on the cover. I never take an assignment that is a pump piece and I rarely review product features. My approach is to explicate a topic or concept that the vendor is aligned with: dynamic load balancing for Teradata, guided analytics for Spotfire, etc. Rarely will the name of the vendor appear in the paper except in the conclusion, if at all. Vendors tend, in the review process, to insert material that violates this approach, but I don’t allow it. If they insist, they are free to publish the paper without my byline.

    I’m not an analyst firm, so I am able to be very selective who I work for and under what terms. Unlike an analyst at one of the big firms, I’m not forced to produce material in subjects I have only a superficial knowledge of. How some of these “analysts” can be spread so thin and pose as experts really troubles me. To me, that is an even greater ethics problem than pay to play, which is usually pretty transparent.

    Neil Raden
    Hired Brains

  3. Sorry that I missed this piece, Jonny. Well done, thoughtful discussion – and thanks for referencing the discussion Curt and I had. Some of this is going to be moot for me after Jan 3 when I start at Gartner, but I’ve made certain all the posts on my blog clearly indicate whether vendors I write about are clients or not.

    Neil’s final point above is also a good one – we have some pundits whose expertise is indeed spread thinly. Still, in training, I used to tell analysts we hired at Giga and Forrester that “it doesn’t say ‘expert’ on the card, it says ‘analyst.'” The point is that analysts bring some expertise to the table – but that can erode with time as analysts get farther away from practice. (Unlike consultant/analysts like Neil, who is a practitioner as well.)

    Analysts combine the expertise they do have with methodology that is honed over time, a connection to data about real practice through interactions with users, formal processes for primary data collection, and a full-time association with the market they play in. Their value is in outsourcing the work a purchasing firm might have to try to reproduce without most of those assets to rely on.

    If we do our work well, are diligent in our sourcing of information, and adhere to the standards that we espouse, then we deserve the confidence our clients place in us. And if we fail, then we get what we deserve. Hold us to those standards, and demand that transparency.

    Best,
    Merv


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: