In praise of open source analysis

05Jun07

Robert Scoble in his blog today discussed how Hugh Macleod gives his cartoons away for free. As a result, he got so popular that now companies are willing to pay for him to draw cartoons for them.

Does this sound familiar?

Analyst houses such as RedMonk, Quocirca, MWD and Freeform Dynamics have been using this model successfully for years.

What amazes me is not that these companies do this but that so many others choose not to. In my opinion people the real value of an “analyst report” is the 80% of data that is not included in it.

To put it bluntly an analyst report is little more than a press release that screams to the reader – come and talk to the author if you want to know more.

James blogged about this earlier this year (along with James McGovern and Barbara at Tekrati) when they highlighted the open source analysis analysts (OSAA) wiki. I will be joining this debate but to kick-off here are a few of my answers to James’ questions:

What is open source analysis?
Analysts who give away their research for free (e.g. via blogs, free downloads, journals etc)
Why is it different from traditional industry analyst models?
Some analyst companies (like Datamonitor) have a transactional methodology as their business model and rely on selling their research
How do we best engage with open source analysts?
The same way as you normally would in so far as the focus of this engagement centres around what is important and timely for the analyst.

I believe that the dated model of transactional-based research will come to an end. Unless you are a strong quantitative player (e.g. IDC) or have a unique value proposition (e.g. Gartner’s Magic Quadrant) then there is little reason why someone in search of data will not use the seemingly infinite resources given away for free on the web as oppose to purchasing it directly.

My advice to analyst houses is to recognise why people come to you in the first place – they are after bespoke information and not generic research. It is the analysts and what’s in their head that counts not the small amount they can put onto paper.

Open source analysis perhaps is the only way that smaller companies will be successful in a market that is dominated by a few big names. They will never be able to compete in the scale and scope of what these companies can offer. If this means that I am now able to access what the analyst thinks for free then I applaud this move… I may even pay a few for more specific information.

Technorati Tags: gartner, mwd, quocirca, freeform dynamics, redmonk, datamonitor, idc, magic quadrant, analyst relations, ar, open source, osaa, open source analysis analysts, free lunch, analyst

Filed under: analyst relations, noteworthy | 22 Comments

22 Responses to “In praise of open source analysis”

Feed for this Entry Trackback Address

1 ARonaut on June 6, 2007 said:

There’s a big difference between direct analysis and reports requiring primary research. Not that James is not relevant but his cost of production (and salary) are way lower than IDC’s or Gartner/Dataquest. This might help to explain why IDC, the Borg and Datamonibores are transactional.

Reply
2 James Governor on June 6, 2007 said:

ARonaut – righto, we have a different model! Just out of interest: how do you know what my salary is? I was talking to a heavy hitter from Microsoft yesterday. He said some people from IDC had complained to him recently that I was undermining the industry by giving stuff away. It may be that we’re not so orthogonal if my model concerns the incumbents. But its all good.

Reply
3 James Governor on June 6, 2007 said:

and i meant to say cheers Johnny!

Reply
4 theARpro on June 6, 2007 said:

“Analyst houses such as RedMonk, Quocirca, MWD and Freeform Dynamics have been using this model successfully for years.”

Seriously?

Perhaps it depends on your measure of success, but none of these firms are making a lot of money — or even the industry average. How much *research* are they doing? RedMonk just seems to be a blog.

Is it really open source? If a vendor pays Freeform Dynamics to do research that can be given away, is that open source? When Aberdeen Group did that, you and your pals weren’t quite so kindly.

Reply
5 Jonny on June 7, 2007 said:

Good comments raised by the ARmadgeddon team. To answer them in turn:

ARonaut:

I agree the transactional-based analyst houses have a much higher cost of production. My point though, is that much of the data they create through research can often be found elsewhere for free and (more importantly) has far less value than the consulting money their research consequently brings in. To reiterate, companies want bespoke information that is of relevance to them – not some generic report.

Yes, analyst houses need to cover their costs but I don’t believe that in this ‘web 2.0’ world this business-model has long term viability. There are of course caveats as I mentioned previously where brands have been created (e.g. MQ or IDC for number-crunching). Nevertheless I foresee many analyst houses relying far more on generating income though consulting and Inquiry rather than flogging reports.

theARpro:

How much research are the other houses doing? Very little compared to the likes of Gartner, DM etc. They may not be making the same amount of money as the established Tier 1’s but they have successfully adapted a new business model and have done OK for themselves. They may not be rolling in it but they aren’t going bust either. In fact, several have been aggressively hiring.

Is it really open source? If a vendor pays Freeform Dynamics to do research that can be given away, is that open source? When Aberdeen Group did that, you and your pals weren’t quite so kindly.

If it’s given away – it’s open source. Period.

Does this liken to Aberdeen being ‘gun’s for hire’ – that’s a different argument entirely. The market will dictate that – if people do not believe the trustworthiness or independence of Freeform then they will go bust. If indeed they or anyone else can be bought I will celebrate their demise too. However, this isn’t the case at the moment and I hope it never becomes that way.

Reply
6 Duncan Chapple on June 8, 2007 said:

Jonny,

You have opened up an interesting discussion.

I wonder, do you want to clarify what you mean when you write that “If it’s given away – it’s open source. Period.” For example, Internet Explorer is free, as is all the research on vantello.com. However, surely the point about open source is that the process of production is transparent and collective. The consumers are also co-producers. That mean that the flaws and weaknesses of the work are open to discussion and correction. The whole product is open. There are not secrets.

In contrast, it seems to be, the research methods and objectives of a firm like Freeform Dynamics are specified by the vendors who sponsor the research. The survey tools and scope will be determined by the vendor. The process is not open. We don’t know the approaches, question and responses that have not been used in the final report.

With open source, users are free to modify the produce, or to take only the parts they want. They can incorporate it into their own work without needing the permission of the original author. They can take the product and host it on their own machines. They can engage in mass distribution. Finally, open source carries with it a certain ‘warranty of quality’ because of the open source process. None of these things are the case with Freeform Dynamics, to take your point.

Duncan.

Reply
7 Jonny on June 8, 2007 said:

Thanks Duncan – interesting comments.

To paraphrase George Orwell… “all open source is equal, but some open source is more equal than others”. I still believe that if it is given away – it is open source. However, it is only natural that some versions of this are far more mature than others.

What cannot be denied though is that analyst houses like Freeform Dynamics provide a service to the mid-market that previously haven’t been addressed. This group, either through ignorance or budget, has no access to IDC, Gartner etc but still need to get their hands on data.

Freeform Dynamics operate under a patronage model whereby they are paid by 3rd parties to research a topic but have full freedom to create the methodology themselves and draw their own conclusions. This in turn is fed into the community. IBM are big believers in this model and hope that this ‘free research’ helps the community, drives change and innovation. I do not for one moment liken this to Aberdeen’s White Paper creation where the entire scope is proposed and agreed up front.

Reply
8 Jonny on June 8, 2007 said:

Edit – with Dale’s permission I have copied some information which directly addresses his patronage model and his open methodology.

We are planning a website upgrade soon during which we will be discussing the patronage model in those terms, though the existing “Services” page on the site does actually describe the approach pretty accurately, with all the safeguards (so I won’t repeat them here):

The topic of sampling and sample bias is quite a different one, and IDC/Forrester/Gartner are not always squeaky clean on this and I would say are generally far worse than smaller firms. I have seen research, for example, saying n% of people will be investing in X, then you read the fine print and it says that the research was conducted at a Forrester conference on X – without highlighting that this means n% is totally invalid as the sample would have been significantly biased towards those with an interest in or knowledge of X.

We don’t labour the methodology thing when writing reports, but we do make comments on sampling where appropriate, i.e. where it could affect interpretation – e.g. in our latest report on Vista adoption, we say:

“The study was designed, executed and analysed on an independent basis by Freeform Dynamics, and as the core theme was generic, i.e. not to do with Windows Vista per se, we can consider the results as being properly representative of the enterprise IT management community, with no significant bias towards those either with or without an interest in or experience of Vista specifically.”

That was to reassure people.

Conversely, here is a note from an SOA study we conducted to warn people:

A Note on Methodology Information was gathered during this study via a Web based questionnaire, promoted to a broad cross section of IT and business professionals through a variety of means. Participation in the study was based on self-selection, i.e. the questionnaire was not directed at specific individuals in the same way as would occur with telephone interviewing. This means the sample is likely to be skewed towards those with an interest in and/or knowledge of SOA, who are more likely to respond. This bias in no way undermines the validity of comparing different groups within the sample, e.g. those from larger organisations versus those from smaller ones, those with experience versus those without, etc. It does, however, mean that absolute percentages relating the level of knowledge and activity are probably an over estimation of the true levels in the population as a whole. Attention has been drawn to this limitation in the commentary where relevant. Finally, we should note that responses were provided anonymously and segmentation is based on information volunteered by the respondents themselves.

There are actually a couple of references in the report, in context, to self selection bias to avoid people misinterpreting charts.

Bottom line, we are highly professional and open about the way we conduct and report our research, regardless of who funded it. I would argue, in fact, that because our first responsibility (culturally) is to the community of buyers and users, rather than a vendor or analyst subscription agenda, our stuff is much more open and transparent than the big analyst firms, whose methods are often very opaque and obviously engineered towards creating the next bandwagon-driven subscription revenue stream.

Reply
9 Dale Vile on June 8, 2007 said:

I have read some of the stuff being discussed on the blog network today on this open source analyst topic, most of which is interesting and useful to further the debate. However, some of the postings portray a lack of understanding of how businesses like ours (Freeform Dynamics) operate, and why funding via the patronage model does not equate to vendor bias. I won’t repeat any of the points I discussed with Jonny earlier as quite a bit has been has already been posted.

For the avoidance of doubt, however, while what we do has some similarities to the open source software model – we publish stuff free of charge, try to be as open as possible, and embrace a similar patronage model – there are some important differences.

When we publish our research, while we encourage viral distribution with no licence fees, we retain copyright and do not allow research deliverables to be taken apart and incorporated into derivative works without our permission. This is not because we want to stop people using the research in whichever non-commercial way they wish (i.e. we invariably grant permission for reuse when asked), it is because we do not want to risk research results being taken out of context. We all know how easy it is to take one chart, present it in isolation, and misrepresent what itmeans, either deliberately or inadvertently.

While this may not be in keeping with the purist view of “open”, it is a practical necessity in our view which is just responsible common sense given the type of material we are publishing. I would be interested in any ideas of how we might go more “open source” without incurring this misrepresentation risk. It would be great to take it to the next step, but we just haven’t figured out a way to do it safely yet.

Anyway, this whole debate on analyst research/business models is something I encourage, but I would ask that if anyone is unfamiliar with some of the newer approaches that have come into being over the past few years, that they drop an email or pick up the phone to myself or any of the other smaller firms mentioned in this thread before making assumptions or passing judgement.

After all, contrary to some of the ill-informed views I have read today, we are all very open about what we do, how we do it, and even how we make a living. We really don’t have anything to hide.

Reply
10 Dale Vile on June 9, 2007 said:

Just a quick note to acknowledge Duncan’s clarification:

http://analystrelations.blogspot.com/2007/06/open-source-or-crowd-surfing_08.html

Please bear this in mind when reading some of the comments in the thread above too.

Thanks for putting the record straight Duncan.

Reply
11 alan pelz-sharpe on June 20, 2007 said:

I think I agree with Duncan on this one – the common understanding of Open Source is not so much that it is free (in fact Open Source options can be quite expensive) but that the process is open, and that you can contribute to it.
Hence this term is not (IMHO) applicable to what these firms are doing – no matter how worthy or good the research. It is simply free research.

The other thing is that there seems to be an assumption here that analyst research is lightweight and simply an appetizer suggesting more substance to come. Though I agree that this may commonly be how things work, it is not universal – take our reports for example they are typically 350 pages up – you get a lot of substance, and there is not that much that you could get for free on the web in there.

In addition there are numerous examples of analyst reports that have been VERY successful with end users/buyers that have (within the industry) zero credibility. In my past consulting work I saw many buyers quoting from vendor sponsored reports from Butler, Aberdeen, F&S etc as if they were quotes from the Bible. Few buyers understand the analyst community and the varying business models. Now that is something I would love to see changed, true Open Source Analysis – transparent methodologies, clarity about stakeholder involvement and funding etc.

Best
Alan

Reply
12 Martin Hingley on July 6, 2009 said:

A bit late to join the debate, but…
Having come from a transactional research house to setting up ITCandor my business model has changed a bit. Of course we give things away for free when setting things up. It’s the only way to go. That doesn’t mean we don’t intend to challenge the major research houses. They were once small and didn’t have the rich infrastructure of the Web to help them. I’d love to have interviewed Pat McGovern in 1964 as he was starting IDC to ask him what he thought about Nielsen and the others.
As for opinions and data… You can build a better top-down model on your own than the majors and they can be more consistent, because you don’t have to negotiate definitions and forecasts – which often say more about the analysts outlook on life than the opportunities for growth.
On blogging, we wouldn’t do so much if customers were paying for our analysis. Watch the posts diminish as business comes in. The major houses can’t blog coherently; there are just too many distractions.
I’m sure Dale and James (and me for that matter) intend to become the incumbents in future. In the meantime giving free analysis and ideas is hopefully a boon for our industry. It’s not Open Source, it’s just free.

Oh and finally a shameless plug for my stuff on WordPress at http://martinhingley.wordpress.com/!

Best Wishes
Martin

Reply