The other thing is that there seems to be an assumption here that analyst research is lightweight and simply an appetizer suggesting more substance to come. Though I agree that this may commonly be how things work, it is not universal - take our reports for example they are typically 350 pages up - you get a lot of substance, and there is not that much that you could get for free on the web in there.

In addition there are numerous examples of analyst reports that have been VERY successful with end users/buyers that have (within the industry) zero credibility. In my past consulting work I saw many buyers quoting from vendor sponsored reports from Butler, Aberdeen, F&S etc as if they were quotes from the Bible. Few buyers understand the analyst community and the varying business models. Now that is something I would love to see changed, true Open Source Analysis - transparent methodologies, clarity about stakeholder involvement and funding etc.

Best
Alan

http://analystrelations.blogspot.com/2007/06/open-source-or-crowd-surfing_08.html

Please bear this in mind when reading some of the comments in the thread above too.

Thanks for putting the record straight Duncan.

For the avoidance of doubt, however, while what we do has some similarities to the open source software model - we publish stuff free of charge, try to be as open as possible, and embrace a similar patronage model - there are some important differences.

When we publish our research, while we encourage viral distribution with no licence fees, we retain copyright and do not allow research deliverables to be taken apart and incorporated into derivative works without our permission. This is not because we want to stop people using the research in whichever non-commercial way they wish (i.e. we invariably grant permission for reuse when asked), it is because we do not want to risk research results being taken out of context. We all know how easy it is to take one chart, present it in isolation, and misrepresent what itmeans, either deliberately or inadvertently.

While this may not be in keeping with the purist view of “open”, it is a practical necessity in our view which is just responsible common sense given the type of material we are publishing. I would be interested in any ideas of how we might go more “open source” without incurring this misrepresentation risk. It would be great to take it to the next step, but we just haven’t figured out a way to do it safely yet.

Anyway, this whole debate on analyst research/business models is something I encourage, but I would ask that if anyone is unfamiliar with some of the newer approaches that have come into being over the past few years, that they drop an email or pick up the phone to myself or any of the other smaller firms mentioned in this thread before making assumptions or passing judgement.

After all, contrary to some of the ill-informed views I have read today, we are all very open about what we do, how we do it, and even how we make a living. We really don’t have anything to hide.

We are planning a website upgrade soon during which we will be discussing the patronage model in those terms, though the existing “Services” page on the site does actually describe the approach pretty accurately, with all the safeguards (so I won’t repeat them here):

The topic of sampling and sample bias is quite a different one, and IDC/Forrester/Gartner are not always squeaky clean on this and I would say are generally far worse than smaller firms. I have seen research, for example, saying n% of people will be investing in X, then you read the fine print and it says that the research was conducted at a Forrester conference on X – without highlighting that this means n% is totally invalid as the sample would have been significantly biased towards those with an interest in or knowledge of X.

We don’t labour the methodology thing when writing reports, but we do make comments on sampling where appropriate, i.e. where it could affect interpretation – e.g. in our latest report on Vista adoption, we say:

“The study was designed, executed and analysed on an independent basis by Freeform Dynamics, and as the core theme was generic, i.e. not to do with Windows Vista per se, we can consider the results as being properly representative of the enterprise IT management community, with no significant bias towards those either with or without an interest in or experience of Vista specifically.”

That was to reassure people.

Conversely, here is a note from an SOA study we conducted to warn people:

A Note on Methodology Information was gathered during this study via a Web based questionnaire, promoted to a broad cross section of IT and business professionals through a variety of means. Participation in the study was based on self-selection, i.e. the questionnaire was not directed at specific individuals in the same way as would occur with telephone interviewing. This means the sample is likely to be skewed towards those with an interest in and/or knowledge of SOA, who are more likely to respond. This bias in no way undermines the validity of comparing different groups within the sample, e.g. those from larger organisations versus those from smaller ones, those with experience versus those without, etc. It does, however, mean that absolute percentages relating the level of knowledge and activity are probably an over estimation of the true levels in the population as a whole. Attention has been drawn to this limitation in the commentary where relevant. Finally, we should note that responses were provided anonymously and segmentation is based on information volunteered by the respondents themselves.

There are actually a couple of references in the report, in context, to self selection bias to avoid people misinterpreting charts.

Bottom line, we are highly professional and open about the way we conduct and report our research, regardless of who funded it. I would argue, in fact, that because our first responsibility (culturally) is to the community of buyers and users, rather than a vendor or analyst subscription agenda, our stuff is much more open and transparent than the big analyst firms, whose methods are often very opaque and obviously engineered towards creating the next bandwagon-driven subscription revenue stream.

To paraphrase George Orwell… “all open source is equal, but some open source is more equal than others”. I still believe that if it is given away - it is open source. However, it is only natural that some versions of this are far more mature than others.

What cannot be denied though is that analyst houses like Freeform Dynamics provide a service to the mid-market that previously haven’t been addressed. This group, either through ignorance or budget, has no access to IDC, Gartner etc but still need to get their hands on data.

Freeform Dynamics operate under a patronage model whereby they are paid by 3rd parties to research a topic but have full freedom to create the methodology themselves and draw their own conclusions. This in turn is fed into the community. IBM are big believers in this model and hope that this ‘free research’ helps the community, drives change and innovation. I do not for one moment liken this to Aberdeen’s White Paper creation where the entire scope is proposed and agreed up front.