TweetLevel – how to measure ‘importance’ on Twitter

20Dec09

image

Edit: reposted as original blog published on 11 November has been deleted. (cached copy)

TweetLevel is a new measurement tool that calculates someone’s importance on Twitter. In other words it is the Twitter equivalent to Google’s PageRank.

The explosive growth of Twitter has massively impacted the nature of communications. Initially the domain of the tech-minority, it has now reached a maturity level where it has branched out to multiple demographics.

imageimage

Tables: Nielsen Twitter usage increase Feb 08 to Feb 09 and Nielsen Twitter usage age groups

My personal view is that as long as Twitter is the tool that key audiences use then I will use it too – if everyone changes next week to product X, then I will too. However, as a communications professional, it is my job to take part in the conversations that  my key audiences are having no matter where they take place.

Jeremiah Owyang of Altimeter Group sums it up:

I’ll be very clear on this. If you want to influence me, be in a conversation and dialog with me, in person, online, and wherever I go.

One of the common questions Edelman is asked is ‘who should I spend my limited time with?’. It is this question that has forced companies to tier their audiences depending upon their impact within the wider community. One of the common problems associated with ranking twitter users is the misconception that popularity = influence. This couldn’t be further from the truth. Popularity has its place but it should not be the only metric that defines ‘importance’.

TwitterLevel aims to answer this question.

This unique tool compiles twitter data from over 30 sources and feeds the data through an algorithm to rank an individual according to four weightings:

  1. Popularity (i.e. How many people follow you)
  2. Influence (i.e. What you say is interesting, relevant and many people listen)
  3. Engaged (i.e. You actively participate within your community)
  4. Trusted (i.e. People believe what you say)

Of course, the explanation above is a simplified definition of a complex algorithm(the full methodology of this is shown at the bottom of this post).

imageTo illustrate how this works, I have selected four individuals who each excel in different areas. What this shows is not a measure of an individuals ‘importance’ in the world or even social media but simply how they use Twitter.

Popular CNN Breaking News

tmp25D

With close to 2.8 million followers this account is exceptionally popular. However, by contrast its engagement score is very low – this is because all the account does is broadcast its opinion (and links) without engaging in conversations with other people. However, without doubt it is exceptionally important as people believe what it says (as shown by its trust score) and its content is retweeted frequently.

InfluentialCalvin Lee

tmp268

Calvin Lee at Mayhem Studios has a high influence score, but a low number of followers as compared to other people in the top 20 TweetLevel names ranked by influence. With less than 40,000 followers Calvin understands that within Twitter it is micro-communities that count not necessarily who can win the popularity contest. Calvin also has a very high engagement score as a result of interactively discussing frequent, relevant and interesting content.

EngagementT-Mobile USA

tmp276

Congratulations to T-Mobile for understanding that the key to using Twitter as a tool is to have two-way conversations that are informative with their community. They have mastered this medium and have been ranked top over a list of millions when it comes to who has the most engaged Twitter account.

Just as PR has gone to public engagement, and AR has gone to analyst engagement, it is only natural that I believe that instead of mass-broadcast (shown by popularity) or mass-amplification (shown by influence), I believe that the purest form of interaction is via multiple targeted micro-conversations where people actively engage and interact with the niche community. This is why when scores are weighted for ‘engagement’ the ‘involvement index’ becomes the most important factor.

Robert Scoble in his recent post discussing the merits of Twitter’s Suggested Users touches on engagement. He explains that the number of followers someone has (i.e. their popularity on Twitter) should be turned off and the answer… engagement.

Turn off follower counts for everyone and come up with a new “engagement score” that is more focused on how you use Twitter and how people engage with you.

Couldn’t agree more which is why TweetLevel has such a high weighting on this metric.

TrustPaulo Coelhlo

tmp28C

The extra measurement of trust has been included as this factor is the key to action. When people are reviewing how they should act, they may take into consideration someone who is popular, influential or engaged – however, this final metric is the decisive or tipping point. This view is backed-up by the Edelman trust barometer which stated that:

77% of people refused to buy products or services from a company they distrusted

It is this final measure of trust that has been the most discussed during beta testing. After all how do you measure trust on Twitter? Although not the only factor that decides this ranking, in my opinion if someone is prepared to associate what someone else has said with them (via retweeting for example), then this indicates a measure of trust. Whereas some Twitter users achieve this via posting news (such as CNN), others such as Paulo Coehlo have achieved a magnificent trust score by tweeting content that other people wish to spread throughout the twittersphere.

As a final point, I know that when discussing this people tend to be far more interested in ‘influence’ rather than engagement or trust. My view was nicely echoed by AdAge when commenting on Ashton Kutcher’s use of bill board advertising when trying to win the (in)formal ‘first to 1million followers’ against CNN competition. In this piece they quoted a New York commenter who goes by Stevewax:

Seems to me what’s useful with Twitter is creating a small, two-way community with people who aren’t busy running a Twitter team and who have time to SHARE ideas. Rather than broadcast them.

Find out what your TweetLevel is

Simply go to: www.tweetlevel.com now and find out your score. While you are at it, try a few other people and see how you compare to them. I welcome your feedback as to how this can be improved or your comments in general.

Algorithm and Methodology

tmp9CD

Following – Twitter lists the number of people each user follows. The tendency for most celebrities is to only follow a few individuals. The more people that someone follows, there is an increased likelihood of them actively participating in conversations with the community instead of simply broadcasting to it. Following ranges were determined (i.e. more than 20, more than 30, etc.) and each range was assigned a number (0 to 30) that was used as part of the algorithm. Note: Twitter opened its API to TweetLevel so that data could be sourced easily and quickly to benefit the user.

Followers – Twitter lists the number of people that follow each user. Like subscribing to a feed, this is a clear indication of ‘popularity’ as it requires someone to actively request participation. Even though TweetLevel has a ranking of people based upon popularity, it is influence, engagement and trust that is more important. Due to the nature of logarithmic ranges, a change in the number of people that follow someone, such as from 500 – 1000, will give a far higher change in score than a move from 180K – 200K. Following ranges were determined (i.e. more than 20, more than 30, etc.) and each range was assigned a number (0 to 30) that was used as part of the algorithm. Note: Twitter opened its API to TweetLevel so that data could be sourced easily and quickly to benefit the user.</p>

Twitter Lists – without a doubt this feature addition to Twitter will significantly change the influence score. Even though Twitter has released their API to us, this particular metric is not yet included. When it is, a TweetLevel score will increase depending upon the number of times a user is included in a list, the number of people who follow that list and the authority of those people.

Updates – How often does someone update what they are doing. This number is purely objective as it scores someone highly no matter what the content of their post (i.e. how relevant is it). Nevertheless it is assumed that if someone posts frequently but has poor content then their ‘followers’ will decrease. Update ranges were determined (i.e. more than 20, more than 30, etc.) and each range was assigned a number (0 to 30) that was used as part of the algorithm.

Name Pointing – e.g. @name – How many people engage in conversation with a celebrity or point to their name. The clearest way to establish this is to run a search on the number of people who reference @username in a message. This calculation is based upon a one month period combined with a 24 hour period. The number of times this happens is calculated with each range was assigned a number (0 to 30) – again this was then used as part of the algorithm.

Retweets – Has a tweet caused sufficient interest that it is worth re-submitting by others? Despite a great deal of ‘noise’ (i.e. posts that are not relevant or interesting), when someone sees something that is of high interest, their post can be re-tweeted. The clearest way to establish this is to run a search on the number of people who reference RT @username in a message. This calculation is based upon a one month period combined with a 24 hour period. The number of times this happens is calculated with each range was assigned a number (0 to 50) – again this was then used as part of the algorithm.

Twitalyzer – “This is a unique (and online) tool to evaluate the activity of any Twitter user and report on relative influence, signal-to-noise ratio, generosity, velocity, clout, and other useful measures of success in social media.” This 3rd party tool is a useful method to combine automated metrics dependent upon criteria within posts and publicly available numbers. Where tools such as this are available, we incorporate them into the algorithm to achieve a more confident score. Twitalyzer gives users scores from 0 to 100. Ranges were determined (i.e. more than 20, more than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Twitalyzer noise to signal ratioSignal-to-noise ratio is a measure of the tendency for people to pass information, as opposed to anecdote. Signal can be references to other people (defined by the use of “@” followed by text), links to URLs you can visit (defined by the use of “http://&#8221; followed by text), hashtags you can explore and participate with (defined by the use of “#” followed by text), retweets of other people, passing along information (defined by the use of “rt”, “r/t/”, “retweet” or “via”). If you take the sum of these four elements and divide that by the number of updates published, you get the “signal to noise” ratio. Twitalyzer gives users scores from 0 to 100. Ranges were determined (i.e. more than 20, more than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Twinfluence RankTwinfluence is an automated 3rd party tool that uses APIs to measure influence. For example: “Imagine Twitterer1, who has 10,000 followers – most of which are bots and inactives with no followers of their own. Now imagine Twitterer2, who only has 10 followers – but each of them has 5,000 followers. Who has the most real “influence?” Twitterer2, of course.” As with Twitalyzer, this index uses 3rd party tools to add greater confidence in the overall Twitter score. Similar to the other criteria, ranges were determined (i.e. less than 20, less than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Twitter GraderTwitter Grader is the final automated tool to add greater confidence to the final index. This site creates a score by evaluating a twitter profile. Similar to the other criteria, ranges were determined (i.e. less than 20, less than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Involvement Index – The Involvement Index is unique Edelman IP that calculates a score based upon how an individual engages with their community. It is calculated by analysing the content of an individual posts. People who score highest in this category have frequent, relevant, high-quality content that actively involved the twitter community (asking questions, posting links or commenting on discussions) and did not purely consist of broadcasting. Ranges were determined (i.e. less than 20, less than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Velocity Index – As more people engage on Twitter, it may become harder to keep activity going. The velocity index measures changes on a regular basis and assigns a score based on increased or decreased participation. Ranges were determined (i.e. less than 20, less than 30, etc.) and each range was assigned a number (0 to 20) that was used as part of the algorithm.

Weighting Each specific variable listed above was given a standard score out of 10. Using a weighting scale I varied the importance of the each metric to establish an individual’s total score.

Weighted for Popularity – the key variable is the number of people someone has following them. There are many online tools that show this such as Twitterholic.

Weighted for Engagement – the key variables are an individual’s participation with the Twitter community (as measured by the Involvement Index), with additional emphasis on the frequency of people name pointing an individual (via @username), the numbers of followers and the signal to noise ratio. Other attributes were included in the final score but were given a lower weighting.

Weighted for Influence – the key variables in this instance is a combination of the number and authority of someone’s followers together with the frequency of people name pointing an individual (via @username) and the how many times and individuals posts are re-tweeted. Other attributes were included in the final score but were given a lower weighting.

Weighted for trust – the best measure of trust is whether an in individual is will to ‘trust’ what someone else has said sufficiently that they are also prepared to have what they tweeted associated with them. The key metric in this instance are a combination of retweets and number of followers. Other attributes were included in the final score but were given a lower weighting.

In the true spirit of ‘open sourcing’ this work, I welcome your comments, views and criticisms in how this approach can be as accurate as possible. Whereas I don’t believe for one moment that TweetLevel has found the holy grail of social media measurement, I think it is a good step forward and look forward to discussing this with you.



6 Responses to “TweetLevel – how to measure ‘importance’ on Twitter”

  1. I think this is a very helpful tool. Will try it for sure!

    I echo Jeremiah Owyang’s quote: “I’ll be very clear on this. If you want to influence me, be in a conversation and dialog with me, in person, online, and wherever I go.”

    I guess we can say “adios” to the bots who helps a Tweeple gather ten thousands of followers without them interacting and engaging – because your measure will catch them ineffective, correct?

  2. One of the key influence data points that doesn’t seem to get factored into most of these tools and formulas is # of Favorites.

    There are many followers who will Fav a Tweet, but not ReTweet. Often I find value in tweets and fav it rather than RT it as the latter may put my neck on the line. The originators of my favs are of significant “true” influence on me as I value the piece of content enough to fav & save it.

    Additionally, people RT not necessarily for reasons of wanting to share/atomize quality content or opinions, but to show props or “tweetservice” (realtime Twitter version of realworld “Lip Service”). Therefor, RT’s can be somewhat misleading as a “true” weighting factor.

    Do you agree with this view?

  3. Nice job. Looks like a lot of work/thought went into this. You’ve really made sure no single variable could dominate the results, which is important, given the variation in possible reasons for high/low scores of those factors. Very much looking forward to seeing how Lists are integrated and how they change results.

    A previous poster mentioned Favorites. Any thoughts on that? I had mostly used this as a bookmarking tool until I came up with other ways to achieve that functionality.

    The number of people someone follows is of limited value. As you mentioned, some people follow only a few but lots of people follow hundreds or thousands, most of which, despite filtering tools like Tweedeck, I’m pretty certain they never read. So, what does the number really indicate unless less engagement the higher it gets?

    Then again some accounts are managed by teams or people with more time. I guess you’ll never know whether anyone reads their stream. Maybe if they retweet a wide variety of people versus the same small set, who knows?

    Either way, it’s these kinda of debates you could have over many different metrics, but your methodology would seem to mitigate those discrepancies, and I like that.

    Looking forward to seeing how you modify the weighting over time.

  4. Interesting post, but I’m not convinced on how you measure trust. As you stated, “others such as Paulo Coehlo have achieved a magnificent trust score by tweeting content that other people wish to spread throughout the twittersphere.” I think you have to break down what you mean by trust and the context it is in. For instance, I tend to retweet stuff that Steve Rubel mentions because I think he is a smart guy, I find his tweets and posts interesting, and I think others in my social network would also find it interesting. What would be the trust in this case? Would it be that I trust Steve Rubel or that others who follow me, trust me enough to read something that I retweeted by Steve Rubel? I think it is less about trust, and more about matching preferences and relevancy.

    Do you think you could put together a visual to explain what you mean by trust?

  5. Tweetlevel is taking the average of a whole bunch of different measurements and calling it a rating tool. Better than nothing– but why are these the right measurements? Why do they all have the same way? What is the relationship between the intuitive definition of terms like “engagement” and the formulas? How are the words that have been chosen to represent the numbers appropriate, and how will they be misleading?

    I don’t think we even understand what the number of followers really means, let alone the complex formulas you are discussing.

    Tweetlevel is, both literally and figuratively, selling something.

    I agree we need metrics. But these are not yet metrics. So far, they are mostly buzzwords.

  6. TweetLevel seems like a great tool. It is kind of sad that we need such a tool.

    I would advocate for a Twitter BlackList hell where all the spammers get sent🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: