A Step-by-step Guide to Visualising Tweets

In response to psychemedia's request, I've put up a commented version of the code used to generate Wordles and Google Timelines.

The very rough and ready process was along the lines of:

1. Run the Perl script to output Tweets into CSV format. These will be ordered in forwards chronological order. The script also generates a count for the number of tweets in each 10 minute time slot, to give some idea of activity over the course of time.


2. Import the CSV into a fresh Google docs spreadsheet.


3. I found it handy to duplicate this sheet, to play with subsets of the data e.g. just tweets for certain days.


4. For the Wordles, I simply selected the relevant column (i.e. usernames, or tweet texts), and pasted into a decent text editor. I removed half the #ukgc09 tags for tweet texts, to stop it overpowering the rest. Then I just headed over to http://www.wordle.net/create, pasted the text, and played with formatting until I got something I like the look of.


5. For the Google Timeline, I created a version of the sheet with 4 columns: tweet date/time (to become the X axis), number of tweets for this 10-minute time band (for Y axis = activity), tweet author and tweet content (for the notations). Then it was just a case of opening the "Insert gadget" menu, choosing the "Interactive Time Series Chart" gadget, and setting the Range to include all the data in these columns.
   
   I found it easiest to limit tweets to 3 days (otherwise the amount of notations causes the browser to get very slow), and to move the gadget to a separate sheet.
   
   Then I published the whole document as a web page (see the "Share" menu in the top right of the spreadsheet - you need to publish the data for the chart to work).
   

And that's it. As yet, I haven't found a way to publish just the gadget - Google have code to embed it in a page, but this doesn't seem to work.

I'm pretty sure there's a whole lot more you could do - I was just intrigued as to how activity varied through the weekend. (Perhaps it's good that the wifi on the day was down - I'm not sure how well that Time Series gadget scales...) The Wordles also seem quite a nice way to remember the day. Perhaps it might be possible to generate a similar, animated version to view word/author proliferation throughout a day as well?

Anyway, if you have any questions, leave a comment or get in touch via Twitter.

Are we competitors? Or collaborators?

This week I've managed to catch up a little with the Power of Information Taskforce "beta" report, geniously (not a real word) put into a Wordpress site allowing anyone to head over and comment on each section.

Today I also had a quick squint at the Digital Britain interim report - or the executive summary a least. Those of you wanting to check this one out in Wordpress will want to run over to the rather less official version at writetoreply.org.

Comments and access aside, what struck me was the division in attitude taken by each. On the one hand, the POIT report seems to be about working out how we can start opening the doors, giving people access to data, encouraging experimentation, and shift data from where it's created to where people want it.

On the other hand, the exec summary for the Digital Britain report seems ensconced in the idea that we need to keep up with other countries - or, preferably, lead them in all the league tables we're able to league in.

From experience and instinct, it seems to me that those who are open to collaboration are more likely to a) produce things and b) collaborate again in future. On the other hand, seeing the world as a race just means we worry more about how we're doing in relation to others, rather than in relation to ourselves.

Perhaps, in other words, we're falling behind digitally precisely because we want to keep up with others, rather than work on what actually needs fixing, and what people actually want. Competition gives us excuses, collaboration gives us energy.

If we're going to change things, we need to start seeing everyone - and by this I mean our neighbours, our councillors, our politicians, and our friends in other countries and other industries - as potential energy, as possible links. We need more collaboration, but as long as we think of collaborators as potential competitors in some made-up chart that really doesn't mean anything, we're never going to seize the full potential of those links.

I'm not saying competition is a lie, or isn't sometimes useful. Just that being "better" than others shouldn't be our motivation - we should instead simply try to be better than ourselves.

Progress doesn't care about league tables.

Playing with UKGC09 tweets and Wordle/Google

As I said before, UKGovCamp '09 was very inspiring - and got me thinking that in this day of mash-ups and widget-gidgets, one probably doesn't need to actually do much coding at all. I put my theory to the test, wrote a quick Perl script to chirp all the #ukgc09 tweets for the day of the event, the day before, and the day after, and shoe-horned the data into a couple of places which could be prettier if I had more time.

Here's a Wordle tag cloud of all tweets over the 3-day period:


(Click to view large, you'll need Java.)

For fun, here's a similar Wordle for the Tweet authors themselves:



Finally, here's a time-series plot of all the tweets over the 3 day period, on a Google graph so you can zoom in and out and click on tweets. The y-axis indicates roughly how much twitter activity there is in a 5-minute slot.

All good, clean fun.

Update Feb 16th: Here's the code I used to grab the #ukgc09 tweets - fairly simple Perl, requires a system with the JSON library and command line Curl installed (see code for links). I used a Mac with curl already installed, and added the JSON library using "perl -MCPAN -e 'install JSON'" from the terminal. YMMV.

Post-GovCamp Thoughts: What is Trust?

Rather than re-cap my sporadic notes from UK GovWeb Barcamp '09, I thought I'd try to pick out some of the more intriguing thoughts that occurred to me during the day. This is the first, and you should be able to track them through the "ukgc09" label below...

What is Trust?

Everyone agrees that Trust is Good. But can we really leave it at that? Trust in what? Why? And how? There are, I think, different answers to each of these, and those answers depend - or inform - the type of political system in play.

Trust seems to overlap a lot with Transparency these days. But I'm not sure it's as simple as that. Take a simple analogy - would you trust your friends, even if hey didn't tell you what you were doing? In fact, wouldn't you have to trust them if you couldn't see what they were doing? Is that the definition of trust? And if so, then...

a) Can we really talk about "trust" in a political system that encourages transparency? Does transparency come about precisely because we can't trust our politicians?

b) Why do we trust our friends, or others that we deem to be "trustworthy"? If we want to trust our politicians (because, let's face it, we don't just want to watch over them all the time like some kind of nanny - we want them to get on the job we've entrusted them with), then how do we go about it? What systems do we need in place to build that trust?

Maybe Trust is a judgement, based on experience, character, reputation. Sometimes we get it wrong, and someone pulls the wool over our eyes. Sometimes we need evidence to start trusting someone again. But I'm not convinced that rushing to more "openness" and transparency is necessarily the best answer. We just need to be more careful about who we trust, and ask why our eyes were covered. And we need to force politicians who do betray our trust to prove they can be trusted again, but not through openness.

We have openness. And yet we still have no trust.

Debategraph

Debategraph is totally awesome. (Via rohan)

Hugely interesting and thought-provoking vague-future-of-gov-on-web event up at Nesta last night, all thanks to dominic. Time is tight and parties are loose this week, and there are so many (relatively) disjointed points to break down, mash back up again, I'm going to have to get this blog back on track over Christmas... In the meantime, Rohan has a summary and some video.

Wispas or Tweets: What have we achieved today?

The recent events in Mumbai were curious from an infotech point of view - standing watching a silent screen in Specsavers, I noticed one news channel (perhaps Sky, perhaps CNN or CBS) were actually showing "live" Twitter coverage of thoughts and comments. It looks like the BBC were also experimenting with integrating "Tweets" with a "live updates" page.

The comments on the Beeb article above are an interesting and considered mix. The Independent also followed up on it proclaiming the downfall of British journalism etc. But that's a debate I'm not getting into here.

For me, the interesting thing is that Twitter is being used in this way. In America, hack the debate spliced Tweets into the US election. This would be difficult to do with blogs - content is harder to scan, and then has to be summarised to make it ready for simple broadcast. Twitter "bypasses" this. It's already all ready to just be spat out. In other words, Twitter is broadcast ready.

When investigating the power of communication media, we learn that "impact effectiveness" - how much information you can gleam in as short a time as possible - is essential when the comms channel is already rammed with a million voices. (OK, I kind of just made that up, but it makes sense.) Ideas put more simply and more effectively will take precedence when the cost of communication tends towards zero, and when time is of the essence. This blog would be dead in the water if everyone was concerned about the time taken to scan it.

Politics is ideas as well as discussion, and everyone has a lot of both. A lot of people say that Twitter is the next "fad", and for a while - when it first came along, when I first signed up - I was cautious about it for the very same reason.

But where politicians have turned to Facebook as a way of re-engaging, we must ask "What really has Facebook achieved?" The answer may come back in the form of a Wispa.

But while politics remains a hierarchical structure based on representation, networking and media attention-grabbing, it's stomping ground will be the micro-bite attention worlds that creatures such as Twitter thrive in.

New Report - Progress in Online Services

Sphereless... it lives! I've been having many ongoing thoughts buzzing round my head for a while, thoughts which would squeeze nicely into this blog's techno-leotard. Unfortunately, they'll have to wait a little.

In the meantime, to get things re-rolling, the Public Accounts Committee has published their report on Government on the Internet, concerning the progress made (or not) in delivering government services by hooking an "e-" up to them.

The Register has a handy summary. Highlights include:

• 25% of .gov organisations couldn't say how much electronic services were costing them to run, and 40% of the rest could only estimate this. (That's an inverse total - 60% of 75% of... hmm... 45% of everyone, I think, who kept track of these things.)


• One third of sites aren't up to accessibility standards. That seems quite a lot still - and I'd be interested in which sites don't meet the standards: old ones no longer under development? Particular sectors? I wonder if there's a chance of mapping these things out.


• The question of the digital divide is raised yet again, probably relevant in light of the recent announcement of some kind of inclusion strategy.


• Interestingly, the Government "aims to close 951 websites by 2011". Furthermore: "the Committee recommends that no new government websites should be set up without permission from the Government's CIO." There are some interesting questions about centrality here - control over what people want, but also over the best way to navigate through both sites and site content. Maybe that's a future blog post, right there.

One for further bedtime reading, I think.