Showing posts with label open data. Show all posts
Showing posts with label open data. Show all posts

Monday, July 09, 2012

Kasabi, Nuclear Power, and the Data Developer Dichotomy

Today's big news is that Talis are shutting down Kasabi, their linked data platform, moving away from semantic data, and losing Leigh Dodds. Talis have probably been the most visible entity in the commercial linked-data realm, and with good reason. They showcased an impressive local data portal for the Open Data Cities Conference a while back in Brighton, and drive various high-profile services like Fix My Street.

But it turns out the demand for the linked data platform just isn't enough to support business. As a data developer, I'm obviously intrigued by this move from Talis.
(Spoiler: There's a lot of background here, and I don't even answer my original question. Feel free to skip to the end.)

How I Learned to Stop Worrying And Love Data


Your front room? [img]
Back at Uni, I read that nuclear power isn't just about physics, technology and war. To wield nukes, a state needs certain organisational structures to support their centralised nature - security structures to protect them, political structures to commission, deploy or maintain them, market structures to permit or prevent sales, and so on. In short, technologies and their usage depend on social-political contexts - and vice versa.

At the other extreme, well away from nuclear power, micro-generation of power is slowly taking a footing. Feeding power into the grid from solar and other electricity generators also requires contexts to be put into place, but the nature and distribution of these are different to those of nuclear power. The consultations yield different emotions, the networks require bi-directional structures, as do tariffs and security measures.

Overall, there is a big difference between the "central" nature of nuclear power, and the "decentralised" nature of micro-generation.This maze of contexts is what Open Data is trying to navigate right now. And Linked Data is part of that, often trying to straddle worlds.

Platform Non-Wars


Once upon a time, XML was the saviour of the data world. Up until then, people had put up with CSVs and related 2x2 grid matrices. Data was a person-to-person communication tool, not a machine-to-machine one. Machines kept their own data, and made it available through their own interface. So passing data around was a presentation job, and people liked doing that in 2D.

XML changed that by structuring data differently - the context of XML was a machine-to-machine medium. Some simple XML can be coded by hand, such as HTML. Other XML (and lots of HTML) is really intended only to be generated and read by machines, which will do the validation, parsing and rendering without the aid of human hands.

XML can afford to be complex then, because the context is strictly defined by standards and parsers. Maybe XML is the nuclear power of the data format world.

Then people discovered JSON, a much more lightweight format which was still supposed to be machine-readable, but which also caters more to a second context - person-readable. Because JSON is so lightweight, it's much easier to debug by hand. And while there are huge industries booming around XML, the lure of JSON appeals to those who don't have rigid structures and definitions in place, who need to do something quickly and often by hand, at least at first.

Like nuclear and micro-generation though, XML and JSON aren't a "versus" or a battle. They're both just about what's appropriate for what context. This is an Important Point.

Open Wide, Please


What could possibly go wrong? [img]
This is where we return to Open Data, and the position of Linked Data within.

Right now, I argue that Open Data is trying too hard to cover everything in one breath. It's doing this because it very rapidly became a cultural symbol rather than an engineering symbol. With the agenda being led by "transparency" and its associated technical underpinnings (yes, the 5-stars of open data are a political quest, not a technical one), we tend to overlook the idea that Data - the other part - is not a standard but something which requires context.

Without context, data is just entropy, and entropy is already handled pretty well by our underlying electronic veins: the binary transistor.

In other words, is "Open Data" an oxymoron? If something is to build on its openness, it needs to flow, and if it is to flow from place to place, it needs to acquire meaning - at which point, is it no longer "Data"? Are we better off talking about "Open Information" - and if so, what does that mean for our tools?

So where does Linked Data and the plight of Kasabi fit into this? From the above, we can see that as "data" moves around the ecosystem, the people using it want different things from it, and will do so using different tools. This is a translation task - to adopt someone else's data, not only do you need to know how the data works in itself, you also need to know how it integrates with your own data. (This is also why Data Engagement is so important.)

The data developer's instinct is to build something generic and beautiful. This is further impounded by a commercial instinct that conforms to economies of both scale and scope.

But in reality data often resists the genericism that technical and economic efficiency loves. A generic data handling system would need to be so generic that it could do anything - at which point, is it any different to any database? Any computer? In her Open Data Cities Conference talk a few months ago, Emer Coleman said that "People are messy." And if people are messy, than people talking to other people is even messier - think Tower of Babel. In the real world, data is equally messier than we think it is.


One Chance Out Between Two Worlds


Not your usual armchair auditor [img]
This is why it's difficult to talk about Open Data and Linked Data in the same sentence, or sell all-encompassing data tools, or come up with "universal" standards that try to transcend contexts. The contexts differ, and translating data from one to another also means translating mindsets, working practices, learning processes, and organisational structures. Data Relativity is still being ignored.

Right now, Linked Data and the Semantic Web are in a funny position. They aim big, and are trying to solve an important problem about data quality. But this big aim means big technology and paradigm shifts - putting Linked Data much more into the realm of an "Enterprise" app in the same way that nuclear power and XML require a certain, quite hefty, amount of planning and structuring to achieve.

A lot of large organisations with "heavy", nuclear-style data have a lot of systems already in place, and a lot of knowledge and resources tied into these - in other words, they have a lot of momentum. This is where Linked Data could make a real difference I think, but the inertia that needs to be overcome is a fundamental issue. Not only that, but because the success of Linked Data is inherently tied to network externalities, that inertia is multiplied up through inter-organisational lines. To be good, many people need to adopt it at the same time.

On the other hand, the economic agenda of Open Data wants armchair auditors and clever freelance developers to innovate quickly for low cost. And the tools, knowledge and skills these kind of people have are geared up not toward "Enterprise-level" data, but toward quick-fire, loosely-knit, human-readable innovation. Find a small problem, and solve it quickly.

These two worlds are currently circling each other. All too often "Open Data" becomes confused with "Big Data" because there's a lot of it. Some of the challenge involves filtering out "Open Data" so much that it's invisible - background noise - leaving just the "Small Data", the useful stuff. Learning a 5-page API reference doesn't help this.

The other challenge is to make the link between "Big Data" and "Small Data" bi-directional. Or omni-directional. There's still massive amounts of work to do around microgeneration of data and feeding user-generated data into "the grid". There's still no real work around standards for data sharing, as far as I know, other than those that exist for content such as RSS, etc.

Right now though, the main challenge is just getting out of the Open Data mindset that data needs to be Big and Open in order to work. It's really important not to get too caught up in the lure of simplicity and easy wins for quick economic and political gain. Ecosystems are not built on top of economies of scale and scope.


Remember Babel.



Friday, April 27, 2012

Dissecting the still-breathing body of "Open Data"

Benjamin Welby has a great 5-part series on Open Data, summarising a lot of the current state in local government, and raising some good questions about how local government can lead on a lot of this. His first article picks up on my "Open Data must die!" post, and got me thinking about the term itself again.

Why is this an interesting thought exercise? Maybe the names don't matter, and function is everything. But at the same time, names crystallise our shared understanding of why we're doing something.

So why do we use the term "Open"?

Partly legacy - open source, open access, open-not-closed. All good stuff.

But in the context of governance, what role does the term "open" play? In fact, if we look at the two aspects of a) service delivery and b) democratic engagement, why is openness important? Answering this question, I believe, is key to moving open data forward - by clarifying new purposes other than openness in its own right. By setting an agenda other than "being open" we can start to look at being smarter and more productive, but within a context of accountability and shared decision-making.


Service Delivery

Let's take a break for a moment and consider the commercial world of "open data" - the Flickrs and the Googles and the like. A lot of the push for open government data came from a shift in the web paradigm from "data-as-a-website" to "data-as-an-API". Suddenly control of the data opened up - an essential inevitability as websites moved to a social model of importing data from users more and more. The website became a store, rather than a presence in itself.

Openness in this commercial world meant (or means) that developers were no longer restricted to the path to access this data that the website builders had pre-determined. You could suddenly write your own slideshow script based on your (or someone else's) Flickr photo stream, for example. Or print it out. Or turn it into text.

To return to government services, we have two types of user though, and two types of data.

First, we need to differentiate between data used by individuals and data used by policy-makers.

Second, we need to compare data about individuals with data about systems. By systems, I mean any organisation or structure that can be seen as a "whole" - such as transport systems, finance systems, demographic systems and geographic areas, etc.

This gives us a basic 2-way matrix on to which we can position "data" (which, after all, is ubiquitous):

Box 3 is greyed out because it doesn't count, imho. Ignore it?

The commercial data mentioned above falls into box 1 - a user accessing their own data. This is largely ignored (or discussed in fairly esoteric circles) under the governmental "Open Data" banner because it's hard. It's probably also the most useful, but maybe one to come back to. People tend to think of health data and benefit information, but one could also include library data (reading histories, current books, etc) and so forth in here.

Box 2 - systemic data used by individuals - is where a lot of people would like to position Open Data, whether it fits or not. The idea of the "individual" accessing data appeals to our consumerist lifestyle, like governments fulfilling their public-good role for the ultimate satisfaction of the private citizen.

(The "Armchair Auditor" idea falls into this precise trap of supposing that individuals will hold Government to account. There is nothing more romantic than a single person overcoming Universal Might.)

Some of this data is actually really, really useful. Transport data, opening times, prices, etc - this is the realm of information that we all use on a day-to-day basis. But that kind of data is "small" information - we consume something very specific in very short bursts, such as "next bus from stop X", or "who is my councillor?". These could almost be considered "facts" about the world.

Other data is also relevant to individuals, but from more of a research perspective. Being able to present data differently, such as with TheyWorkForYou, or aggregating data up to make travel-time-vs-house-price maps, starts to address the notion of how we can interact with the data in a user-experience kind of way. But to engage, users must not only interact with the data, but also with the democratic process (see box 4). The boundary between these two seems relatively unexplored so far.

Box 3 - policy-makers accessing data about individuals. I'm actually going to call this a void box, because I think at this point we transmogrify into anecdotal evidence, rather than data. Policy-makers love using stories about individuals, far more than comparing their stats in a Top-Trumps-esque kind of way. Comment if you feel differently.

Box 4 - systemic data used by policy-makers - really leads us into the idea of "democratic engagement", because it's where the overarching power lies. This is the data/evidence used to make decisions at a higher level. It needs to be aggregated to resolve complexity (and gain understandability), but be reliable and resilient enough for decisions to be made confidently.

This box includes data about government activities, such as spending data, as well as data about geographical areas, such as the IMD and National Indicators and demographics and so on.


Opening up Box 4

Box 4 is the most interesting, because it hasn't really been done before. Or rather, not in an open sense. Plenty of companies use "systemic" data - or "big data" if you like - to explore trends and make decisions. But all that data is valuable and therefore closed.

This is where and why Open Data hits challenges: the systemic data in this box is fundamentally tied to how administrations work and think. That is, it represents a worldview upon which certain types of decision can be made - accountable ones, justifiable ones, trackable ones. Data here, whether it's financial, organisational, demographic statistics or what-have-you, has gone through a long history of hammering out. To be usable, it must be comparable, and to be comparable it must be rigourously defined, in terms of what's collected and what it means.

It's why we have National Indicators and the ESD and consultations on standards within government.

And this leads to a bit of a paradox, which the Open Data world is currently trying to grapple with:

First, the data needs to be understood, because it's so well defined.

Second, the data - even after being understood - is only useful to its original context (the policy-making hierarchy), even after being understood.

Both of these together form an interlocking puzzle: If people don't understand the data, how can they use it? And if not's useful to them in their context, why would they want to understand it?

(Additionally, this paradox is semi-locked inside a Japanese puzzle box. Even if you get both, do you necessarily have the power to influence decisions?)


Democratic Engagement

This gives us effectively a 3-way combo lock on Open Data, which immediately answers our second purpose - that of democratic engagement. There is a philosophical argument for democracy, and indeed a philosophical argument for "openness". But I would say we have that already - people want to engage, even if out of a sense of ownership ("I pay my taxes and vote, therefore I should have a say.").

But people are also busy and/or lazy - there's no philosophical drive to see how government works any more than I have a philosophical drive to see how my computer works while I'm using it. Rather, I look at source code when I want to change it or write my own. The philosophy of "openness" rapidly transforms into a reality of "effect". And it's this sense of being able to "effect" that the idea Open Data comes down to.

Next time you see a dataset being opened up, ask ye 3 questions of it:
  1. Can I understand it?
  2. Is it useful to me in my life?
  3. Who's listening for answers that might lie within it?
Too often "Open" data ignores all 3 of these questions. "Raw data now" ignores these questions. You cannot separate data from policy and expect engagement.

"Open" is a mindset, not a piece of content.

"Open" cannot be an end in its own right. 

Wednesday, February 15, 2012

Data Engagement as a Healthy Ecosystem

Writing up the "Data Engagement (Everyday Data)" session from #ukgc12 is tricky, because it opened up a whole set of new questions and thoughts for me. I'm still trying to tie these thoughts together, and might break it down into a series of posts. This one starts out by looking at the many uses and users of data, and how this fits in with the Data Engagement Charter (see Tim's post) that others started to put together in the Saturday session.

Data as an Ecosystem

A year ago, I asked some questions around the relativity of data. It feels like I have some vague answers now, even if they're shifting about still.

The BBC are currently showing an awesome series about the history of plants on Earth, called "How to Grow a Planet" - but it's not just about plants, it's about how plants developed alongside rocks and Sunlight, and then how they developed alongside animals. It's a story of opportunity and of symbiotic relationships - in other words, of how co-dependence can lead to vitality and huge, thriving ecosystems.

The idea of data as an ecosystem isn't a new one. But it is one that's easy to forget, especially when we're staring at technical formats or CSV files, or when we're on a crusade for a much larger philosophy. However, understanding how different data users interact is key to a data itself thriving. During the session, I drew something like this:



At the time, I wanted to capture the important thing - the links from one group to another. In other words, data flows from user to user, from group to group, and gets transformed as it does so. This is the fundamental reason why data is good - not because it is easily automate-able, or easily packageable into binary, but because it can and does mean different things to different people. And yet we are also intrinsically linked through that flow of data at the same time. All different, yet all the same.

Each group has different needs and different backgrounds, and so each link in the diagram, at a data-level, will involve different...
  • Access needs
  • Numerical understanding
  • Contextual, real-world understanding
  • Quality needs
  • Reliability needs
  • etc.
A systemic/network view of data moving around means we can move from a "one size fits all" perspective to a narrower, more specific viewpoint. This means we can ask questions such as:
  1. What is our role and our position in the system?
  2. Who are our audiences (current and potential) and what do they want?
This gives us a less scary starting point than the idea that every data-handling party needs to cater to every other data-wanting party. Not everyone needs to do everything. We've come to grapple with this the last few years, with the shift from central authorities specifically providing data to an "end user" (if such a user exists) to providing "raw" data for a different audience. But the debate is still murky and needs to be made more explicit.

Data Engagement Charter - A Star Map?

Which is where a Charter can really help, I think. Although not just in terms of giving people an easy-to-follow guide, but also as a way to really map out the different uses and users of data. Si Whitehouse's blogpost on the subject has a 4-way use-case diagram positing 4 different types of people who want to find data:


Can we use this to start working out what the most important needs of each are, and more importantly, who could be filling those needs? And vice versa - if you have an existing audience who are struggling to engage with your data, what can you do to make it meet their needs?

I wanted to use this image to illustrate this post, for some reason:

The Gatekeeper

At some level it represents an opening up of barriers, which can summarise the ongoing efforts to open up data. But at another level it's a great image because it mirrors both the links in the first image above, and the two dimensional axis sitting nonchalantly in the Use Cases diagram. It not-so-neatly sums up the way in which data flow changes according to what the data is being used for right now. Sometimes cars need to flow. Sometimes trains need to flow. With the right barriers and the right flexibility, we can have both. Or all, depending on how many users we're talking about.

So there's no "right" answer to how data should be opened up, any more than there's a "right" answer to which programming language I should use. There is what's most appropriate, perhaps, and certainly what's most fun. Linked data and SPARQL are good in some cases. Excel and Word reports are fine in other cases.

How to Have a Healthy Data Ecosystem

Having said that, it's not quite true. To keep an ecosystem alive means sticking to some rules, otherwise we end up with fragmentation, infestation, and/or the possible collapse of large parts of that ecosystem.

The 2 main components of a healthy system are, for me, openness and feedback.

Openness allows for flexibility and the ability to rapidly make new connections - vital when the environment changes (climate-wise, but also economically, politically, and technologically). Without openness, things can exist for a while, sure, but there is little resilience as time goes on.

Which standard gets used for data doesn't matter as much as whether we can interface with it in the future. We need to avoid data lock-in (or lock-out, depending on where you're standing) - whether that comes from legal, technical or economic aspects. There may be reasons for certain barriers (such as privacy), but these shouldn't translate into a general attitude of making data difficult to obtain. Legal barriers should not prevent technical access from being as easy as possible, for example.

Intertwined

Feedback is also vital, as it provides a way to negotiate the value of data without resorting to a "survival of the fittest" regime. Feedback in a data sense means that the fruits of Data Engagement benefit both parties involved - and that the link between parties gets stronger as a result of it being used.

In other words, we need to consistently and continually make sure that data is useful, otherwise the disadvantages of providing it and acquiring it will outweigh the benefits.

A Data Engagement Charter is an awesome first step to understanding and realising this. Such a Charter, alongside technical prowess and legal openness (another post), can be a very real call-to-arms for this recipe for an ecosystem. To make data successful, we have to understand that we are co-dependent on each other, and that it cannot be a one-way flow, even if we try to make it one.

Monday, January 23, 2012

"Open Data" Needs to Die

Amongst all the UK GovCamp 2012 buzz, point #18 from Tom Sprints' write-up caught me as being one of the more curious:
18. A lot of “open data” sessions just seemed to me to be variations on a theme, and didn’t sell themselves to me at all. I am therefore worried that some of those discussions are either very esoteric, or insufficiently informed by people who understand the issues rather than the tech.

Where have we come from?

As a data geek (I like the word "mechanic" myself), it's been intriguing to see the conversation around "open data" change over successive GovCamps. A few years back, the question was heartily "How can we get hold of data?" - Tim Berners-Lee was starting out on his comeback tour, and mySociety were beginning to show that data could be made useful with some clever tools.

As I remember it (likely in a fairly biased narrative way), the conversation then switched fairly rapidly into "What's the best way to open up data?" - in terms of what data and what platforms were most useful to developers. Suddenly data stores had (experimental) APIs, and the public realm had massive amounts of spending data. There was some loose rhetoric about transparency and accountability, while developers picked things apart with fine Excel toothcombs.

Then things got more interesting, as it turned out everything that had happened so far didn't automagically lead to Amazing Stuff Happening. The question became a necessary "So what?" - as if transparency and accountability weren't enough by themselves! The topic turned to users and reasons and (more often) to interesting examples. Surely, somebody was clamouring for this stuff after all this?

I'm kind of hoping this explains something about why "open data" sessions are a bit fumbly-jumbly now.

Open data got complicated, quickly. Because data is complicated. Jump to the present, and conversations rapidly flit between all of the above either because everybody is involved at the same time, or the people who should be involved, aren't.

"Open Data" is harmful

Or both. The paradox is that it's become difficult to talk about open data firstly because those who were talking about it from one point of view are now talking about it from many points of view. And secondly because those who weren't talking about it before aren't talking about it now. Data silos still exist. Most people still use Excel. Statisticians still output reports.

The term "open data" is meaningless now. Not just meaningless - actively harmful. If you're used to talking about it, then the conversation has begun to fragment and coalesce around more subtle outcrops. And if you're not used to talking about it, then you're put off because nobody can explain what it means - and more importantly, what it means to you. So you carry on as normal.

My session at GovCamp on Data Engagement was, in retrospect, an attempt to get back to the previous question of "So what?". What I really want to do is fence the conversation off from the technical, economic and political aspects of data (although I'm still into all these things) and focus on the why. I desperately tried not to use the term "open data" because I think it would have distracted the discussion. (To be honest, I wanted to find something better than "data engagement" too, hence the phrase "Everyday data".)

And I'm really glad that some of the idea got taken up on day 2 by Tim Davies and others. A "Charter" for engaging with data really starts to delve into how we think about how to make data useful.

I admit I'm a little afraid that the term "Open Data Engagement" just makes the discussion even more vague. What does that mean to you if you have no idea what it is, or what Open Data is supposed to be? Is it all at risk of becoming another buzzword? What about "Data Usability", or "Public Data Engagement"? I'm still aware just how much I hate the terms "Public Understanding of Science" and "Public Engagement with Science". Are we going round in circles?

Should we call a Stats Spade a Stats Spade?

Many people with useful, everyday data and databases really don't think in terms of data. Because the data is about stuff they know, they think of it as "information". Maybe even a "resource". But ask them what "data" they have and they'll probably give you a back-up of their website.

One of the interesting points coming out of the Data Engagement session was that people deal with data all the time - think football, Formula 1, house prices, etc. But do people even refer to this as "data"? Or - more likely - do they call them "stats"? Mention "stats" and people think of tables, averages, and counts.

In a way, "stats" makes sense where "data" doesn't. "Information" makes sense where "data" doesn't. "Data" is tricky because it's all of this and more. It's figures, it's formats, it's visualisations. No wonder even those who understand this get confused when talking to each other. The more you try to take "Data" into the real world, the less the term applies.

Should the "open data" moniker be scrapped instead of more "useful" terms like these? Would this make talking about implementing it more difficult, or easier? After all, any conversation on how to make data useful quickly turns away from talk of even databases and on to other issues (standards, protocols, best practice, comprehension).

Maybe if we talk about our bus times as "public information", and spending figures as "spending figures" then people will be interested in it, and we can stop trying to work out what "open" means.

Maybe.

Friday, July 15, 2011

"Forget the data."

Holy crap, Emma Mulqueeny's (@hubmum) blog from yesterday on the next challenge for Open Data is possibly the. Best. Thing. I. Have. Read. In. A. Long. Time.

In particular:
Open data? Awesome, and we are making tracks.

Open Government? HARD, and we are not banging on that door yet."
and:
So what’s the next challenge for Open Government data?

Forget the data.

Find a way to enable these revolutionary ideas, apps, websites and widgets that save time, money and mind-numbing frustration from those who have to engage with government.

Do that, and only that.
This is the conversation we need to be having. Why? Not to work out "how to do it", but because it questions what is valuable and necessary in government.

Open data isn't a technical thing. It's about relevance. If you could do everything, what would you do? If you were hungry, would you eat, or would you talk about how to find out what the best thing to eat is and what the best way of eating it is?

"Open data" that lacks a medium for turning creative use into real-world change is irrelevant. It's what bad businesses do - they invent a million great things, but never actually want people to use them. Instead they use them as examples to tout how great and creative they are, in the strange hope that a people will think a symbol of progress is as good as progress itself.

Until, that is, someone comes along and not only has a better idea, but also actually builds it. For everyone to use.

Is that difficult? Of course - building stuff requires foresight, management, flexibility and the wisdom of knowing what your goal is. Do people do it all the time? Look around you.

Open data needs to be about other things now - including how it's funded, what the audiences are, and what the future holds. But none of these are about data. None of these are technical. We already have a society that runs on data, so data itself isn't a new paradigm.

We can't keep thinking of open data - and possibly even our entire creative efforts - as some kind of "continual prototype". We need to apply it like we applied sewage systems and electricity.

We need to understand that this isn't just about making the game easier to play, but about a whole new game.

Saturday, January 22, 2011

UKGC11: A General Theory of Data Relativity

First of a small series of small thoughts coming out of UKGovCamp 2011.

One of the key points emerging for me is just how much “data” is tied to the groups or people using it - not just the content, but the structure of it, the tools used to manage it, the background of the data, the assumptions behind it, and so on and so on. This comes up all over the place - standardised, central taxonomies often fall out of favour for being a jack-of-all-trades, useful to none. File formats are a direct result of people wanting data as an easy-to-edit spreadsheet, an easy-to-email PDF, or an easy-to-parse data file.

More fundamentally, even the understanding of what a dataset means becomes embedded in the structure of the data. If something is being measured , what defines that thing? What assumptions are inherent to the way data is measured? Is a van a form of car? More importantly, why are these definitions in place? Reasons are forgotten long before hard drives expire.

If you assume that all data is “relative” - I.e. a combination of the data itself and the people viewing it - what does this mean for linking it? Do we need more effort on translation? Or do we need more effort on fuzzy inferences between metadata, rather than direct mapping? (I suspect the Semantic Web rears its head here, but to me it always feels like a simpler solution is waiting to be seized on.)

Knowing how and where to ask questions about a dataset is a huge part of this - metadata about origins and background is vital; questions help build up an idea of how to fit a dataset into your own world view, your own data model, or your own database. Perhaps development needs to focus on making these links between contexts as transparent as possible, rather than fixing a single, over-arching context in place to fit all.

Wednesday, December 15, 2010

Tis the Season To Be Open

I apologise in advance - this post was originally a small comment on Paul's post, then turned into a blogpost here, and a bit of a rambling one at that. Hopefully it raises some points of interest, and hopefully I can return to some of them in future posts. The area is, ironically, complex in itself.

Paul Clarke has a nice summary of systems complexity generally, and the eternal battle between getting things done, and pleasing everyone. Fortunately I haven't seen the original stories on the Christmas Tree in question, so have no idea what the context is. I prefer it like that.

I think Paul is right to highlight the role of open data as we move forwards into a technical democracy, and the possible solutions/problems coming in as a result - I think there's a good chance that transparency can lead to ever-decreasing circles of receipt-checking, process justification etc, and the whole country implodes in a swamp of exclamation marks and daily mail headlines.

These 2 questions seem rather pertinent, IMHO: "And with what discretion? Authorised by whom?" - Are these the same issues we've been grappling with for years anyway, in the form of representative democracy? On a broad picture, it's not necessary for all citizens to be involved in all decisions all of the time - so we vote for the person we think we can trust most with power. We, as voters, are handing over discretionary power so that someone else is creating a world we want to live in. I call this "trust", because even today there's no way I can know (or want to know) everything my MP is up to. I have a wife and kids and a job.

panopticon

I've yet to be convinced that the drive for more transparency isn't just a way of getting us to trust politicians less. The over-riding message from on high seems to be that transparency is there to hold people to account - which I think is a real shame, as open data is far more powerful as a platform for collaboration than accusation.

Transparency as distrust leads to a bizarre situation in which people we've "trusted" via our vote are then afraid to apply that power - especially considering a vote is local, while headlines are national or global. Worse, it can drive important decisions further into obscurity and complexity to avoid such scrutiny (and here it's hard not to draw comparisons with the banking industry as a warning).

Perhaps part of the problem is believing that cost is the deciding factor in how accountable (and hence transparent) a decision-process should be. But cost says nothing of either complexity or impact - both of which are much more important in deciding the "suitability" of decisions, I would say.

Cf. two other realms - banking, as mentioned already, and open-source software.

On one hand, the uselessness of auditors in predicting the collapse of banks serves to show how bad it is to have systems that can rapidly create complex models around themselves. Compare this to how open-source software operates - for a project to be sustainable, it is vital that complexity is managed, and that the code is readable by anybody. If the code is unreadable, it grows more slowly, is more prone to bugs and security risks, and is less maintainable. Both designing and refactoring code are essential to ensure a solid output.

Recycling

Can we apply these lessons to government decision-processes? If transparency is the way forwards, then I think we have to - sure, there are fundamental differences between software (which, for instance, can be forked) and a democracy (which can't, quite so easily). But as things become more open and "many eyes" start taking peeks, the productivity-gains and effectiveness of open data mean that we cna't just assume that openness is enough. Openness needs to be accompanied by feedback - the same constant re-factoring process that goes into software engineering.

In other words, it is not enough to use transparency to justify decisions already made, and to prevent bad decisions being made in future through the threat of later accountability. Openness in data needs to go hand-in-hand with an openness to change - to influence new ways of contributing, of collaborating, and of voting for those who we trust. Even new ways of thinking and feeling about why the decisions are being made in the first place.