Hackers, transparency, and the zen of failure

This week I picked up the Hacker Ethic from my library (remember them?) by Pekka Himanen, with an intro by Linus Torvalds and an outro by Manuel Castells. It was hard to resist it with names like that splashed across it.

The book was written in 2001, back when the browser wars were in full swing and streamed video was still a bit of a novelty (so nothing's changed that much). But the theme addressed by Himanen is anything but dated - and contains some key threads which I want to think about and blog about more.

A lot of the text deals with the idea of what makes a "hacker" tick -and not just geek hackers, but anyone with a passion for what they do, rather than a bitter feeling that they work because they have to. Central to this hackerness is a passionate creativity, and a desire to share knowledge - including the results of that knowledge, such as code.

The truth and untruth of progress

This act of sharing knowledge is partially a form of status, true. But when you read articles like this about incorrect data being published, you start to notice what else open knowledge (including data) is about - social learning.

Hackers and open government are both (now) keen on sharing data - knowledge, code, ideas. But the real difference is in how they learn - for the hacker, openness brings about learning and improvement through public failure - there is an assumption that what you create can be improved, and an attitude that anyone else is welcome to improve it.

Now compare this to CLG's response to the LGC article above - a response filled with defensive language and finger-pointing. There is something rather scientific - or, rather, legal - about this discourse: claims are made by one party and refuted by another. Slowly the "truth" is "sculpted" from what is left.

But for the hacker, the truth is only what is created - not what is undisputed. Hackers fork code, create new communities, start new websites, run unconferences. If "truth" exists, then it is what emerges, not what is discovered, or what remains.

What do hackers sit on?

Can a highly hierarchical structure such as our democracy adapt to be creative rather than competitive? The open data movement is driven by both of these - data for transparency can be thought of as "evidence" in a legal bid for the justification of an organisation's existence. Data for new apps, on the other hand, only needs a use, and to be useful in a creative context.

This split in attitude is key when considering efforts like the recent consultation on local data transparency, which clearly puts "open data" into the evidential context:

The Government wants to place more power into people’s hands to increase transparency by seeing how their money is spent.

img by AMagill

My fear is that this inherently makes "open data" unuseful to the hacker crowd - an essential crowd to interest when the data is being released in CSV files, or other formats that require some parsing. If hacker's can't create something with the data, they won't do anything with it. The idea of an "army of armchair auditors" becomes a functional paradox, as the people the Government has in mind for the data apparently sit in armchairs, while the hackers sit in cafes, meet in pubs, and generally find comfy chairs far too comfy to code in.

To return to this post's title, what role will failure and learning play in this paradox? Looking at the draft code, we can see a desire to use the "many eyes" approach to fixing data:

18. Data should be as accurate as possible at first publication. While errors may occur the publication of information should not be unduly delayed to rectify mistakes. Instead, publication and use of the data should be used to help address any imperfections and deficiencies.

The hacker approach agrees with this - fix things as we go along. But does this fit with the idea of "armchair auditors"? As we saw in the LGC article, how can an auditor tell the difference between what is incorrect, and what is wholely disagreeable? And if they can't, why should they trust any of it?

(Maybe we need a "stable release" system like that of open source projects? Maybe, like the Linux kernel or desktop distributions, data can be released with an "unstable/testing" tag, then marked up as "stable/trustable" after enough testing has been done on it.)

Transparency is a lovely thing - but everyone has different uses for it. If it's used for creativity, then there is, perhaps, an implicit assumption that things can and will change. If, on the other hand it's to be used for accountability, then there needs to be trust in it.

GovFiction: The Beast Made Transparent

Richard picked up the flaming Torch of Transparency and held it in front of him.

"I can see it," he whispered. "I can see everything."

The creature lay before him, barely breathing. Its belly was half-open, but Richard couldn't tell which parts had been clawed apart by an unknown swarm, and which had been torn open by the creature itself. Quick, but deep cuts littered the remaining flesh, but it was clear that most of the creature's lifeblood had leaked away long before.

Richard looked on guiltily - he couldn't suppress a faint sense of pity for this beast which for so long had had so much control over him. He felt a sense of anti-climax - a romantic desire that such a mighty being should meet its demise in some proud battle, rather than like this. Rather than in dark obscurity.

With slow, steady breaths, he put out the flame, and headed back towards the surface. Somewhere, there was still daylight.

Tis the Season To Be Open

I apologise in advance - this post was originally a small comment on Paul's post, then turned into a blogpost here, and a bit of a rambling one at that. Hopefully it raises some points of interest, and hopefully I can return to some of them in future posts. The area is, ironically, complex in itself.

Paul Clarke has a nice summary of systems complexity generally, and the eternal battle between getting things done, and pleasing everyone. Fortunately I haven't seen the original stories on the Christmas Tree in question, so have no idea what the context is. I prefer it like that.

I think Paul is right to highlight the role of open data as we move forwards into a technical democracy, and the possible solutions/problems coming in as a result - I think there's a good chance that transparency can lead to ever-decreasing circles of receipt-checking, process justification etc, and the whole country implodes in a swamp of exclamation marks and daily mail headlines.

These 2 questions seem rather pertinent, IMHO: "And with what discretion? Authorised by whom?" - Are these the same issues we've been grappling with for years anyway, in the form of representative democracy? On a broad picture, it's not necessary for all citizens to be involved in all decisions all of the time - so we vote for the person we think we can trust most with power. We, as voters, are handing over discretionary power so that someone else is creating a world we want to live in. I call this "trust", because even today there's no way I can know (or want to know) everything my MP is up to. I have a wife and kids and a job.

I've yet to be convinced that the drive for more transparency isn't just a way of getting us to trust politicians less. The over-riding message from on high seems to be that transparency is there to hold people to account - which I think is a real shame, as open data is far more powerful as a platform for collaboration than accusation.

Transparency as distrust leads to a bizarre situation in which people we've "trusted" via our vote are then afraid to apply that power - especially considering a vote is local, while headlines are national or global. Worse, it can drive important decisions further into obscurity and complexity to avoid such scrutiny (and here it's hard not to draw comparisons with the banking industry as a warning).

Perhaps part of the problem is believing that cost is the deciding factor in how accountable (and hence transparent) a decision-process should be. But cost says nothing of either complexity or impact - both of which are much more important in deciding the "suitability" of decisions, I would say.

Cf. two other realms - banking, as mentioned already, and open-source software.

On one hand, the uselessness of auditors in predicting the collapse of banks serves to show how bad it is to have systems that can rapidly create complex models around themselves. Compare this to how open-source software operates - for a project to be sustainable, it is vital that complexity is managed, and that the code is readable by anybody. If the code is unreadable, it grows more slowly, is more prone to bugs and security risks, and is less maintainable. Both designing and refactoring code are essential to ensure a solid output.

Can we apply these lessons to government decision-processes? If transparency is the way forwards, then I think we have to - sure, there are fundamental differences between software (which, for instance, can be forked) and a democracy (which can't, quite so easily). But as things become more open and "many eyes" start taking peeks, the productivity-gains and effectiveness of open data mean that we cna't just assume that openness is enough. Openness needs to be accompanied by feedback - the same constant re-factoring process that goes into software engineering.

In other words, it is not enough to use transparency to justify decisions already made, and to prevent bad decisions being made in future through the threat of later accountability. Openness in data needs to go hand-in-hand with an openness to change - to influence new ways of contributing, of collaborating, and of voting for those who we trust. Even new ways of thinking and feeling about why the decisions are being made in the first place.

Post-GovCamp Thoughts: What is Trust?

Rather than re-cap my sporadic notes from UK GovWeb Barcamp '09, I thought I'd try to pick out some of the more intriguing thoughts that occurred to me during the day. This is the first, and you should be able to track them through the "ukgc09" label below...

What is Trust?

Everyone agrees that Trust is Good. But can we really leave it at that? Trust in what? Why? And how? There are, I think, different answers to each of these, and those answers depend - or inform - the type of political system in play.

Trust seems to overlap a lot with Transparency these days. But I'm not sure it's as simple as that. Take a simple analogy - would you trust your friends, even if hey didn't tell you what you were doing? In fact, wouldn't you have to trust them if you couldn't see what they were doing? Is that the definition of trust? And if so, then...

a) Can we really talk about "trust" in a political system that encourages transparency? Does transparency come about precisely because we can't trust our politicians?

b) Why do we trust our friends, or others that we deem to be "trustworthy"? If we want to trust our politicians (because, let's face it, we don't just want to watch over them all the time like some kind of nanny - we want them to get on the job we've entrusted them with), then how do we go about it? What systems do we need in place to build that trust?

Maybe Trust is a judgement, based on experience, character, reputation. Sometimes we get it wrong, and someone pulls the wool over our eyes. Sometimes we need evidence to start trusting someone again. But I'm not convinced that rushing to more "openness" and transparency is necessarily the best answer. We just need to be more careful about who we trust, and ask why our eyes were covered. And we need to force politicians who do betray our trust to prove they can be trusted again, but not through openness.

We have openness. And yet we still have no trust.