How Management Ratonale fails at IT (and what to do instead)

Apparently 70% of the Met Police's IT systems are obsolete, and it takes people half an hour to log in. But can the sustained, ongoing perspective of driving up "efficiency" by driving down costs bring a solution about? What does this quote, from the Chair of the Budget and Performance Committee, really mean:

The Met cannot afford to go on like this. Its forthcoming strategy must address these problems while focusing on the potential that new technology offers, to drive down costs while increasing productivity and boosting public confidence.

 As an engineer, it feels like none of these are particularly good primary reasons to improve the performance of a system.

For a start, potential is always around us, as the very nature of technology and progress. Making things worse is generally only done as an experiment in failure, in order to learn. Not as an end in itself. So duh, citing potential doesn't really mean much.

"Driving down costs", "Increasing productivity" and "Boosting confidence" - aren't these all just the effect of technology on other domains? As an engineer, there is no inherent economic value of making something "better" - but others may (and will) apply translations to turn change into measurement. Time is money, sure, but engineers focusing on saving money are not doing an engineering job.

Similarly, you wouldn't get a very useful system if you started out with "public confidence" as a primary use case.

So what does make a good rationale for planning an IT system? How can we bring about useful change without getting sidetracked into secondary effects? What model of management might encourage and reinforce this?

Sword and Shield

From a client-facing engineer's perspective, there are two key processes/perspectives needed, if engineering is to be a success. I'd say these are true of any engineering task, or project.

The first is having a clear goal - and if not, then deliberately specifying resources as experimental. Knowing what you want to achieve is the first key to getting a system running. Without this, how you get there becomes confused. As above, this needs to be a practical goal, not an abstract side effect. "Save me money" is not a technical goal. Nor is "Fix this problem". Users taking half an hour to log in is not a system problem, but just the way the system runs right now. It's only a problem if you're in a hurry.

The second is puzzle solving ability. If you're in a pickle, then you need a way to work out a solution, rather than wandering blindly around hoping for a solution based on luck. Puzzle solving is lacking in a world full of politics and plans. But without it, any attempt to change the system is worthless - even dangerous.

Problem solving is about analysing the system, and working out why it's doing what it's doing. It's about understanding, rather than wishful thinking. In fact, it's the opposite of wishful thinking. There are no wishes - and once you understand what's going on, there's not much thinking to do either.

Management as Spies

It seems amazing that management love plans, dashboards and reviews in order to find out what their people are doing. But have little in the way of understanding what the system that's being built by those people is doing. This is like an army general sending spies into their own camps to root out dissent, rather than finding out what the forces across the river are intending.

If management want to really improve productivity and cut costs etc etc, then it's not just a long term IT strategy they need, but a long term puzzle solving strategy. Costs are always relative, to both short and long term. Finding a solution that works isn't based on numbers though.

Tackling 400+ systems is an insane task, ie. it would send anyone insane. And having a crazy general leading you is fair grounds for becoming a Conscientous Objector. Any management looking to make real, sustainable changes needs to a) know what its "battles" are - ie. use the puzzle solving aspect to figure out just what the key problems affecting work are, and b) wield wisdom in choosing which battles to pick first - ie. which clear goals are the most important.

The second of these might well be based on other needs - deadlines, cash flow, etc. But in identifying such battles, one needs to be careful not to confuse needs with engineering process (that is, the two should be separate), and secondly to be clear on where that needs come from, to avoid confusion in the future (or during the work to unpick it).

Everything else will probably fail.

"BBAs" et al: Do policy buzzwords attract or filter people?

I tweeted this after the recent elections:

Having a hard time taking elections seriously.
— Scribe (@6loss) May 3, 2013

Now I'm reading this speech on bus subsidy reform, and just realising why it's so hard to get my head into elections - or a lot of "reform" and "policy".

"Better Bus Areas". "Portas Pilots". "Big Society". "Pasty Tax". Oh God, the list of buzzwords uh, I mean hypewagons, uh I mean "policies" just goes on and on.

Democracy - or rather democratic discourse (which is a different thing, but includes speeches, media coverage, and general efforts to increase enthusiasm) - has become a boring hamster-wheel of phrases and symbols.

Taking a lead from the private sector, it feels like politicians are wont to sell society in more ways than one - not only in the practical terms of outsourcing, but also (and perhaps more irritatingly and dangerously) with regard to how customers citizens buy into engage with the process.

Yes, there's some kind of thought-process going on behing all of these, some kind of "actual" attempt to solve the real-world problems we do need to address. I appreciate that, really.

But by the time this thought-process has become a "programme", it's been so tarred with political context, existing decisions, and PR "acceptance" efforts ("What's the hashtag for this new policy?") that engaging is about as engaging as a Microsoft sales chat. (And sure, some people like that. Some people enjoy politics too.)

The interesting aspect of this is whether such a technique is to entice people in with a brushing of faux-simplicity, or to discourage them, leaving the nitty-gritty to people with the time and background to understand the real talk and consequence behind the symbols.

Should policy be a user-friendly, read-it-on-the-loo affair? Or an academic, know-what-you're-talking-about conversation? Both have advantages.

Maybe I'll just carry on grumbling when the price of a single bus ticket goes up again.

UKGC13: Together we have the tools and guts to kill off antiquated democracies

This year's UKGovCamp was a week ago, but Kids and Side Projects have made it difficult to write it up. Here goes though.

I semi-deliberately didn't stand up to pitch a session this year - partly because I was coming out of a too-busy fortnight, and partly I wanted to spend this year worrying less about running a session, and listening more to what people wanted to talk about; over the years, govcamp has taken on a bit of a “zeitgeist” role for me, offering a chance to gauge the ongoing mood.

I think not pitching was actually a good decision - this year's event seemed (to me; YMMV) to be a bit less all over the shop, ideas-wise. Several main threads ended up stringing the day together, which I'll go through below.

For ref, the sessions I wandered into (ripped from the sweet Google Spreadsheet) were:

1. Identity in general & philosophical + Managing personas and identities in personal and civi spaces, with @curiousc and @pubstrat
2. Open data in 5 years time + structure in open data community, with @jenit and @hadeybeeman
3. Real, grass roots collaboration, with @shortblue
4. Open data skills in communities + taking council information out into communities, with @podnosh and @tiffanystjames
5. Open Government and the National Action plan, with @timdavies

Notes on each session are gradually going up here.

(photo by alexljackson)

More than once it felt like the same conversation was taking place in more than one room at a time. As I said, several (really importantt) threads seemed to emerge as being stronger and differenter:

1. Open Data and Social Meda are converging

Gone are the days where it feels like “tech” types are in one set of rooms, and “social” types in another. Or maybe that was just me. But it’s often felt like sessions have fallen either into a “how to do conversation” camp, or “how to do tech” one.

This year, the two camps felt closer, like an estranged couple gradually getting to know each other again after being separated. At one point, I found myself - without shame using data and statistical models as an analogy for individual identity. And the idea of "personal data" (energy efficiency, money saved, etc) makes it even harder to separate figures from debate. Are we at a point where “the network” is becoming less niche about specialist subjects?

I can't work out if geeks are actually cool yet though.

2. Democracy is killing openness

A biggie this one, even worth a separate post. The discussion in the “Grass Roots Collaboration" session was small but fascinating, and delved into the contrast between how the private sector deal with failure, in comparison to the public sector, and what the latter could learn perhaps.

In fact, the theme of failure was the main thread running through the day for me, from social media crisis to how to spin Open Data case studies. And in particular, how failure interacts with democratic process, and vice versa. “Democracy”, often shied away from for being too big a target (too big to fail?), was really the elephant in the room(s).

Can it be a coincidence that “private” companies are often likely to say sorry for public cock-ups, while supposedly-transparent “public” bodies go to great lengths to sweep such events under the rug? Maybe (like yin and yang creating each other) closed groups tend towards (selective) openness, while open groups tend towards (selective) secrecy.

 

(photo by alexljackson)

Which is basically to say, no wonder the "Open policy-making" / Gov 2.0 movement is an uphill struggle. No wonder that ideas of genuinely open debate or technology get spun into poor, PR-driven facsimiles. Right now, Democracy, in its Representative and Accountable form, is all about hiding failure instead of learning from it.

On the Open Data front (which is probably my main interest, it's telling - and, yes, extremely encouraging - that the debate at govcamp has moved on from choosing data and getting it in machine-readable formats, to how to embed date into debate, decision-making, and the wider world.

Coming back to theme 1 above, we are no longer having 'technical" discussions about what web services to use. We are talking about how to adapt to existing democratic processes.

Or if that doesn't cut it, how to change democracy itself.

3. Govcamp Got Guts

... which is where I waffle on about how, like every year, govcamp was unexpectedly not what I was expecting and how full of enthusiasm and renewed vigour I am again etc. (And also mention Podnosh and Delib (and all the other sponsors) for the awesome free bar. Please do go and listen to some Neutral Milk Hotel.)

But unlike previous years, the fallout seemed more... "constructive" this time. Maybe it's an effect of a post-recession economy, some inspiring central government work, the election cycle, Ant n Dec, or those mice.

But maybe it's finally enough years of everyone asking "so what (now)?" afterwards. Maybe it was IBM's pop-up plug-points and sci-fi projector screens, or the snazzy T- shirts, or the return to a one-day format, or the electronic session list or those Bytemark mugs with Your Name on.

(photo by sharonodea)

I’m not the only one. Do go and read this post from LouLouK too, where she writes:

“Are you happy sitting in a room, being brilliant [but] never letting anyone else actually benefit from that brilliance, or are you going to stick your head above the parapet?”

Whatever it was, it feels like the ideas and debates at govcamp now have much heavier implications outside its gathering. I wonder if what people saw, heard and discussed made them feel more able to challenge the status quo, at a time when new questions, new answers, and fundamentally new ways of doing "government" are correctly needed.

Maybe we’ll find out next year.

"Local Government has been Emasculated" (But so has everything else)

From a systemic perspective, the central-vs-local government clash continues to be pretty bemusing.

On the one hand, central government decides how much localism there should be, and giveths and takeths away from local or super-local (regional/LEPs, etc, whichever the trendier?) all the time.

On the other hand, the hierarchy at all levels is being infiltrated by both the private sector, and the "new", networked form of group organisation which has managed to somehow avoid the "postcode lottery" moniker so far.

Lord Heseltine is advocating even more decentralisation away from central government, and offers some small looks into where local-interest power means that Stuff Gets Done, ie. that the hierarchy needs to get out of the way of people just getting together to achieve something.

A few days ago I started thinking about how a community wifi effort (Google Doc) could benefit and be hampered by a "network" approach vs a "hierarchical" approach. (Hint: there's no straightforward answer.) Richard Veryard suggested that the network/hierarchy split (or spectrum) wasn't too useful, which I'm still unpicking, but it did get me thinking about why we organise into different types of group.

It is perhaps most true to say that democratic models are the most complex they have ever been. This is different to saying that "we need to move to a network democracy" or that "it is local government's time".

It is understanding instead that we have a very real situation of multiple, overlapping, integrating models of democracy organisation/objective/philosophy. There is no "one approach fits all" solution because there never was - we just managed to shoehorn the world into the latest mainstream model through a heavy imbalance of power.

Local Government has been emasculated. But so has central government. And perhaps so have ordinary citizens along the way.

I'm going meta. There is no such thing as democracy, but there is such a thing as a democracy network. Which, confusingly, includes networked democracy - but also centralised democracy, European democracy, local democracy and hyperlocal democracy.

Which means there's no point arguing over which one's best, only over how we make sure they all get along.

Why the Public Sector Needs to Get Stupider

...or "From Intellect to Intelligence".

There are a handful of people whom I've never met in person, but talked to a lot over the years online, and who have influenced my thinking more than I could suspect in that time. Richard Veryard is one of those people. Even back when my blog "Into the Machine" was called "Blunkett is an Arse", Richard prompted me to think heavily about the role of power in society - everything from CCTV through to newspaper headlines and reasons for going to war.

Richard introduced me to Stafford Beer's concept of POSIWID (The Purpose Of the System Is What It Does) which trips off the tongue more easily and addictedly than you'd expect. Recently, he's focused his efforts on Enterprise Architecture ("#entarch") and Organisational Intelligence ("OI"), and I'm very much enjoying shamefully yanking ideas out of his OI Primer e-book.

I don't plug things very often on my blogs (and it's horrifically short notice) but I really do want to mention his 2 courses this week: on Business Architecture and Organisational Intelligence. I think there are still spaces left. And obviously as I've never actually met Richard, I'm not guaranteeing anything. But if you're interested in this post, or this blog, check em out. There's also an Org Intelligence group on LinkedIn for good discussion.

But it's more than just a shameless plug - and more than just a way of paying him back for the insight over the years. Fundamentally, his driving factor (IMHO) is around something which is not just lacking in many groups, but dangerously lacking across society. In fact, my whole last post on the PCC elections, nuclear subs and mythological engineering was about this lack.

In short, we lack systemic intelligence.

What does that really mean though? And more importantly, so what? Many smart people instinctively know that our most common modes of organisation lack an inherent "togetherness". I'm not saying anything new here.

But what we can learn from Richard - and from looking around us, and from taking a moment to think clearly - is how to move from just knowing there is a lack of systems-thinking, to doing something about it. To rejoining the things which are fragmented.

Departments are Inherently Silo Factories

In the public sector, the defragmenting process is bubbling under like the power behind the throne. It's why the Patchwork project makes sense to people. It's why everyone goes on and on about breaking down data silos. It's why joined-up government arose - but ultimately, its fall also highlights not how difficult it is, but how badly prepared we are for it.

Our pin-factory-style, categorical hierarchy mentality likes to divide work up into efficient processes. Many people claim this to be an "efficient" way to focus on a particular labour task - which it can be, to an extent. But this also leads directly to those data silos we mentioned earlier - because it's not the data which is inherently silo'd, but the labour itself. Once a person or a group becomes responsible for an organisational function, both the data and the importance that result from it become an asset - and assets are there to be protected, especially under competitive pressures.

The same effect is at play whether it's one person keeping their spreadsheet private, a department keeping its data in obscure formats, or a company slurping personal data on millions of customers. Data goes hand-in-hand with function, and divided data leads to divided functions.

The Road to Recovery is Balance

The first challenge in reconnecting these functions (and therefore departments, and data) within a system is to get out of the existing model. It's important to remember that an fragmented viewpoint is mutually exclusive to systems thinking. Systems thinking is not about the best way to get two existing units (departments, groups, people) to talk to each other, nor about opening everything up to the glorious light of transparency.

Systems thinking is about how the units depend on each other and what it makes sense for them to share in order to do their (otherwise standalone) job. Object-oriented coders Get This.

In other words, the underlying mental model for a more systemic form of organisation needs to shift to one of multiple contexts - firstly, the overall purpose of the entire organisation, and secondly, how the needs (data, assets, etc) of each unit within that organisation break up and overlap.

That is, there needs to be:

1. Balance between the output of each unit within the organisation, and the organisation as a whole - no one unit should be seen as "more" important, in the same way that the brain is not "more important" than the stomach. The priority is that the organismisation acts as a whole.
   
2. Balance between the "internal" and "external" roles of each unit, so that effort is not wasted on either information over- or under-provision. Each unit should have access to the information it needs to do its job. And yes, this changes over time. Get used to it.

Forwards, stupid workers! 

"Architecting" this is not just a role for a head honcho or a business consultant. Ignore what well-paid people tell you. It's not even a bottom-up process.

The needs of units and the whole organisation change over time - technology relating to certain tasks changes, staff change, external factors mean new pressure, political goals shift. As a result, this process has to be infused into the whole of the organisation. It is a culture and a strategy on one level, but on another level it is also an individual choice. 

The divided labour approach to work - ie. what you have in your head the moment you get to your desk in the morning - intrinsically ties together function with knowledge. "What do you need to be told in order to do you job?" - or from a more negative, jobsworthy viewpoint, "I can't do that because nobody has told me why or how."

Divided labour resists flexibility. And this is why joining everything together takes more than someone banging a gong or drafting a "data sharing" proposal - the rules themselves need to be flexible for flexible working to work.

But so often, the body of working knowledge we hold - and the reasons why we hold it and where we should apply it - are valued. They form a set of ideas centred on our job function that I call "Intellect" - in other words, a specialised definition that takes the immediate environment, and hardens it - formalises it. This means we can know if what we're doing is "right" or "wrong". And we can't be judged for not doing stuff that lies outside of that context. Nobody ever got fired for buying IBM.

Losing this "Intellect", this hard-wired context, is key to making our organisations smarter.

We need to become stupider.

Photo: San Diego Shooter

The Dashboard Wisdom of Stupidity

Being stupid is not a weakness. I heartily wish that I had been more stupid in school, and wasn't so afraid to ask questions when I didn't fully understand something. Because when all the parameters around you are shifting, guess what? Nobody really understands anything. And even more so, nobody can predict what is happening to the people they're connected to.

Asking questions gives us feedback. One of the fascinating areas I looked into after reading Richard's book was the usefulness of dashboards - which are basically designed to answer that perennial question, "What's going on right now?" 

Feedback is only useful if it's fast - fast enough to allow us to relate our actions to their consequences.

And immediate feedback is awesome because it gives us an answer before we've even had time to ask the question. A dashboard, an ever-present source of up-to-date information, assumes we are stupid - or, at least, could turn stupid at any moment. "Hey! Driver! Sooner or later, you're going to need this information, you know?"

And when you're sitting there, protecting your body of knowledge and your data in order to protect your job, hanging on to that "Intellect" you've carefully gathered doesn't make you "Intelligent". It makes you "Wrong".

Your Silo is My Silo

The world is changing at a much faster pace than we're ready for. No, scrap that. The world is change, and we don't like change. Everything we do is about controlling change. Our policies are direct descendants of the 17th century,and hence our mental models are struggling to keep up with the change going on - change that we've created, ironically. (Remember, pin factories didn't always exist.)

"Systems" intelligence and "Organisational" intelligence are misnomers, because people are individuals, not organisations - and organisations only really exist on paper and in law courts, and can't make me a cup of tea in the morning.

But individuals exist within organisations, within systems. There is little to differentiate the two except perhaps where we stand. 

Accepting that we are part of a system is essential to realising that this "systemic" intelligence is also a part of us. It is an aggregative effect - a flow of cause and effect from person to person, and unit to unit. It is not a "culture" or a "programme" or any "Thing", because it is always a dynamicism that acts on every level.

Asking questions is key to opening up that flow - the flow of information and understanding between units, between silos, and between contexts. It breaks down the protection of Intellect. It is a form of empathic efficiency. It is flexibility. 

It is necessary.

Want to know more? Richard's courses on Enterprise Architecture and Organisational Intelligence.

PCCs, Nuclear Subs, and Mythological Engineering

With the low turnout in the PCC elections yesterday,  and this interesting piece on the construction problems with the UK's new nuclear subs, I can't help but think of Barthes' "Mythologies" and the state we're in regarding symbolism vs engineering.

i. asking the right questions

Barthes lays out a sketchy-but-useful/interesting framework for signs and symbols, and in an ever more mediated world (first through broadcast media such as TV, and increasingly through short-attention information-overload) this framework seems more relevant.

The 3 main questions that a mythological viewpoint helps get at are:

1. Is what we're doing actually working?
2. If not, can we find out why not?
3. And if not, are we willing to identify where our own beliefs are causing those reasons?

None of these are significantly mythological - we can fail at something because we lack understanding or experience, for instance. Indeed failing is also learning, if the context and expectations are set out properly.

But the PCC and Nuclear Sub cases, when viewed from a symbolic, mythological viewpoint, become a lot more understandable.

ii. nothing to see here

In the case of the PCC elections, it's telling that many people voted not to improve the operation and efficiency of their Police force, but to keep out people who they think would make their Police force worse. Question #1 has moved from "is it working?" to "is it not breaking?". Or, in other words, from "how can we improve things?" to "how can we stop things from getting worse?"

This shift in rationale is essential to understand. It shows that politics, a symbol in itself, has become a stronger signifier for maintaining the status quo than for setting out sustainable solutions that involve cross-society engagement. We can say that Democracy has become about keeping people out rather than integrating people in.

The PCC elections are clearly to be seen in a light of localism and "democratic accountability". The elections are a symbol for democratic principles, but ones which have left behind everyday operations of how policy becomes policing. The "polis", in every sense, has become an abstract notion, and can only surely be less empowered because of that.

Not only was it seen to be more important to make the system "votable" than have a working system, the idea of the vote itself took precedence over the voting process.

iii. an aside: symbolic actions

Barthes contrasts the object as a "symbol" to the object as part of an "action" - eg. a woodcutter chopping down a tree performs an action on the tree, and so there is a real-world interaction and effect with the tree. However, the image of a woodcutter cutting down a tree, or a generic image of a tree in itself, is "symbolic" in that it has no direct, real-world consequences. A logo of a tree may act as a standard, but is an indirect effect, and can certainly be detached from the actions it inspires. (Many people find it easier to be inspired by a logo than to act based on inspiration.)

iv. unclear power

Nuclear power is another symbol, as any fan of Dr Strangelove will understand. The paradox with nuclear power is that it is "inherently" (or "naturally", as Barthes would put it) powerful and therefore dangerous - we accept this without question. By wielding "Nuclear power" as a symbol, we also wield an intrinsic argument for more control and global sanctions - because power is concerned with potential rather than actual abilities.

This paradox leads to the problems seen under the HMS Astute nuclear programme. Building nuclear power as a symbol is very different to building nuclear power as a "thing" in the real world.

The paradox is this: You cannot design a machine (or an organisation, or a person) based on symbolic or economic principles.Sure, they can feed into the process, but engineering needs to maintain an inherent, internal coherence to provide a "thing" which functions as a whole. Allow a fragmented economic-political process to take priority, and you will end up with a fragmented machine.

Ironically, the nuclear machine does not depend on the nuclear symbol in order to work, but the symbol is impacted by the effectiveness of the machine. In undermining the effectiveness of the machine, the symbol also devalues itself. 

v. tappity-tap

Churchill said that "Scientists should be on tap, but not on top". I don't believe it's a case of scientists (and engineers, and anyone that designs any kind of process) being on top of  politicians, or vice versa though. The two "sides" have very different briefs and contexts that need to work with each other. There is a "what" and a "how", and the twain will always meet somewhere.

Kasabi, Nuclear Power, and the Data Developer Dichotomy

Today's big news is that Talis are shutting down Kasabi, their linked data platform, moving away from semantic data, and losing Leigh Dodds. Talis have probably been the most visible entity in the commercial linked-data realm, and with good reason. They showcased an impressive local data portal for the Open Data Cities Conference a while back in Brighton, and drive various high-profile services like Fix My Street.

But it turns out the demand for the linked data platform just isn't enough to support business. As a data developer, I'm obviously intrigued by this move from Talis.
(Spoiler: There's a lot of background here, and I don't even answer my original question. Feel free to skip to the end.)

How I Learned to Stop Worrying And Love Data

Your front room? [img]

Back at Uni, I read that nuclear power isn't just about physics, technology and war. To wield nukes, a state needs certain organisational structures to support their centralised nature - security structures to protect them, political structures to commission, deploy or maintain them, market structures to permit or prevent sales, and so on. In short, technologies and their usage depend on social-political contexts - and vice versa.

At the other extreme, well away from nuclear power, micro-generation of power is slowly taking a footing. Feeding power into the grid from solar and other electricity generators also requires contexts to be put into place, but the nature and distribution of these are different to those of nuclear power. The consultations yield different emotions, the networks require bi-directional structures, as do tariffs and security measures.

Overall, there is a big difference between the "central" nature of nuclear power, and the "decentralised" nature of micro-generation.This maze of contexts is what Open Data is trying to navigate right now. And Linked Data is part of that, often trying to straddle worlds.

Platform Non-Wars

Once upon a time, XML was the saviour of the data world. Up until then, people had put up with CSVs and related 2x2 grid matrices. Data was a person-to-person communication tool, not a machine-to-machine one. Machines kept their own data, and made it available through their own interface. So passing data around was a presentation job, and people liked doing that in 2D.

XML changed that by structuring data differently - the context of XML was a machine-to-machine medium. Some simple XML can be coded by hand, such as HTML. Other XML (and lots of HTML) is really intended only to be generated and read by machines, which will do the validation, parsing and rendering without the aid of human hands.

XML can afford to be complex then, because the context is strictly defined by standards and parsers. Maybe XML is the nuclear power of the data format world.

Then people discovered JSON, a much more lightweight format which was still supposed to be machine-readable, but which also caters more to a second context - person-readable. Because JSON is so lightweight, it's much easier to debug by hand. And while there are huge industries booming around XML, the lure of JSON appeals to those who don't have rigid structures and definitions in place, who need to do something quickly and often by hand, at least at first.

Like nuclear and micro-generation though, XML and JSON aren't a "versus" or a battle. They're both just about what's appropriate for what context. This is an Important Point.

Open Wide, Please

What could possibly go wrong? [img]

This is where we return to Open Data, and the position of Linked Data within.

Right now, I argue that Open Data is trying too hard to cover everything in one breath. It's doing this because it very rapidly became a cultural symbol rather than an engineering symbol. With the agenda being led by "transparency" and its associated technical underpinnings (yes, the 5-stars of open data are a political quest, not a technical one), we tend to overlook the idea that Data - the other part - is not a standard but something which requires context.

Without context, data is just entropy, and entropy is already handled pretty well by our underlying electronic veins: the binary transistor.

In other words, is "Open Data" an oxymoron? If something is to build on its openness, it needs to flow, and if it is to flow from place to place, it needs to acquire meaning - at which point, is it no longer "Data"? Are we better off talking about "Open Information" - and if so, what does that mean for our tools?

So where does Linked Data and the plight of Kasabi fit into this? From the above, we can see that as "data" moves around the ecosystem, the people using it want different things from it, and will do so using different tools. This is a translation task - to adopt someone else's data, not only do you need to know how the data works in itself, you also need to know how it integrates with your own data. (This is also why Data Engagement is so important.)

The data developer's instinct is to build something generic and beautiful. This is further impounded by a commercial instinct that conforms to economies of both scale and scope.

But in reality data often resists the genericism that technical and economic efficiency loves. A generic data handling system would need to be so generic that it could do anything - at which point, is it any different to any database? Any computer? In her Open Data Cities Conference talk a few months ago, Emer Coleman said that "People are messy." And if people are messy, than people talking to other people is even messier - think Tower of Babel. In the real world, data is equally messier than we think it is.

One Chance Out Between Two Worlds

Not your usual armchair auditor [img]

This is why it's difficult to talk about Open Data and Linked Data in the same sentence, or sell all-encompassing data tools, or come up with "universal" standards that try to transcend contexts. The contexts differ, and translating data from one to another also means translating mindsets, working practices, learning processes, and organisational structures. Data Relativity is still being ignored.

Right now, Linked Data and the Semantic Web are in a funny position. They aim big, and are trying to solve an important problem about data quality. But this big aim means big technology and paradigm shifts - putting Linked Data much more into the realm of an "Enterprise" app in the same way that nuclear power and XML require a certain, quite hefty, amount of planning and structuring to achieve.

A lot of large organisations with "heavy", nuclear-style data have a lot of systems already in place, and a lot of knowledge and resources tied into these - in other words, they have a lot of momentum. This is where Linked Data could make a real difference I think, but the inertia that needs to be overcome is a fundamental issue. Not only that, but because the success of Linked Data is inherently tied to network externalities, that inertia is multiplied up through inter-organisational lines. To be good, many people need to adopt it at the same time.

On the other hand, the economic agenda of Open Data wants armchair auditors and clever freelance developers to innovate quickly for low cost. And the tools, knowledge and skills these kind of people have are geared up not toward "Enterprise-level" data, but toward quick-fire, loosely-knit, human-readable innovation. Find a small problem, and solve it quickly.

These two worlds are currently circling each other. All too often "Open Data" becomes confused with "Big Data" because there's a lot of it. Some of the challenge involves filtering out "Open Data" so much that it's invisible - background noise - leaving just the "Small Data", the useful stuff. Learning a 5-page API reference doesn't help this.

The other challenge is to make the link between "Big Data" and "Small Data" bi-directional. Or omni-directional. There's still massive amounts of work to do around microgeneration of data and feeding user-generated data into "the grid". There's still no real work around standards for data sharing, as far as I know, other than those that exist for content such as RSS, etc.

Right now though, the main challenge is just getting out of the Open Data mindset that data needs to be Big and Open in order to work. It's really important not to get too caught up in the lure of simplicity and easy wins for quick economic and political gain. Ecosystems are not built on top of economies of scale and scope.


Remember Babel.