There is a more distinctly human dimension to the crisis of complexity that I wrote about last time, but I’m not as confident making declarations about it. The more technical, methodological questions I have long, tedious arguments about – people are trickier.

What I have in mind is: How do you get people sharing knowledge in the right ways? For centuries, the problem was getting enough knowledge passed from person to person. In some forms, that problem still exists. But there is also a new problem. Now (and by now I mean the last few decades, a century at the most) we’re faced with too much knowledge.

Liminal - By Andrew Rose

I purposefully don’t say “too much information”, because that word has taken on too much, and information is too cheap. There certainly is more data crashing around the globe than there ever was before, but almost all of it is not practically important. And furthermore, we have some exceptionally good methods for sorting relevant information from irrelevant. Anyone who has spent even a short time with Google cannot help but be impressed with this. No, the problem is more insidious than that. There is too much relevant information – too much that we really ought to know.

My encounter with this problem has mainly been through a Research Assistant job I did back in 2009. I was handed off by the professor to a friend of his, a global health researcher. This gentleman explained to me his idea for a collaborative tool, a kind of knowledge base that would go online to help Canadian global health researchers work together more effectively. A few of the big Canadian funding agencies were on board. I was tasked with part of the initial assessment – I was to pick a few low-to-middle-income countries, and put together some kind of survey of who was doing health research there. Easy, right?

During my initial research, I quickly came across the CCGHR, the Canadian Coalition for Global Health Research. Their mandate was to do exactly what I was tasked with – to facilitate coordination and collaboration between global health researchers in Canada. How exciting! They had even done a similar project, their Country Focus Strategy. It had been, as far as I could tell, a disaster. They had a low response rate, and ended up with more or less nothing useful. How the hell was I supposed to do better?

I liked this project, and really wanted to do something useful to move it forward. As I saw it, this was the practical version of my questions about emergence and complexity. How do you bring a group of people together in such a way that their knowledge becomes more than the sum of its parts? We have all seen it happen – online forums dot the landscape of the internet, where people spontaneously come together, organize themselves into little knowledge-hierarchies (newbs are quickly separated from old-hands) and information and skills are effectively disseminated. Setting up a forum seemed to me a low-cost, high-yield way of getting people together.

Of course, as Dilbert helpfully explains, it doesn’t work that way in a professional environment. As I got deeper into researching the idea of a professional forum, I found a sad, sad thing. The web is littered with the decaying husks of empty professional fora. Some kind-hearted soul like myself thought that, if only people had the chance they’d start sharing ideas right away, and poured a few thousand dollars of someone’s money into a slick custom interface. Then, they made one or two optimistic posts like “Well here we are! Pretty soon this site is going to be full of people sharing this and that!”. And that was back in 2008, and no one has posted anything since. There’s nothing sadder than a failed, empty forum.

I spent a year or so trying to pin down what magic it is that makes some forums flourish and others die. Clay Shirky’s excellent book Cognitive Surplus (he did a TED talk here) gave me some clues. For one, participation must be voluntary. As soon as you make it an assigned task, people will put in the minimum effort that they can. People need to feel a sense of ownership of the project, or else they simply won’t engage with it in the same way. But that means that if a forum is to be an effective collaborative tool, it needs to be, in a very concrete sense, self-organized. I was shocked, but my interesting, human-scale research had come around and made contact with my boring, technical research.

a simulation of self-organized fluid convection

I have to admit, I don’t know what to do with all this. I don’t think I pinned down the magic ingredient for successful collaboration (if I had, I would have bottled and branded it by now, and would be selling it to the highest bidder). I guess what I found was that there is no magic ingredient, in the same sense that there is no vital force or mysterious organizing power behind self-organized convection. If you want to make it happen, the only way to do it is to set up the boundary conditions properly, and wait. It’s perhaps more like gardening than engineering.

There is a more immediate instance of this human problem of complexity in my life. I study at the Institute for the History and Philosophy of Science and Technology. If that sounds to you like too many conjunctions for the title of anything useful, you may have put your finger on the problem. It’s a lifelong project to learn how to be a good Historian – you have to learn other languages, immerse yourself in a period or theme, and read mountains of primary sources. It’s also a lifelong project to become a good philosopher. Hell, just getting a grip on Kant is a lifelong project, and he’s only one of many thinkers that a philosopher worth anything will have a mastery of. Furthermore, its a lifelong project to master any one of the sciences! And we at the Institute are being asked to do all three – to be Historians and Philosophers of Science. It’s an impossible problem. There is no way to do it, yet we do.

Furthermore, there need to be people like us. The crisis is that we have mountains of knowledge, but no idea how to put it all together. If no one does that liminal work, the efforts of the 20th century to stack up more and more knowledge about the world will be for nothing. It needs to be integrated into a meaningful whole if knowledge is to function as a basis for action. But for that to happen, we need to solve the impossible problem, and make the whole more than the sum of its parts.


4 thoughts on “

  1. I’ll see your Dilbert and raise you one XKCD:

    Particularly the alt text: “Our brains have just one scale, and we resize our experiences to fit.” Sure, you can study Kant all your life — but opportunity cost lurks ever in the background. I actually violently disagree with the idea that there’s too much relevant knowledge out there, and wonder if I can tempt you with a different vision: that we are simply mistaken in most of our judgments about relevance. And so is Google. To say there’s “too much that we really ought to know” is contradictory — ought implies can, as they say.

    I’ve found that my relevance filters automatically get more restrictive the clearer and more precise a notion I have about what I’m doing, and in fact have started using the amount of material I choose not to read (and how fast I decide) as a rough index of how focused I am.

    Not that letting your focus fuzz out isn’t a crucial part of the creative cycle — breadth-first search (play) has to precede depth-first search (work) for you to figure out what’s worth focusing on in the first place. But if you think there’s too many things you need to learn then you either don’t actually know what you’re doing or don’t know how to learn effectively (this is isomorphic to the point I made in the other thread about asking the right questions). I sympathize — I’ve spent the last several years like that, and this hardass line is the outcome of several years of major-swapping and meandering self-study. A common mistake here is to confuse “interesting” with “relevant”, which are definitely not at all the same thing.

    Kurt Lewin used to say that there’s nothing so practical as a good theory. One measure of smartness is how little information you need to make a correct judgment. And if I need to know the digits of pi, it’s a lot easier for me to remember an algorithm that generates them to whatever degree of precision I need than to remember hundreds of digits. The core idea here is that there are ways of tacitly encoding a lot of knowledge into a highly compact form, and how to do this effectively is a huge part of the art of learning.

    Here’s an example: lately I’ve been obsessively learning to use Clifford algebras as a formalism for physics, because it turns out they’re enormously powerful — trigonometric theorems fall out so trivially that they don’t even merit special attention, you can condense Maxwell’s equations down to one equation in two variables and a single operator, {vector, matrix, tensor, spinor} can all be subsumed under a more general class of objects (multivectors), etc etc etc. This makes it enormously useful for everything from quantum mechanics to general relativity to control theory and robotics, etc. It’s a powerful secret weapon that will provide a basis for understanding Lie groups, differential forms, etc. as and when I need to, and it allows me to completely bypass all the kludgery associated with Cartesian coordinate systems and think directly in terms of invariant geometric structures. Spending one year on this stuff is literally worth an entire undergrad degree in math for me.

    I became convinced a long time ago that science is suffering from a Babel problem and that the fragmentation of knowledge tells us more about poor language design than about the world. Which turns out to be a special case of what I’m working on …

  2. OTOH, I commiserate completely over the mystery of how to get effective knowledge-sharing communities going and self-sustaining. I’ve been banging my head against this problem a fair bit and have yet to hit upon the secret sauce recipe.

    • So I do find your reformulation at least tempting, especially given your clever is-ought move. But before I buy it wholesale, let me ask a few questions. For example, how would you diagnose my situation with respect to category theory? I suspect (and let us say for the sake of argument) that it would be genuinely useful to me to learn category theory. Maybe I could say what I want to more clearly, and connect it to more technical literatures. My life would be better, my spiritual goals closer to attainment, and so on. It is, in that sense, actually relevant, and not merely interesting. Furthermore, I suspect (and let us suppose) that I am competent to learn category theory, if I really put my mind to it – we therefore don’t have to worry that I am simply unable to do it.

      But I’m in a similar situation with respect to thermodynamics, which I really would benefit from learning the mathy bits of, as a lot of important work on part-whole explanation goes on in the literature on thermodynamics. I would also really benefit from learning more about evolutionary developmental biology (again, rich fodder for part-whole talk). I could multiply examples of things I would genuinely benefit from mastering, and could feasibly learn in a reasonable amount of time. But obviously, I can’t learn them all.

      So how would you diagnose my situation? Am I mistaken that these things are genuinely relevant? I am able to do any one of them, but clearly not all of them, so what should we say about the is/ought problem?

      I mean, I suspect that we agree on the basic human dilemma, but want to talk about it slightly differently. I want to say that that stuff I wish I could learn but don’t have time to really is relevant, and you’d rather deny that. Your argument about Clifford algebras strikes me as an example of the incredible successes in dealing with complexity that are possible by pursuing an analytic cognitive strategy – sharping and straightening your frame as it were. But surely you’d agree that being analytic doesn’t always work, and that we’ll always be re-framing our experience as circumstances change.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s