Storing our future knowledge?

Over the summer, I read a couple of blog posts about knowledge storage that I marked to come back and comment on. Separately, Mary Abraham and Greg Lambert have suggested a fairly traditional approach to selection of key knowledge for storage and later access.

First, Greg issued a clarion call for selectivity in information storage:

Knowledge Management should not be based on a “cast a wide net” approach to the information that flows in and out of our firms. In fact, most information should be ephemeral in nature; addressing only the specific need of the moment and not be thought of as a permanent addition to the knowledge of the firm. When we try to capture everything, we end up capturing nothing. In the end we end up losing the important pieces of knowledge because they are buried in a mountain of useless data filed under the topic of “CYA”.

I had to Google “CYA”. And thereby hangs a lesson. How can we know when we make a decision about recording the present for posterity that the things we choose will be (a) comprehensible to those who come after us and (b) meet their as yet unknowable needs?

For centuries, the study of history relied on official records and was therefore a story of kings and queens, emperors and presidents, politicians and popes. The things that were left behind — castles, cathedrals, palaces and monuments as well as documents — actually provided us with only slender insight into the real lives of the majority of people who lived at any given point in time. Only when archaeologists and social historians started to untangle more trivial artefacts like potsherds, clay pipes, bone pits and everyday documents like manorial rolls, diaries, or graffiti were we given a more rounded picture of the world of our predecessors. At the time, those things were ephemeral — not created for posterity. The lesson we always forget to learn is that we don’t get to write our history — the future does.

Because Google has access to a vast mass of ephemera, I was able to learn what “CYA” means. In Greg’s context, it is the stuff we think we might need to keep to protect ourselves — it is an information security blanket.

Mary Abraham picked up the thread by addressing the Google question:

Folks who drink the super search kool-aid will say that the cost of saving and searching data is becoming increasingly trivial, so why spend any time at all trying to weed the collection?  Rather, save it all and then try Filtering on the Way Out.  On the other hand, look at the search engine so many of us envy — Google.  It indexes and searches enormous amounts of data, but even Google doesn’t try to do it all.  Google doesn’t tackle the Deep Web.

So why are we trying to do it all?

That’s a good question, and one that Greg challenged as well. I want to come to that, but first the Deep Web issue needs to be dealt with.

As I understand it, the problem for Google is that many useful web resources are stored in ways that exclude it — in databases, behind paywalls, or by using robots.txt files. That may be a problem on the public web, but it shouldn’t be in the enterprise context. By definition, an properly set up enterprise search engine is able to get access to anything that the user can see. If there is material in a subscription service like Westlaw or Lexis Nexis, then searches can be federated so that the result set includes links into those services as well as a firm’s own know-how. Alternatively, a firm or search provider can make special arrangements to index content through a paywall. There simply should not be a Deep Web problem in the enterprise context.

But what of the main issue — by storing too much, we lose our ability to find what is important? I think Greg and Mary are right to challenge the “store everything” model. There is much that is truly ephemeral — the e-mail that simply says “Thanks” or the doodles from that boring meeting. The problem with those, though is not that keep them, but that we created them in the first place. If the meeting was that boring, should the doodler not have gone and done something else instead? Isn’t there a better way of showing appreciation than sending an e-mail (especially if it was a reply-to-all)? I think that is the bit that is broken. Some other things are ephemeral even though they do need to be captured formally. Once an expenses claim has been paid, and the taxman is satisfied, there is little need to keep the claim forms available for searching. (Although there may be other reasons why they should not be discarded completely.)

However, I am still concerned that we cannot know what will be useful in the future, or why it might have a use. At the heart of an organisation like a law firm there are two strands of information/knowledge. The first is a body of technical material. Some of this is universally available (even if not comprehensible) — statutes, cases, codes, textbooks, journal articles: documents created externally that we all have to understand. Some is specific to the firm — standard documents, briefing notes, drafting guides: our internal know-how. I think this is the material that Greg and Mary are concerned with. And they are right that we should be critical about the potential immensity of these resources. Does that new journal article say anything new? Is that textbook worth the space that it takes on our shelves? Is our know-how really unique to us, or is it just a reflection of market practice? These are all crucial questions. However, almost by definition, as soon as we fix this material in some form it is of mainly historical interest — it is dying information. The older it gets, the less value it will have for our practice and our clients.

The other strand is intangible, amorphous, constantly shifting. It is the living knowledge embodied in our people, their relationships with each other and our clients, and their reactions to formal information. That changing body is not just responsible for the knowledge of the firm, but its direction and focus. At any time, it is the people and their connections that actually define the firm and its strategic preoccupations. In particular, what our clients want will drive our future knowledge needs. If we can predict what our future clients commercial concerns and drivers will be, then we can confidently know what we should store, and what to discard. I don’t think I can do that. As a result, we need to retain access to more than might seem useful today.

Patrick Lambe catches this tension neatly in his post “The War Between Awareness and Memory.” I looked at that in my last post (five weeks ago — August really isn’t conducive to blogging). As I was writing this one, I recalled words I last read thirty years ago. This is how John Dos Passos caught the same mood in the closing words of the eponymous prose poem that opens the single volume edition of his great novel U.S.A.

It was not in the long walks through jostling crowds at night that he was less alone, or in the training camp at Allentown, …

but in his mother’s words telling about longago, in his father’s telling about when I was a boy, in the kidding stories of uncles, …

it was in the speech that clung to the ears, the link that tingled in the blood; U.S.A.

U.S.A. is the slice of a continent. U.S.A. is a group of holding companies, some aggregations of trade unions, a set of laws bound in calf, a radio network, a chain of moving picture theatres, a column of stock quotations rubbed out and written in by a Western Union boy on a blackboard, a publiclibrary full of old newspapers and dogeared historybooks with protests scrawled on the margins in pencil. U.S.A. is the world’s greatest rivervalley finged with mountains and hills. U.S.A. is a set of bigmouthed officials with too many bankaccounts. U.S.A. is a lot of men buried in their uniforms at Arlington Cemetery. U.S.A. is the letters at the end of an address when you are away from home.But mostly U.S.A. is the speech of the people.

Sometimes questions about the “laws bound in calf” or “dogeared historybooks” are less important, maybe even a distraction. The real life and future knowledge of the firm is the speech of the people. That cannot be reconstructed. We need to be aware of all the ways in which we can preserve and retain access to it, for use when a client comes up with a new conundrum for us to help them resolve.