The New Yorker, Kindergarten Edition

The New Yorker is by far and away my favorite magazine, which is why it was disappointing to read Nicholas Lemann's adolescent screed under the label "Fact". Unfortunately Lemann was clearly hoping that by savaging blogging he would get a loud round of denunciations. But well, the New Yorker's cycle time is so long that the "oh blogging is just little people scribbling and it will always be us professionals" article merely makes people roll their eyes - point out a few of the obvious errors, for example bloggers don't call reporters "the priesthood", that is strictly a pseudo-libertarianism, but instead far more often used is the term MSM, for "Mainstream Media" or "Media Whores" for the specific individual. Atrios might just have a link and say "Leeman calls for a blogger ethics panel!" Id Est an example of poor reporting in top down media circles which self-servingly defends the money that its members make...

Lemann's work is so bad, that it isn't even worth linking too. If the New Yorker wants to show off how well its kindergarten is doing, well, that's the New Yorker's business. It is sad and lamentable that even the best magazine in the English language cannot help itself when it comes to protecting economic rents, but it isn't surprising at this point. But since they don't want to even make a pretense of engaging in discussion, my guess is that Lemann won't listen until he sees a guilliotine going up outside of his office, he even attempts to goad on an electronic version of this. Truly sad, yes, but worth an explanation of what he gets so badly wrong on a fundamental basis.

In the interests of full disclosure I'll note that he cites fellow bopnews founder Jay Rosen, but I rather suspect it is the New York connection that is the point, Lemann is the dean of the journalism school at Columbia, and is, thus, protecting guild privilege. In the sciences we used to have a joke that an error was so bad that it would cause students to be dismissed, and professors to be promoted to head of the department, and Lemann's given an example of exactly what that means.

However, what's obvious when the best magazine in the English language can publish such an incoherent and self-contradictory screed, that something is terribly amiss. My favorite two moments being where, on one hand, he states that internet journalism supporters argue that amateurs can be journalists, and then says that the number of internet journalists has expanded the "profession" of journalism greatly as a way of sneering at it. The clatter of this oxymoronic argument is followed by another argument that doesn't even have the oxygen in it, when he simply tries to pigeon hole the most ambitious blogs as being merely cross referenced penny dreadfuls, while implying that bloggers don't know this. If whatever passes for a fact checker over there at the New Yorker had looked, he or she would have found numerous examples of how blogging is aware of the penny dreadful press as part of its antecedants, including Marcy Wheeler's excellent post from some years ago comparing blogging to feuilletons.

The non-arguments in the piece have been disposed of before, and like most self-serving economically motivated hatchet jobs, it engages in personal insults and disputatious attacks, in hopes of baiting people into anger. Which can then be used as "proof" of the social unfitness of the people who were attacked in the first place. Again, tiresome, old, juevenile, and so obviously motivated by Lemann's economic interests as to be cliché.

However, clearly since Prof. Lemann does not understand the basics of information theory, or more probably was so blinded by his own self-interest as he chugged out his prose - it seems worthwhile to go over the really unforgiveable mistake in his piece, that of making an assertion about credibility and authority which is contrary to the entire theory of credibility as it has evolved over the last 50 years. Errors of fact, logic and civility are a dime of dozen, and can be handled by anyone who wants to refute nonsense in detail - but errors of concept and paradigm indicate that there is a need to, again, patiently explain fundamental ideas. I don't expect Lemann to understand, his insulting attack clearly means he's enraged at the threat to his position beyond the ability to engage in any sort of rational discussion, sadly, the graveyard still makes more converts to an idea that the blackboard. but it would be nice if people defending their economic interests showed some passing interest in the economic system which those interests are embedded in.

The Theory of Credibility

Societies create structures of authority for producing and distributing knowledge, information, and opinion.

This is simply wrong. It is one of those statements so contrary to fact and practice as to make me wonder which planet its author comes from, but, given that by this point in the article, a careful reader would know that it is the same planet where neither the financial scandals of the NASDAQ crash, nor the Iraq war, happened, it is at least obvious that it isn't the little place we call earth.

Authority is not about producing and distributing knowledge and information, instead, whether one is a post-structuralist in the mold of Derrida, who wrote a very thick book about how authority works to counteract dissemination. The reason is that one of the fundamental signs of academic authority is that a group of people given the same inputs will produce the same outputs, that is how we test what we teach. Dissemination, however, does not return to any stability, because words are seminal - the produce in those that read them new ideas, which cannot be forced back to the original without some form of power, that is authority, to establish some set of original statements as better, more correct, more privileged than others. In short, one job of authority is to squash out the heresies that inevitably rise up.

Nor does authority produce information, instead, authority produces credibility. That is, it reduces the amount of information produced.

The development of credibility as an idea is one which has received more and more attention in the last 25 years, in no small part because the question of central bank credibility in a floating currency monetary era has been one of intense study. Credibility rests on restraint of freedom of action, and the need for that restraint rests, in turn, on the nature of printed currency - since printed currency doesn't mean anything other than some monetary authority blessed it, there is a natural tension between those that produce currency, and therefore hold authority, and those who use it. It is admitted that having one monetary authority is preferable, because if everyone produced currency, then the costs of deciding which currency is credible and which currency is not credible would consume all the economic activity of society - and it is further admited that the monetary authority must limit the amount of economic information carried in currency terms, or else each individual note will reduce to carrying no information at all. The moments where these failures happen are called a financial panic, and an episode of hyper-inflation respectively.

The job of authority is then to weed out poor quality information, information that is not worth wasting time over, because the reduction in uncertainty or the increase in area of coherent certainty is too small to be worth the energy of processing it, and to make sure that information is interoperable, because it is the ability to transmit information which, in fact, is the defining quality of information.

A simple way of putting this is that the Federal Reserve doesn't create economic activity, nor does it distribute goods or even money, but, instead, creates a restricted bandwidth of currency within which institutions must decide how much credibility, that is credit, to provide to those who attempt to prove that a particular asset is worth a particular amount of currency, and therefore qualifies for a loan. The federal reserve doesn't make anything, it gives permission to banks to declare that something already made has value measured in dollars. The process by which this happens, as Galbraith notes, "is so simple as to repel the intelligent mind."

The purpose of authority then is to reduce the quantity of information to manageable amounts, and to increase the volume, which this information then fills. That is, it assures a certain quality of information so that users do not have to spend large amounts of their own time and effort in doing so.

Authority, then, is about generating a heuristic, or series of heuristics, from which people can construct maps which allow them to judge facts and assertions which they cannot themselves directly test and verify. A good work to start with is Thomas F. Gieryn's Cultural Boundaries of Science which talks about the heuristics that people evolve to judge the credibility of information.

Open and Closed Models

There are two theories of how to do this, one is the proprietary theory. Proprietary information asserts a chain of argument. The first step is that information quality, absent authority, is so bad that unblessed information should be excluded from use, lest it produce chaos and havoc. The second is that there is such a high cost to maintaining quality of information that those that do it should have an economic rent that they derive from control of the information - this follows from the first, if information without authority is low quality and destructive, it must be expensive to create high quality information. The third is that, in order to defend this economic rent, and in order to prevent the disorderly society from abusing key pieces of information, that the bulk of how information is created and processes must be kept invisible from the public.

This is the guild system, but it is also the system of Microsoft. It is the theory of spying, and of alchemy. The public cannot look into the sausage factory, and the public is incapable of producing anything which is anywhere near as good as what the guild/agency/OS monopoly can produce. Therefore, we must protect the authority by fiat, and by "Security through obscurity".

The other theory of information is the open theory of information. This states that those who promulgate putative information cannot be trusted with a monopoly. They make mistakes, they will be tempted to warp results towards their own interests, and there is a cost for disinforming a wider public, in that who knows what would have been discovered had more people not been "reinventing the digital wheel". This is the theory of open source software, but it is also the theory of the scientific revolution, where work must be shown, data sets published and experiments duplicated.

In practice we use both theories, and we use mechanisms such as the law and the market to mediate between how much needs to be open, and how much can be held as proprietary. Patents provide an economic rent to pay for the costs of creating credible information, but they also require disclosure to expand the sphere of interoperability.

As you can see, there is an entire world of error in Lemann's off handed assertion of top down authority - authorities produce information, and the rest of us merely consume it. It is, in fact, in contradiction with the system which has profitted so greatly from. In the system which he, and I, and all of the readers of both the article and the blog post live - the centers of authority manage, direct and keep interoperable the vast effort and information creation of the wider society. The failure of systems where authority tries to be the only source of information and activity is one of the few points upon which everyone agrees since the collapse of communism. Authority pays for itself by appropriating some of the efforts of people under its authority, on the argument that they would have produced so much less without that authority. The fight over exactly how much that surplus is, and how it should be divided, is the subject of economic competition, for example "the consumer surplus".

The conflicts between these two models often end up creating an ornate dance. For example, people are not supposed to trade stock based on "material non-public information", that is, the theory which we run our equities markets on is an open one, and yet corporations have large amounts of proprietary information, which we allow them to extract economic advantage from, and which we require their officers not disclose. The two parts are in conflict, and require administrative law, backed by physical sanctions, to resolve.

Just as there are two models, there are two default positions. One position is that open is better, unless there is a compelling case for proprietary, the other is that proprietary is better, except to the extent that open is needed for the purposes of exchange. Which camp you belong to tends to rest on what you think the proprietary value of the information you hold is worth relative to the costs you incur because other people are hiding information from you. Where you stand, depends on where your information sits.

The Rise of the Open Model (1500-1850)

In general, however, the modern world rests on the open model, allowing proprietary information as an incentive to economic activity - that is as a marginal incentive to work rather than engage in leisure - or when there is a general belief that the terms of the proprietary argument - poor information quality leading to high costs - are clearly met. Hence we license doctors and lawyers, because the costs of having a bad doctor or lawyer are quite high. This development proceded slowly from the occult roots of Western science. In the beginning there was astrology and alchemy, both of which rested on secreting information away from the outside. However, both required the publication of material, and this openly exchanged material began to produce huge gains in productivity. Mathematics, before people published for tenure, had prizes which people won by actually exposing their methods to the public. This was the state of affairs in the 1500's and 1600's.

Sir Isaac Newton sat on the cusp, where prizes and royal grants were still used as incentive to publish methods, and where reconciling coinage systems and other forms of interoperable economic information was a major undertaking - Newton himself spent a great deal of time on an assay of coin values to create a table of relative worth between them. He, himself, kept key aspects of the calculus secret for years. He was also a dedicated alchemist, who believed in mystical forces and specific creation.

However, the open part of what Newton did, particularly with the publication of Principia and Opticks overwhelmed the secret world. Suddenly people who could never before have calculated an orbit, had the mental tools to do so from a few observations. We have the name "Halley's Comet" from a friend of Newton who used the then new methods to predict the return of that famous heavenly body.

Newton himself created a large incentive to openness - his theory of kinematics is far from complete, it makes a huge fundamental prediction, and that is that the deviations from pure motion that we see around us, and which had prevented a general theory of motion for centuries - would turn out to be explicable in terms of motion an heat. In short, for Newton to be right, one had to be able to prove that lost motion became heat, and that the energy that became heat, if it could perfectly be transformed back into motion, was exactly the same as the motion lost. Newton's mechanics created the need to prove a far from certain assertion. This assertion would require a long series of experiments - including the Cavendish experiment to measure the force of gravity, and the work of James Prescott Joule to determine the equivalence of mechanical and heat energy.

To prove this project required that dozens of people communicate, and expose their work to each other. This created an incentive to publish beyond economic rent - status within a community, and the profits that might come from others being able to employ an idea commercially.

The commercial pressure for an open society went hand in hand with the credibility within the scientific discipline. Authority began to be derived, not from a structure, but as an emergent property of debate and competition. This mirrored Adam Smith's contention that economic equilibrium could emerge from the interplay of open market forces, and the Jefferson-Madison contention that political equilibrium could be maintained through the conflict of interest against interest in an open arena of politics, writing and debate.

It also parallels the rise of western classical music of the same time, as composers began to publish scores in the hopes of gaining more and more performances, and thus being required to put the information required for a performance in the score. Bach and Mozart both left behind scores without key information, since in many cases they assumed that they would be present to lead the performance. But by the late 1790's the movement towards the score as an open source document had begun.

Authoritative editions were not created, at this point, by composers, but by publishers and by supporters who wished to see works have a wider dissemination. Again the point that authority tweezes information already produced in order to create interoperability and therefore expand the sphere of use, rather than producing information an sich.

A good small documentation of how this played out in a specific sense can be found the rise of precision optics in Germany, as chronicled in The Spectrum of Belief by Myles W. Jackson - where the guild system of glass making evolved into an open system of precision opticks, both under the pressure of scientific credibility, and under the economic pressure of being able to sell products.

Thus the open model is not merely a model of information, it is a theory that ordered societies and activities are emergent forms. This is the fundamental conflict between proprietists, and open source. The proprietist has a fundamentally paranoid and tragic narrative of society, where the dragons of disorder and the ate of greek drama wait to consume the products of man, and only by iron clad hierarchical authority can this degenerate state be held at bay. It's quite Victorian and quaint in its own way.

The open society believes that the boundaries of chaos lie much farther out, that the abuse of proprietary and hierarchical position is greater than the benefits it creates, and that limiting the production of the society by who is socially acceptable to the top of the pyramid will, over time, remove all, or almost all, of the people who are capable of being highly productive. Or to put it another way, the open society asks Aristotle's question about whether good behavior in an aristocracy is behavior that aristocrats like, or behavior that preserves the aristocracy, and then dryly note that the repeated collapse of empires in history shows that the first drives out the second remorselessly.

The open society argues, from such events as the NASDAQ crash and the Iraq War, that people in the position to profit from authority and propriety do so, and that the costs run to trillions - trillion with a trill - of dollars. Those that wonder why journalism is held in such low regard by the reality based community must come to terms with the reality that Enron, Adelphia, MCI-WorldCom and the dot com bubble were blessed, backed and shilled for by supposed journalists in the supposedly authoritative business press. This was followed up by the march to war in Iraq based on yellow cake forgeries, assertions of proprietary information about the threat Saddam posed, and the authority of the unitary executive. It suprises me to see the New Yorker implicitly endorse the Iraq War, and the deceptions which led up to it, but obviously, money is at stake, and all other principles go out the window at that point.

Confirmation and Information

The two theories then represent two theories of order in society. The open society argues that order is emergent – that provided certain boundary conditions are met, and that there are tests of efficacy, that order, not disorder, will prevail. Hence order is imposed when there is high consequence for low quality of information, when boundary conditions are being threatened, or when there is a strong asymmetry of information. In short, the reason to impose order, is to reduce the total amount of imposed order.

The theory of imposed order as a default is often on a rhetorical disadvantage against the open theory. First because the worst abuses of history are generally the results of attempts to impose order – the atrocities of the totalitarian states, slavery, apartheid – all not good bed fellows to have. Worse still, defenders of the extreme forms of proprietism, such as Prof. Lemann, often take the "look – the niggers are getting uppity!" tone in their screeds. Prof Lemann attacks bloggers wholesale by saying they can make people's lives uncomfortable, but he does so from the sinecure of a deanship, and is being paid quite well by the New Yorker to smear his economic opponents. While comparisons to racist behavior and bigotry are invidious, the mendacity of someone who is well paid and completely secure in his position attacking people who are often scraping by paycheck to paycheck invites exactly such comparisons.

But one of the most important dynamics of the emergence of an open challenge to proprietary models of news gathering is not merely the crass failures of Judith Miller and the rest of the rush to the Iraq War, it is the nature of information and confirmation itself.

As Kuhn makes clear in The Stucture of Scientific Revolutions most of the work of a given scientific discipline – and therefore of authority – is not involved in producing information or disproofs of existing beliefs, but of producing confirmation. There is a good pragmatic reason behind this, confirmations are required in a Bayesian sense. One good example of why this is so can be seen any time an observation seems to contradict established theory. On one hand it is a great deal of unpleasant grinding to check instruments, check experiment design for uncontrolled variables, check for contamination, check concept to make sure that no effects predicted in present theory were left out, and then to check calculations for possible error – in short, the road to staying within established theory is both burdensome and assumes a path of intense self-criticism, which makes it doubly unpleasant, where as the path of asserting that theory is wrong leaves all of these problems to others, and has the additional reward of potentially making the researcher well known in the field. Thus credibility and confirmation serve as counterweights to keep researchers on the straight and narrow.

The market for confirmation is also important in the political and economic realm. People don't make decisions based on information, but on when information has been confirmed to such an extent as to be actionable. Action follows from confirmation, which is why people so often by stocks far too late, and hold on to them far too long. It is why government stimulus in a downturn often arrives late, because by the time it has been confirmed that stimulus really is needed, the downturn is quite likely already over.

Part of the attack on traditional concepts of journalism comes from the right, and it is now about the desire for information, but a demand that right wing points of view – including anti-scientific notions of Ignorant Denial – be confirmed by the press. The right wing wants the press not to merely uncritically print wildly inaccurate claims from the executive about the threat of Saddam, but to participate as cheerleaders in those claims. Objectively speaking, there is no more evidence for Saddam having a WMD program or being involved in 911, than there is that Bush was involved in 911. The difference is that the first, a right wing point of view, is more or less the official position of the US military, and the second is a fringe position which is actively banned from sites like DailyKos. That Lemann equates DailyKos with Newsmax is telling, his interests are threatened and to him, open source community information is the same as top down disinformation aimed at circumventing standards.

The economic value of providing this confirmation is fairly evident, one can see how CNN and other cable news networks have significant jumps in viewership when there are bombs flying, and how there is a direct correlation between this activity and rises in Bush's support. In short, there is a market segment that wants more confirmation of the Bushite view of the world, and they only watch, and therefore their attention can only be sold to advertisers, when their world view is being confirmed by images. Traditional outlets such as the New York Times, realizing that these people are both identifiable, and vocally hostile, took steps such as appointing a "public editor" whose job is, more or less, to appease the right wing populists, because it certainly hasn't been about providing more information that contradicts the party in power in Washington DC.

Since this post is already nearly 5,000 words long, and has covered topics from modern macro-economic central banking theory; the philosophy of science; the emergent world view and its relationship to Madisonian political theory, Smithian economic theory, open source software, classical music and the scientific method; the information/confirmation duality and its relationship to emergent as opposed to imposed order – it seems time to rest for the moment and take up the track later on.

But a summary is in order. Lemann and the New Yorker have violated the trust of the readers, in presenting an inaccurate, insulting and unbalanced article purporting to be fact which, instead, is a self-serving and crassly self-interested apologia for the economic interests, as they see them, of the author and the magazine. The deep fault in the article is an indefensible assertion, even in its own framework, of the purpose of authority in society. Authority, contra- the New Yorker, does not produce information, but, instead is focused on producing confirmation. The purpose of this is to reduce the uncertainty in those who act, by providing a powerful Bayesian base of confirmation which will allow individuals to ignore observations which seem to contradict that backdrop.

The lack of credibility of the top down system of information production is rooted in two separate, and in fact polar opposite, lines of critique. The line of critique from the left is that the top down information system provided disinformation on critical stories, most particularly the stock market bubble and the threat posed by Iraq, and that the results of the media malfeasance and lack of integrity have been profits for top down media, and death and disaster for ordinary people. The line of critique from the right is that the top down media has been insufficiently enthusiastic in supporting right wing narratives.

These two lines of critique represent two fundamentally different views of the problems of our current information system. Lemann's piece –as so much else of top down thinking on this subject – is directed at defending against the economic consequences of not providing sufficient confirmation to the right wing world view, and at the same time defending the personal privileges of people like Lemann from the left wing critique. This rests, in turn, on the dual nature of authority itself. One on hand authority is based on some paradigm which provides superior handling of the external world, on the other hand, it is a social system which rewards those who join it. The second drives out the first, and eventually authority capsizes, because anyone smart enough to handle the paradigm has been ostracized or punished for being insufficiently obsequious to the social norms of those who have the ability to admit, or deny, individuals from the charmed circle of those who can make their living as handlers of information.

From here it is only left to prove that, from the examples at hand, that the economic rent being protected by so called "journalists" is far more important than any nominal standard of ethics or discipline which they supposedly rest their absolute superior claim to economic privilege upon. This attraction is so strong that the New Yorker, and its editors and writers, are willing to abandon all other principles, and even endorse the means by which the public has been deceived about Iraq, in order to protect it.

A very sad day for "journalism," however constructed.