Good post. What are your thoughts about factoring in the human analysis of this data? When all this semi-raw data comes down to the hands of an individual there's a whole element of how his/her perspectives.  The Brits have introduced an interesting wrinkle by monioring all cars which says to visitors & residents alike, "Live here and you will be monitored, we know where you live ..." It's right there in your face. Lastly, the use of "wiretapping" seems Caponesque to me and I'm wondering when Bush will hide behind what he sees as a limited definition of what he's doing. He isn't, in my book, doing warrantless wiretaps. I want my Republican friends to understand the scope of this undertaking. Bush isn't just doing this based on his narrow definition, no he's blanketed us all, no one is immune. One piece, one element of the Bill of Rights at a time until our rights are scraps in a scrap yard.

Look, Jerry, obviously you need to know the error rate and exactly what's done with the data in order to evaluate it. Indeed, if you read my post I think you'll see that I already say as much. That's my point. You need to know more.

But my other point is simply that even an error rate that looks low at first glance (10 percent, say) can actually turn out to be much more problematic than it initially seems depending on various other factors. Thus, a lot of the merits of the program winds up turning on the issue of roughly how many terrorists are "out there" overall. If terrorists are relatively common, then data mining is a promising way of trying to find them. But if terrorists are very rare, then for a data mining program to catch more terrorists than false positives, it's error rate is going to need to be implausibly low. My sense is that we're in the second situation -- 9/11 was perpetrated by just a few dozen people.

If that's the case, then probabalistic seives aren't a good way to catch terrorists, and it seems that that's what the NSA is doing. Maybe it isn't -- secret programs are hard to evaluate. And maybe I'm wrong and there are tons of terrorists out there but I think that's wrong.

I just posted an historical take on this problem at eurotrib.

Surveillance vs Civil Liberties 

The main point is that there are fundamental information gathering problems with surveillance when the target is an unknown actor.

However, the same techniques work well when aimed at domestic activists who are addressing social problems. So historically once a secret police is set up it tends to go after a country's own citizens.

I argue that this may be part of the motivation from the start. 

 

Good summary.  And it obviously goes farther - a 99% solution sounds great until you do that same math for finding 50 people out of a million.

One detail - generally you don't have a single error rate - you have a false positive rate and a false negative rate.  Not only are these not generally identical, they're often inversely related.  Lowering your terroristness threshold will reduce your false negative rate (you'll catch more bad guys), but will also increase your false positive rate (you'll catch more guys who just happened to email out "Afghan music is DA bomb!").

Any program of this type has an "adjustable" alert level.  The adjustment sets the balance between correct detection and false alarm rates. 

Conceivably, the threshold could be set for extremely low false alarm rates.  This would inevitably lower the correct detection rate also.

A more interesting point:  it seems to me that it is quite likely that they don't know the false alarm rates.  Every alarm is flagged for followup, and the alarm threshold would be set to match the followup capacity.

It seems to me that the likelihood is that the vast majority of alarms would be false alarms.  We have the suggestion that there have been a few high profile successes, and that this justifies the program.

Matt--

Great to bring this up.  It's actually a classic problem in public health--specificity vs. sensitivity.  

And your conclusion is exactly right--testing (or, in counter-terrorism, spying) has a cost.  

That's why in public health (where making the right decision has far greater life-or-death consequences than counter-terrorism), few screening tests are used on the entire population.  Even a "low" rate of false positives can mean that the test causes greater harm than not having the test.

Mammograms, for example, are only performed on high-risk populations.  Because, the fact of the matter is, most women don't have breast cancer.  But if you gave the test to everyone and then followed-up aggressively based on the restuls, you'd have a lot more healthy 25-year old women walking around with one breast lopped off than you'd have 25-year old women with cancer who were saved by the test.

This isn't to excuse racial profiling as a means of "targeting."  It's to point out that there is a cost to testing--to me, the cost of pissing off patriotic muslims who we will need as translators and operatives in order to win the war on terror is far higher than the reward of catching a mentally-ill Muslim who wants to take down the Brooklyn Bridge with a blowtorch.  

What it says to me is that we have to be very honest with ourselves about what each type of targeting yields, and how it should be applied.  We should know sensitivity tests from specificity tests, and their power to label someone as a "bad guy" should be limited the less sensitive they are.  

And it makes civil rights MUCH more important, because that's the untimate specificity test.  If your case can survive a writ of habeas corpus suit, that's a really specific test, and we're far more confident that we're not persecuting the innocent.  

I mean, if we're willing to NOT test people when life and death is actually on the line, why aren't we willing to NOT imprison people when life and death isn't on the line?

Interesting post. What you've touched on is the curious fact that lawyers don't quantify important legal concepts.

What is an acceptable error rate for "reasonable suspicion", "probable cause" or "beyond a reasonable doubt?

Reasonable Suspicion is the threshold the cop must pass before he can stop you on the street to ask for your ID, or pull you over to check your blood alcohol level. I'd say this requires about a 25% certainty.

Probable Cause is the threshold for a search warrant, arrest warrant or indictment. This would mean at least 51% certainty

Beyond a reasonable doubt, of course is what a criminal must find you guilty beyond. I'd guess 95%
certainty (but I've never seen this percentage spelled out in a case before).

I think if the government can show that data mining is only minimally intrusive (the equivalent of a cop asking for your ID on a city street), then maybe they could get away with it on merely "reasonable suspicion". But the courts will look to see how how often someone who's searched turns out to be a "bad guy", the Feds better be able to show at least a 25% accuracy rate. If its lower than that, or the Feds just start data mining everyone, that can't possibly be constitutional.

This is an Administration about which the most charitable possible description, giving full benefit of every doubt and then some, is that their intelligence was inaccurate.

We're being asked to trust a President who's gives no hint of trustworthiness.

Some people might think they have nothing to hide, so what's the big deal. To them, I say it's only possible to feel that way because we're American.

Our sense of security comes from the very rights that George Bush took away. If this is allowed to stand, our kids and grandkids won't be nearly so carefree about "what they have to hide."

for the scenario where they're spying on all phone calls, not just overseas calls.

Let's say there are 250 million americans who make phone calls on a given day, and make an average of 4 calls per day. That's a billion calls.

Now let's say the algorithm has a .01% false positive rate. That's 25,000 calls falsely flagged as terrorist.  Even if it's .001% falls positive -- 99.999% accurate -- that's still 2500 innocent communications being reviewed without a warrant every day.

That's the problem with vast fishing expeditions. If you are looking for a small enough needle in a large enough haystack, the noise overwhelms the signal.

I'm pretty skeptical that this sort of fishing expedition is worthwhile, even if a law could be crafted that could satisfy civil liberties concerned.  Traditional law enforcement, where you start with real leads, has a much higher probability of success at catching the bad guys.

 

The same concern applies with the use of biometrics databases for law enforcement.

A digital photo works well to confirm that you are who you say you are. But searching a digital photo database works poorly to pick a suspected criminal out a football crowd, for the same reason.

Local governments aren't always smart buyers of technology (also see voting machines). They buy based on glowing promises from vendors. When there is secrecy in the procurement process, they are even less likely to get good analytical advice.

I'm more familiar with local/state issues than federal issues, but it is not hard to imagine that a lot of homeland security technology is oversold at the federal level. 

 

The example of using facial recognition for identifying terrorists in a crowd is very apt here.

As best as I can make out, virtually no one uses this technology anymore, not because it necessarily raises any civil liberties issues, but rather because it was operationally a total bust, producing no positive results, while being a huge sink for law enforcement resources, which were obliged to follow up all the false leads.

I can't imagine any reason in the world to think that monitoring hundreds of millions (billions?) of phone calls a day -- as opposed to only tens of thousands of people in a stadium -- could possibly prove any better. Speech recognition generally sucks, to begin with, at least as bad as facial recognition. Just this one really weak link in the chain would probably do the whole program in by itself, not to mention all the stupidities introduced down the line by any natural language processing.

I had originally thought too that the program was somehow restricted in its application.

Apparently, though, that isn't true -- it's much wider than that, including, as best I can make out, EVERY conversation, potentially.

Times link:

http://www.nytimes.com/2005/12/24/politics/24spy.html

 

I'll bet if we put a surveillance camera on every block we'd catch more criminals, and perhaps a few terrorists too. And I'll bet if we assigned a government monitor to every citizen and resident alien in America we'd have less crime, and terrorism probably too. I'll bet the suburbs would even accept the former at least, just as their ancestors tried to stay out of that silly revolution their neighbors we're holding.

I want to launch into some kind of diatribe about America's newfound imperial tendencies, and how there will continue to be barbarians at the gates until this country rediscovers its republican roots, but really these tendencies are as old as the republic. Eliot and Pound dated the death of the republic to the 1820s or 1830s, Henry Adams to the 1860s or 1870s, others to the 1890s and 1900s, and still others to the 1940s, but the revolution never really resolved the conflict between Federalism and Antifederalism, and even as the Antifederalists manage to prevent imperial drift for a few decades eventually the Federalists (sometimes called Democrats, and sometimes including lately called Republicans) get back in power, and it becomes awful hard to fully undo what they have done (the bad or good parts). No one should forget that the legacy of the Civil War is not just an end to slavery but the beginning of the national police state. No one should forget that the legacy of the New Deal and second world war is not just social security and a victory over fascism but a lasting expansion of those police state powers. You bribe and cajole the people with some greater good, and take away their freedoms, and their privacy, in the process.

I don't claim to be a legal or constitutional scholar. I have little idea whether this president will be so much as scolded by Congress, let alone impeached, for his actions, or granted broader statutory powers to listen to our phone sex. But the precedents strike me as grim. Eventually the Republicans or the Democrats who replace them will get around to bribing us with health care or free handjobs in the restroom of your local megachurch (just a thought; they appear to be running out of things to bribe us with) so they can have their continued war on terror and Very Important Place in History, taking away a little more of our freedom in the process. In the midst of imperial drift, the police state replaces culture as the primary arbiter of good behavior.

Perhaps eventually there will secession movements over these things. As a matter of fact there already are. But someone should tell the Second Vermont republicans that should they utlimately attain their goal of independence the NSA will almost certainly be watching them still.

A smarter way to use this kind of program is to recognize the uncertainty in the measurement and develop a score.  Say it is the probability that this person is worth questioning.  People trying to stop terrorist attacks have a finite amount of resources.  You want to use data mining to focus the resources most efficiently.   

  Some of the recent leaks have already talked about, how the "call" patterns are the most valuable data.  Many messages are encrypted and it takes more effort to eavesdrop.  So a first pass analysis might take a list of email addresses, phone numbers, IP addresses and web sites that are known to be Al-Quaeda.  Call them level 0.   Who do they contact and who contacts them?  Call them level 1.  Then take those addresses and see who they contact.  As the network is constructed and correlated with data from investigators, they are able to see which branches from level 0 are the live ones.  Mark the bad leads as no good and create the scoring system.

  If you think what has happened over the last few years, there is a lot of data to work with.  Computers seized when the training camps were overrun in Afghanistan.  Zarqawi’s laptop  seized in Iraq.  Lots of interrogations all over the world, like the cell in Lackawana, New York, to Gitmo, etc.
 
Do the Dems want to run against the government using this kind of information efficiently to stop attacks in the US?  It does not seem like a strategy that will take the back the House of Reps, Senate or White House

<i>While the issues of false positives and false negatives is a real one that the NSA or someone needs to take up, you really don't know what the actual error rates are of the actual NSA technology involved.  </i>

Really, this just misses the point.

The point is that ANY feasible technology is bound to produce "crappy" results, simply because of the numbers involved.

The error rates could not POSSIBLY be so low as not to create the sorts of problems people have been talking about here. Anyone with a small acquaintance with the kinds of technology involved, such as speech recognition, or natural language processing, or "pattern matching" understands that there's no way they can be made reliable on the scale involved without producing overwhelming numbers of false positives. You need an algorithm that might produce, say, a false positive error rate of less than 0.01%. I have never heard of such an algorithm in either of those fields, or anything remotely approaching such an algorithm.

My guess is that the place where this will break down most dramatically is in the natural language processing and/or pattern matching area. (As I think about it, unreliability in the speech processing part might only, say, double or triple the number of false positives. That's where the real work of winnowing by many orders of magnitude would have to take place, yet where real unreliability in prediction would certainly still exist.)

Powers used to find so-called terrorists will be used to find other people doing things the government doesn't approve above and that (in some government officials' eyes) might be dangerous.

It is easy to imagine one of those phone taps finding someone who might have been discussing a terrorist act, but actually discussing a small plot of marijuana. So, does anyone think that person would not be prosecuted as a result of that discovery? Of course further investigating would be done and the phone tap would never see the light of day.

Or, perhaps the discussion included some joking about hiding under the table income from the IRS. Or, about "borrowing" some materials from one's job. In all of those cases the Constitution would be grossly violated, but those people would be prosecuted.

We must use a "zero defects" policy when it comes to violations of the Bill of Rights.

Do the Dems want to run against the government using this kind of information efficiently to stop attacks in the US?

Winning elections is important, but not the most important thing in the world. If it were, there are plenty of illegal means available to increase the odds on winning.

What we are discussing here is not a political problem. It is a problem of a rogue president, using illegal means not just to get elected, but to dominate the citizens of the country. The smart and, more importantly, the legal way to use this system is to junk it totally. It is illegal for the government to spy on American citizens. I can't see why that is so hard to understand. If we were at war it might be interesting to discuss this again, but the answer would still be that it is illegal. And, no we are not at war - if Bush says we are, of course we aren't.

Re: Quaker Peace Activists?------------
How in the heck can this community pose a threat? That's just ridiculous.


 

Since all mechanical test systems, whether medical, police, judicial, or intel, generate error, follow-up tests are applied.

Medically, a positive leads to other tests. Police evidence is tested in court. Judicial tests (trials) are tested on appeal. The best argument against the death penalty is the combination of high error rate and no way back after execution.

If it's deep-black intel, you're screwed. Without a FISA warrant there are no grounds and no procedures to catch error. With a FISA warrant there would be some hope.

Apropos that, I find this item very intriguing. The AG office is asking the FISA court in March 2002 if the Act can now "be used primarily for a law enforcement purpose, so long as a significant foreign intelligence purpose remains."  Odd, considering that law enforcement would be difficult without a FISA warrant (as has been the reported practice since then, for some cases.) Also, there is little evidence of administration interest in prosecution and no successful prosecutions to date. The court rejected the request, on statutory grounds.

If that's the case, then probabilistic sieves aren't a good way to catch terrorists, and it seems that that's what the NSA is doing. Maybe it isn't -- secret programs are hard to evaluate. And maybe I'm wrong and there are tons of terrorists out there but I think that's wrong
Tons is really not a good metric for quantifying terrorist activity, since we know that 20 men with some support network can kill thousands in a matter of a few hours.  Or maybe it is, 20 men would be about 2 tons of terrorists, I guess, if they each weighed 200 lbs. heh.
Anyway, it is interesting to note two things:
1.       Since 9-11 there have been mass casualty attackes in Bali, Madrid, London, Russia and all over the middle east but none in the US. This despite tons of threat by Bin Laden, Zawahiri and others.

1. Most the civil rights and rights of privacy issues we are debating are rights in most of Europe.  Generally, we are much more civil libertarian than Europe.
   

Probabilistic sieves seems like a very rational way to conduct counterterrorism operations, so I support them.  The fact that there is a risk they will be abused is not, in and of itself, a reason to abolish them, anymore than the fact that there are welfare cheats is a reason to end welfare.  My biggest problem with the Bush Admin on the prevention side has been the wasted money trying to keep tweezers, screwdrivers etc. off the airplanes.
I share the concern about growth of government power and prosecutorial abuse, and want to see it limited.  I think ending the war on drugs is a good place to start.  But I don’t see anyone in either party pushing for that, because a big majority of the American people don’t agree with me.
But I believe the threat of jihadi attack here is very real.  If one happens I expect most of the posters on this board will blame the Bush Admin and that’s OK with me,

No, the false positives and false negatives tend to go in opposite directions.

Not quite. That is only true for a fixed population of people being snooped on. If you cast a really big net, vs. a targeted net, you get both more false positives and false negatives.

Aren't we forgetting that the problem behind 9/11 wasn't the lack of data, but the lack of an ability to analyze and coordinate with that data? There were lots of data points that could have been put together to predict the 9/11 attacks, but no mechanism for doing that. With the NSA generating orders of magnitude more data, the problem gets worse, not better. But, the ability to target pesky political opposition does get better. That kinda makes one wonder, doesn't it?

Jesus H Christ Matt...what a logician's nightmare

A fuggit about it..I ain't botherin to try to pick out the pearls out of that garbage..

When you put the law aside and went into "arguably" mode, I couldn't make it past the second paragraph for all the side splitting laughter

So let's return to some clear-head thinking shall we...

Putting the law first and last...

Not Authorized By Law: Domestic Spying and Congressional Consent


UofPitt Law School's JURIST Guest Columnist Jordan Paust of the University of Houston Law Center says that contrary to assertions by President Bush and the US Department of Justice, post-9/11 Congressional legislation on the use of military force against terrorists does not authorize domestic spying...





George W. Bush and US Attorney General Alberto Gonzales claim that domestic spying in manifest violation of the Foreign Intelligence Surveillance Act (FISA) was authorized by Congress in broad language in the 2001 Authorization for Use of Military Force (AUMF) regarding persons responsible for the 9/11 attacks. Similar claims have been made in a December 22 letter from Assistant Attorney General William Moschella to the leaders of the House and Senate Intelligence Committees. The claims are patently false.

MW's comments above point the way to lowering the false alarms, but I'd assert that this comes at the cost of lowering the correct detection rate.

Once the pool is restricted in various (obvious) ways, the terrorists will take elementary precautions to avoid being swept up.  Some of the restrictions may be clear violation of civil rights - "profiling" based on religious affiliation, etc.

I'd say that in the end the program has to be expansive to catch terrorists that are trying to be covert.  Then we're back to relying on a wide net, which will generate comparatively more false alarms.

Groups may be monitored because some of their members may have used verbal and physical abuse. This is an even likelier consequence in cliques because some of its members can become compulsive or obsessional about other people. As I said this situation is more a function of culture than agency. Perhaps a crisis like 9-11 will bring these behaviours out starkly. For some reason, they get away with them.

Reasonable Suspicion is the threshold the cop must pass before he can stop you on the street to ask for your ID, or pull you over to check your blood alcohol level. I'd say this requires about a 25% certainty. Probable Cause is the threshold for a search warrant, arrest warrant or indictment. This would mean at least 51% certainty Beyond a reasonable doubt, of course is what a criminal must find you guilty beyond. I'd guess 95% certainty (but I've never seen this percentage spelled out in a case before).

 Alas, surveys have consistently shown that most people are both very bad at estimating and applying probabilities, and are not very generous with respect to these legal concepts.  Most people ascribe to "Reasonable Suspicion" anything that is possible and plausible, or in other words, anything greater than 0% chance.  Sometimes tempered by a requirement of first-hand knowledge.  "Probable cause" is equated in most peoples mind with dismally low chances, generally on the order of 25%.  Meanwhile, the real shocker is that  people are all over the map as to what "Beyond a reasonable doubt" means.  Some peg it at 95%, fewer still at 99%.  Some at 50% or even below.  The consensus is in the 75% ballpark.

Given this discussion about error rates for tests, consider the number of criminal trials resulting in convictions at error rates of 25% false positives*.

 *  hopefully other mechanisms conbine to ensure lower error rates.

This post of Matthew's makes no sense at all.  I mean, yeah, false positives and false negatives.  Right.  So how does this distinguish the NSA program from every single other method of detection ever created in the history of mankind?

I mean, let's just do away with every bit of electronics ever created, and we can detect terrorists using only human beings and their eyes, ears, and brains.  OK?  Now, every single point Matthew made in his entire post is exactly the same!  We still get false positives.  We still get false negatives.  It still matters what we do with our deteminations (whether we pick up people and torture them right away, e.g.).

So, what the hell is Matthew's point???

He does not serve at the pleasure of Congress or the Courts -- he serves at the pleasure of the People.

 

 

Not so. A President also serves, theoretically, at the pleasure of the group with the power of removal: the Congress. A Congress, protective and jealous of its power, would have already removed Bush. 

That the Republican Congress hasn't already removed Bush is an indication that an American Caesar is not unthinkable. 

When evaluating the effectiveness of any screening program, you also need to consider one other factor that I did not see mentioned: the extent of the consequences when a false positive or false negative is the result.
In testing for breast cancer, for example, getting a false negative (failing to detect someone with the disease) is serious, but it only effects that one individual. In that case, you probably do not want to have a large number of false negatives just to catch that one missed positive. You will discard the test.
In terrorism, however, the consequences are a little different. To use your example of screening 10,000 people, of whom 200 are terrorists and the accruacy of the test is 90%, you correctly identify 180 of the terrorists, while 20 go undetected. This result is weighed against 980 innocent people being falsely labelled as suspects.
Is that acceptable? If you are screening for a disease, or DUI, probably not. If you are trying to identify someone who is planning and capable of killing perhaps 5,000 people or more, you might evaluate the test differently. You must weigh the number of people ‘saved’ by catching the 180 terrorists, which might be as many as 100,000,  against the unfortunate result of 980 innocent people being falsely accused. Should you disregard catching those 180 because of the 980?
I cannot answer that question for everyone, but at some point the number of potential lives saved outweighs the ‘rights’ of the falsely accused, and you then proceed with the screening. It is a variation on the current torture debate, where the question becomes how big a potential catastrophe must be to justify torturing a suspect in order to prevent it? Under no circumstances?

Individual rights are critical in our democracy, but I consider the right to live as the most important. Depending on the extent of the threat to each, compromising my right to live by upholding my right to privacy is a bad trade-off.