« Obama Adminstration Expands DOD Budget Secrecy. Wait, what? | Jonathan Eyler-Werve's Blog | Sen. Lieberman Calls for Release of CRS Library »
Premier/Diebold Voting Machines Deleted Ballots
This week, the State of California completed its
investigation into why their electronic voting machines quietly deleted
197 ballots in the November 2008 election in Humboldt County. The loss
was discovered only after discrepancies in the vote count were found by
the Humboldt County Election Transparency Project, a local watchdog.
The investigation report (also: background docs) throws the blame solidly on junk software provided by Premier Election Systems, a company so widely ridiculed that it abandoned it old name, Diebold, in 2007.
The fundamental problem with the systems used in Humboldt County is that the software creates no permanent records of votes cast (i.e. paper receipts).
Instead everything is tossed in a database, which as this case ably demonstrates, is only as good as its software. When Premier/Diebold found a flaw that could lead to a user unknowingly erasing a stack of votes, they sent an ambiguous email warning to election officials... in 2004. They didn't however, correct the software in the counties where it was deployed.
The problems don't stop there. The investigation also noted that the Premier/Diebold software has, nestled between the "Print", "Save As" and "Close" buttons, a button labeled "Clear". This button deletes the permanent audit logs which record (in theory) everything that happens on the machine. There is no "Are you sure?" confirmation or notification of what has happened after the button is pressed. It just wipes out, with one stray click, the federally mandated log files. According to California's report, a 2001 internal email discussing the addition of the button noted that "there are too many reasons why doing that is a bad idea." They did it anyway.
The report concludes:
The Global Integrity Commons covered Premier/Diebold before the election, citing "serious doubts" on Premier/Diebold voting machines.
They aren't doubts anymore: it's public record that these things are zapping votes into the ether, and without attentive watchdog groups, there is no indication whatsoever that anything is amiss. How many times has this happened without getting noticed? What happens when someone wants to delete votes?
We can only hope that more states using these systems will follow California's lead and hold public hearings on whether to decertify the Premier/Diebold systems in use.
I'll say it again: if buying a medium nonfat latte merits a paper receipt, so does the ballot.
-- Jonathan Eyler-Werve, reporting for the Global Integrity Commons
The investigation report (also: background docs) throws the blame solidly on junk software provided by Premier Election Systems, a company so widely ridiculed that it abandoned it old name, Diebold, in 2007.
The fundamental problem with the systems used in Humboldt County is that the software creates no permanent records of votes cast (i.e. paper receipts).
Instead everything is tossed in a database, which as this case ably demonstrates, is only as good as its software. When Premier/Diebold found a flaw that could lead to a user unknowingly erasing a stack of votes, they sent an ambiguous email warning to election officials... in 2004. They didn't however, correct the software in the counties where it was deployed.
The problems don't stop there. The investigation also noted that the Premier/Diebold software has, nestled between the "Print", "Save As" and "Close" buttons, a button labeled "Clear". This button deletes the permanent audit logs which record (in theory) everything that happens on the machine. There is no "Are you sure?" confirmation or notification of what has happened after the button is pressed. It just wipes out, with one stray click, the federally mandated log files. According to California's report, a 2001 internal email discussing the addition of the button noted that "there are too many reasons why doing that is a bad idea." They did it anyway.
The report concludes:
GEMS version 1.18.19 [the software running the election] contains a serious software error that caused the omission of 197 ballots from the official results... Key audit trail logs in GEMS version 1.18.19 do not record important operator interventions such as deletion of decks of ballots, assign inaccurate date and time stamps to events that are recorded, and can be deleted by the operator. The number of votes erroneously deleted from the election results reported by GEMS in this case greatly exceeds the maximum allowable error rate established by HAVA [Help America Vote Act of 2002].It's logging inaccurate dates? How sloppy is this product?
The Global Integrity Commons covered Premier/Diebold before the election, citing "serious doubts" on Premier/Diebold voting machines.
They aren't doubts anymore: it's public record that these things are zapping votes into the ether, and without attentive watchdog groups, there is no indication whatsoever that anything is amiss. How many times has this happened without getting noticed? What happens when someone wants to delete votes?
We can only hope that more states using these systems will follow California's lead and hold public hearings on whether to decertify the Premier/Diebold systems in use.
I'll say it again: if buying a medium nonfat latte merits a paper receipt, so does the ballot.
-- Jonathan Eyler-Werve, reporting for the Global Integrity Commons
Advertisement
Diebold/PES sued for stealing... CODE
And the beauty of this case? They allegedly stole what is likely open source code. Why is that important?
1) Because they will have to make all of their source code for the machines public if this is true.
2) You will legally be able to take all of their code and use it yourself in any way you want if this is true.
Just for a laugh and in case you were interested. :)
March 5, 2009 1:59 PM | Reply | Permalink
Almost all elections in the world are done with hand writen paper ballots. These ballots are then counted by hand. They are usually counted right where they were cast. No moving them to a warehouse. No storing them in insecure locations. No sticking them in a car trunk to be "forgotten".
No chance to "mislay" them.
People in civilised, industrial countries carry out national elections with hand filled out paper ballots that everyone can see, understand, are kept in plain view until they get counted in plain view of evry interested party.
I just really don't understand why this is so impossible for America. We don't need a Ron Popiel Vote-O-Matic. We just need plain old secure elections. KISS! KIPS! (keep it paper stupid) We don't need a recipt. We need a ballot.
March 5, 2009 4:46 PM | Reply | Permalink
I don't mind the machines. But we need machines that can give receipts so voters can double check the ballots and so poll workers can do an accurate handcount. Let the media have their super fast machine results BUT... Then do a handcount of everything before it is certified.
Just my thoughts on it.
March 5, 2009 6:35 PM | Reply | Permalink
If -- if we have machines, then these are the bare minimum requirements:
1) The software must be completely open-source. No proprietary code that only insiders can view.
2) The machine must produce, or read, a paper ballot, and the paper ballot must be the official ballot for legal purposes.
3) Hand recounts of paper ballots must be required by law.
March 6, 2009 1:55 AM | Reply | Permalink
Absolutely.
March 6, 2009 7:47 AM | Reply | Permalink
I'd like to add there must be strict chain of custody procedures to ensure the counted paper ballots are not tampered with prior to the official count. It would be best if the ballots were counted at the precinct level with full public witness (even video recording of the process) to allow the public to have faith in the process and the outcome of elections.
March 6, 2009 8:18 AM | Reply | Permalink
Two thoughts:
If the software is open source, does that make it easier for someone to tamper with? I don't know this and am asking the computer savvy. Doesn't open source make the process less secure?
The Constitution calls for recounts under certain conditions. If there are no paper ballots, what can be recounted? Therefore, is paperless voting unconstitutional? Seems like it should be to me. Anyone want to launch a court case?
March 6, 2009 11:32 AM | Reply | Permalink
I don't believe there's anything in the US Constitution regarding recounts. I could be wrong. Can you cite article/section or amendment?
You are right; without paper ballots there is little to "recount", and if the integrity of the electronic voting system is the issue, an "electronic recount" is essentially useless -- garbage in, garbage out, as they say.
Regarding open source software being less secure -- no. First of all, the fact that everyone can view the code does not mean that everyone has permission to modify the code willy-nilly -- or more precisely, permission to get their modified code re-incorporated into the code base. New code would have to be vetted before it got re-incorporated. In any event, the "many eyes" phenomenon means that bad/buggy or deliberately-sabotaged code gets caught and fixed more quickly.
Tampering/sabotage of code is actually much easier when the code is proprietary and secret. Any insider, and any outsider with sufficient skills to gain access (sometimes a not of skill is required and some of the proprietary voting software and hardware is notoriously insecure) can modify code and/or data when it's proprietary -- and it's much harder to detect since a very limited number of people have authorization to look at it.
There are videos out there of university computer scientists using very simple tools to hack into some of these voting systems in a matter of minutes. I don't remember if it was Diebold, Sequoia, ES&S, some other outfit, or a combination of these. The "proprietary code is more secure" idea, pushed by the voting machine racket, is just wrong.
March 6, 2009 1:39 PM | Reply | Permalink
RE: hardware hacking.
We've previously covered a case where 1) a product line of (then) Diebold boxes used a low-tech vending machine key 2) the machines all used the same key 3) Diebold published photos of the keys on its website, allowing voter security activists to make working copies from the photos.
Yikes.
March 9, 2009 3:27 PM | Reply | Permalink
@fpie - I agree.
I've seen some touch-screen + paper systems that I thought were ok -- here in Chicago, I vote with a touchscreen. Once I confirm the results are valid (no "lizard people" here), it prints out a paper receipt behind a clear window. I see the receipt, and once I'm happy, another button scrolls it into the box. If the election is disputed, you junk the electronic tally, pull out the paper rolls and start counting.
It's not an improvement over a paper ballot, but it's not the nightmare that the no-paper systems are.
March 5, 2009 5:29 PM | Reply | Permalink
So why isn't it in place? This shit has been going on for more than a decade! What does it take to change it?
I simply don't trust our voting system, and I am actually amazed that Barack Obama was allowed to actually win last time. Next time it could be less obvious, and easier to cheat.
There MUST be a national standard, and that must include a paper proof! I was saying this afer the 2000 election (where there was plenty of proof that the results were faked) and still nothing has changed.
So, how do we get this movement moving?
March 5, 2009 8:10 PM | Reply | Permalink
The Voter-Verified Paper Trail. We have such a system in central Ohio (ES&S iVotronic).
The adding-machine tape "paper trail" is very difficult to audit properly due to its format (as opposed to ballots formatted as sheets of paper).
The result of this audit difficulty is that the secretary of state is constantly under pressure to reduce the number of audited precincts, which limits the statistical strength of the audit process as a whole.
I have been a software developer for 31 years (B.S.E.E. Ohio State, 1977) who believes that ballots should be paper, durable, easily auditable and thoroughly audited.
Standard business practices of verification and audit have existed for centuries. Voting, that most central act of democratic process, deserves no less.
March 6, 2009 7:45 AM | Reply | Permalink
The really dumb part of all this is computer software is available that can meet all the required criteria. There have been choices made that had their decision proceses decided by money rather than meeting the necessary specifications. Frankly, congress dropped the ball on this one. We have a federal agency, NIST, that is absolutely up on all this stuff and should have been the agency guiding the establishment of criteria and a certification process for vendors. This didn't happen and the only reason it didn't happen is guess what?
March 6, 2009 1:49 AM | Reply | Permalink
The voter must be able to verify that the ballot which is used for the official vote tally reflects this or her choices. This can only be done when the voter can see those choices documented on a durable, auditable (recountable) ballot.
There is no process, no software, whether qualified by NIST or not, that can provide the security that is guaranteed when voters can verify that the legal ballot reflects their choices.
The difficulty with all electronic voting systems is that a successful attack would compromise the entire system. The fact that there is one "program" running on many machines results in effective centralization of the counting mechanism that single program, which makes such a program an exceptionally valuable target of attack.
We all know that the reality is that every nontrivial system is not a "program" but rather a large set of interacting layers and tools including bootloader code, compilers, kernels, libraries, applications, configuration files, etc. etc. It's too easy to hide a trojan horse in one of these many component layers. Inspecting every one of those many components with the thoroughness necessary to satisfy the security requirements would be prohibitively expensive.
DO-178B DOD software inspection, validation and documentation requirements serve as a good example here. Boards of elections cannot afford to buy systems built under such rules.
I have developed code for avionics systems as well as the SS7 signalling network and know the value of the inspections and validation of such critical code. I am also a poll worker and know that boards of elections are underfunded and understaffed. There is a major impedance mismatch between cost of proper system security and acceptable cost of election systems.
Hand-recountable, durable paper ballots are well matched to the capabilities of voters to see their choices, as well as being low-tech enough to be affordable to underfunded boards of elections.
March 6, 2009 8:18 AM | Reply | Permalink
An aspect of this I didn't mention is that all elections in the US are locally administered. Congress sets guidelines (which they have) but it's up to counties to meet them.
For a brush up on election law in the US, see:
http://report.globalintegrity.org/united%20states
March 6, 2009 9:48 AM | Reply | Permalink
thats the insane part of all of this. people are acting like we have to re-invent the wheel here. we don't.
ATM's have been functioning perfectly fine for over 30 years now, and never give out the same money twice. standards. theyre already written. the rest is just smoke, mirrors, and theft -- of democracy.
March 6, 2009 6:12 AM | Reply | Permalink
ATM code is written under standards of validation and audit that would price them far far outside the limited budgets of boards of elections.
Additionally and much more importantly, ATMs are monitored thoroughly and their accuracy verified by external balancing/reconciliation processes that (again) far exceed the budgets of boards of elections.
With that said, I agree that modern honest business practices can serve as a guide that show us what is possible.
March 6, 2009 7:54 AM | Reply | Permalink
Diebold is primarily an ATM company. Premier Election Systems is a relatively small part of their business. :(
March 6, 2009 4:37 PM | Reply | Permalink
Kudos for pointing this out!
March 6, 2009 9:04 AM | Reply | Permalink
For what they're worth, here are three emails from Diebold in 2001 where they are clearly aware of one of the problem, which is the ability to alter the audit log.
[Begin emails] EMAIL 1
To: "support"
Subject: alteration of Audit Log in Access
From: "Nel Finberg"
Date: Tue, 16 Oct 2001 23:31:30 -0700
Importance: Normal
Jennifer Price at Metamor (about to be Ciber) has indicated that she can access the GEMS Access database and alter the Audit log without entering a password. What is the position of our development staff on this issue? Can we justify this? Or should this be anathema?
Nel
EMAIL 2
To: "support"
Subject: RE: alteration of Audit Log in Access
From: "Ken Clark"
Date: Thu, 18 Oct 2001 09:55:02 -0700
Importance: Normal
Its a tough question, and it has a lot to do with perception. Of course everyone knows perception is reality.
Right now you can open GEMS' .mdb file with MS-Access, and alter its contents. That includes the audit log. This isn't anything new. In VTS, you can open the database with progress and do the same. The same would go for anyone else's system using whatever database they are using. Hard drives are read-write entities. You can change their contents.
Now, where the perception comes in is that its right now very *easy* to change the contents. Double click the .mdb file. Even technical wizards at Metamor (or Ciber, or whatever) can figure that one out.
It is possible to put a secret password on the .mdb file to prevent Metamor from opening it with Access. I've threatened to put a password on the .mdb before when dealers/customers/support have done stupid things with the GEMS database structure using Access. Being able to end-run the database has admittedly got people out of a bind though. Jane (I think it was Jane) did some fancy footwork on the .mdb file in Gaston recently. I know our dealers do it. King County is famous for it. That's why we've never put a password on the file before.
Note however that even if we put a password on the file, it doesn't really prove much. Someone has to know the password, else how would GEMS open it. So this technically brings us back to square one: the audit log is modifiable by that person at least (read, me). Back to perception though, if you don't bring this up you might skate through Metamor.
There might be some clever crypto techniques to make it even harder to change the log (for me, they guy with the password that is). We're talking big changes here though, and at the moment largely theoretical ones. I'd doubt that any of our competitors are that clever.
By the way, all of this is why Texas gets its sh*t in a knot over the log printer. Log printers are not read-write, so you don't have the problem. Of course if I were Texas I would be more worried about modifications to our electronic ballots than to our electron logs, but that is another story I guess.
Bottom line on Metamor is to find out what it is going to take to make them happy. You can try the old standard of the NT password gains access to the operating system, and that after that point all bets are off. You have to trust the person with the NT password at least. This is all about Florida, and we have had VTS certified in Florida under the status quo for nearly ten years.
I sense a loosing battle here though. The changes to put a password on the .mdb file are not trivial and probably not even backward compatible, but we'll do it if that is what it is going to take.
Ken
EMAIL 3
To: "support"
Subject: RE: alteration of Audit Log in Access
From: "Nel Finberg"
Date: Wed, 17 Oct 2001 14:48:16 -0700
Importance: Normal
Thanks for the response, Ken. For now Metamor accepts the requirement to restrict the server password to authorized staff in the jurisdiction, and that it should be the responsibility of the jurisdiction to restrict knowledge of this password. So no action is necessary in this matter, at this time.
Nel
[End emails]
The same company whose President told Ohio Republicans in 2003 that he was "committed to helping Ohio deliver its electoral votes to the president next year."
And deliver he did.
March 6, 2009 10:51 AM | Reply | Permalink
That whole Ohio 2004 election is very suspicious.
March 6, 2009 11:18 AM | Reply | Permalink