Obama, Clinton finance reports overwhelm FEC computers
What a terrible problem to have...
Yet it’s had another consequence that has gone all but unnoticed. The campaign finance reports filed by Obama and Clinton have grown so massive that they’ve strained the capacity of the Federal Election Commission, good government groups, the media and even software applications to process and make sense of the data.
A milestone of sorts was reached earlier this year, when Obama, the Illinois senator whose revolutionary online fundraising has overwhelmed Clinton, filed an electronic fundraising report so large it could not be processed by popular basic spreadsheet applications like Microsoft Excel 2003 and Lotus 1-2-3.
Those programs can’t download data files with more than 65,536 rows or 256 columns.
This is astounding on two levels. For one, the huge influx of money to both campaigns is absolutely unheard of. Can you even imagine how much cash the eventual nominee will have to throw around this fall once we combine forces? Couple that with McCain's pathetic fundraising appeal, and we have a huge money advantage right off the bat, not to mention the luxury of running against the third term of the most unpopular President in US history. Secondly, is the FEC so outdated that they're really still using Excel 2003? They can't be that incompetent, can they?
well, just ask the FBI about their nifty 386's...
Once grandpa McCain gets to start using the GOP money for himself, he will have plenty. The military industrial complex will never allow the election of a populist president!
How long is it before Rove and McCain see this and accuse Obama of campaign finance violations?
"Secondly, is the FEC so outdated that they’re really still using Excel 2003? They can’t be that incompetent, can they?"
The fact that they used Excel 2003 gives me faith in them. The 2007 version is horrible.
Dennis @ 4:
2003 still uses a 16-bit integer to count rows.
Intel started selling 32-bit processors in the 1980's. So there you go.
If the Obama and Clinton supporters don't act like spoiled brats if their candidate doesn't win . . . it is the end of the Republican Party. If any supporter refuses to vote for the other one . . . then you are worse than the War Criminals that currently dominate our government. Think of it . . . band together and it is THE END of the Republicans. Over. Done. Gone forever. A new world. Or, pitch a fit and this Nation gets exactly what it then deserves.
Dennis @ 4:
I would agree, but I took a quickly course in 2007 and it isn't as horrible is it first appears. It has some quirks I don't like but I could grow fond of it.
But back to the fundraising issue - since a lot of these contributors were only contributing $25-$50, can you imagine the number of people that means that contributed to both camps?
This is not a good sign. Having to raise half a billion to run for the presidency is exactly what corporate America wants.
kakistocracy
or
Idiocracy
take your choice
I would be more concerned if the FEC was running the latest and greatest versions of everything. Upgrading for the sake of upgrading is a waste of time and money and rolling out a new version of any enterprise app is a pain in the ass and invariably breaks things. I would posit that Excel 97 would be perfectly adequate for the task. (Yeah, I work in IT)
Dennis @ 4:
I agree: on a corporate point of view, to switch form Excel 2003 to 2007 is a nightmare.
However I think the incompetence resides in the utilization of a spreadsheet to handle these data: how is it that they aren't using a real database ? Using Excel file to handle such critical data is much more then incompetence, it's a IT sin.
the government might change but the policies and the bureacracy will remain the same virtually nullifying the change in government
Comrade Rou @ 8:
And if you you can raise it a couple hundred bucks at a time from a couple million people .. that is just what gives Corporate America nightmares.
this proves they need to team up and be unbeatable.
FC @ 11:
That's what I'm wondering... a SPREADSHEET?!?! Seriously, WTF people?! Using MS Access would even be a step in the right direction.
It is nice to see so many Americans putting their money where their votes will be. Yippee!
FC @ 11:
This is the USA, remember?
The only thing about elections that concern the powers that be is how they can be manipulated to favor the GOP.
once we combine forces?
Ha ha ha ha ha ha!
You speak of a land which does not exist. The Clintons will be standing there, side by side, on October 29, in some gymnasium in Ohio or Pennsylvania, convinced she can still be the nominee and reminding people that Obama can't beat McCain.
Sorry, but "money" used to mean something BEFORE the media decided to give John McCain a billion dollars worth of free love called their eminent blessing.
I've moved on--it's 2012 or bust and make sure the next nominee has never been to church!
And yet Hilary's campaign is in debt?
Blue Taliban Osama Buddha @ 15:
It's called SQL. You can even get a free version of it. Database sizes are in terabytes now, not . . . kilobytes.
i havent seen a lotus spread sheet for 4 years
how does our government still function?
[Deleted. Off Topic-Sitemonitor]
Blue Taliban Osama Buddha @ 15:
On the contrary, as a means to get them to submit the information Excel is a perfectly adequate medium. There is nothing to say that they spreadsheets arent immediately imported into some large and magical application that properly categorizes and archives the information and even if they aren't there is nothing to say that they arent immediately copied and saved into a secure location.
If all that is required is to catalogue the contributions and do some simple calculations prior to submission then Excel, or another spreadsheet, is exactly the right medium. Why make it any more difficult for folks than it has to be. Every office has Excel, or Lotus, or OpenOffice and every office has someone that knows how to use it. Remember that those filing requirements are the same for all candidates, not just the whales.
And no, I cannot believe I am defending a Microsoft product... I have spent 20 years as a Oracle DBA and developer cursing Microsoft at every turn.. but sometimes the simple solutions are the right ones.
But there is no functioning FEC. It's the wild West, where McSame spokeswoman Fiorina can claim he has "more money" than Obama. Sure. Nobody can prove otherwise. See http://thehill.com/campaign-2008/mccain-camp-claims-it-has-money-lead-20...
Blue Taliban Osama Buddha @ 15:
M$ Access=a shitty wizard for SQL.
Isn't the other factoid to be taken from this backlog the sheer -number- of donors there has been? The Obama campaign (not sure about Clinton) frequently touts how much money it has raised from high numbers of small donors. I think this by itself is a great sign for democracy but I am also depressed by it.
Part of me is also saddened by the sheer amount of money that is collected and spent on running campaigns. Imagine what other things we could do with all that money (see also: Iraq). If every taxpayer would just contribute $5-10 per year we could have completely publicly financed elections and remove the influence of lobbyists and special interest groups.
CoIntelPro for Democratic Party Victory @ 9:
That's a false dichotomy, and you know it. In actual fact, we already have both.
To be fair, it seems that the referenced article does not say that the FEC uses spreadsheet program to handle the data, just that these data cannot be downloaded on Excel 2003 or Lotus 1-2-3 (OMG, who is using Lotus 1-2-3 in 2008 ?) format by the public on the their website and the massive amount strains their computing capacity.
DeLaRocha @ 26:
When you think that the candidates have to reach a couple hundred million people with their message it really isnt unreasonable to expect that it is going to cost a $ or 2 per voter to run. It is only when you do the math and total it up that it sounds horrifying.
This is good news, however, it remains to be seen how much McCain can bum off the rich.
McPapen will not win a fair election, no matter who the Dem nominee is!
No, they are not. Read the article! Nowhere did it say that the FEC is using Excel 2003. Why are you bashing them?
You can't be that incompetent, can you? (I'm not trying to flame here, just gently mock you). :-)
uncle joe hussein mccarthy @ 21:
badly
Sorry,
But even as much as I would like the hyperbole to fit, you can't know if bush is the least liked in history since polling is mostly a modern invention. That said I donated to Obama not Ubama.
NoTorture @ 31:
Obama, the Illinois senator ... ...filed an electronic fundraising report so large it could not be processed by popular basic spreadsheet applications like Microsoft Excel 2003 and Lotus 1-2-3.
Obama's people obviously use Open Office or something modern,
the FEC uses Excel to process submissions before the data goes into their database (of whatever type).
Thats how I read it
and just think its was only a few months agao that mccain was washed up ,no money on his way to the trash heap! it only shows you never can tell whos going to wind up with plenty of money and at the top of the heap, now even obama and clintons rolling in dough, no wonder the fcc lost!
too many moneys! oh noooooes!!!!!!
metroplexual @ 33:
You know the orgins of polling is medieval, its started as the practice of the king putting his enemies heads on poles,
then people would either spit on or nod to them as they went by, hence 'polling'
Breaking: McCain Campaign General Co-Chair At Heart Of Foreclosure Crisis
Site monitor: Wow, so surprised that my first ever post to my favorite website has be declared OT.
Let me try again:
This political season has clearly demonstrated a sea-change.
For the first time, I have given my hard-earned money to a candidate, and, evidently, I'm not alone.
This is it ya'll. Barack Obama has effectively utilized the intertubes to give us a way to effectively make ourselves heard.
Money talks. (and of course: bullshit walks)
T
T
cf @ 27:
LOL OK.
ferrofluid @ 34:
Well you misread it then ferrofluid because that is not what the article says. People are easily confused by selective bolding.
Many campaigns file FEC reports using FEC File, the software developed by the FEC to file campaign contribution and expense reports. Others use COTS products that integrate with the FEC online filing system (much like TurboTax does with eFile).
NoTorture @ 41:
Excel. Lotus 123.
Oh my God.
Then we wonder how Enron, Bear Stearns got away with it.
How about using a *database* to import, store and process this kind of data?
It's one thing for the electorate to be "dumbed down". It's quite another for the government bean counters to be in the same shape.
Had a similar issue at work today. Guy had a humongous csv file that wouldn't load into Excel. Told him to either import it into MySQL or parse it out with a Perl script. He decided to chunk it out and read it in pieces.
Old habits die hard. Especially when they come with a cute little paperclip thingy that pops up on your screen.
We're doomed.
once we combine forces?
I thought Hillary was free to roll that cash over into her 2010 reelection campaign? She's going to need it. New York is in search of a Ned Lamont to call our own.
What is the rule? She doesn't have to give her pennies to Obama, does she?
Used Excel 97 at my last job, maxed it out using to many pages and too many graphs per page.
We were testing a module calibration chamber we had built.
Phoned up It and asked "any way around this problem, maybe Office 2000!" reply was "nope"
"Can you even imagine how much cash the eventual nominee will have to throw around this fall once we combine forces?"
No, I can't.
And also -- any projections on when that nominee will be chosen/elected ... already ...?!
I mean, holy crap!
Dhalgren @ 44:
from the last figures I saw, Obama has 50M in the bank, 700K in debt, Clinton 30M in the bank, 15M in debt, McCain 11M in the bank, 600K in debt.
For God's sake. Get OpenOffice and dump M$.
Yes.
This has been another episode of simple answers to simple questions.
Just remember who has been president for the past 8 years.
B-, b-, b-, baby, you just ain't seen na-, na-, nothin yet.
... when it comes down to Obama vs McSame the flow of $$ will open up even more!
THE 1992 CAMPAIGN: Pennsylvania; Pennsylvania Governor Criticizes Process That's Turning to Clinton
By MICHAEL DECOURCY HINDS,
Published: April 24, 1992
With Pennsylvania's primary only days away, the state's Democratic Governor today criticized his party's Presidential primary process -- and its likely nominee, Bill Clinton -- saying the process had produced a front-runner who could not win. He urged the party's uncommitted delegates to remain neutral so that a stronger nominee could be selected at the convention this summer.
"We have to recognize reality," Gov. Robert P. Casey said in an interview today. "The primary process is not producing someone who has a good crack at winning in November." Pennsylvania Democrats hold their primary next Tuesday.
Referring to low turnouts in earlier primaries, Governor Casey said, "We've got a tiny minority of Democrats voting for Bill Clinton, and he's winning every race without generating any sparks, any enthusiasm, any momentum."
Mr. Casey spoke highly of the Arkansas Governor's resilience and intelligence, but added: "People have a tremendous unease about him. He's got a tiny, fly speck of support."
He said the Democratic Party should consider a strategy of having all uncommitted delegates to the convention in New York in July remain uncommitted so that they could consider nominating another candidate should Mr. Clinton fail to win more support."
Oh the irony.
ferrofluid @ 48:
People are often confused about the difference between primary funds and election funds. Hillary has a pile of the latter but there are strict rules about how these may be used if she is not the nominee. Too bad Karen isn't here tonight. She seems to have all this election arcana down pat.
"Secondly, is the FEC so outdated that they’re really still using Excel 2003?"
And another one voting on how dismayed, but not very surprised, he is that they are using a desktop spreadsheet to store and analyze a database. If money is an issue, they could use a FREE client/server database like PostgreSQL. PL/pgSQL is quite similar to Oracle and there is a simple OpenOffice.org reporting tool for the Program Assistant types that links via ODBC or java.
Well sure our candidates are raising unheard of amounts of money for their campaigns, but at least the Republicans can claim to be raising more money for their convention. Personally, I'd rather have a cheap convention without all of the excess and hoopla and have plenty of cash for the actual presidential campaign. If the amounts raised for the primaries are any indication, outraising McCain shouldn't be an issue.
John Tighe @ 19:
Precisely...the MSM has added Clinton into this headline. IT"S OBAMA that caused the problem. How did Hillary get added? Spin Baby Spin to have us think she has got something too. WTF
This isn't a huge deal. A lot of organizations use Excel as a quick-and-dirty database frontend. No need to overengineer a solution if quick-and-dirty gets the job done.
Now they have a job that may actually require a real database. Whoo! That doesn't mean their "computers" are overwhelmed; they just have to import the CSV or XML or whatever they got from the campaigns into something more tailor-made for data processing than Excel. It is a milestone, but one that is crossed by almost all organizations of any size. It doesn't make the FEC a bunch of incompetent doofuses. (doofi?)
slippy hussein toad @ 20:
1) Has anyone considered the idea of just using 2 files?
2) SSIS and other database integration solutions commonly ship with major databases and provide easy automated tools for importing excel files.
3) Excel gives campaigns large and small with an easily portable file format that doesn't require additional software or training than what most entry level office managers have right at their fingertips.
4) There's no reason for Excel to use anything more than a 16 bit integer for the row counter. It's already a memory and processing hog on most of the machines where it's getting deployed and for 99% of its' intended use, this isn't necessary.
5) 2003 is still the de facto format for exchanging spreadsheets since the majority of computers don't have 2007 yet.
That said, it's great that this problem exists :)
Oh - one more thing... if this was never a problem before, why would anyone expect a government office to shell out for a business class database solution? Even when databases are free, they need someone to design the schema and write some software. Believe me... I'm a software engineer, and I'm at the point of concluding that 95% of the folks who write code out there are fully incompetent. My office receptionist does a better job managing our time sheets with Excel than some of the developers can do with their project work.
I worry that the long primaries mean donations that would have otherwise been earmarked for the Presidential race have instead been ponied up for the primaries. If the donors' purse is zero-sum, that will mean a lot less Democratic donations for the Presidential race, no?
Uh.. Yeah .. WE the Left... Will combine forces under the Democratic Nominee .. Barrack Obama...
Who has won.. and earned it.
Hillary... has lost.. she does not get the automatic VP spot for coming in second place.
The winner should be able to pick their VP...
Unless of course you want to encourage the guys in the smokey rooms calling the shots type of government...
Seriously.. this is getting VERY old...
Alright, as an IT guy I'm going to explain this as simply as I possibly can for the non Technical Elite (I mean no disrespect).
MS Excel IS an analysis tool.
It should not be used to actually store data, it should only be used to evaluate that data in a simple detailed fashion using a bar chart, a line graph, or any other substantial charting mechanism.
Since MS Excel uses VBA as a Code/Macro system, columns and rows are limited to 65535 (2^16; No matter what OS or Version) items.
You should be using a Database system to retain the records of each transaction. Then you can organize the transactions via some sort of Database query. Those results can then be put into Excel, or any other analysis tool, and charted to review the significance of the data.
I will repeat this: A spreadsheet is an analysis tool. It should never be used to store substantial data that can not be derived from another data source.
AceGentlly @ 62:
I would not have been so kind; Why on Earth would anyone think that the Presidential candidates would use the same technology to database their donations as we would toward our own - comparatively mediocre - purposes. Does anyone seriously think that Obama made a stop at Best Buy so that he could keep track of the people that gave him money? Perhaps Clinton at Circuit City?!? I don't think that anything about the Clinton or Obama campaign has costed as little as a copy of Microsoft Excel. Sometimes it seems like the authors on this page will grasp for straws, in order to manipulate the news to fit their own narrative.
Actually, one market that uses Lotus 1-2-3 extensively is mining; the standard macros for estimating mining projects were originally written for 1-2-3 and were in widespread use as late as 2000. Lotus is a good spreadsheet, just not popular. It's sort of like WordPerfect; it barely exists outside the legal community, but in that community it's dominant.
As the poster above mentioned, a spreadsheet's not bad as a submission medium, though if I were handling the input I'd rather have CSV files; they're much easier to manipulate into something that you'd import into a real system rather than those Wintel toys.
Certainly, none of this indicates incompetence on the part of the FEC; conservatism in software upgrades is far from a sin in this sysadmin and computer programmer's book.
Dude! Why are you getting excited about so much campaign cash for Obama and Clinton and smearing McCain's campaign as woefully underfunded and requiring a pathetic appeal. Don't count your chickens before they hatch. The mainstream media is already bought and paid for and it wasn't from Obama or Clinton coffers. Chill out before you ejaculate all over yourself. If truth, honor and vision amount to nothing in a presidential campaign, what makes you think "a huge money advantage right off the bat" might?
Jack @ 64:
I've read the entire politico.com article. Where in that article does it say that the FEC is actually *USING* Excel or Lotus 1-2-3 to process/store/report on this data??? The article simply states that the amount of data related to Obama/Clinton campaign fund raising now exceeds the limits of those two spreadsheet programs. The limitations mentioned were 65,536 rows or 256 columns.
The politico.com article also states that the problem was fixed by noon the next day. Quite likely the FEC admins simply made an indexing change to their *database* where these records are stored to make data access more efficient. The FEC isn't "incompetent" or "outdated" as the author of this article suggests. They've simply encountered, for the first time, a new requirement of their data processing system. Companies grow, data grows, databases change to meet new demands: this is IT - it happens.
Good for Obama and Clinton that they have collected +66K online donations, but let's keep this in perspective, shall we?
As mentioned by several, the article is simply indicating the large number of rows, and not indicating actual use.
It would take me 2 minutes total to import all of the data into FileMaker (otherwise knows as Access done properly) and send it back with user privileges. I cannot imagine asking someone to move to SQL (complete overkill) or the dreaded Access (complete nonsense).
Excel 2007 doesn't work either. Spreadsheets are stupid. They don't work at all with big data sets. I downloaded this file and I've been poking at it here and there with Unix shell tools like cut, tr, sort, and uniq. This is the fastest and easiest way to deal with data like this. As others have said, it's a waste of time to use a spreadsheet or an SQL database for this. You need the right tools for the job. If you're using Windows, you're already on the wrong track.
"Can you even imagine how much cash the eventual nominee will have to throw around this fall once we combine forces?"
McCain isn't really running against anyone yet, since the Democratic nominee hasn't been officially chosen. Campaign money spent on Democratic infighting doesn't help beat McCain at all. Neither does money donatied to the losing Democratic contender. If Obama can keep outpacing McCain's fundraising after the conventions, then we can start counting chickens.
Guys, guys, guys,
I happen to know that they are using a database, though I'm not sure which one. (Well, at least they were four years ago.) But for viewing the data, sorting it various ways, and preparing analysis on it, there are a lot of people who are very, very familiar with Excel and how to make it jump through hoops. When they want to analyze the data, I assume they would have some pre-built analyses in the database/front end, but if they want something that isn't in there, they most likely take the list and put it in Excel. It's a hell of a lot easier than getting the SQL programmer to dick around for a few hours.
Now they can't do this. And that's annoying to them. Do you really think that's somehow unreasonable?
God, there are plenty of things to complain about with this government. Let's not get side-tracked on red herrings.
-fred
OpenOffice.org only goes up to 65536 rows, 256 columns as well, for compatibility with Excel (though OOo 3.0 will go to 1024 columns). However, geeks can easily recompile it to go up to 2,000,000 by 32,000. For what that's worth :-)
>Secondly, is the FEC so outdated that they’re really still using Excel 2003? They can’t be that incompetent, can they?
Incompetent? Come on! If they've never had this problem before than their solution is working for soemthing like 99.999% cases right? I'm still using Excel 2003 here at my small business. Not because we are a bunch of incompetents, but because there's no urgent need to upgrade. When we need to process hundreds of thousands of records, we use Access or SQL Server.
Yeah, because they must use or convert them to Microsoft Excel. Big deal, use multiple files.
Comments are closed on this entry