During the 1,5 year prior to the Election Day in November 2012 in total over $ 1.5 billion was collected and spent during the Obama campaign. In addition, over 1.000 paid staff worked on the campaign, 2,2 million volunteers and in total more than 100 data analysis who ran more than 66,000 computer simulations every day. The objective of the campaign set out by Jim Messina was to measure everything. The idea was to demand data on everything that happened during the campaign in order to measure everything and ensure that they were being smart about everything.
One Metric
During a webinar organised by HP Vertica, Chris Wegrzyn – Director of Data Architecture of the DNC, explained that in order to do this, they had defined three major ways to influence the campaign:
- Registration: increase the amount of voters who were eligible to vote;
- Persuasion: Convince voters to vote for Obama;
- Turnout: Increase the turnout on the actual Election Day.
Each potential swing-state voter would be assigned one number, ranging from 0-100. There were four different scores based on the three different ways to influence:
- The likelihood that they would support Obama;
- The prospect that they would show up at the poll;
- The odds that an Obama supporter who was an inconsistent voter could be nudged to the polls;
- How persuadable someone was by a conversation on a particular issue.
- This metric would be at the heart of the campaign and would influence the message send to a swing-state voter.
Fragmented Data Sources
In order to effectively manage all this during the campaign they divided the campaign team in different channels:
- The Field channel (actively approach voters in the field);
- Digital channel (focusing on recruitment of staff and volunteers and fundraising);
- Comms / Press (focusing on the persuasion aspect);
- Media (persuasion by buying media time);
- Finance (fundraising).
The problem with these different channels was however, that all data was also managed fragmented and that an overview was difficult to achieve. That was when big data made its appearance in the campaign.
During the previous campaign they had learned a lot already regarding new technologies and usage of social media and now it was time to move forward. The new technologies used in 2008 and analytics captured during that campaign allowed them in 2008 to build an unprecedented massive efficient measurable program and as such were all field staff evaluated based on data entered. From the introduction of new technologies and a data focus in 2008, it was now time to move to data modelling and deep analytics. In 2012 the objective was to build an analyst driven organisation and an environment for smart people to freely pursue their (data-driven) ideas.
Three Dimensions
The DNC determined three dimensions to focus upon:
- Volume: according to big data standards, the amount of raw data they had, was small. They had under 10 TB to start with, but because they led their analysts pursue their data-driven ideas they created a tenfold of this in a short time frame.
- Variety: there were many sources of data; much of it was new to the DNC. Because of the short time span, they did not have time to build ETL processes to bring it all together nicely.
- Velocity, the data analysts, staff and volunteers created new data at high-speed and that needed to be taken care of.
MPP Database
In order to cope with all this they decided to use a MPP Database, a Massively Parallel Processing database built by HP Vertica. They used this because of familiarity and simplicity, as it uses an SQL Model. It has a high-speed performance, it is stable and it is very scalable, meaning it could easily grow with the needs of the DNC.
On top of this they had built a positive feedback loop, so that the engineers could build on top of each other. This proved to be a powerful tool and it led to unexpected innovations. Such as that potential voters could receive tailored news information on a topic they had said to be interested in when a volunteer had come by their house. Such an email would have been sent by the local field agent to keep things personal. This was all done automatically.
In the end, the decision to move to big data was a very good and once again, just as Obama did in 2008, the campaign changed the playing field and raised the bar for future campaigns. What will happen to the massive amounts of data collected is yet unclear. The Washington Post reported earlier that other Democratic candidates are eager to use that data for their own campaigns, however it is unclear whether the DNC has sufficient resource (financial and technological) to manage and maintain all data produced.