The commercial music streaming service Spotify was launched in 2008 and since then is has registered over 24 million active users of which 6 million are paying users. The have 3.7 million Facebook fans. It has over 20 million songs online and every day 20.000 new songs are added to the database. Users created over 1 billion playlists and over $ 500 million has been paid out to rights holders since the launch of Spotify. It may be clear that without big data techniques and tools used, Spotify would not be able to exist.
Spotify is a data-driven company, meaning that data is used in almost any part of the organisation. The numbers confirm this: Spotify users create 600 Gigabyte of data per day and 150 Gigabyte of data per day via different services. Every day 4 Terabyte of data is generated in Hadoop, a 700-node cluster running over 2.000 jobs per day. They currently have 28 Petabytes of storage, spread out over 4 data centres across the world.
They also developed a workflow manager, Luigi, which they open sourced. Luigi is a Python framework for data flow definition and execution. Luigi is used to crunch a lot of data. Most of the data is user-centric data, such as billions of log messages that allows Spotify to provide music recommendations or select for example the next song heard on the radio. The data however is also used in decision-making, providing forecasting information and business analytics. According to the Jason Palmer from Spotify Labs, data is part of their culture.
Just recently, Spotify updated its discovery page to a Pinterest-like design. This feature uses a lot of data to create the recommendations for the user. Data such as user profiles, which music played and playlists made and other historical data are used for this feature. With millions of users, a user does not have to have a large playlist or extensive profile to receive qualitative recommendations.
Spotify uses all that data also for other, fun and interesting, facts. In the beginning of 2013, Spotify used streaming data to predict the Grammy Awards winners. Spotify did this by breaking down its users listening habit, taking into account song and album streaming, to determine the popularity of the music. In the end, 4 out of the 6 predictions made by Spotify turned out correctly.
Without big data, Spotify would not have turned out the way it did. With an ever-growing presence in many countries and a growing listeners base only more data will be created in the coming years. More data will mean better recommendations, better predictions, more users and thus more payouts to the rights holders. Big data truly enabled Spotify to change the music industry.
Image: Courtesy of Spotify
