Why Perl 6 is Remarkably Robust at Handling Big Data Sets

Perl has undergone a massive overhaul over the last year.

Perl 6 was released last December. It’s virtually unrecognizable from previous versions of the 15-year-old programming language.

There are a number of updates Perl programmers should be aware of. Among the most significant is the emphasis on big data. Brian Kelly, The main developer of FullAuto said these applications will be tremendously useful in many verticals.

Perl has a huge community of avid users that continues to thrive in spite of detractors, said Brian Kelly, the main developer behind a configuration management tool written with Perl 5. This community like communities for every language in computing, is being pulled into the Big Data world, like it or not.

Tom Radcliffe, the Director of Engineering at ActiveState, concurs that Perl 6 will lead the way with big data analytics. He said the new language will be particularly valuable for his clients in the financial industry.

Many of our clients in the finance industry are using ActivePerl to pull data from various databases and process it, said Radcliffe. Its being used as big-data lite or as a way to load up big-data Hadoop systems.

How Does Perl 6 Handle Big Data?

Perl 6 is capable of handling data queries much more efficiently. The language relies on lightweight arrays, which dont require much attention from the programmer.

These arrays dont have properties, which limits their applications. However, they save considerable memory for applications where such properties are not needed. This isnt really a shortcoming though, because programmers usually work with data that doesnt have to exactly match built-in data types.

Perl 6 is Late to the Party, But May Be a New Leader in Big Data

A couple years ago, if you asked data engineers which programming language they relied on, they would claim Python or R. R was preferred by programmers that enjoyed a robust ecosystem and visualized data, while Python programmers enjoyed the short learning curve and general purpose applications.

Perl wouldnt even be on their list of languages for big data applications until Perl 5 was released. Perl 5 was able to handle large data sets much better than its predecessors. However, the functionality still left a lot to be desired. Programmers needed to work with very resource intensive arrays to process such data, which consumed a lot of memory.

In 2012, Perl developers predicted that they would be one of the pioneers of big data. Here is what Martin Holste said during YAPC::NA 2012:

Hadoop is overrated. Come see what modern Perl can do with map/reduce on terabytes of data with an extremely simple, maintainable architecture by orchestrating the inserting and querying of data on enormous scales. This talk will deconstruct the Enterprise Log Search and Archive (ELSA) project which is fully written in asynchronous, object-oriented Perl and provides a framework for Big Data analytics in a modular, pluggable architecture with the flexibility and customization that only Perl can provide.

Despite these promises, Perl was a late adopter of big data. The developers realized they couldn’t ignore it any longer. The language was quickly eroding as an industry standard, largely due to its inability to handle large data sets. This shortcoming threatened its very existence, as Conor Myhrvold pointed out a couple years ago.

Here is an excerpt from Myhrvolds post, discussing why he chose to transition from Perl to Python.

Why Python, and not Perl? Perhaps an illustrative example of what happened to Perl is my own experience with the language.

In college, I still stuck to the contained environments of Matlab and Mathematica, but my programming perspective changed dramatically in 2012. I realized lacking knowledge of structured computer code outside the “walled garden” of a desktop application prevented me from fully simulating hypotheses about the natural world, let alone analyzing data sets using the web, which was also becoming an increasingly intellectual and financially lucrative skill set.In 2013, Gregory Piatesky discussed the top languages for analytics, data mining and data science. He noted that most data mining applications were written in R, SQL and python. The usage of Perl for data mining decline 50% that year.

The launch or Perl 6 was intended to stave off the languages otherwise inevitable obsolescence. Considering how robust many of the new methods are for handling large data queries, Perl could very well be a leader in the big data revolution in the coming years.