Supercomputers and department servers have something in common with smartphones: no one really wants to buy more than they must, but the real power really comes from what applications can do with a combined platform. We now know that no one wants to carry multiple electronic organizers – that’s how we ended up with so many of us having smartphones. Good bye MP3 players, GPS devices, pocket cameras, navigation devices, single function phones, and electronic organizers! Even membership cards, credit cards and paper tickets are on the way out. The really amazing applications take advantage of the converging of all these capabilities into a single platform.
Today, we still speak often about different systems being used for HPC, Big Data, AI, and machine learning. Convergence is often predicted, but the reason that is cited is cost efficiency (purchase of a machine, building to house it, etc.)
I don’t mean to trivialize the cost factor – but the real reason for convergence is applications. Today very few applications combine HPC algorithms, big data, and AI/ML techniques. However, anyone who sees what is happening, when these techniques are combined, will quickly realize this is the future of applications. It is easy to realize computers will need to support this future of application!
To support this premise – I will share my thinking, and examples that I see supporting my contention that applications are what will really seal-the-deal on convergence.
Deploying Varieties of Computer Systems
Those who deploy computer systems, large and small, are on a quest to accelerate their time to new discoveries. For decades, High-Performance Computing (HPC) has been a broad term for such endeavors. Today, the goals of Artificial Intelligence (AI) fit perfectly into that big picture with their growing hunger for computer cycles. Such convergence offers opportunities for us all. What the big guys are doing, and thinking, can inspire us all.
In many organizations, expertise in HPC and AI are aligned with different decision makers who find themselves in competition within their organization instead of aligned in discovering the possibilities for their company. There is a huge opportunity for those who bring these factions together successfully. Such visionaries seem most likely to be in the best position to beat their competition!
I love AI in our HPC center, we can get wrong answers, and nobody seems upset. – anonymous
Insights
I will share some insights that can help your organization tackle the challenges related to the convergence of HPC and AI. I have had the good fortune to talk with a number of experts, including Robert (Bob) Wisniewski, who is Intel’s Chief Software Architect for Extreme Scale Computing. I hope to help inspire us all to look for ways to harness HPC and AI together, with their ever-increasing performance requirements to accelerate discovery and technology innovations. This will enable leveraging current HPC investments, as well as mapping out future requirements.
AI is a powerful tool for HPC workloads, and vice-versa; many future HPC machine/installations cycles will be dedicated to AI. – Robert Wisniewski, Intel Extreme Scale Computing
Thinking about convergence
It is easy to pen predictions of convergence, but reality can often be a different beast. While there are certain similarities between HPC workloads and AI workloads, there are also differences. I asked Intel’s Robert Wisniewski for his take on convergence. Bob told me, There are challenges to produce a single streamlined software stack for HPC and AI because their runtimes and frameworks were borne in different environments and require a disjointed set of libraries and frameworks. However, there are significant synergies between the two, and their co-existence and co-utilization are imminently imaginable. By focusing on what capabilities, they need to leverage from each other, it is easier to find a path forward and map out what needs to occur for their convergence.
When HPC and AI techniques come together, they often benefit significantly from tight coupling. This encourages convergence. Instead of having two distinct machines connected to each other, one doing HPC while the other does AI , convergence allows them to be run on a single machine. Bob pointed out that large data sets are one of the synergies between the two. Large data sets drive the need for massive amounts of compute, storage, and networking to be brought together in the right way to drive application performance. A second motivator for convergence is the significant performance gains that occur when HPC and AI are tightly coupled.
Fusion simulations
A specific example, which Wisniewski pointed me to, that illustrates his thinking, is the work by William (Bill) Tang as a computational scientist focused on fusion energy. Bill is a Principal Research Physicist at the Princeton Plasma Physics Laboratory at Princeton University. Containment of plasma, using magnetic fields, has remained a key challenge in developing fusion reactors.
The international ITER facility is the world’s largest experimental nuclear fusion reactor. They plan to be the first controlled device capable of maintaining fusion reactions for long periods of time. In order to sustain burning plasma reactions, disruptive events that cause plasma escapes must be predicted in order to minimize or avoid such disruptions.To predict and steer fusion reactions to avoid such disruptions, researchers at Princeton University have developed the advanced machine learning Fusion Recurrent Neural Network (FRNN) predictive code. FRNN uses deep learning methods to predict the onset of highly deleterious disruption events under reactor-relevant conditions in magnetically-confined fusion tokamak devices. These predictions from deep learning, augment the more traditional HPC simulations in a very important way. It’s a powerful combination, and interesting how AI techniques can help HPC simulations focus on what matters. Simulations in climate forecasts, weather forecasts, and pricing predictions, as we will discuss, can also benefit for AI guiding HPC simulations.
HPC drives or AI drives?
Perhaps unsurprisingly, Wisniewski categorizes some of what he has seen as HPC driving a project, and AI augmenting it and some as AI driving with HPC augmenting. The plasma escape problem, is one that Bob characterizes as an HPC workload that found a fantastic way to be augmented using AI. Bob cited the CANDLE project (Exascale Deep Learning and Simulation Enabled Precision Medicine for Cancer) as a project with AI driving the core work but with HPC in a strong role to augment the work well beyond the AI work itself. Traditionally, human experts have directed the use of HPC toward what they felt would be more promising investigations. AI can, in a sense, play the role of the human expert in the directing what HPC models to be run. Bringing these areas together is central to how the CANDLE project aims to bring the full promise of exascale computing to the problem of cancer and precision medicine.
Climate informatics
Climate informatics refers to innovation at the intersection of data science and climate science and is an emerging discipline that highlights how HPC and AI techniques combine to form important opportunities for advancement of science. In a published interview with Claire Monteleoni, a fellow at the University of Paris-Saclay, and an associate professor of computer science at George Washington University, she summed a key hope saying When you can massively process data at scale, you can study things in a more dispassionate way and find other trends. Monteleoni went on to cite examples of work also bringing AI techniques to bear for guiding work and finding patterns in data that had previously gone unnoticed.
Ensemble selection
Ensemble forecasting is a widely used technique in modern weather forecasting, and it is used in climate models as well (a significantly different field from weather prediction, with longer time scales, but with many similarities too). Instead of making a single forecast, an ensemble (a set, or collection) of predictions (forecasts) are produced by varying the initial conditions within their range of possible values (they have ranges due to uncertainties in what the initial conditions actually are), or certain parameters during a simulation (to reflect uncertainty inherent in the actual numerical models used), or which models are used in various ensembles (multi-model ensembles). Therefore, ensembles are a form of Monte Carlo analysis. Such ensembles help give an indication of the range of possible future states for weather or climate. For instance, if all outputs are essentially the same, then we can have higher confidence in the output.
Ensemble selection has a degree of art to it. Like the fusion simulation work, climatology work deals with too much data to ever hope to simulate at a sufficiently small granularity to find answers simply by brute force. Instead, we can look to AI to help guide ensemble selection. There are a number of innovative ideas being tried by researchers to use AI to help pick ensembles, and to help analyze the ensemble results. Some usage of machine learning work aims to integrate global observations and local high-resolution simulations into a comprehensive Earth system model (ESM).
AI is helping uncover phenomenon from climate models, such as discussed in a paper Unsupervised Discovery of El Nino Using Causal Feature Learning on Microlevel Climate Data. This paper makes the case that the climate phenomena of El Nino and La Nina arise naturally when certain levels of zonal wind (ZW) and sea surface temperatures (SST) over the equatorial band of the Pacific Ocean exist. The casual relationship was discovered without any input about past occurrences of El Nino or La Nina.
Hurricane targets, electricity, fuel and crop prices
Weather predictions of where a major storm will make landfall, such as a hurricane, is an example where an ensemble will give a range of possible answers. Hurricane forecasting is indeed drawing the attention of AI experts to attempt to help guide the ensembles with the hope of reducing the range of error in the results, or at least increase the certainty of the prediction probabilities.
Ensemble models, or Monte Carlo simulations, have applications well beyond weather and climate simulations. One example is electricity price forecasting. It is not hard to imagine other pricing models that could do similar work, including fuel prices and commodity prices. The addition of AI in all these approaches promises similar opportunities for improvement.
Vision from U.S. National Labs “ the CORAL Collaboration
CORAL stands for, and is, a Collaboration of Oak Ridge, Argonne and Livermore national laboratories. The draft CORAL-2 statement of work (SOW) was released earlier this year and highlights areas of interest for leading scientists. Included in the SOW are a few key statements regarding convergence:
- exascale systems will need to be designed to address the emerging convergence of data science, machine learning, and simulation science. ¦ A few examples of this convergence are: supervised learning methods used to capture the complex, non-linear relationships in the output of large multi-physics simulations and large science experiments; unsupervised learning used to guide multi-scale physics runs through a large-scale state-space; in situ analysis and extraction of information while a simulation is running.
- offeror shall describe ¦ how the technologies, architecture, and programming model address the emerging convergence of data science, machine learning, and simulation science.
Certain benchmark categories are established by the SOW, including Data Science and Deep Learning Benchmarks. The SOW says that these will represent the operations and algorithms more common in data analytics and machine learning. They stress lower precision floating point as well as integer operations, instruction throughput, and indirect addressing capabilities, among other system characteristics.
New capabilities will revolutionize areas such as energy production, materials design, chemistry, and precision health care.
The desire to do well in Data Science and Deep Learning Benchmarks does not replace the need for ever increasing performance on traditional HPC workloads. The SOW has benchmarks for the more traditional needs called out as Scalable Science Benchmarks and Throughput Benchmarks. The vision for convergence of AI and HPC, for the national labs, definitely looks like an HPC machine enhanced with high-performance capabilities added for AI workloads. This will, in turn, maximize the ability for AI to augment HPC, and HPC to augment AI within workflows and on a single machine. No wonder the SOW says that the capabilities of these systems will revolutionize areas such as energy production, materials design, chemistry, and precision health care.
Trends
When I asked Wisniewski to give a concise pitch for the convergence of AI and HPC, he simply said 1 + 1 is greater than 2. He clearly sees enormous potential, but he cautioned that it is a fast-moving field so the target is shifting. He prefers to speak of AI rather than Deep Learning in the same way he tends to describe HPC broadly rather than by individual fields. Deep Learning is definitely critical to support today, and well into the future, but Wisniewski emphasizes that AI is much more than Deep Learning. Architecting software or hardware without grasping that would be short-sighted.
AI is much more than Deep Learning.
In ten years, I will be surprised if deep learning will be as hot as it is now. Something else [AI related] will be hot instead. – Robert Wisniewski, Intel Extreme Scale Computing
Data Movement and Power
Unsurprisingly, we always come back to issues of power consumption and the role of data movement. Reducing data movement can positively impact power and performance.
Wisniewski shared an example of a common model for combining HPC and AI today: A simulation application completes its work and dumps output to a disk. This write operation will take several hours to complete for a moderate simulation. After the writes complete, an AI application running on different (or even the same) parts (nodes) of the machine reads the data back in to perform its calculations.
Immediately one can grasp the opportunities for improvements, but the solution space is non-trivial. It includes options for hardware convergence, finer grained storage, in-memory storage, object-based storage, and much more.
Coupling of AI and HPC is still in the art vs. engineering phase. – Robert Wisniewski, Intel Extreme Scale Computing
Exciting times: Art Integrating with Engineering
The convergence of HPC and AI is a hot topic, and published results are hot topics for publications and conferences alike. Wisniewski told me that he sees three key challenges in the convergence:
- architecting and building machines to meet the needs of both HPC and AI
- bringing frameworks to serve HPC and AI into the same software stack and getting them to work seamlessly across both disciplines
- training for us all (e.g., Intel AI Academy, and classes by Stanford’s Andrew Ng, I’m a huge fan of both of these resources)
Combining the strengths of HPC and the strengths of AI, offers opportunities to accelerate the path to new discoveries. I hope this article helps inspire thinking in these exciting times, and I hope the challenge of organizational divides between HPC and AI expertise will be bridged. We all have a role to play “ gain experience and reach out to the other side to find grounds to collaborate. That way, we can make 1+1 > 2.
Transparency
I have accepted compensation for my time interviewing Bob and writing my perspective (in this blog). Bob is a former co-worker (Intel) and a friend. This blog came about shortly after Bob visited our home in Oregon. I’m fascinated with what makes the world go around, and Bob’s frank input helped me articulate how applications are making this world go around. I came up with the title and content on my own using Bob’s input as a solid basis for this piece. I welcome any feedback.
