There are now many examples of how Cloud Computing infrastructure has delivered real-world benefits in the operation and management of large-scale distributed applications. Awareness of the technology continues to grow but there are still many areas where this awareness has not yet been translated into widespread use. One such area, we feel, is HPC-based scientific research. This post introduces the RAPPORT (Robust Application Porting for HPC in the Cloud) project and the work we are doing to identify and address the challenges of porting a group of scientific applications from different domains to a Cloud environment.
RAPPORT is a 6-month projectfunded under the JISC/EPSRC Pilot Projects in Cloud Computing for Research programme. The project came about as the result of a number of conversations between Computing researchers and application scientists. Links with groups originally made during the UK e-Science Programme resulted in some general discussions about Cloud Computing-related research and the use of scientific codes in a Cloud environment. The view expressed by many domain scientists was that they were aware of recent developments but didn’t have the time or resources to gain a detailed understanding of the technologies involved and how these may be leveraged to aid their research.
Distilling these discussions, we selected a set of three domain areas that could provide a group of applications covering a range of different scientific HPC application profiles. The selected applications were drawn from a range of interesting and high-profile research areas – Physics, Bioinformatics and the Arts & Humanities (Classics). RAPPORT is a collaboration between the London e-Science Centre (LeSC), academics and researchers from the application domains based at Imperial College London, the Software Sustainability Institute at the University of Edinburgh and Imperial’sICT HPC service. A description of each of our target domains and applications follows:
Particle Physics: The CMS experiment based at the Large Hadron Collider (http://cms.cern.ch/) is concerned with the study of the property of elementary particles. The data collected runs into tens of petabytes a year. To first order the data are processed centrally and grouped into datasets according to Physics criteria which are then distributed and analysed by physicists at the universities. Imperial College, as a major contributor to the CMS experiment, hosts several of these datasets. The smallest unit within a dataset is an ‘event’ – i.e. a collision of particles in the detector which can be analysed independently. We will be looking at two processes used for analysis of the data:Monte Carlo production used to generate a set of simulated events representing production and decay of elementary particles and data analysis based on software developed by the CMS team to analyse detector data or corresponding Monte Carlo data. This field presents a number of challenges in terms of its very large data requirements and the high-throughput nature of the computation requiring processing of large numbers of jobs to analyse a data set.
Bioinformatics:Several bioinformatics tools have been considered for evaluation in aCloud environment. Candidate software has been identified through asurvey of the software used within existing collaborations bythe Bioinformatics Support Service at Imperial. We have decided to focus initially on MrBayes(http://mrbayes.csit.fsu.edu/), anopen source, tightly coupled MPI code used by computational biologists to perform Bayesianinference of phylogeny (the deduction of evolutionary relationshipsbetween organisms). We also hope, term permitting, to profile theperformance of GenomeThreader (http://www.genomethreader.org/), which is used to predict the structure ofgenes. Both of these applications have been developed by academic groupsto analyse large data sets, and MrBayes in particular is designed toexploit the parallelism offered by compute clusters. However, manypotential users may not have access to suitable dedicated hardwareresources – hence our interest in provisioning such software for on-demandaccess via a Cloud-based environment.
Classics: The Arts & Humanities encompass a number of fields that are increasingly making use of high-performance computational infrastructure to manage and analyse large quantities of data. The Dynamic Variorum Editions (DVE) project (http://dynamicvariorum.perseus.tufts.edu/), part of the JISC/NEH/NSF/SSRC-funded Digging in to Data challenge (http://www.diggingintodata.org/) is a collaboration between LeSC, Imperial College, the Perseus Digital Library, Tufts University, MA, USA and the Department of Classics, Mount Allison University, Sackville, New Brunswick, Canada. The project is developing a framework for undertaking large-scale OCR recognition of Greek and Latin texts in order to identify and semantically track references to the Greco-Roman world. The project is making use of large numbers of publicly available scanned texts stored in repositories such as the Internet Archive. We believe that the computationally intensive, “embarrassingly parallel” nature of the application will be well suited to execution in a Cloud environment. The potential to scale significantly, beyond the number of local resources available to anindividual group, means that Cloud resources could dramatically reduce the time taken to analyse and index the target texts.
There are a range of Cloud platforms available to us and we intend to look at internal (Eucalyptus), external community-based (NGS Cloud prototype) and Public Cloud (Amazon EC2) platforms. LeSC have an existing Eucalyptus test-bed that is available for initial prototyping work and we are deploying an updated Eucalyptus test-bed based on more modern resources that have been made available by the High Energy Physics group at Imperial as part of the project.
The project is formed of 3 phase. Phase 1 will cover infrastructure provisioning and pilot application analysis in parallel. In phase 2, groups will work to port applications to a Cloud environment, looking at the range of available platforms to identify the most appropriate.Phase 3 covers evaluation of the work. As part of the evaluation phase we’ll also be working with a member of the Imperial College HPC Service team to look at how the profile of our jobs compares to those run on an established HPC infrastructure.We look forward to an exciting few months ahead and the opportunity to report some potentially valuable outcomes.