Amidst searching for a new job, I have been diving deep in to the emerging world of High performance (Super)Computing on the Cloud. It’s been fascinating world and the impression I have now is that Cloud computing will make super computing a commodity in the not too distant future.
Cloud computing is really disruptive and will bring revolution of sorts from this perspective to enable and cater to the growing super computing demands in the fields of Science,Engineering ,Biology and Information Technology. It is already happening infact.
Today,One can deploy a Cloud computing system that can process at Teraflops scale. Amazon web services cloud was ranked 231 in the Top 500 list of Elite for its ability to process at speeds of 41.82 teraflops.
The Economics of Super computing for this very interesting and inviting. One can run a complex Fluid dynamics CAE simulation on a 100 node system for say $200 an hour or a 10,000 core system for $2000 an hour. Even in the recent past ,say 1-2 years ago, Companies especially small and mid size ones, couldn’t even imagine about accessing or leasing a 1000 core computing cluster to do Scientific or Engineering computations.
Today, there is drastic change in this scenario.One of the things Cloud computing is big and disruptive is just because of the paradigm shift it is bringing about in this realm.
What is driving this paradigm shift :: Super Computing on the Cloud
* The Availability of High powered computers,storage and networks on the Cloud and several players evolving in the cloud Eco-system with competing technologies and services,
* The on-demand nature/model of the cloud with lower costs
* The availability of numerous and proven complex scientific processing software on a open source model (e.g. BLAST,OpenFOAM etc)
* The Availability of proven and complex open-source software meant for enabling supercomputing models. The existence of proven reference models.
* Availability of Innovative ,Opensource Cloud management software (e.g. OpenNebula)
* Some of the Vendors of large scale Clustering software offering their service on the cloud
* Initiatives by companies in simplifying and democratizing supercomputing as a service (very few companies ,but this is a very interesting )
* Needless to mention the huge need/demand to research, engineer and develop complex products in the areas such as Fundamental research in Science, Life sciences, Aerospace,Automobiles,IT and Telecommunications so on on…
The very interesting fact here to be noted is the open, democratization initiatives by a very few ,innovative companies to “really offer an end user everything” that is required to do a meaningful supercomputing effortlessly. They offer this in an on demand model, be it to an individual or to a Fortune 500 organization.
For instance an engineer or a scientist or a small upstart company with the right domain knowledge and resources may be in a position to afford and use such a facility to research discover and develop revolutionary products and can compete with a large MNC . Also, revolutionary changes could be brought about from the real possibilities of drastically slashing the time for ideation to implementation (i.e : Shortened Time to discover and implement) .This is a very interesting and disruptive phenomenon of the cloud.
Imagine the paradigm shift this could bring about! For instance, a Pharma Scientist with no upfront infrastructure and investments may leverage the cloud based supercomputing to discover and develop the formula for making antibiotic drugs for a complex disease for say at $ 50,000 or less. Such a person may have the potential to productize and commercialize his Intellectual properly for several million dollors and save the lifes of millions of people. There is clearly a potential for mass impact!!
This is a small glimpse of what Cloud computing in the future will bring about. One of the many scenarios that will drive the large scale adoption of cloud (super)computing . Hence my viewpoint that not too distant in the future, the possibility of Super computing to get commoditised.
In essence the global availability combined with growing power and capacities of Cloud computing services in the form of hardware , software and easy to use solutions available and accessible to anyone in the world on a open (democratic),low cost model has significant potential to commoditise super computing in the years to come. The biggest beneficiaries of this will be the Cloud service providers (Amazon,RackSpace,Go Grid ,Microsoft Azure etc) . (Good to invest and hold some stocks in such select companies on a long term?!)
Potential and Opportunities :- The emerging “Cloud-Supercomputing” road to Commoditisation presents very interesting commercial possibilities and opportunities to individuals and organisations. In the context of IT & Software development which is my profession, there are significant opportunities to leverage this at this point of time, when things are at incipient stages. Here are a few possibilities and Scenarios
Sample Scenario -1 (An Easy to use, Supercomputing Enablement on the Cloud)
* An automotive company wants to develop next generation car models faster than its comptetitors through a large scale Aero and Fluid dynamics simulation and trials
* It cannot afford to buy lease or rent “Boutique” super computing facilities or may have constraints in buying expensive commercial software.
* Its key challenges include inhouse IT resources and capabilities to enable this and ofcourse budgets
Opportunity :: An Upstart Software company or a large IT services company integrates a number of available opensource and/or commercial software to form a complete solution on the cloud at a supercomputing scale, that facilitates the Automotive firm to run their Simulations seamlessly at low costs without upfront investments,overheads and liabilities!
The opportunities are in providing System and Application level software frameworks and solutions for Automatic provisioning,deployment ,control and monitoring .
(The beauty here is you as the enabler and service provider also donot incur upfront costs and overheads)
Also, if you try to be open and democratic in your model of offering, you could better cater to the demands and reap the benefits of Super computing demands on the cloud .
Sample Scenario -2 ( Economic Optimisation of Super Computing the Cloud) A Bio-Sciences company wanting to develop a new drug intends to run BLAST (Basic Alignment and Search Tool) on the cloud ,say that may potentially demand several hundred or thousands of cores during a peak BLAST operation.
One of the challenges associated with running BLAST is, you dont know how long the query and search operation will run to determine certain bio patterns and out comes. Will it run for 2 days ? Will my cloud cost me a Bomb ? This means the end user doesnot have a clear picture on what is the computing capacity needed to run it or what is going to be the cost of running it , if all they want is to run it in a few hours or days in a predictable or deterministic fashion.
Opportunity :- (Software based HPC Optimisation on the Cloud)
* Develop a Software that does an intelligent auto-scaling for specialised super computing use-case scenarios and related software tools on the cloud
* Implement intelligent algorithms (existing or that you may develop) in your software that can estimate the runtimes and capacities in advance for specific simulation tools such as BLAST and accordingly implement the auto- scaling of computing elements on a cloud. Provide real time visibility though a dynamic software interface.
* Implement this software on a SAAS model on different cloud platforms/services. Opportunities for Scientists , Engineers , Academic Researchers and Small to medium size organisations
The notion of Supercomputing on the Cloud is already a reality. Reality to a certain degree from the real supercomputing forms of usage and performance benchmarks. Nonethless, Sooner or later ,one may be able to sign up and deploy Supercomputers on an unimaginable scale on the cloud .
This paradigm shift and the emerging landscape is going to be an exciting period for Cloud services providers,IT practitioners and End users to leverage this significant opportunity .
The progression of computing needs in this 21st Century combined with the open,democratic, globalized,low cost structure of the Clouds will fuel the “Commoditisation” of Super Computing. One day, I have a dream to gift a super computer for the Science Lab of my school located in a small town in South India and I know now this is going to be possible!!
Finally , This is just a post to share my thoughts on what is the envisaged big picture. The technical aspects of to how Cloud computing enables Super computing , what are its various ingredients,approach and how to leverage them for commercially to build Software products,solutions or services or How to run a large scale CAE simulation is beyond the scope of discussion for this.
If you are interested in understanding it further in your commercial initiatives or leveraging this , you may contact me at nagarajansankar[at]gmail[dot]com
I am currently involved in researching and trying out cloud based super computing for specific identified commercial models and usecases. I will be happy to collaborate with like minded people or organisations with initiatives on this front.