Amazon AWS’s role in my Data Science pursuits

In a matter if two months, I climbed to the top 1% of Kaggle by solving some very interesting problems provided by leading organisations and through applying various Machine learning techniques to Complex data. If you haven’t known about Kaggle, it is a global platform that connects Machine Learning Scientists and Engineers with Organisations that wants to solve their data science problems in the form of  competitions.

While I do have some exposure in the AI related areas several years ago, I am neither a real Data Scientists holding a Phd or a Post-doc researcher or an Industry Veteran working in the field of Analytics except the fact I have been learning and working on some of the connected areas offlate. When I started at Kaggle initially, I quickly realised that Solving complex machine learning problems in its true sense is not for the weak hearted!  and I am one of those in the process of getting stronger over every weekend hacks these days and its been an exciting intellectually rewarding journey!

Several times over these weekend pursuits, I had to run algorithms on machines that required very high capacity and I had to do it the lowest cost. Amazon AWS so far has helped me address both these problems with its high memory XL and spot instances combined with the ability to quickly launch different sets of pre-baked machine learning run times through AWS machine images and Cloud-formation deployment.

In essence AWS is significantly helping me to leap forward in my data science pursuits.

Cloud Based IP Video Surveillance – A demo

There is a slow and steady proliferation of IP based Video surveillance around the World. Particularly in the US and Europe where advanced wired and wireless broadband IP networks have been rapidly gaining a stronghold . My belief is that the confluence of this  IP network proliferation combined with Megapixel High resolution HD Cameras integrated to On-demand Cloud services provides a  significant opportunity to create niche IP Video surveillance solutions on the Cloud.

I believe , Cloud combined with video software technologies has the potential to  become the largest and ‘defacto’ distribution hub for IP Video surveillance data in the days to come. Cloud based storage and content distribution services will bring a paradigm shift in this landscape.  The Elastic nature of the Cloud will bring petabyte scale DVRs recording and storing live IP surveillance Video, scalable cloud servers and CDN can broadcast hundreds and thousands of live streams to global consumers for instance….

As a demonstration of this concept, Here is a demo of a solution I created on Amazon AWS cloud.

The figure below provides high level overview of the Concept/Solution

Here is the Live Video of a Demo IP CAM Video in HD format transmitted and distributed from the Cloud in real time!

AWS Virtual Private Cloud(VPC) NAT Server Extended utilization – Part 1

Sharing an Interesting blog post from our company. This is about the possible ways to  leveraging the NAT instance that you would have to typically set up for VPC beyond its basic use.

http://cloudkinetics.wordpress.com/

Running a Hadoop Cluster on Amazon AWS Virtual Private Cloud (VPC)

Setting up and running a Hadoop Cluster for Big data processing on Amazon AWS’ Virtual Private Cloud (VPC) has specific advantages such as the ability to run a cluster in a much secure and isolated part of the AWS which is the VPC and in the case of enterprises where AWS VPC is integrated to act as an extended virtual data center to its on-premise IT environment, a hadoop cluster on AWS VPC could be seamlessly integrated to on-premise data and analytics systems.

However,, Amazon AWS’ EMR and other Hadoop vendor’s systems are typically designed to  be run on the AWS Public Cloud.

AWS VPC based Hadoop cluster is tricky to set up and run.

Here is a blog post of what we did to run a Hadoop cluster at our company to enable this on AWS VPC.

http://cloudkinetics.wordpress.com

For any questions or feedback email contactus@cloud-kinetics.com

Transcoding a 1080P HD Video on ‘Amazon Elastic Cloud Video transcoder’ in 30 seconds

Amazon web services today introduced an Elastic Video Transcoding service. I set out to quickly try a 1080P HD Video transcoding on it. Basically , I got a HD Video transcoded to  a 480 P ,SD Video in Mp4(H.264 codec) ,  and FLV (Flash video format) . Here is how i did it…..

1. I downloaded a publicly available 1080P HD Video [.mp4] ,~ 500 MB in size of  a space launch and uploaded it to one of my AWS S3 Buckets .

2. This bucket also served as the input bucket for the AWS Elastic transcoder

3.  I created a new Pipeline for AWS Elastic transcoding, specifying the name, input and output buckets

4. Next create a  transcoding job with the above pipeline and the 1080P HD Video file we had . Specified 480P ,SD Video as the video preset for transcoder output.  AWS provides various presets (i,e for Screen /device targets..)

5.  Start this job and the status of the transcoding job can be monitored. progressing now…

6. Next, I check the status again and the transcoding is completed . I think it just took about 30 seconds for this entire pipeline job to be completed

7. Next,Checked if the transcoded video has been saved at the designated S3 Bucket. Yes, It has indeed been processed and saved. The output is a 15.8 MB , mp4 video file with 480P resolution. (As you can see here, I had also transcoded a similar job with the output transcoding as an Flash media file (FLV 480P))

I then made the 480P transcoded video as public (s3 ACL) for playing and copied the URL

8.  To play this transcoded file, I used the the great JW Player’s Online wizard accessible at

http://www.longtailvideo.com/jw-player/wizard/     and configured its player mode to Flash player.[other mode being HTML5]

9. JW Player played the 480P transcoded MP4 and FLV videos of the spacecraft launch with a great outcome without a noticeable loss in quality.

MP4 Format play

FLV  format Play

I believe it is a great new service by AWS and the pricing at the outset seems to be cheaper than the some of the 3rd party vendors providing similar services.

I can only imagine, Amazon has set its sight on the Hollywood studios ! One of the slides I  used to see in some of the AWS conferences ,I had attended. I believe it is going to be forth coming. Studios could get thousands of their videos up in to S3 through AWS Import/Export…-:)  and get this done fast and cheap.

Recollecting Dr.Werner Vogels quote ”If you have a great idea, the cloud (aws) will execute it for you!”

Looks like the cost of video post-production for studios and media companies is drastically going to fall if AWS Elastic transcoding services could be leveraged in a suitable manner.

Why Amazon AWS Cloud is seamless to try than Softlayer Cloud

To Sign up with Amazon AWS , All you need is a valid credit card. If  you provide the relevant information, you can sign up in a matter of minutes or within an hour.

With Softlayer , After you submit the registration , you will have to go through a process with their sales to process your order . You have to submit relevant documents that they ask for.

The surprising thing that had happened to me with Softlayer when I had applied for a trial account was, After providing the relevant information, they responded to me saying that they could not process my order because of “Billing inconsistencies” .  So, what does “billing inconsistencies” mean? there is no specific information, so I asked them. They said, they are apologizing and that any further thing on this, I have to contact my bank.

With any credit card processing today globally, either a vendor says that your card has been accepted or rejected. How would a Bank be able to correlate a generic term such as ” billing consistencies” and associating it with a specific service provider’s order processing, If I were to go and ask them with out any basis or a reference or to a transaction ? As a matter of fact, the credit card that I had registered during the sign up with Softlayer is from an MNC bank and I have been using the same card with different global cloud service providers including amazon and a host of others since many years. I dont have any negative credit rating since many years.Hence there is no reason for me believe that there is a real issue.

I had further responded  to Softlayer , that there is no specific reason for me to pursue a conversation with my Bank taking an abstract reason given to me with out any specifics. Until now there has been no response .

Anyhow, the process of opening a trial account has been significantly overwhelming  based on my experience .

In contrast to this ,Based on the standard model of on-line purchase through a valid Visa or Master credit card, trying out /purchasing with Amazon AWS Cloud would be a breeze. Amazon AWS also provides a FREE tier to try your application deployments on the cloud or if you have play around with AWS services. Even updating your account information on Amazon takes a few minutes later when your personal information changes.

Amazon AWS  Cloud is awesome! Go for it!  It would be seamless and painless experience to explore your cloud pursuits.

OpenNebula Cloud Architecture Survey

OpenNebula released an interesting survey recently reflecting  the Cloud computing usage and adoption trends.

One the key response was that 58% of the respondents surveyed are running their workloads for  non-critical environment or peripheral installations for running testing or development applications, while 42% are using the cloud for running production workloads.  This kind of validates the expected trend about which I had blogged in the Yr 2010 !

It makes sense most of the time as non-mission critical or non-value add IT assets which doesn’t contribute to the ROI and/or which has lower risks are first class candidates to be evaluated for cloud migration.