Amazon AWS’s role in my Data Science pursuits

In a matter if two months, I climbed to the top 1% of Kaggle by solving some very interesting problems provided by leading organisations and through applying various Machine learning techniques to Complex data. If you haven’t known about Kaggle, it is a global platform that connects Machine Learning Scientists and Engineers with Organisations that wants to solve their data science problems in the form of  competitions.

While I do have some exposure in the AI related areas several years ago, I am neither a real Data Scientists holding a Phd or a Post-doc researcher or an Industry Veteran working in the field of Analytics except the fact I have been learning and working on some of the connected areas offlate. When I started at Kaggle initially, I quickly realised that Solving complex machine learning problems in its true sense is not for the weak hearted!  and I am one of those in the process of getting stronger over every weekend hacks these days and its been an exciting intellectually rewarding journey!

Several times over these weekend pursuits, I had to run algorithms on machines that required very high capacity and I had to do it the lowest cost. Amazon AWS so far has helped me address both these problems with its high memory XL and spot instances combined with the ability to quickly launch different sets of pre-baked machine learning run times through AWS machine images and Cloud-formation deployment.

In essence AWS is significantly helping me to leap forward in my data science pursuits.

Image

Posted in Uncategorized | Tagged , , | Leave a comment

Cloud Based IP Video Surveillance – A demo

There is a slow and steady proliferation of IP based Video surveillance around the World. Particularly in the US and Europe where advanced wired and wireless broadband IP networks have been rapidly gaining a stronghold . My belief is that the confluence of this  IP network proliferation combined with Megapixel High resolution HD Cameras integrated to On-demand Cloud services provides a  significant opportunity to create niche IP Video surveillance solutions on the Cloud.

I believe , Cloud combined with video software technologies has the potential to  become the largest and ‘defacto’ distribution hub for IP Video surveillance data in the days to come. Cloud based storage and content distribution services will bring a paradigm shift in this landscape.  The Elastic nature of the Cloud will bring petabyte scale DVRs recording and storing live IP surveillance Video, scalable cloud servers and CDN can broadcast hundreds and thousands of live streams to global consumers for instance….

As a demonstration of this concept, Here is a demo of a solution I created on Amazon AWS cloud.

The figure below provides high level overview of the Concept/Solution

IP CAM CLOUD

Here is the Live Video of a Demo IP CAM Video in HD format transmitted and distributed from the Cloud in real time!

Posted in Cloud Computing | Tagged , , , , , | Leave a comment

AWS Virtual Private Cloud(VPC) NAT Server Extended utilization – Part 1

Sharing an Interesting blog post from our company. This is about the possible ways to  leveraging the NAT instance that you would have to typically set up for VPC beyond its basic use.

http://cloudkinetics.wordpress.com/

Posted in Uncategorized | Tagged , , , | Leave a comment

Running a Hadoop Cluster on Amazon AWS Virtual Private Cloud (VPC)

Setting up and running a Hadoop Cluster for Big data processing on Amazon AWS’ Virtual Private Cloud (VPC) has specific advantages such as the ability to run a cluster in a much secure and isolated part of the AWS which is the VPC and in the case of enterprises where AWS VPC is integrated to act as an extended virtual data center to its on-premise IT environment, a hadoop cluster on AWS VPC could be seamlessly integrated to on-premise data and analytics systems.

However,, Amazon AWS’ EMR and other Hadoop vendor’s systems are typically designed to  be run on the AWS Public Cloud.

AWS VPC based Hadoop cluster is tricky to set up and run.

Here is a blog post of what we did to run a Hadoop cluster at our company to enable this on AWS VPC.

http://cloudkinetics.wordpress.com

For any questions or feedback email contactus@cloud-kinetics.com

Posted in Cloud Computing | Tagged , , , , , , | Leave a comment

Transcoding a 1080P HD Video on ‘Amazon Elastic Cloud Video transcoder’ in 30 seconds

Amazon web services today introduced an Elastic Video Transcoding service. I set out to quickly try a 1080P HD Video transcoding on it. Basically , I got a HD Video transcoded to  a 480 P ,SD Video in Mp4(H.264 codec) ,  and FLV (Flash video format) . Here is how i did it…..

1. I downloaded a publicly available 1080P HD Video [.mp4] ,~ 500 MB in size of  a space launch and uploaded it to one of my AWS S3 Buckets .

transcode1

2. This bucket also served as the input bucket for the AWS Elastic transcoder

transcode2

3.  I created a new Pipeline for AWS Elastic transcoding, specifying the name, input and output buckets

trancoder3

4. Next create a  transcoding job with the above pipeline and the 1080P HD Video file we had . Specified 480P ,SD Video as the video preset for transcoder output.  AWS provides various presets (i,e for Screen /device targets..)

transcode-480-mp4-in

5.  Start this job and the status of the transcoding job can be monitored. progressing now…transcoder5

6. Next, I check the status again and the transcoding is completed . I think it just took about 30 seconds for this entire pipeline job to be completed

transcode-480-mp4-complete

7. Next,Checked if the transcoded video has been saved at the designated S3 Bucket. Yes, It has indeed been processed and saved. The output is a 15.8 MB , mp4 video file with 480P resolution. (As you can see here, I had also transcoded a similar job with the output transcoding as an Flash media file (FLV 480P))

I then made the 480P transcoded video as public (s3 ACL) for playing and copied the URL

transcode-480-mp4-s3-out

8.  To play this transcoded file, I used the the great JW Player’s Online wizard accessible at

http://www.longtailvideo.com/jw-player/wizard/     and configured its player mode to Flash player.[other mode being HTML5]

9. JW Player played the 480P transcoded MP4 and FLV videos of the spacecraft launch with a great outcome without a noticeable loss in quality.

MP4 Format play

transcode-jwplay

FLV  format Play

transcode-jwplay-flv

I believe it is a great new service by AWS and the pricing at the outset seems to be cheaper than the some of the 3rd party vendors providing similar services.

I can only imagine, Amazon has set its sight on the Hollywood studios ! One of the slides I  used to see in some of the AWS conferences ,I had attended. I believe it is going to be forth coming. Studios could get thousands of their videos up in to S3 through AWS Import/Export…-:)  and get this done fast and cheap.

Recollecting Dr.Werner Vogels quote ”If you have a great idea, the cloud (aws) will execute it for you!”

Looks like the cost of video post-production for studios and media companies is drastically going to fall if AWS Elastic transcoding services could be leveraged in a suitable manner.

Posted in Cloud Computing | Tagged , , , , | 3 Comments

OpenNebula Cloud Architecture Survey

OpenNebula released an interesting survey recently reflecting  the Cloud computing usage and adoption trends.

One the key response was that 58% of the respondents surveyed are running their workloads for  non-critical environment or peripheral installations for running testing or development applications, while 42% are using the cloud for running production workloads.  This kind of validates the expected trend about which I had blogged in the Yr 2010 !

It makes sense most of the time as non-mission critical or non-value add IT assets which doesn’t contribute to the ROI and/or which has lower risks are first class candidates to be evaluated for cloud migration.

Posted in Uncategorized | Tagged , , , , , , | Leave a comment