Tech Primer: Big Data

Welcome to the Big World of Data

The recent fad in IT is ‘Big Data’. It is famous and commands money & respect, so everyone wants to be ‘in’ it in the IT world. But to those who are not yet engulfed by this trend, this gives a primer to what it is in basic terms.

Lets start with the scenario

For a large global company, the data generated by its systems is unbelievably massive – from tens of Terabytes to Petabytes. It could be detailed sales data, transactions, instrument readings, website access & usage, customers, social media, search engine data etc.

The systems at the company keep accumulating more and more data over time. Over a period, it starts becoming huge, and keeps growing bigger and bigger and massive. The data could come in torrents and in disparate formats like text, images and videos, not just structured RDBMS type data.

This information is valuable. Companies need to analyse this data to understand their performance and make ‘learned’ strategic decisions for future.

If they ignore, their strategy becomes skewed, ‘base-less’ (because it is not based on data research), and hence risky. They can ignore this only at the cost of loosing out to competition. And this is a continuous process because they need to keep making course corrections at regular intervals to align with the actual performance(vs expected) and market movements.

Related: Beginners guide to Cloud Computing

Traditional Methods

But traditional technologies and database servers cannot handle and process such data with respect to the combination of huge size, great variety and high speed.

So Big Data technologies bring in capabilities to

• Take-in massive amounts of disparate data at high speed
• Hold in a distributed environment (distributed storage)
• Process it by horizontally distributing the load across clusters (distributed processing) and,
• Be continuously scalable (scale-out), available anytime (least downtime), robust, responsive, affordable and secure.

Traditional ‘scaling-up’ means adding more resources (memory, storage, processing) to a single node in a system, to which there are obvious limitations.

‘Scaling-out’ means to add more nodes, like adding more servers and distribute storage and processing. The advantage is that logically, it can be expanded to a limitless extent.

Related: Software Testing: A Time-Tested Career

Big Data Solutions

Hadoop is one such Big Data solution. It is an Apache open source framework written in java. Hadoop is designed to scale up from a single server to thousands of machines, each offering local computation and storage.

Opportunities

Big Data has its own challenges and complexities, so it may not be suited unless the requirements have reached or are expected to reach those proportions of size, variety and speed.

The transition from the Enterprise Data Warehouse centric architecture to the Big Data Management System – on Premise, on the Cloud, or in hybrid Cloud systems – is going to revolutionise any company’s information management architecture.

The demand for Big Data professionals is rising significantly. To learn Hadoop and become a developer, knowledge of Java basics is crucial. Plus knowledge of basic Linux is helpful. Existing System Administrators, DBAs, BI/DW analysts, ETL architects etc have a good scope to move their career path.

If you are an aspirant, you can find options to join Big Data courses in your city or online at AddMyLearning.com. If you don’t find enough options, please contact us by submitting a query on the site. We will help you with offline options. If you are a Trainer or run a Training Institute, we can enable featuring and online enrollment of your courses through AddMyLearning.com.

Inviting your comments on this article. I will try to simplify some more of the niche IT trends in future. Keep following.