Welcome to the Big World of Data
The recent fad in IT is ‘Big Data’. It is famous and commands money & respect, so everyone wants to be ‘in’ it in the IT world. But to those who are not yet engulfed by this trend, this gives a primer to what it is in basic terms.
Lets start with the scenario
For a large global company, the data generated by its systems is unbelievably massive – from tens of Terabytes to Petabytes. It could be detailed sales data, transactions, instrument readings, website access & usage, customers, social media, search engine data etc.
The systems at the company keep accumulating more and more data over time. Over a period, it starts becoming huge, and keeps growing bigger and bigger and massive. The data could come in torrents and in disparate formats like text, images and videos, not just structured RDBMS type data.
This information is valuable. Companies need to analyse this data to understand their performance and make ‘learned’ strategic decisions for future.
If they ignore, their strategy becomes skewed, ‘base-less’ (because it is not based on data research), and hence risky. They can ignore this only at the cost of loosing out to competition. And this is a continuous process because they need to keep making course corrections at regular intervals to align with the actual performance(vs expected) and market movements.
Related: Beginners guide to Cloud Computing
But traditional technologies and database servers cannot handle and process such data with respect to the combination of huge size, great variety and high speed.
So Big Data technologies bring in capabilities to
- Take-in massive amounts of disparate data at high speed
- Hold in a distributed environment (distributed storage)
- Process it by horizontally distributing the load across clusters (distributed processing) and,
- Be continuously scalable (scale-out), available anytime (least downtime), robust, responsive, affordable and secure.
Traditional ‘scaling-up’ means adding more resources (memory, storage, processing) to a single node in a system, to which there are obvious limitations.
‘Scaling-out’ means to add more nodes, like adding more servers and distribute storage and processing. The advantage is that logically, it can be expanded to a limitless extent.
Big Data Solutions
Hadoop is one such Big Data solution. It is an Apache open source framework written in java. Hadoop is designed to scale up from a single server to thousands of machines, each offering local computation and storage.
Big Data has its own challenges and complexities, so it may not be suited unless the requirements have reached or are expected to reach those proportions of size, variety and speed.
The transition from the Enterprise Data Warehouse centric architecture to the Big Data Management System – on Premise, on the Cloud, or in hybrid Cloud systems – is going to revolutionise any company’s information management architecture.
The demand for Big Data professionals is rising significantly. To learn Hadoop and become a developer, knowledge of Java basics is crucial. Plus knowledge of basic Linux is helpful. Existing System Administrators, DBAs, BI/DW analysts, ETL architects etc have a good scope to move their career path.
If you are an aspirant, you can find options to join Big Data courses in your city or online at AddMyLearning.com. If you don’t find enough options, please contact us by submitting a query on the site. We will help you with offline options. If you are a Trainer or run a Training Institute, we can enable featuring and online enrollment of your courses through AddMyLearning.com.
Inviting your comments on this article. I will try to simplify some more of the niche IT trends in future. Keep following.
AddMyLearning.com provides you
- Discovr: A comprehensive directory of Training courses in any skill or location
- Counslr: A system for online guidance in making Career decisions by connecting you with experts.
- Fulflr: A chat & bid based Learning needs Fulfillment system.
We are in a non-stop continuous improvement mode ourselves to enhance your knowledge, skills and career, in every domain of knowledge. Be it IT, Software, Hardware, Managerial, Personality, Business, Vocational, Tuitions, Entrance Exams and Competitive Exams, Sports, Music, Dance, Arts, Cooking, anything, everything.
You are also invited to post guest articles on our blog.