This is the latest version of our course on big data:   Watch the full course on coursera at …  

The following diagram shows a typical Big Data Infrastructure Design. This is from one of Allied Consultant’s Big Data works.

  Cluster Expected Volume Benchmark hardware Project Hardware requirements Cores RAM # nodes Disk Source 6 Million records / month ~ 3 records per second HDFS 6 million/month 1 namenode, 20 datanodes, 2 CPU/node, 64GB RAM/node   1 6G 1 : Master 3: Slaves 120% of 6G =7.2GB/month Kafka 4 topics 6 million/month per topic 1 nodes @ 4 GB RAM, 1 CPU,200 GB disk each 16,000 Msg/sec 1 4G 1 24 GB/month Kafka Connector for HBASE N/A Kafka Connector for HDFS N/A Logstash 6 million/month 1 node, 3.75GB RAM, 1 CPU Cores 180 Events/sec 1 4G 1 Hbase 6GB/month 7 nodes, 32gb RAM, 8 CPU cores 60240 req/sec – 200 req/sec 1 8G 1 4GB/month

With the help of Big data, Small Businesses can gain the competitive edge they require to stay ahead of the curve. For beginners, Big data includes large data sets of information which can reveal insights about your customers to help you make valuable business decisions. Data can help you to compose overall strategy for your business & add incremental value. All you need to know is to look for the right kind of data which you already possess related to your customers without spending much money on buying data. Here are several reasons to use Big Data for Small Businesses. Setting Your Objectives Start off by defining objectives which you want to achieve with the help of Big Data. For a website, one goal could be improving traffic of your website. This shouldn’t only mean that people should be searching for hits to your site. Users visiting your website should be converted into leads. By setting objectives, small businesses need to decide what exactly they want to track — downloads, leads, purchases etc. You can track your results, for example the number of leads generated from the content you produce on your website in a given timeline & then compare results.  Make Decisions with real-time Customer Data Maintaining & Analyzing Data for Business Analysis is changing the competitive landscape for small businesses as well. With the help of a standardized plan, your insights will be more elegant & well tailored to your business needs by appearing with more predictable and enabling you to be on par with others in the competitive landscape. For problem-solving Business Analytics help understand real customer behavior on website. They also help you to look for problems (such as pages with high exit rates on your site) and finding ways to solve it. Analytics help you understand of what part of the website’s content needs improvement & how it will later on help you convert that customer into a lead. Tools such as GA help you understand metrics such as bounce rate etc. which enable you to know what turns them off & more importantly, what keeps them coming back to your website. Evaluate Your Competition Avail free tools such as Google Trends along with other social media analytics to show how popular a product or brand is and see what others are saying about you. You can also use these types of tools to sneakily check out your competitors. For example, if you notice a rival is getting more media coverage and social mentions, you can compare their content and campaigns to see what they’re doing better than you. Understanding Your Consumer Small businesses must use public data sets to their advantage as these tools are freely available for use. They can use them to identify & draw insights about any drivers which cause change. For example, a hotel service could use social media to identify containers of potential customers closest to their business premises. A food delivery service app could get data on the consumer

Data Scientists are known for having a knack for statistics, data analysis etc. in order to understand and obtain insights from a given dataset, usually quite enormous in quantity. Here are some fundamentally important data science skills that are absolutely necessary for a Data Scientist. This list is not a conclusive one, as it only provides a general review of essential skills that a data scientists needs: Passion for Problem Solving Problem-solving skills Data Scientists are critical in solving their day to day business problems. As a data scientist, you’ll be performing in-depth analysis of data with the knowledge of the relevant industry you’re working in. Business problems must be solved in a systematic manner, in a way that is critical for the business. Data Scientists also invent improved ways of how the business should use its data for better decision making. It is important to know what business problems your company is trying to solve, as well as practical strategies to solve them. Programming Familiarity with at least one programming language. It’s the most fundamental of a data scientist’s skill set. Problems for Data scientist are much more practical than theoretical in nature. It is also nice to have fundamental knowledge of algorithms and data structures for writing efficient code. Knowledge of an open source statistical computing package software such as a statistical package like R would help you with this. Python is also a popular choice among companies looking for data scientists. Also, anyone wanting to get into data science also needs to learn about databases. Without learning tools like Hadoop & SQL, you won’t be able to do much. Ability to Communicate Solutions To Problems This skill complements your technical skills in communicating the solution to the decision makers in a concise, effective manner. For this to happen, it is absolutely necessary to have good communication skills. Possessing good soft skills will help you in presenting your critical observations, as well as it will make your presentation impressive enough to convince the management. Good interpersonal skills are required to communicate all around within the organization hierarchy, including the non-technical staff e.g. Marketing & Sales department. It is often branded as storytelling because it simplifies all the complexities to communicate insights in a clear, comprehensive manner for others to act on the instructions. Knowledge of Statistics/Mathematics Most of the hard work is done by software, but it only makes sense if a data scientist has the ability to choose which statistical test to run when and what insights to gather from the results. To be a data scientist, one has to think like a researcher while dealing with your company’s data. Most of the interpretations will be done of data, and you will be expected to implement a solution that will improve the decision making of the business. For this, you’ll need to have mastered the basics of descriptive and inferential statistics. It is also important to have strong analytical skills by learning about multi variable calculus, Linear algebra etc.

Resource Management in Information Technology There is a whole host of technology available now a days to ensure that your IT hardware resources are managed efficiently. You may have a data center in house, a few cloud nodes/services/apps which together may constitute your investment in hardware. That would translate to resource capacity : memory, disk, processor. In that past you’d purchase a machine and deploy an application on it. It may, or may not consume the resources of the application. Typically these resources were set for peak capacity and most of the rest of the time the machines would sit idle accruing capital investment cost. Resource Management through Virtualization Then came virtualization. Virtual machines would typically contain an OS, app and their dependencies. Microsoft shipped special versions of windows to manage and host these technologies under the Hyper-V brand. But even with this, the VMs were bulky chunks of repetitive OS bits that weren’t really required on all nodes. The stingy guys are resource central felt a finder level of management was possible and introduced containers. A container is a miniature version of the VM in that it (generally) doesn’t contain an OS. IT really only contains the bits required to run an application and little more. But each container is a complete environment on its own (usually stateless). That made it lightweight and that meant you could fit more of these into the same resources. It also made application deployment and scale out much simpler by viewing the app as a self-sufficient, completely independent black box (container) that a resource manager (like docker) can move around, start and stop as needed. Key technologies here are docker (for container creation and execution), docker swarm (to automatically create a cluster of containers that act as one) and Kuberneties. There is much excitement around Kuberneties because it promises to provide a single platform to host containers that can run on multiple cloud providers, open stack and bare metal. This brings about an interesting possibility of having a certain (cheaper) fixed capacity in house along with some elastic cloud capacity all consolidated under one platform. This is still in the making though (a few years old) and is still very fixated on Linux (unlike VMs) But they didn’t really stop there. Frameworks like YARN and Mesos take the idea of application scale out further by having even more fine grained control of how an application scales out and how a clusters resources are used. YARN for e.g. typically would restrict the idea of an app to a java JAR file (a map reduce or Spark compiled code) , which it can then micro-manage. YARN lets you manage in what sequences, priority etc. the resources should be made available between jobs. E.g. Round Robin, Fair or capacity schedulers. Separate but tightly integrated with these are service registries like zoo-keeper that let you keep some sanity about all this through its service registry. Conclusion Now, practically speaking, most of the apps out in the business world (at