Simple demo for spark streaming

USING PYSPARK

1. Login to your shell and open pyspark

[cloudera@quickstart ~]$ pyspark
Python 2.6.6 (r266:84292, Jul 23 2015, 15:22:56) 

2. Run Netcat

[cloudera@quickstart ~]$  nc -l localhost 2182

Receivers in Spark Streaming

• Task which collects input data from different sources
• Spark allocates a receiver for each input source
• Special task that run on the executors 

Overview of Spark streaming

How much data does google  deal?

• Stores about 15 exabytes ( 1000000000000000000B )of data 
• Process 100 petabytes of data per day
• 60 trillion pages are indexed
• 1 billion google search users per month

Note: Fraud detection is an example of real time processing

Limitations of Map reduce

• Entire Map reduce job is a batch processing job
• Does not allow real time processing of the data

Modules in Apache Spark

Spark SQL

• Is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as distributed SQL query engine.
• DataFrames 
• Is a distributed collection of data organized into named columns. 
• It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. 
• DataFrames can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs.
• DataFrame API is available in Scala, Java, and Python.