Friday, February 9, 2018

Receivers in Spark Streaming

  • Task which collects input data from different sources
  • Spark allocates a receiver for each input source
  • Special task that run on the executors 

  1. Task run within executors
  2. Collect data from input source and save them as RDDs in memory, so that spark can replicate the collected data to another executor for fault tolerance  
  3. If we have 2 sources then 2 exectors would be assigned for the streaming data

No comments:

Post a Comment