Tuesday, February 27, 2018

Team formation in Kafka

  • Hierarchy starts with a controller/supervisor 
  • It is a worker node elected by its peers to officiate in the administrative capacity of a controller
  • The worker node selected as controller is the one that is been around the longest

  • Maintain inventory of what workers are available to take on work.
  • Maintain a list of work items that has been committed to and assigned to workers
  • Maintain active status of the staff and their progress on assigned tasks

  • Once a controller is established and the workers are assigned and available team is formed
  • Team formation is the cluster and its members brokers that have assigned themselves to a designated controller within the cluster.
  • When a task comes in from a kafka producer, the controller has to make a decision which worker should take it. There are lot of factor to play
The controller within kafka has the following resposibilities

  •  Work availability and health
    • Needs to know who is available in good health
  •  Task redundancy
    •  What risk policy should govern its assignment decisions.Thing that determines what level of  replication to employ in case as assigned worker fails
    • It has to ensure that if work is assigned to a worker,and that worker becomes unavailable, the work assigned or that work already done is not lost
    • Each task given to a worker must also be given to at least one of the workers peers in the event of an unexpected catastrophe, but among a group 
  • Promotion
    •  For an assignment, if the controller determines redundancy is required, it will promote a worker into a leader, which will take direct ownership of the task assigned.
  •  It would be leaders job to recruit two of its peers to take part in the replication
  •  The risk factor to protect against loss is known as its replication factor
  •   Once peers have committed to a leader a Quorom is formed.


No comments:

Post a Comment