Sunday, April 22, 2018

Kafka partitions

  • Each topic has one or more partitions
  • The no of topics in kafka is dependent on the circumstances in which Apache Kafka is intended to be used.It can be configurable
  • A partition is the basis for which kafka can
    • Scale
    • Become fault-tolerant
    • Achieve higher level of throughput
  • Each partitions are maintained at at-least one or more brokers
 Note: Each partition must fit on an entire machine. If we have one partition for a large and growing topic, we would be limited by the one broker node's ability to capture and retain messages being published to that topic. We would also run into IO constraints



 Partition management in Kafka
  • When the command to create 3 partitions is issued, it  is handled by a Zookeeper (Maintains meta data regarding the cluster)
  • ZooKeeper looks into the available brokers and decide which brokers will be made the responsible leaders for managing a single partition within a topic
  • When that assignment is made, each unique kafka broker will create a log for the newly assigned partition.
  • As partition assignments are broadcast, each individual broker maintains a subset of the metadata that Zookeeper does,particularly the mapping of what partitions are being managed by that brokers.This enables any individual broker to direct a producer client to the appropriate broker for producing message to a specific partition.
  • Status is sent by each broker to the Zookeeper
Producer Messages
1) When a producer is ready to send messages to a topic. it must have knowledge of at-least one broker in the cluster, so it can find the leaders of the topics partitions.
2) Each broker knows which partitions are owned by which leader. 
3) The metadata related to the topic is sent back to the producer so it can send messages to the individual brokers participating in managing the topic.

Consumer Inquires
1)When consuming message from the cluster, the consumer inquires of Zookeeper which brokers own which partitions and gets additional metadata that affects the consumers consumption behavior
2) Once the consumer knows the brokers, with the partitions that make up the topic, it will pull the messages from the brokers based on the message offset per partition. 

Partitioning tradeoffs
  • The more partitions the greater the Zookeeper overhead
    • With large partition numbers ensure proper ZK capacity
  • Message ordering can become complex
    • Single partition for global ordering
    • Consumer-handling for ordering
  • The more the partitions the longer the leader fail-over time






21 comments:

  1. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. freelance web designer

    ReplyDelete
  2. This is a nice and informative, containing all information and also has a great impact on the new technology. Thanks for sharing it
    website designing Service

    ReplyDelete
  3. Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. Web Designing

    ReplyDelete
  4. When you use a genuine service, you will be able to provide instructions, share materials and choose the formatting style. website design agency lakeland fl

    ReplyDelete
  5. شرکت تلکا هاست پیشرو در میزبانی انواع وب سایت و هم چنین دارای انواع خرید هاست ارزان با کنترل پنل سی پنل میباشد.حتما از سایت ما دیدن کنید و از قیمت های مناسب برای انواع سرویس های وب و ثبت انواع دامنه شگفت زده شوید.

    ReplyDelete
  6. I have recently started a blog, the info you provide on this site has helped me greatly. Thanks for all of your time & work Professionele webshop laten bouwen

    ReplyDelete
  7. I found that site very usefull and this survey is very cirious, I ' ve never seen a blog that demand a survey for this actions, very curious... website development companies in Egypt

    ReplyDelete
  8. I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. interface companies

    ReplyDelete
  9. Nice post. I learn some thing tougher on distinct blogs everyday. Most commonly it is stimulating to learn to read content from other writers and exercise a specific thing there. I’d would rather use some together with the content in my weblog no matter whether you don’t mind. Natually I’ll provide you with a link in your web weblog. Many thanks for sharing. website designers san francisco

    ReplyDelete
  10. Just wanna comment that you have a very nice site, I the style and design it actually stands out. converting websites

    ReplyDelete
  11. You lost me, friend. I am talking about, I imagine I purchase what youre saying. I’m sure what you’re saying, nevertheless, you just appear to have forgotten that you will find a few other folks from the world who view this trouble for it is really and could perhaps not agree with you. You could be turning away many people that was lovers of the website. brand identity los angeles

    ReplyDelete