Manual Offset Commits & Exactly-Once Once Processing in Kafka Consumer using Python

In this video Manual Offset Commits & Exactly-Once Once Processing in Kafka Consumer is covered in detail using Python. Note: ---------- When we spin up a consumer and the consumer has subscribed to a topic which already has some messages in diff partitions then the consumptions of messages will not be in order in newly spinned up consumer as the single consumer is consuming messages from more than one partition and message ordering not guaranteed across multiple partitions-- same phenomenon we observed in this demo also... Also note , kafka is maintaining offset in partition level for a topic as different consumers are consuming messages from different partitions , so committed offsets for diff consumers will not be same , so it is not possible to have a single offset in topic level. As a result when consumer rebalancing happens , then for any partition , the new consumer start consuming the messages where earlier consumer stopped & ensure no messages are missed. Producer Code used: ---------------------------------------- from time import sleep from json import dumps from kafka import KafkaProducer topic_name='hello_world1' producer = KafkaProducer(bootstrap_servers=[&x: dumps(x).encode('utf-8')) for e in range(1000): data = {&: e} print(data) producer.send(topic_name, value=data) sleep(2) Consumer Code: ------------------------------- from kafka import KafkaConsumer from kafka import TopicPartition , OffsetAndMetadata import json consumer =...