There exists only one way to avoid message loss: use the tools that banks use.
They loose the money and feel the pain if a message dies and are seriuos about this. If you want kafka with no message loss so you need to :
- implement your own protection layer around Kafka
- monitor everything and everywhere.
Monitor is a beast to do operations fast. The SLAs are out of its possibility.
We find that there is a lot of unconsistency. The same concept is exposed more than one time in a different way at different places. Some descriptions are just impossible to understand. LinkedIn guys are good at creating a huge cloud of blogs , benchmarks and assertions that no one can check. They also assert that it fulfils all the three requirements of CAP theorem. And are completely confused where they explain this. It tries to be many things contemporary. No university department has made any research trying to classify and find its nature just because it is a bag containing many things. There is plenty of performance benchmark instead which are done in friendly environments where all the nodes, consumers, zookeeper , producers, sinks stay well and never fail. The benchmarks are done from Linkedin girls or are based on their assertions.