Comments on: Building a Distributed Log from Scratch, Part 1: Storage Mechanics

By: iwa2no

iwa2no — Thu, 29 Aug 2024 12:38:19 +0000

Hi, I want to ask why access ordered log in large file is not succifient. AFAIK, if we want to find a log in 10^6 logs, we just need log2(10^6) which approximates 20 times, which I believe is a small number. Have I oversimplified anything ?

By: Dio

Dio — Wed, 06 Apr 2022 23:36:28 +0000

Thanks for this wonderful series of posts! I have some question on the performance of WAL. Apparently Kafka’s strength lies in the fact that the log is append only, therefore there is no disk seek hence being super fast.

However, my question is:
1. AFAIK, two files are written for a single , one is for the actual messages, the other is an index.
2. Besides, each broker server (one machine) can hosts many topic + partition, and IIRC Kafka keeps the logs for different separated.
#1 combined with #2, I don’t see how it optimizes for disk access, because on a single machine we are still writing to multiple files which now needs disk seek.

Any insight? Thanks!

By: Rohan

Rohan — Mon, 20 Aug 2018 21:22:20 +0000

Understood it myself on thinking more…:)
Doing a binary search that refers an index for it’s jumps would still involve logN seeks in the file which’d be expensive.
Hence we can’t use that to just “find the last message older than”

Thanks Tyler!

By: Rohan

Rohan — Sun, 05 Aug 2018 20:00:14 +0000

Hi Tyler,
Thanks for the wonderful series.

I have a question regarding the idea of splitting the log into segments:
Even if we had one single long log file and an index file accompanying it, couldn’t we refer the index file for an offset, that’d give us the byte position and seek to that position?

By: Ahmed

Ahmed — Mon, 25 Dec 2017 10:53:22 +0000

Great

By: Philip Enrico

Philip Enrico — Fri, 22 Dec 2017 18:21:27 +0000

Any free online course so i can learn more deeply