Part 5 - Persistence to Disk

Part 4 - Our First Tests (and Bugs)

Part 6 - The Cursor Abstraction

“Nothing in the world can take the place of persistence.” – Calvin Coolidge

Our database lets you insert records and read them back out, but only as long as you keep the program running. If you kill the program and start it back up, all your records are gone. Here’s a spec for the behavior we want:

  1. it 'keeps data after closing connection' do
  2.   result1 = run_script([
  3.     "insert 1 user1 person1@example.com",
  4.     ".exit",
  5.   ])
  6.   expect(result1).to match_array([
  7.     "db > Executed.",
  8.     "db > ",
  9.   ])
  10.   result2 = run_script([
  11.     "select",
  12.     ".exit",
  13.   ])
  14.   expect(result2).to match_array([
  15.     "db > (1, user1, person1@example.com)",
  16.     "Executed.",
  17.     "db > ",
  18.   ])
  19. end

Like sqlite, we’re going to persist records by saving the entire database to a file.

We already set ourselves up to do that by serializing rows into page-sized memory blocks. To add persistence, we can simply write those blocks of memory to a file, and read them back into memory the next time the program starts up.

To make this easier, we’re going to make an abstraction called the pager. We ask the pager for page number x, and the pager gives us back a block of memory. It first looks in its cache. On a cache miss, it copies data from disk into memory (by reading the database file).

|How our program matches up with SQLite architecture

The Pager accesses the page cache and the file. The Table object makes requests for pages through the pager:

  1. +struct Pager_t {
  2. +  int file_descriptor;
  3. +  uint32_t file_length;
  4. +  void* pages[TABLE_MAX_PAGES];
  5. +};
  6. +typedef struct Pager_t Pager;
  7. +
  8.  struct Table_t {
  9. -  void* pages[TABLE_MAX_PAGES];
  10. +  Pager* pager;
  11.    uint32_t num_rows;
  12.  };

I’m renaming new_table() to db_open() because it now has the effect of opening a connection to the database. By opening a connection, I mean:

  • opening the database file
  • initializing a pager data structure
  • initializing a table data structure
  1. -Table* new_table() {
  2. +Table* db_open(const char* filename) {
  3. +  Pager* pager = pager_open(filename);
  4. +  uint32_t num_rows = pager->file_length / ROW_SIZE;
  5. +
  6.    Table* table = malloc(sizeof(Table));
  7. -  table->num_rows = 0;
  8. +  table->pager = pager;
  9. +  table->num_rows = num_rows;
  11.    return table;
  12.  }

db_open() in turn calls pager_open(), which opens the database file and keeps track of its size. It also initializes the page cache to all NULLs.

  1. +Pager* pager_open(const char* filename) {
  2. +  int fd = open(filename,
  3. +                O_RDWR |      // Read/Write mode
  4. +                    O_CREAT,  // Create file if it does not exist
  5. +                S_IWUSR |     // User write permission
  6. +                    S_IRUSR   // User read permission
  7. +                );
  8. +
  9. +  if (fd == -1) {
  10. +    printf("Unable to open file\n");
  11. +    exit(EXIT_FAILURE);
  12. +  }
  13. +
  14. +  off_t file_length = lseek(fd, 0, SEEK_END);
  15. +
  16. +  Pager* pager = malloc(sizeof(Pager));
  17. +  pager->file_descriptor = fd;
  18. +  pager->file_length = file_length;
  19. +
  20. +  for (uint32_t i = 0; i < TABLE_MAX_PAGES; i++) {
  21. +    pager->pages[i] = NULL;
  22. +  }
  23. +
  24. +  return pager;
  25. +}

Following our new abstraction, we move the logic for fetching a page into its own method:

  1.  void* row_slot(Table* table, uint32_t row_num) {
  2.    uint32_t page_num = row_num / ROWS_PER_PAGE;
  3. -  void* page = table->pages[page_num];
  4. -  if (!page) {
  5. -    // Allocate memory only when we try to access page
  6. -    page = table->pages[page_num] = malloc(PAGE_SIZE);
  7. -  }
  8. +  void* page = get_page(table->pager, page_num);
  9.    uint32_t row_offset = row_num % ROWS_PER_PAGE;
  10.    uint32_t byte_offset = row_offset * ROW_SIZE;
  11.    return page + byte_offset;
  12.  }

The get_page() method has the logic for handling a cache miss. We assume pages are saved one after the other in the database file: Page 0 at offset 0, page 1 at offset 4096, page 2 at offset 8192, etc. If the requested page lies outside the bounds of the file, we know it should be blank, so we just allocate some memory and return it. The page will be added to the file when we flush the cache to disk later.

  1. +void* get_page(Pager* pager, uint32_t page_num) {
  2. +  if (page_num > TABLE_MAX_PAGES) {
  3. +    printf("Tried to fetch page number out of bounds. %d > %d\n", page_num,
  4. +           TABLE_MAX_PAGES);
  5. +    exit(EXIT_FAILURE);
  6. +  }
  7. +
  8. +  if (pager->pages[page_num] == NULL) {
  9. +    // Cache miss. Allocate memory and load from file.
  10. +    void* page = malloc(PAGE_SIZE);
  11. +    uint32_t num_pages = pager->file_length / PAGE_SIZE;
  12. +
  13. +    // We might save a partial page at the end of the file
  14. +    if (pager->file_length % PAGE_SIZE) {
  15. +      num_pages += 1;
  16. +    }
  17. +
  18. +    if (page_num <= num_pages) {
  19. +      lseek(pager->file_descriptor, page_num * PAGE_SIZE, SEEK_SET);
  20. +      ssize_t bytes_read = read(pager->file_descriptor, page, PAGE_SIZE);
  21. +      if (bytes_read == -1) {
  22. +        printf("Error reading file: %d\n", errno);
  23. +        exit(EXIT_FAILURE);
  24. +      }
  25. +    }
  26. +
  27. +    pager->pages[page_num] = page;
  28. +  }
  29. +
  30. +  return pager->pages[page_num];
  31. +}

For now, we’ll wait to flush the cache to disk until the user closes the connection to the database. When the user exits, we’ll call a new method called db_close(), which

  • flushes the page cache to disk
  • closes the database file
  • frees the memory for the Pager and Table data structures
  1. +void db_close(Table* table) {
  2. +  Pager* pager = table->pager;
  3. +  uint32_t num_full_pages = table->num_rows / ROWS_PER_PAGE;
  4. +
  5. +  for (uint32_t i = 0; i < num_full_pages; i++) {
  6. +    if (pager->pages[i] == NULL) {
  7. +      continue;
  8. +    }
  9. +    pager_flush(pager, i, PAGE_SIZE);
  10. +    free(pager->pages[i]);
  11. +    pager->pages[i] = NULL;
  12. +  }
  13. +
  14. +  // There may be a partial page to write to the end of the file
  15. +  // This should not be needed after we switch to a B-tree
  16. +  uint32_t num_additional_rows = table->num_rows % ROWS_PER_PAGE;
  17. +  if (num_additional_rows > 0) {
  18. +    uint32_t page_num = num_full_pages;
  19. +    if (pager->pages[page_num] != NULL) {
  20. +      pager_flush(pager, page_num, num_additional_rows * ROW_SIZE);
  21. +      free(pager->pages[page_num]);
  22. +      pager->pages[page_num] = NULL;
  23. +    }
  24. +  }
  25. +
  26. +  int result = close(pager->file_descriptor);
  27. +  if (result == -1) {
  28. +    printf("Error closing db file.\n");
  29. +    exit(EXIT_FAILURE);
  30. +  }
  31. +  for (uint32_t i = 0; i < TABLE_MAX_PAGES; i++) {
  32. +    void* page = pager->pages[i];
  33. +    if (page) {
  34. +      free(page);
  35. +      pager->pages[i] = NULL;
  36. +    }
  37. +  }
  38. +  free(pager);
  39. +}
  40. +
  41. -MetaCommandResult do_meta_command(InputBuffer* input_buffer) {
  42. +MetaCommandResult do_meta_command(InputBuffer* input_buffer, Table* table) {
  43.    if (strcmp(input_buffer->buffer, ".exit") == 0) {
  44. +    db_close(table);
  45.      exit(EXIT_SUCCESS);
  46.    } else {
  47.      return META_COMMAND_UNRECOGNIZED_COMMAND;

In our current design, the length of the file encodes how many rows are in the database, so we need to write a partial page at the end of the file. That’s why pager_flush() takes both a page number and a size. It’s not the greatest design, but it will go away pretty quickly when we start implementing the B-tree.

  1. +void pager_flush(Pager* pager, uint32_t page_num, uint32_t size) {
  2. +  if (pager->pages[page_num] == NULL) {
  3. +    printf("Tried to flush null page\n");
  4. +    exit(EXIT_FAILURE);
  5. +  }
  6. +
  7. +  off_t offset = lseek(pager->file_descriptor, page_num * PAGE_SIZE, SEEK_SET);
  8. +
  9. +  if (offset == -1) {
  10. +    printf("Error seeking: %d\n", errno);
  11. +    exit(EXIT_FAILURE);
  12. +  }
  13. +
  14. +  ssize_t bytes_written =
  15. +      write(pager->file_descriptor, pager->pages[page_num], size);
  16. +
  17. +  if (bytes_written == -1) {
  18. +    printf("Error writing: %d\n", errno);
  19. +    exit(EXIT_FAILURE);
  20. +  }
  21. +}

Lastly, we need to accept the filename as a command-line argument. Don’t forget to also add the extra argument to do_meta_command:

  1.  int main(int argc, char* argv[]) {
  2. -  Table* table = new_table();
  3. +  if (argc < 2) {
  4. +    printf("Must supply a database filename.\n");
  5. +    exit(EXIT_FAILURE);
  6. +  }
  7. +
  8. +  char* filename = argv[1];
  9. +  Table* table = db_open(filename);
  10. +
  11.    InputBuffer* input_buffer = new_input_buffer();
  12.    while (true) {
  13.      print_prompt();
  14.      read_input(input_buffer);
  16.      if (input_buffer->buffer[0] == '.') {
  17. -      switch (do_meta_command(input_buffer)) {
  18. +      switch (do_meta_command(input_buffer, table)) {

With these changes, we’re able to close then reopen the database, and our records are still there!

  1. ~ ./db mydb.db
  2. db > insert 1 cstack foo@bar.com
  3. Executed.
  4. db > insert 2 voltorb volty@example.com
  5. Executed.
  6. db > .exit
  7. ~
  8. ~ ./db mydb.db
  9. db > select
  10. (1, cstack, foo@bar.com)
  11. (2, voltorb, volty@example.com)
  12. Executed.
  13. db > .exit
  14. ~

For extra fun, let’s take a look at mydb.db to see how our data is being stored. I’ll use vim as a hex editor to look at the memory layout of the file:

  1. vim mydb.db
  2. :%!xxd

|Current File Format

The first four bytes are the id of the first row (4 bytes because we store a uint32_t). It’s stored in little-endian byte order, so the least significant byte comes first (01), followed by the higher-order bytes (00 00 00). We used memcpy() to copy bytes from our Row struct into the page cache, so that means the struct was laid out in memory in little-endian byte order. That’s an attribute of the machine I compiled the program for. If we wanted to write a database file on my machine, then read it on a big-endian machine, we’d have to change our serialize_row() and deserialize_row() methods to always store and read bytes in the same order.

The next 33 bytes store the username as a null-terminated string. Apparently “cstack” in ASCII hexadecimal is 63 73 74 61 63 6b, followed by a null character (00). The rest of the 33 bytes are unused.

The next 256 bytes store the email in the same way. Here we can see some random junk after the terminating null character. This is most likely due to uninitialized memory in our Row struct. We copy the entire 256-byte email buffer into the file, including any bytes after the end of the string. Whatever was in memory when we allocated that struct is still there. But since we use a terminating null character, it has no effect on behavior.

Conclusion

Alright! We’ve got persistence. It’s not the greatest. For example if you kill the program without typing .exit, you lose your changes. Additionally, we’re writing all pages back to disk, even pages that haven’t changed since we read them from disk. These are issues we can address later.

Next time we’ll introduce cursors, which should make it easier to implement the B-tree.

Until then!

Complete Diff

  1. +#include <errno.h>
  2. +#include <fcntl.h>
  3.  #include <stdbool.h>
  4.  #include <stdio.h>
  5.  #include <stdlib.h>
  6.  #include <string.h>
  7. +#include <unistd.h>
  8.  
  9.  struct InputBuffer_t {
  10.    char* buffer;
  11. @@ -61,8 +64,15 @@ const uint32_t TABLE_MAX_PAGES = 100;
  12.  const uint32_t ROWS_PER_PAGE = PAGE_SIZE / ROW_SIZE;
  13.  const uint32_t TABLE_MAX_ROWS = ROWS_PER_PAGE * TABLE_MAX_PAGES;
  15. -struct Table_t {
  16. +struct Pager_t {
  17. +  int file_descriptor;
  18. +  uint32_t file_length;
  19.    void* pages[TABLE_MAX_PAGES];
  20. +};
  21. +typedef struct Pager_t Pager;
  22. +
  23. +struct Table_t {
  24. +  Pager* pager;
  25.    uint32_t num_rows;
  26.  };
  27.  typedef struct Table_t Table;
  28. @@ -83,21 +93,79 @@ void deserialize_row(void* source, Row* destination) {
  29.    memcpy(&(destination->email), source + EMAIL_OFFSET, EMAIL_SIZE);
  30.  }
  32. +void* get_page(Pager* pager, uint32_t page_num) {
  33. +  if (page_num > TABLE_MAX_PAGES) {
  34. +    printf("Tried to fetch page number out of bounds. %d > %d\n", page_num,
  35. +           TABLE_MAX_PAGES);
  36. +    exit(EXIT_FAILURE);
  37. +  }
  38. +
  39. +  if (pager->pages[page_num] == NULL) {
  40. +    // Cache miss. Allocate memory and load from file.
  41. +    void* page = malloc(PAGE_SIZE);
  42. +    uint32_t num_pages = pager->file_length / PAGE_SIZE;
  43. +
  44. +    // We might save a partial page at the end of the file
  45. +    if (pager->file_length % PAGE_SIZE) {
  46. +      num_pages += 1;
  47. +    }
  48. +
  49. +    if (page_num <= num_pages) {
  50. +      lseek(pager->file_descriptor, page_num * PAGE_SIZE, SEEK_SET);
  51. +      ssize_t bytes_read = read(pager->file_descriptor, page, PAGE_SIZE);
  52. +      if (bytes_read == -1) {
  53. +        printf("Error reading file: %d\n", errno);
  54. +        exit(EXIT_FAILURE);
  55. +      }
  56. +    }
  57. +
  58. +    pager->pages[page_num] = page;
  59. +  }
  60. +
  61. +  return pager->pages[page_num];
  62. +}
  63. +
  64.  void* row_slot(Table* table, uint32_t row_num) {
  65.    uint32_t page_num = row_num / ROWS_PER_PAGE;
  66. -  void* page = table->pages[page_num];
  67. -  if (!page) {
  68. -    // Allocate memory only when we try to access page
  69. -    page = table->pages[page_num] = malloc(PAGE_SIZE);
  70. -  }
  71. +  void* page = get_page(table->pager, page_num);
  72.    uint32_t row_offset = row_num % ROWS_PER_PAGE;
  73.    uint32_t byte_offset = row_offset * ROW_SIZE;
  74.    return page + byte_offset;
  75.  }
  77. -Table* new_table() {
  78. +Pager* pager_open(const char* filename) {
  79. +  int fd = open(filename,
  80. +                O_RDWR |      // Read/Write mode
  81. +                    O_CREAT,  // Create file if it does not exist
  82. +                S_IWUSR |     // User write permission
  83. +                    S_IRUSR   // User read permission
  84. +                );
  85. +
  86. +  if (fd == -1) {
  87. +    printf("Unable to open file\n");
  88. +    exit(EXIT_FAILURE);
  89. +  }
  90. +
  91. +  off_t file_length = lseek(fd, 0, SEEK_END);
  92. +
  93. +  Pager* pager = malloc(sizeof(Pager));
  94. +  pager->file_descriptor = fd;
  95. +  pager->file_length = file_length;
  96. +
  97. +  for (uint32_t i = 0; i < TABLE_MAX_PAGES; i++) {
  98. +    pager->pages[i] = NULL;
  99. +  }
  100. +
  101. +  return pager;
  102. +}
  103. +
  104. +Table* db_open(const char* filename) {
  105. +  Pager* pager = pager_open(filename);
  106. +  uint32_t num_rows = pager->file_length / ROW_SIZE;
  107. +
  108.    Table* table = malloc(sizeof(Table));
  109. -  table->num_rows = 0;
  110. +  table->pager = pager;
  111. +  table->num_rows = num_rows;
  112.  
  113.    return table;
  114.  }
  115. @@ -127,8 +195,71 @@ void read_input(InputBuffer* input_buffer) {
  116.    input_buffer->buffer[bytes_read - 1] = 0;
  117.  }
  119. -MetaCommandResult do_meta_command(InputBuffer* input_buffer) {
  120. +void pager_flush(Pager* pager, uint32_t page_num, uint32_t size) {
  121. +  if (pager->pages[page_num] == NULL) {
  122. +    printf("Tried to flush null page\n");
  123. +    exit(EXIT_FAILURE);
  124. +  }
  125. +
  126. +  off_t offset = lseek(pager->file_descriptor, page_num * PAGE_SIZE, SEEK_SET);
  127. +
  128. +  if (offset == -1) {
  129. +    printf("Error seeking: %d\n", errno);
  130. +    exit(EXIT_FAILURE);
  131. +  }
  132. +
  133. +  ssize_t bytes_written =
  134. +      write(pager->file_descriptor, pager->pages[page_num], size);
  135. +
  136. +  if (bytes_written == -1) {
  137. +    printf("Error writing: %d\n", errno);
  138. +    exit(EXIT_FAILURE);
  139. +  }
  140. +}
  141. +
  142. +void db_close(Table* table) {
  143. +  Pager* pager = table->pager;
  144. +  uint32_t num_full_pages = table->num_rows / ROWS_PER_PAGE;
  145. +
  146. +  for (uint32_t i = 0; i < num_full_pages; i++) {
  147. +    if (pager->pages[i] == NULL) {
  148. +      continue;
  149. +    }
  150. +    pager_flush(pager, i, PAGE_SIZE);
  151. +    free(pager->pages[i]);
  152. +    pager->pages[i] = NULL;
  153. +  }
  154. +
  155. +  // There may be a partial page to write to the end of the file
  156. +  // This should not be needed after we switch to a B-tree
  157. +  uint32_t num_additional_rows = table->num_rows % ROWS_PER_PAGE;
  158. +  if (num_additional_rows > 0) {
  159. +    uint32_t page_num = num_full_pages;
  160. +    if (pager->pages[page_num] != NULL) {
  161. +      pager_flush(pager, page_num, num_additional_rows * ROW_SIZE);
  162. +      free(pager->pages[page_num]);
  163. +      pager->pages[page_num] = NULL;
  164. +    }
  165. +  }
  166. +
  167. +  int result = close(pager->file_descriptor);
  168. +  if (result == -1) {
  169. +    printf("Error closing db file.\n");
  170. +    exit(EXIT_FAILURE);
  171. +  }
  172. +  for (uint32_t i = 0; i < TABLE_MAX_PAGES; i++) {
  173. +    void* page = pager->pages[i];
  174. +    if (page) {
  175. +      free(page);
  176. +      pager->pages[i] = NULL;
  177. +    }
  178. +  }
  179. +  free(pager);
  180. +}
  181. +
  182. +MetaCommandResult do_meta_command(InputBuffer* input_buffer, Table* table) {
  183.    if (strcmp(input_buffer->buffer, ".exit") == 0) {
  184. +    db_close(table);
  185.      exit(EXIT_SUCCESS);
  186.    } else {
  187.      return META_COMMAND_UNRECOGNIZED_COMMAND;
  188. @@ -210,14 +341,21 @@ ExecuteResult execute_statement(Statement* statement, Table* table) {
  189.  }
  191.  int main(int argc, char* argv[]) {
  192. -  Table* table = new_table();
  193. +  if (argc < 2) {
  194. +    printf("Must supply a database filename.\n");
  195. +    exit(EXIT_FAILURE);
  196. +  }
  197. +
  198. +  char* filename = argv[1];
  199. +  Table* table = db_open(filename);
  200. +
  201.    InputBuffer* input_buffer = new_input_buffer();
  202.    while (true) {
  203.      print_prompt();
  204.      read_input(input_buffer);
  206.      if (input_buffer->buffer[0] == '.') {
  207. -      switch (do_meta_command(input_buffer)) {
  208. +      switch (do_meta_command(input_buffer, table)) {
  209.          case (META_COMMAND_SUCCESS):
  210.            continue;
  211.          case (META_COMMAND_UNRECOGNIZED_COMMAND):
  212. diff --git a/spec/main_spec.rb b/spec/main_spec.rb
  213. index 21561ce..bc0180a 100644
  214. --- a/spec/main_spec.rb
  215. +++ b/spec/main_spec.rb
  216. @@ -1,7 +1,11 @@
  217.  describe 'database' do
  218. +  before do
  219. +    `rm -rf test.db`
  220. +  end
  221. +
  222.    def run_script(commands)
  223.      raw_output = nil
  224. -    IO.popen("./db", "r+") do |pipe|
  225. +    IO.popen("./db test.db", "r+") do |pipe|
  226.        commands.each do |command|
  227.          pipe.puts command
  228.        end
  229. @@ -28,6 +32,27 @@ describe 'database' do
  230.      ])
  231.    end
  233. +  it 'keeps data after closing connection' do
  234. +    result1 = run_script([
  235. +      "insert 1 user1 person1@example.com",
  236. +      ".exit",
  237. +    ])
  238. +    expect(result1).to match_array([
  239. +      "db > Executed.",
  240. +      "db > ",
  241. +    ])
  242. +
  243. +    result2 = run_script([
  244. +      "select",
  245. +      ".exit",
  246. +    ])
  247. +    expect(result2).to match_array([
  248. +      "db > (1, user1, person1@example.com)",
  249. +      "Executed.",
  250. +      "db > ",
  251. +    ])
  252. +  end
  253. +
  254.    it 'prints error message when table is full' do
  255.      script = (1..1401).map do |i|
  256.        "insert #{i} user#{i} person#{i}@example.com"

And the diff to our tests:

  1.  describe 'database' do
  2. +  before do
  3. +    `rm -rf test.db`
  4. +  end
  5. +
  6.    def run_script(commands)
  7.      raw_output = nil
  8. -    IO.popen("./db", "r+") do |pipe|
  9. +    IO.popen("./db test.db", "r+") do |pipe|
  10.        commands.each do |command|
  11.          pipe.puts command
  12.        end
  13. @@ -28,6 +32,27 @@ describe 'database' do
  14.      ])
  15.    end
  17. +  it 'keeps data after closing connection' do
  18. +    result1 = run_script([
  19. +      "insert 1 user1 person1@example.com",
  20. +      ".exit",
  21. +    ])
  22. +    expect(result1).to match_array([
  23. +      "db > Executed.",
  24. +      "db > ",
  25. +    ])
  26. +
  27. +    result2 = run_script([
  28. +      "select",
  29. +      ".exit",
  30. +    ])
  31. +    expect(result2).to match_array([
  32. +      "db > (1, user1, person1@example.com)",
  33. +      "Executed.",
  34. +      "db > ",
  35. +    ])
  36. +  end
  37. +
  38.    it 'prints error message when table is full' do
  39.      script = (1..1401).map do |i|
  40.        "insert #{i} user#{i} person#{i}@example.com"

Part 4 - Our First Tests (and Bugs)

Part 6 - The Cursor Abstraction

原文: https://cstack.github.io/db_tutorial/parts/part5.html