本文分为三节，分别介绍clog的fsync频率，原子操作，与异步提交一致性。

PostgreSQL pg_clog fsync 频率分析

分析一下pg_clog是在什么时候需要调用fsync的？

首先引用wiki里的一段pg_clog的介绍

Some details here are in src/backend/access/transam/README:

  1. “pg_clog records the commit status for each transaction that has been assigned an XID.”
  2. “Transactions and subtransactions are assigned permanent XIDs only when/if they first do something that requires one — typically, insert/update/delete a tuple, though there are a few other places that need an XID assigned.”

pg_clog is updated only at sub or main transaction end. When the transactionid is assigned the page of the clog that contains that transactionid is checked to see if it already exists and if not, it is initialised. pg_clog is allocated in pages of 8kB apiece(和BLOCKSZ一致，所以不一定是8K，见后面的分析). Each transaction needs 2 bits, so on an 8 kB page there is space for 4 transactions/byte * 8k bytes = 32k transactions. On allocation, pages are zeroed, which is the bit pattern for “transaction in progress”. So when a transaction starts, it only needs to ensure that the pg_clog page that contains its status is allocated, but it need not write anything to it. In 8.3 and later, this happens not when the transaction starts, but when the Xid is assigned (i.e. when the transaction first calls a read-write command). In previous versions it happens when the first snapshot is taken, normally on the first command of any type with very few exceptions.

This means that one transaction in every 32K writing transactions does have to do extra work when it assigns itself an XID, namely create and zero out the next page of pg_clog. And that doesn’t just slow down the transaction in question, but the next few guys that would like an XID but arrive on the scene while the zeroing-out is still in progress. This probably contributes to reported behavior that the transaction execution time is subject to unpredictable spikes.

每隔32K个事务，要扩展一个CLOG PAGE，每次扩展需要填充0，同时需要调用PG_FSYNC，这个相比FSYNC XLOG应该是比较轻量级的。但是也可能出现不可预知的响应延迟，因为如果堵塞在扩展CLOG PAGE，所有等待clog PAGE的会话都会受到影响。

这里指当CLOG buffer没有空的SLOT时，会从所有的CLOG buffer SLOT选择一个脏页，将其刷出，这个时候才会产生pg_fsync。 CLOG pages don’t make their way out to disk until the internal CLOG buffers are filled, at which point the least recently used buffer there is evicted to permanent storage.

下面从代码中分析一下pg_clog是如何调用pg_fsync刷脏页的。

每次申请新的事务ID时，都需要调用ExtendCLOG，如果通过事务ID计算得到的CLOG PAGE页不存在，则需要扩展；但是并不是每次扩展都需要调用pg_fsync，因为checkpoint会将clog buffer刷到磁盘，除非在申请新的CLOG PAGE时所有的clog buffer都没有刷出脏页，才需要主动选择一个page并调用pg_fsync刷出对应的pg_clog/file。 src/backend/access/transam/varsup.c

  1. /*
  2.  * Allocate the next XID for a new transaction or subtransaction.
  3.  *
  4.  * The new XID is also stored into MyPgXact before returning.
  5.  *
  6.  * Note: when this is called, we are actually already inside a valid
  7.  * transaction, since XIDs are now not allocated until the transaction
  8.  * does something.  So it is safe to do a database lookup if we want to
  9.  * issue a warning about XID wrap.
  10.  */
  11. TransactionId
  12. GetNewTransactionId(bool isSubXact)
  13. {
  14. ......
  15.         /*
  16.          * If we are allocating the first XID of a new page of the commit log,
  17.          * zero out that commit-log page before returning. We must do this while
  18.          * holding XidGenLock, else another xact could acquire and commit a later
  19.          * XID before we zero the page.  Fortunately, a page of the commit log
  20.          * holds 32K or more transactions, so we don't have to do this very often.
  21.          *
  22.          * Extend pg_subtrans too.
  23.          */
  24.         ExtendCLOG(xid);
  25.         ExtendSUBTRANS(xid);
  26. ......

ExtendCLOG(xid)扩展clog page，调用TransactionIdToPgIndex计算XID和CLOG_XACTS_PER_PAGE的余数，如果不为0，则不需要扩展。 src/backend/access/transam/clog.c

  1. #define TransactionIdToPgIndex(xid) ((xid) % (TransactionId) CLOG_XACTS_PER_PAGE)
  3. /*
  4.  * Make sure that CLOG has room for a newly-allocated XID.
  5.  *
  6.  * NB: this is called while holding XidGenLock.  We want it to be very fast
  7.  * most of the time; even when it's not so fast, no actual I/O need happen
  8.  * unless we're forced to write out a dirty clog or xlog page to make room
  9.  * in shared memory.
  10.  */
  11. void
  12. ExtendCLOG(TransactionId newestXact)
  13. {
  14.         int                     pageno;
  16.         /*
  17.          * No work except at first XID of a page.  But beware: just after
  18.          * wraparound, the first XID of page zero is FirstNormalTransactionId.
  19.          */
  20.         if (TransactionIdToPgIndex(newestXact) != 0 &&    // 余数不为0，说明不需要扩展。
  21.                 !TransactionIdEquals(newestXact, FirstNormalTransactionId))
  22.                 return;
  24.         pageno = TransactionIdToPage(newestXact);
  26.         LWLockAcquire(CLogControlLock, LW_EXCLUSIVE);
  28.         /* Zero the page and make an XLOG entry about it */
  29.         ZeroCLOGPage(pageno, true);
  31.         LWLockRelease(CLogControlLock);
  32. }

ZeroCLOGPage(pageno, true)，调用SimpleLruZeroPage，扩展并初始化CLOG PAGE，写XLOG日志。

  1. /*
  2.  * Initialize (or reinitialize) a page of CLOG to zeroes.
  3.  * If writeXlog is TRUE, also emit an XLOG record saying we did this.
  4.  *
  5.  * The page is not actually written, just set up in shared memory.
  6.  * The slot number of the new page is returned.
  7.  *
  8.  * Control lock must be held at entry, and will be held at exit.
  9.  */
  10. static int
  11. ZeroCLOGPage(int pageno, bool writeXlog)
  12. {
  13.         int                     slotno;
  15.         slotno = SimpleLruZeroPage(ClogCtl, pageno);
  17.         if (writeXlog)
  18.                 WriteZeroPageXlogRec(pageno);
  20.         return slotno;
  21. }

SimpleLruZeroPage(ClogCtl, pageno)，调用SlruSelectLRUPage(ctl, pageno)，从clog shared buffer中选择SLOT。 src/backend/access/transam/slru.c

  1. /*
  2.  * Initialize (or reinitialize) a page to zeroes.
  3.  *
  4.  * The page is not actually written, just set up in shared memory.
  5.  * The slot number of the new page is returned.
  6.  *
  7.  * Control lock must be held at entry, and will be held at exit.
  8.  */
  9. int
  10. SimpleLruZeroPage(SlruCtl ctl, int pageno)
  11. {
  12.         SlruShared      shared = ctl->shared;
  13.         int                     slotno;
  15.         /* Find a suitable buffer slot for the page */
  16.         slotno = SlruSelectLRUPage(ctl, pageno);
  17.         Assert(shared->page_status[slotno] == SLRU_PAGE_EMPTY ||
  18.                    (shared->page_status[slotno] == SLRU_PAGE_VALID &&
  19.                         !shared->page_dirty[slotno]) ||
  20.                    shared->page_number[slotno] == pageno);
  22.         /* Mark the slot as containing this page */
  23.         shared->page_number[slotno] = pageno;
  24.         shared->page_status[slotno] = SLRU_PAGE_VALID;
  25.         shared->page_dirty[slotno] = true;
  26.         SlruRecentlyUsed(shared, slotno);
  28.         /* Set the buffer to zeroes */
  29.         MemSet(shared->page_buffer[slotno], 0, BLCKSZ);
  31.         /* Set the LSNs for this new page to zero */
  32.         SimpleLruZeroLSNs(ctl, slotno);
  34.         /* Assume this page is now the latest active page */
  35.         shared->latest_page_number = pageno;
  37.         return slotno;
  38. }

SlruSelectLRUPage(SlruCtl ctl, int pageno)，从clog buffer选择一个空的SLOT，如果没有空的SLOT，则需要调用SlruInternalWritePage(ctl, bestvalidslot, NULL)，写shared buffer page。

  1. /*
  2.  * Select the slot to re-use when we need a free slot.
  3.  *
  4.  * The target page number is passed because we need to consider the
  5.  * possibility that some other process reads in the target page while
  6.  * we are doing I/O to free a slot.  Hence, check or recheck to see if
  7.  * any slot already holds the target page, and return that slot if so.
  8.  * Thus, the returned slot is *either* a slot already holding the pageno
  9.  * (could be any state except EMPTY), *or* a freeable slot (state EMPTY
  10.  * or CLEAN).
  11.  *
  12.  * Control lock must be held at entry, and will be held at exit.
  13.  */
  14. static int
  15. SlruSelectLRUPage(SlruCtl ctl, int pageno)
  16. {
  17. ......
  18.         /* See if page already has a buffer assigned */  先查看clog buffer中是否有空SLOT，有则返回，不需要调pg_fsync
  19.         for (slotno = 0; slotno < shared->num_slots; slotno++)
  20.         {
  21.             if (shared->page_number[slotno] == pageno &&
  22.                 shared->page_status[slotno] != SLRU_PAGE_EMPTY)
  23.                 return slotno;
  24.         }
  25. ...... 
  26.         /*  如果没有找到空SLOT，则需要从clog buffer中选择一个使用最少的PAGE，注意他不会选择最近临近的PAGE，优先选择IO不繁忙的PAGE
  27.          * If we find any EMPTY slot, just select that one. Else choose a
  28.          * victim page to replace.  We normally take the least recently used
  29.          * valid page, but we will never take the slot containing
  30.          * latest_page_number, even if it appears least recently used.  We
  31.          * will select a slot that is already I/O busy only if there is no
  32.          * other choice: a read-busy slot will not be least recently used once
  33.          * the read finishes, and waiting for an I/O on a write-busy slot is
  34.          * inferior to just picking some other slot.  Testing shows the slot
  35.          * we pick instead will often be clean, allowing us to begin a read at
  36.          * once.
  37.          *  
  38.          * Normally the page_lru_count values will all be different and so
  39.          * there will be a well-defined LRU page.  But since we allow
  40.          * concurrent execution of SlruRecentlyUsed() within
  41.          * SimpleLruReadPage_ReadOnly(), it is possible that multiple pages
  42.          * acquire the same lru_count values.  In that case we break ties by
  43.          * choosing the furthest-back page.
  44.          *
  45.          * Notice that this next line forcibly advances cur_lru_count to a
  46.          * value that is certainly beyond any value that will be in the
  47.          * page_lru_count array after the loop finishes.  This ensures that
  48.          * the next execution of SlruRecentlyUsed will mark the page newly
  49.          * used, even if it's for a page that has the current counter value.
  50.          * That gets us back on the path to having good data when there are
  51.          * multiple pages with the same lru_count.
  52.          */
  53.         cur_count = (shared->cur_lru_count)++;
  54.         for (slotno = 0; slotno < shared->num_slots; slotno++)
  55.         {
  56.             int            this_delta;
  57.             int            this_page_number;
  59.             if (shared->page_status[slotno] == SLRU_PAGE_EMPTY)  // 如果在此期间出现了空SLOT，返回这个slotno
  60.                 return slotno;
  61.             this_delta = cur_count - shared->page_lru_count[slotno];
  62.             if (this_delta < 0)
  63.             {
  64.                 /*
  65.                  * Clean up in case shared updates have caused cur_count
  66.                  * increments to get "lost".  We back off the page counts,
  67.                  * rather than trying to increase cur_count, to avoid any
  68.                  * question of infinite loops or failure in the presence of
  69.                  * wrapped-around counts.
  70.                  */
  71.                 shared->page_lru_count[slotno] = cur_count;
  72.                 this_delta = 0;
  73.             }
  74.             this_page_number = shared->page_number[slotno];
  75.             if (this_page_number == shared->latest_page_number)
  76.                 continue;
  77.             if (shared->page_status[slotno] == SLRU_PAGE_VALID)  // IO不繁忙的脏页
  78.             {
  79.                 if (this_delta > best_valid_delta ||
  80.                     (this_delta == best_valid_delta &&
  81.                      ctl->PagePrecedes(this_page_number,
  82.                                        best_valid_page_number)))
  83.                 {
  84.                     bestvalidslot = slotno;
  85.                     best_valid_delta = this_delta;
  86.                     best_valid_page_number = this_page_number;
  87.                 }
  88.             }
  89.             else
  90.             {
  91.                 if (this_delta > best_invalid_delta ||
  92.                     (this_delta == best_invalid_delta &&
  93.                      ctl->PagePrecedes(this_page_number,
  94.                                        best_invalid_page_number)))
  95.                 {
  96.                     bestinvalidslot = slotno;  // 当所有页面IO都繁忙时，无奈只能从IO繁忙中选择一个.
  97.                     best_invalid_delta = this_delta;
  98.                     best_invalid_page_number = this_page_number;
  99.                 }
  100.             }
  101.         }
  103.         /*  如果选择到的PAGE
  104.          * If all pages (except possibly the latest one) are I/O busy, we'll
  105.          * have to wait for an I/O to complete and then retry.  In that
  106.          * unhappy case, we choose to wait for the I/O on the least recently
  107.          * used slot, on the assumption that it was likely initiated first of
  108.          * all the I/Os in progress and may therefore finish first.
  109.          */
  110.         if (best_valid_delta < 0)  // 说明没有找到SLRU_PAGE_VALID的PAGE，所有PAGE都处于IO繁忙的状态。
  111.         {
  112.             SimpleLruWaitIO(ctl, bestinvalidslot);
  113.             continue;
  114.         }
  116.         /*
  117.          * If the selected page is clean, we're set.
  118.          */
  119.         if (!shared->page_dirty[bestvalidslot])  // 如果这个页面已经不是脏页（例如被CHECKPOINT刷出了），那么直接返回
  120.             return bestvalidslot;
  122. ......
  123. 仅仅当以上所有的步骤，都没有找到一个EMPTY SLOT时，才需要主动刷脏页（在SlruInternalWritePage调用pg_fsync)。
  124.                 /*
  125.                  * Write the page.  注意第三个参数为NULL，即fdata
  126.                  */
  127.                 SlruInternalWritePage(ctl, bestvalidslot, NULL);
  128. ......

SlruInternalWritePage(SlruCtl ctl, int slotno, SlruFlush fdata)，调用SlruPhysicalWritePage，执行write。

  1. /*
  2.  * Write a page from a shared buffer, if necessary.
  3.  * Does nothing if the specified slot is not dirty.
  4.  *
  5.  * NOTE: only one write attempt is made here.  Hence, it is possible that
  6.  * the page is still dirty at exit (if someone else re-dirtied it during
  7.  * the write).  However, we *do* attempt a fresh write even if the page
  8.  * is already being written; this is for checkpoints.
  9.  *
  10.  * Control lock must be held at entry, and will be held at exit.
  11.  */
  12. static void
  13. SlruInternalWritePage(SlruCtl ctl, int slotno, SlruFlush fdata)
  14. {
  15. ......
  16.         /* Do the write */
  17.         ok = SlruPhysicalWritePage(ctl, pageno, slotno, fdata);
  18. ......

SLRU PAGE状态

  1. /*
  2.  * Page status codes.  Note that these do not include the "dirty" bit.
  3.  * page_dirty can be TRUE only in the VALID or WRITE_IN_PROGRESS states;
  4.  * in the latter case it implies that the page has been re-dirtied since
  5.  * the write started.
  6.  */
  7. typedef enum
  8. {
  9.     SLRU_PAGE_EMPTY,            /* buffer is not in use */
  10.     SLRU_PAGE_READ_IN_PROGRESS, /* page is being read in */
  11.     SLRU_PAGE_VALID,            /* page is valid and not being written */
  12.     SLRU_PAGE_WRITE_IN_PROGRESS /* page is being written out */
  13. } SlruPageStatus;

SlruPhysicalWritePage(ctl, pageno, slotno, fdata)，这里涉及pg_clog相关的SlruCtlData结构，do_fsync=true。

  1. /*
  2.  * Physical write of a page from a buffer slot
  3.  *
  4.  * On failure, we cannot just ereport(ERROR) since caller has put state in
  5.  * shared memory that must be undone.  So, we return FALSE and save enough
  6.  * info in static variables to let SlruReportIOError make the report.
  7.  *
  8.  * For now, assume it's not worth keeping a file pointer open across
  9.  * independent read/write operations.  We do batch operations during
  10.  * SimpleLruFlush, though.
  11.  *
  12.  * fdata is NULL for a standalone write, pointer to open-file info during
  13.  * SimpleLruFlush.
  14.  */
  15. static bool
  16. SlruPhysicalWritePage(SlruCtl ctl, int pageno, int slotno,
  17.                                           SlruFlush fdata);
  18. ......
  19.         int                     fd = -1;
  20. ......
  21. //   如果文件不存在，自动创建  
  22.         if (fd < 0)
  23.         {
  24.                 /*
  25.                  * If the file doesn't already exist, we should create it.  It is
  26.                  * possible for this to need to happen when writing a page that's not
  27.                  * first in its segment; we assume the OS can cope with that. (Note:
  28.                  * it might seem that it'd be okay to create files only when
  29.                  * SimpleLruZeroPage is called for the first page of a segment.
  30.                  * However, if after a crash and restart the REDO logic elects to
  31.                  * replay the log from a checkpoint before the latest one, then it's
  32.                  * possible that we will get commands to set transaction status of
  33.                  * transactions that have already been truncated from the commit log.
  34.                  * Easiest way to deal with that is to accept references to
  35.                  * nonexistent files here and in SlruPhysicalReadPage.)
  36.                  *
  37.                  * Note: it is possible for more than one backend to be executing this
  38.                  * code simultaneously for different pages of the same file. Hence,
  39.                  * don't use O_EXCL or O_TRUNC or anything like that.
  40.                  */
  41.                 SlruFileName(ctl, path, segno);
  42.                 fd = OpenTransientFile(path, O_RDWR | O_CREAT | PG_BINARY,
  43.                                                            S_IRUSR | S_IWUSR);
  44. ......
  45.         /*
  46.          * If not part of Flush, need to fsync now.  We assume this happens
  47.          * infrequently enough that it's not a performance issue.
  48.          */
  49.         if (!fdata)  // 因为传入的fdata=NULL，并且ctl->do_fsync=true，所以以下pg_fsync被调用。
  50.         {
  51.                 if (ctl->do_fsync && pg_fsync(fd))  // 对于pg_clog和multixact，do_fsync=true。
  52.                 {
  53.                         slru_errcause = SLRU_FSYNC_FAILED;
  54.                         slru_errno = errno;
  55.                         CloseTransientFile(fd);
  56.                         return false;
  57.                 }
  59.                 if (CloseTransientFile(fd))
  60.                 {
  61.                         slru_errcause = SLRU_CLOSE_FAILED;
  62.                         slru_errno = errno;
  63.                         return false;
  64.                 }
  65.         }

ctl->do_fsync && pg_fsync(fd)涉及的代码： src/include/access/slru.h

  1. /*
  2.  * SlruCtlData is an unshared structure that points to the active information
  3.  * in shared memory.
  4.  */
  5. typedef struct SlruCtlData
  6. {
  7.         SlruShared      shared;
  9.         /*
  10.          * This flag tells whether to fsync writes (true for pg_clog and multixact
  11.          * stuff, false for pg_subtrans and pg_notify).
  12.          */
  13.         bool            do_fsync;
  15.         /*
  16.          * Decide which of two page numbers is "older" for truncation purposes. We
  17.          * need to use comparison of TransactionIds here in order to do the right
  18.          * thing with wraparound XID arithmetic.
  19.          */
  20.         bool            (*PagePrecedes) (int, int);
  22.         /*
  23.          * Dir is set during SimpleLruInit and does not change thereafter. Since
  24.          * it's always the same, it doesn't need to be in shared memory.
  25.          */
  26.         char            Dir[64];
  27. } SlruCtlData;
  28. typedef SlruCtlData *SlruCtl;

src/backend/access/transam/slru.c

  1. ......
  2. void
  3. SimpleLruInit(SlruCtl ctl, const char *name, int nslots, int nlsns,
  4.                           LWLock *ctllock, const char *subdir)
  5. ......
  6.         ctl->do_fsync = true;           /* default behavior */  // 初始化LRU时，do_fsync默认是true的。
  7. ......

以下是clog初始化LRU的调用，可以看到它没有修改do_fsync，所以是TURE。 src/backend/access/transam/clog.c

  1. /*
  2.  * Number of shared CLOG buffers.
  3.  *
  4.  * Testing during the PostgreSQL 9.2 development cycle revealed that on a
  5.  * large multi-processor system, it was possible to have more CLOG page
  6.  * requests in flight at one time than the number of CLOG buffers which existed
  7.  * at that time, which was hardcoded to 8.  Further testing revealed that
  8.  * performance dropped off with more than 32 CLOG buffers, possibly because
  9.  * the linear buffer search algorithm doesn't scale well.
  10.  *
  11.  * Unconditionally increasing the number of CLOG buffers to 32 did not seem
  12.  * like a good idea, because it would increase the minimum amount of shared
  13.  * memory required to start, which could be a problem for people running very
  14.  * small configurations.  The following formula seems to represent a reasonable
  15.  * compromise: people with very low values for shared_buffers will get fewer
  16.  * CLOG buffers as well, and everyone else will get 32.
  17.  *
  18.  * It is likely that some further work will be needed here in future releases;
  19.  * for example, on a 64-core server, the maximum number of CLOG requests that
  20.  * can be simultaneously in flight will be even larger.  But that will
  21.  * apparently require more than just changing the formula, so for now we take
  22.  * the easy way out.
  23.  */
  24. Size
  25. CLOGShmemBuffers(void)
  26. {
  27.         return Min(32, Max(4, NBuffers / 512));
  28. }
  30. void
  31. CLOGShmemInit(void)
  32. {
  33.         ClogCtl->PagePrecedes = CLOGPagePrecedes;
  34.         SimpleLruInit(ClogCtl, "CLOG Ctl", CLOGShmemBuffers(), CLOG_LSNS_PER_PAGE,
  35.                                   CLogControlLock, "pg_clog");
  36. }

以下是subtrans初始化LRU的调用，看到它修改了do_fsync=false。所以subtrans扩展PAGE时不需要调用pg_fsync。 src/backend/access/transam/subtrans.c

  1. void
  2. SUBTRANSShmemInit(void)
  3. {
  4.         SubTransCtl->PagePrecedes = SubTransPagePrecedes;
  5.         SimpleLruInit(SubTransCtl, "SUBTRANS Ctl", NUM_SUBTRANS_BUFFERS, 0,
  6.                                   SubtransControlLock, "pg_subtrans");
  7.         /* Override default assumption that writes should be fsync'd */
  8.         SubTransCtl->do_fsync = false;
  9. }

multixact.c也没有修改do_fsync，所以也是需要fsync的。 MultiXactShmemInit(void)@src/backend/access/transam/multixact.c

pg_fsync代码： src/backend/storage/file/fd.c

  1. /*
  2.  * pg_fsync --- do fsync with or without writethrough
  3.  */
  4. int
  5. pg_fsync(int fd)
  6. {
  7.         /* #if is to skip the sync_method test if there's no need for it */
  8. #if defined(HAVE_FSYNC_WRITETHROUGH) && !defined(FSYNC_WRITETHROUGH_IS_FSYNC)
  9.         if (sync_method == SYNC_METHOD_FSYNC_WRITETHROUGH)
  10.                 return pg_fsync_writethrough(fd);
  11.         else
  12. #endif
  13.                 return pg_fsync_no_writethrough(fd);
  14. }
  16. /*
  17.  * pg_fsync_no_writethrough --- same as fsync except does nothing if
  18.  *      enableFsync is off
  19.  */
  20. int
  21. pg_fsync_no_writethrough(int fd)
  22. {
  23.         if (enableFsync)
  24.                 return fsync(fd);
  25.         else
  26.                 return 0;
  27. }
  29. /*
  30.  * pg_fsync_writethrough
  31.  */
  32. int
  33. pg_fsync_writethrough(int fd)
  34. {
  35.         if (enableFsync)
  36.         {
  37. #ifdef WIN32
  38.                 return _commit(fd);
  39. #elif defined(F_FULLFSYNC)
  40.                 return (fcntl(fd, F_FULLFSYNC, 0) == -1) ? -1 : 0;
  41. #else
  42.                 errno = ENOSYS;
  43.                 return -1;
  44. #endif
  45.         }
  46.         else
  47.                 return 0;
  48. }

从上面的代码分析，扩展clog page时，如果在CLOG BUFFER中没有EMPTY SLOT，则需要backend process主动刷CLOG PAGE，所以会有调用pg_fsync的动作。

clog page和数据库BLOCKSZ (database block size)一样大，默认是8K（如果编译数据库软件时没有修改的话，默认是8KB），最大可以设置为32KB。每个事务在pg_clog中需要2个比特位来存储事务信息(xmin commit/abort,xmax commit/abort)。所以8K的clog page可以存储32K个事务信息，换句话说，每32K个事务，需要扩展一次clog page。

下面的代码是clog的一些常用宏。 src/backend/access/transam/clog.c

  1. /*
  2.  * Defines for CLOG page sizes.  A page is the same BLCKSZ as is used
  3.  * everywhere else in Postgres.
  4.  *
  5.  * Note: because TransactionIds are 32 bits and wrap around at 0xFFFFFFFF,
  6.  * CLOG page numbering also wraps around at 0xFFFFFFFF/CLOG_XACTS_PER_PAGE,
  7.  * and CLOG segment numbering at
  8.  * 0xFFFFFFFF/CLOG_XACTS_PER_PAGE/SLRU_PAGES_PER_SEGMENT.  We need take no
  9.  * explicit notice of that fact in this module, except when comparing segment
  10.  * and page numbers in TruncateCLOG (see CLOGPagePrecedes).
  11.  */
  13. /* We need two bits per xact, so four xacts fit in a byte */
  14. #define CLOG_BITS_PER_XACT      2
  15. #define CLOG_XACTS_PER_BYTE 4
  16. #define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
  17. #define CLOG_XACT_BITMASK       ((1 << CLOG_BITS_PER_XACT) - 1)
  19. #define TransactionIdToPage(xid)         ((xid) / (TransactionId) CLOG_XACTS_PER_PAGE)
  20. #define TransactionIdToPgIndex(xid)     ((xid) % (TransactionId) CLOG_XACTS_PER_PAGE)
  21. #define TransactionIdToByte(xid)          (TransactionIdToPgIndex(xid) / CLOG_XACTS_PER_BYTE)
  22. #define TransactionIdToBIndex(xid)       ((xid) % (TransactionId) CLOG_XACTS_PER_BYTE)

查看数据库的block size：

  1. postgres@digoal-> pg_controldata |grep block  
  2. Database block size:                  8192  
  3. WAL block size:                       8192

我们可以使用stap来跟踪是否调用pg_fsync，如果你要观察backend process主动刷clog 脏页，可以把checkpoint间隔开大，同时把clog shared buffer pages。 你就会观察到backend process主动刷clog 脏页。

  1. Size
  2. CLOGShmemBuffers(void)
  3. {
  4.     return Min(32, Max(4, NBuffers / 512));
  5. }

跟踪

  1. src/backend/access/transam/slru.c
  2. SlruPhysicalWritePage
  3. ......
  4.                 SlruFileName(ctl, path, segno);
  5.                 fd = OpenTransientFile(path, O_RDWR | O_CREAT | PG_BINARY,
  6.                                                            S_IRUSR | S_IWUSR);
  7. ......
  8. src/backend/storage/file/fd.c
  9. OpenTransientFile
  10. pg_fsync(fd)

stap脚本

  1. [root@digoal ~]# cat trc.stp
  2. global f_start[999999]
  4. probe process("/opt/pgsql/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c").call { 
  5.    f_start[execname(), pid(), tid(), cpu()] = gettimeofday_ms()
  6.    printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), gettimeofday_ms(), pp(), $$parms$$)
  7.    # printf("%s -> time:%d, pp:%s\n", thread_indent(1), f_start[execname(), pid(), tid(), cpu()], pp() )
  8. }
  10. probe process("/opt/pgsql/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c").return {
  11.   t=gettimeofday_ms()
  12.   a=execname()
  13.   b=cpu()
  14.   c=pid()
  15.   d=pp()
  16.   e=tid()
  17.   if (f_start[a,c,e,b]) {
  18.   printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d, $return$$)
  19.   # printf("%s <- time:%d, pp:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d)
  20.   }
  21. }
  23. probe process("/opt/pgsql/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c").call {
  24.    f_start[execname(), pid(), tid(), cpu()] = gettimeofday_ms()
  25.    printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), gettimeofday_ms(), pp(), $$parms$$)
  26.    # printf("%s -> time:%d, pp:%s\n", thread_indent(1), f_start[execname(), pid(), tid(), cpu()], pp() )
  27. }
  29. probe process("/opt/pgsql/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c").return {
  30.   t=gettimeofday_ms()
  31.   a=execname()
  32.   b=cpu()
  33.   c=pid()
  34.   d=pp()
  35.   e=tid()
  36.   if (f_start[a,c,e,b]) {
  37.   printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d, $return$$)
  38.   # printf("%s <- time:%d, pp:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d)
  39.   }
  40. }
  42. probe process("/opt/pgsql/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c").call {
  43.    f_start[execname(), pid(), tid(), cpu()] = gettimeofday_ms()
  44.    printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), gettimeofday_ms(), pp(), $$parms$$)
  45.    # printf("%s -> time:%d, pp:%s\n", thread_indent(1), f_start[execname(), pid(), tid(), cpu()], pp() )
  46. }
  48. probe process("/opt/pgsql/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c").return {
  49.   t=gettimeofday_ms()
  50.   a=execname()
  51.   b=cpu()
  52.   c=pid()
  53.   d=pp()
  54.   e=tid()
  55.   if (f_start[a,c,e,b]) {
  56.   printf("%s <- time:%d, pp:%s, par:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d, $return$$)
  57.   # printf("%s <- time:%d, pp:%s\n", thread_indent(-1), t - f_start[a,c,e,b], d)
  58.   }
  59. }

开启一个pgbench执行txid_current()函数申请新的事务号。

  1. postgres@digoal-> cat 7.sql
  2. select txid_current();

测试，约每秒产生32K左右的请求。

  1. postgres@digoal-> pgbench -M prepared -n -r -P 1 -f ./7.sql -c 1 -j 1 -T 100000
  2. progress: 240.0 s, 31164.4 tps, lat 0.031 ms stddev 0.183
  3. progress: 241.0 s, 33243.3 tps, lat 0.029 ms stddev 0.127
  4. progress: 242.0 s, 32567.3 tps, lat 0.030 ms stddev 0.179
  5. progress: 243.0 s, 33656.6 tps, lat 0.029 ms stddev 0.038
  6. progress: 244.0 s, 33948.1 tps, lat 0.029 ms stddev 0.021
  7. progress: 245.0 s, 32996.8 tps, lat 0.030 ms stddev 0.046
  8. progress: 246.0 s, 34156.7 tps, lat 0.029 ms stddev 0.015
  9. progress: 247.0 s, 33259.5 tps, lat 0.029 ms stddev 0.074
  10. progress: 248.0 s, 32979.6 tps, lat 0.030 ms stddev 0.043
  11. progress: 249.0 s, 32892.6 tps, lat 0.030 ms stddev 0.039
  12. progress: 250.0 s, 33090.7 tps, lat 0.029 ms stddev 0.020
  13. progress: 251.0 s, 33238.3 tps, lat 0.029 ms stddev 0.017
  14. progress: 252.0 s, 32341.3 tps, lat 0.030 ms stddev 0.045
  15. progress: 253.0 s, 31999.0 tps, lat 0.030 ms stddev 0.167
  16. progress: 254.0 s, 33332.6 tps, lat 0.029 ms stddev 0.056
  17. progress: 255.0 s, 30394.6 tps, lat 0.032 ms stddev 0.027
  18. progress: 256.0 s, 31862.7 tps, lat 0.031 ms stddev 0.023
  19. progress: 257.0 s, 31574.0 tps, lat 0.031 ms stddev 0.112

跟踪backend process

  1. postgres@digoal-> ps -ewf|grep postgres
  2. postgres  2921  1883 29 09:37 pts/1    00:00:05 pgbench -M prepared -n -r -P 1 -f ./7.sql -c 1 -j 1 -T 100000
  3. postgres  2924  1841 66 09:37 ?        00:00:13 postgres: postgres postgres [local] SELECT

从日志中抽取pg_clog相关的跟踪结果。

  1. [root@digoal ~]# stap -vp 5 -DMAXSKIPPED=9999999 -DSTP_NO_OVERLOAD -DMAXTRYLOCK=100 ./trc.stp -x 2924 >./stap.log 2>&1
  3.      0 postgres(2924): -> time:1441503927731, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").call, par:ctl={.shared=0x7f74a9fe39c0, .do_fsync='\001', .PagePrecedes=0x4b1960, .Dir="pg_clog"} pageno=12350 slotno=10 fdata=ERROR
  4.     31 postgres(2924): -> time:1441503927731, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").call, par:fileName="pg_clog/0181" fileFlags=66 fileMode=384
  5.     53 postgres(2924): <- time:0, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").return, par:14
  6.    102 postgres(2924): -> time:1441503927731, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").call, par:fd=14
  7.   1096 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").return, par:0
  8.   1113 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").return, par:'\001'
  10. 1105302 postgres(2924): -> time:1441503928836, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").call, par:ctl={.shared=0x7f74a9fe39c0, .do_fsync='\001', .PagePrecedes=0x4b1960, .Dir="pg_clog"} pageno=12351 slotno=11 fdata=ERROR
  11. 1105329 postgres(2924): -> time:1441503928836, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").call, par:fileName="pg_clog/0181" fileFlags=66 fileMode=384
  12. 1105348 postgres(2924): <- time:0, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").return, par:14
  13. 1105405 postgres(2924): -> time:1441503928836, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").call, par:fd=14
  14. 1106440 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").return, par:0
  15. 1106452 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").return, par:'\001'
  17. 2087891 postgres(2924): -> time:1441503929819, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").call, par:ctl={.shared=0x7f74a9fe39c0, .do_fsync='\001', .PagePrecedes=0x4b1960, .Dir="pg_clog"} pageno=12352 slotno=12 fdata=ERROR
  18. 2087917 postgres(2924): -> time:1441503929819, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").call, par:fileName="pg_clog/0182" fileFlags=66 fileMode=384
  19. 2087958 postgres(2924): <- time:0, pp:process("/opt/pgsql9.4.4/bin/postgres").function("OpenTransientFile@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:1710").return, par:14
  20. 2088013 postgres(2924): -> time:1441503929819, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").call, par:fd=14
  21. 2089250 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("pg_fsync@/opt/soft_bak/postgresql-9.4.4/src/backend/storage/file/fd.c:315").return, par:0
  22. 2089265 postgres(2924): <- time:1, pp:process("/opt/pgsql9.4.4/bin/postgres").function("SlruPhysicalWritePage@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/slru.c:699").return, par:'\001'

计算估计，每隔1秒左右会产生一次fsync。

  1. postgres=# select 1441503928836-1441503927731;
  2.  ?column? 
  3. ----------
  4.      1105
  5. (1 row)
  7. postgres=# select 1441503929819-1441503928836;
  8.  ?column? 
  9. ----------
  10.       983
  11. (1 row)

前面pgbench的输出看到每秒产生约32000个事务，刚好等于一个clog页的事务数(本例数据块大小为8KB)。 每个事务需要2个比特位，每个字节存储4个事务信息，8192*4=32768。

如果你需要观察backend process不刷clog buffer脏页的情况。可以把checkpoint 间隔改小，或者手动执行checkpoint，同时还需要把clog buffer pages改大，例如：

  1. Size
  2. CLOGShmemBuffers(void)
  3. {
  4.     return Min(1024, Max(4, NBuffers / 2));
  5. }

使用同样的stap脚本，你就观察不到backend process主动刷clog dirty page了。

通过以上分析，如果你发现backend process频繁的clog，可以采取一些优化手段。

  1. 因为每次扩展pg_clog文件后，文件大小都会发生变化，此时如果backend process调用pg_fdatasync也会写文件系统metadata journal（以EXT4为例，假设mount参数data不等于writeback），这个操作是整个文件系统串行的，容易产生堵塞； 所以backend process挑选clog page时，不选择最近的page number可以起到一定的效果，（最好是不选择最近的clog file中的pages）； 另一种方法是先调用sync_file_range, SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER，它不需要写metadata。将文件写入后再调用pg_fsync。减少等待data fsync的时间；
  2. pg_clog文件预分配，目前pg_clog单个文件的大小是由CLOGShmemBuffers决定的，为BLOCKSZ的32倍。可以尝试预分配这个文件，而不是每次都扩展，改变它的大小；
  3. 延迟backend process 的 fsync请求到checkpoint处理。

[参考] https://wiki.postgresql.org/wiki/Hint\_Bits http://blog.163.com/digoal@126/blog/static/1638770402015840480734/ src/backend/access/transam/varsup.c src/backend/access/transam/clog.c src/backend/access/transam/slru.c src/include/access/slru.h src/backend/access/transam/subtrans.c src/backend/storage/file/fd.c

pg_clog的原子操作与pg_subtrans(子事务)

如果没有子事务，其实很容易保证pg_clog的原子操作，但是，如果加入了子事务并为子事务分配了XID，并且某些子事务XID和父事务的XID不在同一个CLOG PAGE时，保证事务一致性就涉及CLOG的原子写了。

PostgreSQL是通过2PC来实现CLOG的原子写的：

  1. 首先将主事务以外的CLOG PAGE中的子事务设置为sub-committed状态；
  2. 然后将主事务所在的CLOG PAGE中的子事务设置为sub-committed，同时设置主事务为committed状态，将同页的子事务设置为committed状态；
  3. 将其他CLOG PAGE中的子事务设置为committed状态；

src/backend/access/transam/clog.c

  1. /*
  2.  * TransactionIdSetTreeStatus
  3.  *
  4.  * Record the final state of transaction entries in the commit log for
  5.  * a transaction and its subtransaction tree. Take care to ensure this is
  6.  * efficient, and as atomic as possible.
  7.  *
  8.  * xid is a single xid to set status for. This will typically be
  9.  * the top level transactionid for a top level commit or abort. It can
  10.  * also be a subtransaction when we record transaction aborts.
  11.  *
  12.  * subxids is an array of xids of length nsubxids, representing subtransactions
  13.  * in the tree of xid. In various cases nsubxids may be zero.
  14.  *
  15.  * lsn must be the WAL location of the commit record when recording an async
  16.  * commit.  For a synchronous commit it can be InvalidXLogRecPtr, since the
  17.  * caller guarantees the commit record is already flushed in that case.  It
  18.  * should be InvalidXLogRecPtr for abort cases, too.
  19.  *
  20.  * In the commit case, atomicity is limited by whether all the subxids are in
  21.  * the same CLOG page as xid.  If they all are, then the lock will be grabbed
  22.  * only once, and the status will be set to committed directly.  Otherwise
  23.  * we must
  24.  *       1. set sub-committed all subxids that are not on the same page as the
  25.  *              main xid
  26.  *       2. atomically set committed the main xid and the subxids on the same page
  27.  *       3. go over the first bunch again and set them committed
  28.  * Note that as far as concurrent checkers are concerned, main transaction
  29.  * commit as a whole is still atomic.
  30.  *
  31.  * Example:
  32.  *              TransactionId t commits and has subxids t1, t2, t3, t4
  33.  *              t is on page p1, t1 is also on p1, t2 and t3 are on p2, t4 is on p3
  34.  *              1. update pages2-3:
  35.  *                                      page2: set t2,t3 as sub-committed
  36.  *                                      page3: set t4 as sub-committed
  37.  *              2. update page1:
  38.  *                                      set t1 as sub-committed,
  39.  *                                      then set t as committed,
  40.                                         then set t1 as committed
  41.  *              3. update pages2-3:
  42.  *                                      page2: set t2,t3 as committed
  43.  *                                      page3: set t4 as committed
  44.  *
  45.  * NB: this is a low-level routine and is NOT the preferred entry point
  46.  * for most uses; functions in transam.c are the intended callers.
  47.  *
  48.  * XXX Think about issuing FADVISE_WILLNEED on pages that we will need,
  49.  * but aren't yet in cache, as well as hinting pages not to fall out of
  50.  * cache yet.
  51.  */

实际调用的入口代码在transam.c，subtrans.c中是一些低级接口。

那么什么是subtrans？ 当我们使用savepoint时，会产生子事务。子事务和父事务一样，可能消耗XID。一旦为子事务分配了XID，那么就涉及CLOG的原子操作了，因为要保证父事务和所有的子事务的CLOG一致性。 当不消耗XID时，需要通过SubTransactionId来区分子事务。

  1. src/backend/acp:process("/opt/pgsql9.4.4/bin/postgres").function("SubTransSetParent@/opt/soft_bak/postgresql-9.4.4/src/backend/access/transam/subtrans.c:75").return, par:pageno=? entryno=? slotno=607466858 ptr=0

重新开一个会话，你会发现，子事务也消耗了XID。因为重新分配的XID已经从607466859开始了。

  1. postgres@digoal-> psql
  2. psql (9.4.4)
  3. Type "help" for help.
  4. postgres=# select txid_current();
  5.  txid_current 
  6. --------------
  7.     607466859
  8. (1 row)

[参考] src/backend/access/transam/clog.c src/backend/access/transam/subtrans.c src/backend/access/transam/transam.c src/backend/access/transam/README src/include/c.hr

CLOG一致性和异步提交

异步提交是指不需要等待事务对应的wal buffer fsync到磁盘，即返回，而且写CLOG时也不需要等待XLOG落盘。 而pg_clog和pg_xlog是两部分存储的，那么我们想一想，如果一个已提交事务的pg_clog已经落盘，而XLOG没有落盘，刚好此时数据库CRASH了。数据库恢复时，由于该事务对应的XLOG缺失，数据无法恢复到最终状态，但是PG_CLOG却显示该事务已提交，这就出问题了。

所以对于异步事务，CLOG在write前，务必等待该事务对应的XLOG已经FLUSH到磁盘。

PostgreSQL如何记录事务和它产生的XLOG的LSN的关系呢？ 其实不是一一对应的关系，而是记录了多事务对一个LSN的关系。 src/backend/access/transam/clog.c LSN组，每32个事务，记录它们对应的最大LSN。 也就是32个事务，只记录最大的LSN。节约空间？

  1. /* We store the latest async LSN for each group of transactions */
  2. #define CLOG_XACTS_PER_LSN_GROUP        32      /* keep this a power of 2 */
  4. 每个CLOG页需要分成多少个LSN组。
  5. #define CLOG_LSNS_PER_PAGE      (CLOG_XACTS_PER_PAGE / CLOG_XACTS_PER_LSN_GROUP)
  7. #define GetLSNIndex(slotno, xid)        ((slotno) * CLOG_LSNS_PER_PAGE + \
  8.         ((xid) % (TransactionId) CLOG_XACTS_PER_PAGE) / CLOG_XACTS_PER_LSN_GROUP)

LSN被存储在这个数据结构中 src/include/access/slru.h

  1. /*
  2.  * Shared-memory state
  3.  */
  4. typedef struct SlruSharedData
  5. {
  6. ......
  7.     /*
  8.          * Optional array of WAL flush LSNs associated with entries in the SLRU
  9.          * pages.  If not zero/NULL, we must flush WAL before writing pages (true
  10.          * for pg_clog, false for multixact, pg_subtrans, pg_notify).  group_lsn[]
  11.          * has lsn_groups_per_page entries per buffer slot, each containing the
  12.          * highest LSN known for a contiguous group of SLRU entries on that slot's
  13.          * page.  仅仅pg_clog需要记录group_lsn
  14.          */
  15.         XLogRecPtr *group_lsn;  // 一个数组，存储32个事务组成的组中最大的LSN号。
  16.         int                     lsn_groups_per_page;
  17. ......

src/backend/access/transam/clog.c

  1.  * lsn must be the WAL location of the commit record when recording an async
  2.  * commit.  For a synchronous commit it can be InvalidXLogRecPtr, since the
  3.  * caller guarantees the commit record is already flushed in that case.  It
  4.  * should be InvalidXLogRecPtr for abort cases, too.
  6. void
  7. TransactionIdSetTreeStatus(TransactionId xid, int nsubxids,
  8.                                         TransactionId *subxids, XidStatus status, XLogRecPtr lsn)
  9. {
  10. ......

更新事务状态时，同时更新对应LSN组的LSN为最大LSN值。(CLOG BUFFER中的操作)

  1. /*
  2.  * Sets the commit status of a single transaction.
  3.  *
  4.  * Must be called with CLogControlLock held
  5.  */
  6. static void
  7. TransactionIdSetStatusBit(TransactionId xid, XidStatus status, XLogRecPtr lsn, int slotno)
  8. {
  9. ......
  10.         /*
  11.          * Update the group LSN if the transaction completion LSN is higher.
  12.          *
  13.          * Note: lsn will be invalid when supplied during InRecovery processing,
  14.          * so we don't need to do anything special to avoid LSN updates during
  15.          * recovery. After recovery completes the next clog change will set the
  16.          * LSN correctly.
  17.          */
  18.         if (!XLogR       int                     lsnindex = GetLSNIndex(slotno, xid);
  20.                 if (ClogCtl->shared->group_lsn[lsnindex] < lsn)  // 更新组LSN
  21.                         ClogCtl->shared->group_lsn[lsnindex] = lsn;
  22.         }
  23. ......

将事务标记为commit状态，对于异步事务，多一个LSN参数，用于修改事务组的最大LSN。

  1. /*
  2.  * TransactionIdCommitTree
  3.  *              Marks the given transaction and children as committed
  4.  *
  5.  * "xid" is a toplevel transaction commit, and the xids array contains its
  6.  * committed subtransactions.
  7.  *
  8.  * This commit operation is not guaranteed to be atomic, but if not, subxids
  9.  * are correctly marked subcommit first.
  10.  */
  11. void
  12. TransactionIdCommitTree(TransactionId xid, int nxids, TransactionId *xids)
  13. {
  14.         TransactionIdSetTreeStatus(xid, nxids, xids,
  15.                                                            TRANSACTION_STATUS_COMMITTED,
  16.                                                            InvalidXLogRecPtr);
  17. }
  19. /*
  20.  * TransactionIdAsyncCommitTree
  21.  *              Same as above, but for async commits.  The commit record LSN is needed.
  22.  */
  23. void
  24. TransactionIdAsyncCommitTree(TransactionId xid, int nxids, TransactionId *xids,
  25.                                                          XLogRecPtr lsn)
  26. {
  27.         TransactionIdSetTreeStatus(xid, nxids, xids,
  28.                                                            TRANSACTION_STATUS_COMMITTED, lsn);
  29. }
  31. /*
  32.  * TransactionIdAbortTree
  33.  *              Marks the given transaction and children as aborted.
  34.  *
  35.  * "xid" is a toplevel transaction commit, and the xids array contains its
  36.  * committed subtransactions.
  37.  *
  38.  * We don't need to worry about the non-atomic behavior, since any onlookers
  39.  * will consider all the xacts as not-yet-committed anyway.
  40.  */
  41. void
  42. TransactionIdAbortTree(TransactionId xid, int nxids, TransactionId *xids)
  43. {
  44.         TransactionIdSetTreeStatus(xid, nxids, xids,
  45.                                                            TRANSACTION_STATUS_ABORTED, InvalidXLogRecPtr);
  46. }

从XID号，获取它对应的LSN，需要注意的是，这个XID如果是一个FROZEN XID，则返回一个(XLogRecPtr) invalid lsn。 src/backend/access/transam/transam.c

  1. /*
  2.  * TransactionIdGetCommitLSN
  3.  *
  4.  * This function returns an LSN that is late enough to be able
  5.  * to guarantee that if we flush up to the LSN returned then we
  6.  * will have flushed the transaction's commit record to disk.
  7.  *
  8.  * The result is not necessarily the exact LSN of the transaction's
  9.  * commit record!  For example, for long-past transactions (those whose
  10.  * clog pages already migrated to disk), we'll return InvalidXLogRecPtr.
  11.  * Also, because we group transactions on the same clog page to conserve
  12.  * storage, we might return the LSN of a later transaction that falls into
  13.  * the same group.
  14.  */
  15. XLogRecPtr
  16. TransactionIdGetCommitLSN(TransactionId xid)
  17. {
  18.         XLogRecPtr      result;
  20.         /*
  21.          * Currently, all uses of this function are for xids that were just
  22.          * reported to be committed by TransactionLogFetch, so we expect that
  23.          * checking TransactionLogFetch's cache will usually succeed and avoid an
  24.          * extra trip to shared memory.
  25.          */
  26.         if (TransactionIdEquals(xid, cachedFetchXid))
  27.                 return cachedCommitLSN;
  29.         /* Special XIDs are always known committed */
  30.         if (!TransactionIdIsNormal(xid))
  31.                 return InvalidXLogRecPtr;
  33.         /*
  34.          * Get the transaction status.
  35.          */
  36.         (void) TransactionIdGetStatus(xid, &result);
  38.         return result;
  39. }
  42. /*
  43.  * Interrogate the state of a transaction in the commit log.
  44.  *
  45.  * Aside from the actual commit status, this function returns (into *lsn)
  46.  * an LSN that is late enough to be able to guarantee that if we flush up to
  47.  * that LSN then we will have flushed the transaction's commit record to disk.
  48.  * The result is not necessarily the exact LSN of the transaction's commit
  49.  * record!      For example, for long-past transactions (those whose clog pages  // long-past事务，指非标准事务号。例), we'll return InvalidXLogRecPtr.  Also, because
  50.  * we group transactions on the same clog page to conserve storage, we might
  51.  * return the LSN of a later transaction that falls into the same group.
  52.  *
  53.  * NB: this is a low-level routine and is NOT the preferred entry point
  54.  * for most uses; TransactionLogFetch() in transam.c is the intended caller.
  55.  */
  56. XidStatus
  57. TransactionIdGetStatus(TransactionId xid, XLogRecPtr *lsn)
  58. {
  59.         int                     pageno = TransactionIdToPage(xid);
  60.         int                     byteno = TransactionIdToByte(xid);
  61.         int                     bshift = TransactionIdToBIndex(xid) * CLOG_BITS_PER_XACT;
  62.         int                     slotno;
  63.         int                     lsnindex;
  64.         char       *byteptr;
  65.         XidStatus       status;
  67.         /* lock is acquired by SimpleLruReadPage_ReadOnly */
  69.         slotno = SimpleLruReadPage_ReadOnly(ClogCtl, pageno, xid);
  70.         byteptr = ClogCtl->shared->page_buffer[slotno] + byteno;
  72.         status = (*byteptr >> bshift) & CLOG_XACT_BITMASK;
  74.         lsnindex = GetLSNIndex(slotno, xid);
  75.         *lsn = ClogCtl->shared->group_lsn[lsnindex];
  77.         LWLockRelease(CLogControlLock);
  79.         return status;
  80. }

前面所涉及的都是CLOG BUFFER中的操作，如果要将buffer写到磁盘，则真正需要涉及到一致性的问题，即在将CLOG write到磁盘前，必须先确保对应的事务产生的XLOG已经flush到磁盘。那么这里就需要用到前面每个LSN组中记录的max LSN了。 代码如下： src/backend/access/transam/slru.c

  1. /*
  2.  * Physical write of a page from a buffer slot
  3.  *
  4.  * On failure, we cannot just ereport(ERROR) since caller has put state in
  5.  * shared memory that must be undone.  So, we return FALSE and save enough
  6.  * info in static variables to let SlruReportIOError make the report.
  7.  *
  8.  * For now, assume it's not worth keeping a file pointer open across
  9.  * independent read/write operations.  We do batch operations during
  10.  * SimpleLruFlush, though.
  11.  *
  12.  * fdata is NULL for a standalone write, pointer to open-file info during
  13.  * SimpleLruFlush.
  14.  */
  15. static bool
  16. SlruPhysicalWritePage(SlruCtl ctl, int pageno, int slotno, SlruFlush fdata)
  17. {
  18.         SlruShared      shared = ctl->shared;
  19.         int                     segno = pageno / SLRU_PAGES_PER_SEGMENT;
  20.         int                     rpageno = pageno % SLRU_PAGES_PER_SEGMENT;
  21.         int                     offset = rpageno * BLCKSZ;
  22.         char            path[MAXPGPATH];
  23.         int                     fd = -1;
  25.         /*
  26.          * Honor the write-WAL-before-data rule, if appropriate, so that we do not
  27.          * write out data before associated WAL records.  This is the same action
  28.          * performed during FlushBuffer() in the main buffer manager.
  29.          */
  30.         if (shared->group_lsn != NULL)
  31.         {
  32.                 /*
  33.                  * We must determine the largest async-commit LSN for the page. This
  34.                  * is a bit tedious, but since this entire function is a slow path
  35.                  * anyway, it seems better to do this here than to maintain a per-page
  36.                  * LSN variable (which'd need an extra comparison in the
  37.                  * transaction-commit path).
  38.                  */
  39.                 XLogRecPtr      max_lsn;
  40.                 int                     lsnindex,
  41.                                         lsnoff;
  43.                 lsnindex = slotno * shared->lsn_groups_per_page;
  44.                 max_lsn = shared->group_lsn[lsnindex++];
  45.                 for (lsnoff = 1; lsnoff < shared->lsn_groups_per_page; lsnoff++)
  46.                 {
  47.                         XLogRecPtr      this_lsn = shared->group_lsn[lsnindex++];
  49.                         if (max_lsn < this_lsn)
  50.                                 max_lsn = this_lsn;
  51.                 }
  53.                 if (!XLogRecPtrIsInvalid(max_lsn))  // 判断max_lsn是不是一个有效的LSN，如果是有效的LSN，说明需要先调用xlogflush将wal buffer中小于该LSN以及以前的buffer写入磁盘。
  54.                                                     则。
  55.                 {
  56.                         /*
  57.                          * As noted above, elog(ERROR) is not acceptable here, so if
  58.                          * XLogFlush were to fail, we must PANIC.  This isn't much of a
  59.                          * restriction because XLogFlush is just about all critical
  60.                          * section anyway, but let's make sure.
  61.                          */
  62.                         START_CRIT_SECTION();
  63.                         XLogFlush(max_lsn);
  64.                         END_CRIT_SECTION();
  65.                 }
  66.         }
  67. ......

小结 对于异步事务，如何保证write-WAL-before-data规则？ pg_clog将32个事务分为一组，存储这些事务的最大LSN。存储在SlruSharedData结构中。 在将clog buffer write到磁盘前，需要确保该clog page对应事务的xlog LSN已经flush到磁盘。

[参考] src/backend/access/transam/clog.c src/include/access/slru.h src/backend/access/transam/transam.c src/backend/access/transam/slru.c