Capped Collections

Overview

Capped collections are fixed-sizecollections that support high-throughput operations that insertand retrieve documents based on insertion order. Cappedcollections work in a way similar to circular buffers: once acollection fills its allocated space, it makes room for new documentsby overwriting the oldest documents in the collection.

See createCollection() or createfor more information on creating capped collections.

Tip

As an alternative to capped collections, consider MongoDB’sTTL (Time To Live) indexes. Asdescribed in Expire Data from Collections by Setting TTL, these indexes allow youto expire and remove data from normal collections based on the valueof a date-typed field and a TTL value for the index.

TTL indexes are not compatiblewith capped collections.

Behavior

Insertion Order

Capped collections guarantee preservation of the insertion order. As aresult, queries do not need an index to return documents in insertionorder. Without this indexing overhead, capped collections can supporthigher insertion throughput.

Automatic Removal of Oldest Documents

To make room for new documents, capped collections automatically removethe oldest documents in the collection without requiring scripts orexplicit remove operations.

Consider the following potential use cases for cappedcollections:

  • Store log information generated by high-volume systems. Insertingdocuments in a capped collection without an index is close to thespeed of writing log information directly to a filesystem. Furthermore, the built-in first-in-first-out propertymaintains the order of events, while managing storage use.
  • Cache small amounts of data in a capped collections. Since cachesare read rather than write heavy, you would either need to ensurethat this collection always remains in the working set (i.e. inRAM) or accept some write penalty for the required index orindexes.

For example, the oplog.rs collection that stores a logof the operations in a replica set uses a capped collection.Starting in MongoDB 4.0, unlike other capped collections, the oplog cangrow past its configured size limit to avoid deleting the majoritycommit point.

_id Index

Capped collections have an _id field and an index on the _idfield by default.

Restrictions and Recommendations

Updates

If you plan to update documents in a capped collection, create an indexso that these update operations do not require a collection scan.

Document Size

Changed in version 3.2.

If an update or a replacement operation changes the document size, the operation will fail.

Document Deletion

You cannot delete documents from a capped collection. To remove alldocuments from a collection, use the drop()method to drop the collection and recreate the capped collection.

Sharding

You cannot shard a capped collection.

Query Efficiency

Use natural ordering to retrieve the most recently inserted elementsfrom the collection efficiently. This is (somewhat) analogous to tailon a log file.

Aggregation $out

The aggregation pipeline stage $outcannot write results to a capped collection.

Transactions

Starting in MongoDB 4.2, you cannot write to capped collections in transactions. Reads from capped collections are still supportedin transactions.

Procedures

Create a Capped Collection

You must create capped collections explicitly using thedb.createCollection() method, which is a helper in themongo shell for the create command. Whencreating a capped collection you must specify the maximum size of thecollection in bytes, which MongoDB will pre-allocate for the collection.The size of the capped collection includes a small amount of space forinternal overhead.

  1. db.createCollection( "log", { capped: true, size: 100000 } )

If the size field is less than or equal to 4096, then the collection willhave a cap of 4096 bytes. Otherwise, MongoDB will raise the provided size tomake it an integer multiple of 256.

Additionally, you may also specify a maximum number of documents for thecollection using the max field as in the following document:

  1. db.createCollection("log", { capped : true, size : 5242880, max : 5000 } )

Important

The size argument is always required, even whenyou specify max number of documents. MongoDB will remove olderdocuments if a collection reaches the maximum size limit before itreaches the maximum document count.

See

db.createCollection() and create.

Query a Capped Collection

If you perform a find() on a capped collectionwith no ordering specified, MongoDB guarantees that the ordering ofresults is the same as the insertion order.

To retrieve documents in reverse insertion order, issuefind() along with the sort()method with the $natural parameter set to -1, as shownin the following example:

  1. db.cappedCollection.find().sort( { $natural: -1 } )

Check if a Collection is Capped

Use the isCapped() method to determine if acollection is capped, as follows:

  1. db.collection.isCapped()

Convert a Collection to Capped

You can convert a non-capped collection to a capped collection withthe convertToCapped command:

  1. db.runCommand({"convertToCapped": "mycoll", size: 100000});

The size parameter specifies the size of the capped collection inbytes.

This holds a database exclusive lock for the duration of the operation.Other operations which lock the same database will be blocked until theoperation completes. See What locks are taken by some common client operations? foroperations that lock the database.

Tailable Cursor

You can use a tailable cursor with capped collections. Similar to theUnix tail -f command, the tailable cursor “tails” the end of acapped collection. As new documents are inserted into the cappedcollection, you can use the tailable cursor to continue retrievingdocuments.

See Tailable Cursors for information on creatinga tailable cursor.