3.2.4. View Cookbook for SQL Jockeys

This is a collection of some common SQL queries and how to get the same result in CouchDB. The key to remember here is that CouchDB does not work like an SQL database at all, and that best practices from the SQL world do not translate well or at all to CouchDB. This document’s “cookbook” assumes that you are familiar with the CouchDB basics such as creating and updating databases and documents.

3.2.4.1. Using Views

How you would do this in SQL:

  1. CREATE TABLE

or:

  1. ALTER TABLE

How you can do this in CouchDB?

Using views is a two-step process. First you define a view; then you query it. This is analogous to defining a table structure (with indexes) using CREATE TABLE or ALTER TABLE and querying it using an SQL query.

3.2.4.1.1. Defining a View

Defining a view is done by creating a special document in a CouchDB database. The only real specialness is the _id of the document, which starts with _design/ — for example, _design/application. Other than that, it is just a regular CouchDB document. To make sure CouchDB understands that you are defining a view, you need to prepare the contents of that design document in a special format. Here is an example:

  1. {
  2.     "_id": "_design/application",
  3.     "_rev": "1-C1687D17",
  4.     "views": {
  5.         "viewname": {
  6.             "map": "function(doc) { ... }",
  7.             "reduce": "function(keys, values) { ... }"
  8.         }
  9.     }
  10. }

We are defining a view viewname. The definition of the view consists of two functions: the map function and the reduce function. Specifying a reduce function is optional. We’ll look at the nature of the functions later. Note that viewname can be whatever you like: users, by-name, or by-date are just some examples.

A single design document can also include multiple view definitions, each identified by a unique name:

  1. {
  2.     "_id": "_design/application",
  3.     "_rev": "1-C1687D17",
  4.     "views": {
  5.         "viewname": {
  6.             "map": "function(doc) { ... }",
  7.             "reduce": "function(keys, values) { ... }"
  8.         },
  9.         "anotherview": {
  10.             "map": "function(doc) { ... }",
  11.             "reduce": "function(keys, values) { ... }"
  12.         }
  13.     }
  14. }

3.2.4.1.2. Querying a View

The name of the design document and the name of the view are significant for querying the view. To query the view viewname, you perform an HTTP GET request to the following URI:

  1. /database/_design/application/_view/viewname

database is the name of the database you created your design document in. Next up is the design document name, and then the view name prefixed with _view/. To query anotherview, replace viewname in that URI with anotherview. If you want to query a view in a different design document, adjust the design document name.

3.2.4.1.3. MapReduce Functions

MapReduce is a concept that solves problems by applying a two-step process, aptly named the map phase and the reduce phase. The map phase looks at all documents in CouchDB separately one after the other and creates a map result. The map result is an ordered list of key/value pairs. Both key and value can be specified by the user writing the map function. A map function may call the built-in emit(key, value) function 0 to N times per document, creating a row in the map result per invocation.

CouchDB is smart enough to run a map function only once for every document, even on subsequent queries on a view. Only changes to documents or new documents need to be processed anew.

3.2.4.1.4. Map functions

Map functions run in isolation for every document. They can’t modify the document, and they can’t talk to the outside world—they can’t have side effects. This is required so that CouchDB can guarantee correct results without having to recalculate a complete result when only one document gets changed.

The map result looks like this:

  1. {"total_rows":3,"offset":0,"rows":[
  2. {"id":"fc2636bf50556346f1ce46b4bc01fe30","key":"Lena","value":5},
  3. {"id":"1fb2449f9b9d4e466dbfa47ebe675063","key":"Lisa","value":4},
  4. {"id":"8ede09f6f6aeb35d948485624b28f149","key":"Sarah","value":6}
  5. ]}

It is a list of rows sorted by the value of key. The id is added automatically and refers back to the document that created this row. The value is the data you’re looking for. For example purposes, it’s the girl’s age.

The map function that produces this result is:

  1. function(doc) {
  2.     if(doc.name && doc.age) {
  3.         emit(doc.name, doc.age);
  4.     }
  5. }

It includes the if statement as a sanity check to ensure that we’re operating on the right fields and calls the emit function with the name and age as the key and value.

3.2.4.2. Look Up by Key

How you would do this in SQL:

  1. SELECT field FROM table WHERE value="searchterm"

How you can do this in CouchDB?

Use case: get a result (which can be a record or set of records) associated with a key (“searchterm”).

To look something up quickly, regardless of the storage mechanism, an index is needed. An index is a data structure optimized for quick search and retrieval. CouchDB’s map result is stored in such an index, which happens to be a B+ tree.

To look up a value by “searchterm”, we need to put all values into the key of a view. All we need is a simple map function:

  1. function(doc) {
  2.     if(doc.value) {
  3.         emit(doc.value, null);
  4.     }
  5. }

This creates a list of documents that have a value field sorted by the data in the value field. To find all the records that match “searchterm”, we query the view and specify the search term as a query parameter:

  1. /database/_design/application/_view/viewname?key="searchterm"

Consider the documents from the previous section, and say we’re indexing on the age field of the documents to find all the five-year-olds:

  1. function(doc) {
  2.     if(doc.age && doc.name) {
  3.         emit(doc.age, doc.name);
  4.     }
  5. }

Query:

  1. /ladies/_design/ladies/_view/age?key=5

Result:

  1. {"total_rows":3,"offset":1,"rows":[
  2. {"id":"fc2636bf50556346f1ce46b4bc01fe30","key":5,"value":"Lena"}
  3. ]}

Easy.

Note that you have to emit a value. The view result includes the associated document ID in every row. We can use it to look up more data from the document itself. We can also use the ?include_docs=true parameter to have CouchDB fetch the individual documents for us.

3.2.4.3. Look Up by Prefix

How you would do this in SQL:

  1. SELECT field FROM table WHERE value LIKE "searchterm%"

How you can do this in CouchDB?

Use case: find all documents that have a field value that starts with searchterm. For example, say you stored a MIME type (like text/html or image/jpg) for each document and now you want to find all documents that are images according to the MIME type.

The solution is very similar to the previous example: all we need is a map function that is a little more clever than the first one. But first, an example document:

  1. {
  2.     "_id": "Hugh Laurie",
  3.     "_rev": "1-9fded7deef52ac373119d05435581edf",
  4.     "mime-type": "image/jpg",
  5.     "description": "some dude"
  6. }

The clue lies in extracting the prefix that we want to search for from our document and putting it into our view index. We use a regular expression to match our prefix:

  1. function(doc) {
  2.     if(doc["mime-type"]) {
  3.         // from the start (^) match everything that is not a slash ([^\/]+) until
  4.         // we find a slash (\/). Slashes needs to be escaped with a backslash (\/)
  5.         var prefix = doc["mime-type"].match(/^[^\/]+\//);
  6.         if(prefix) {
  7.           emit(prefix, null);
  8.         }
  9.     }
  10. }

We can now query this view with our desired MIME type prefix and not only find all images, but also text, video, and all other formats:

  1. /files/_design/finder/_view/by-mime-type?key="image/"

3.2.4.4. Aggregate Functions

How you would do this in SQL:

  1. SELECT COUNT(field) FROM table

How you can do this in CouchDB?

Use case: calculate a derived value from your data.

We haven’t explained reduce functions yet. Reduce functions are similar to aggregate functions in SQL. They compute a value over multiple documents.

To explain the mechanics of reduce functions, we’ll create one that doesn’t make a whole lot of sense. But this example is easy to understand. We’ll explore more useful reductions later.

Reduce functions operate on the output of the map function (also called the map result or intermediate result). The reduce function’s job, unsurprisingly, is to reduce the list that the map function produces.

Here’s what our summing reduce function looks like:

  1. function(keys, values) {
  2.     var sum = 0;
  3.     for(var idx in values) {
  4.         sum = sum + values[idx];
  5.     }
  6.     return sum;
  7. }

Here’s an alternate, more idiomatic JavaScript version:

  1. function(keys, values) {
  2.     var sum = 0;
  3.     values.forEach(function(element) {
  4.         sum = sum + element;
  5.     });
  6.     return sum;
  7. }

Note

Don’t miss effective built-in reduce functions like _sum and _count

This reduce function takes two arguments: a list of keys and a list of values. For our summing purposes we can ignore the keys-list and consider only the value list. We’re looping over the list and add each item to a running total that we’re returning at the end of the function.

You’ll see one difference between the map and the reduce function. The map function uses emit() to create its result, whereas the reduce function returns a value.

For example, from a list of integer values that specify the age, calculate the sum of all years of life for the news headline, “786 life years present at event.” A little contrived, but very simple and thus good for demonstration purposes. Consider the documents and the map view we used earlier in this document.

The reduce function to calculate the total age of all girls is:

  1. function(keys, values) {
  2.     return sum(values);
  3. }

Note that, instead of the two earlier versions, we use CouchDB’s predefined sum() function. It does the same thing as the other two, but it is such a common piece of code that CouchDB has it included.

The result for our reduce view now looks like this:

  1. {"rows":[
  2.     {"key":null,"value":15}
  3. ]}

The total sum of all age fields in all our documents is 15. Just what we wanted. The key member of the result object is null, as we can’t know anymore which documents took part in the creation of the reduced result. We’ll cover more advanced reduce cases later on.

As a rule of thumb, the reduce function should reduce to a single scalar value. That is, an integer; a string; or a small, fixed-size list or object that includes an aggregated value (or values) from the values argument. It should never just return values or similar. CouchDB will give you a warning if you try to use reduce “the wrong way”:

  1. {
  2.     "error":"reduce_overflow_error",
  3.     "message":"Reduce output must shrink more rapidly: Current output: ..."
  4. }

3.2.4.5. Get Unique Values

How you would do this in SQL:

  1. SELECT DISTINCT field FROM table

How you can do this in CouchDB?

Getting unique values is not as easy as adding a keyword. But a reduce view and a special query parameter give us the same result. Let’s say you want a list of tags that your users have tagged themselves with and no duplicates.

First, let’s look at the source documents. We punt on _id and _rev attributes here:

  1. {
  2.     "name":"Chris",
  3.     "tags":["mustache", "music", "couchdb"]
  4. }
  1. {
  2.     "name":"Noah",
  3.     "tags":["hypertext", "philosophy", "couchdb"]
  4. }
  1. {
  2.     "name":"Jan",
  3.     "tags":["drums", "bike", "couchdb"]
  4. }

Next, we need a list of all tags. A map function will do the trick:

  1. function(doc) {
  2.     if(doc.name && doc.tags) {
  3.         doc.tags.forEach(function(tag) {
  4.             emit(tag, null);
  5.         });
  6.     }
  7. }

The result will look like this:

  1. {"total_rows":9,"offset":0,"rows":[
  2. {"id":"3525ab874bc4965fa3cda7c549e92d30","key":"bike","value":null},
  3. {"id":"3525ab874bc4965fa3cda7c549e92d30","key":"couchdb","value":null},
  4. {"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"couchdb","value":null},
  5. {"id":"da5ea89448a4506925823f4d985aabbd","key":"couchdb","value":null},
  6. {"id":"3525ab874bc4965fa3cda7c549e92d30","key":"drums","value":null},
  7. {"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"hypertext","value":null},
  8. {"id":"da5ea89448a4506925823f4d985aabbd","key":"music","value":null},
  9. {"id":"da5ea89448a4506925823f4d985aabbd","key":"mustache","value":null},
  10. {"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"philosophy","value":null}
  11. ]}

As promised, these are all the tags, including duplicates. Since each document gets run through the map function in isolation, it cannot know if the same key has been emitted already. At this stage, we need to live with that. To achieve uniqueness, we need a reduce:

  1. function(keys, values) {
  2.     return true;
  3. }

This reduce doesn’t do anything, but it allows us to specify a special query parameter when querying the view:

  1. /dudes/_design/dude-data/_view/tags?group=true

CouchDB replies:

  1. {"rows":[
  2. {"key":"bike","value":true},
  3. {"key":"couchdb","value":true},
  4. {"key":"drums","value":true},
  5. {"key":"hypertext","value":true},
  6. {"key":"music","value":true},
  7. {"key":"mustache","value":true},
  8. {"key":"philosophy","value":true}
  9. ]}

In this case, we can ignore the value part because it is always true, but the result includes a list of all our tags and no duplicates!

With a small change we can put the reduce to good use, too. Let’s see how many of the non-unique tags are there for each tag. To calculate the tag frequency, we just use the summing up we already learned about. In the map function, we emit a 1 instead of null:

  1. function(doc) {
  2.     if(doc.name && doc.tags) {
  3.         doc.tags.forEach(function(tag) {
  4.             emit(tag, 1);
  5.         });
  6.     }
  7. }

In the reduce function, we return the sum of all values:

  1. function(keys, values) {
  2.     return sum(values);
  3. }

Now, if we query the view with the ?group=true parameter, we get back the count for each tag:

  1. {"rows":[
  2. {"key":"bike","value":1},
  3. {"key":"couchdb","value":3},
  4. {"key":"drums","value":1},
  5. {"key":"hypertext","value":1},
  6. {"key":"music","value":1},
  7. {"key":"mustache","value":1},
  8. {"key":"philosophy","value":1}
  9. ]}

3.2.4.6. Enforcing Uniqueness

How you would do this in SQL:

  1. UNIQUE KEY(column)

How you can do this in CouchDB?

Use case: your applications require that a certain value exists only once in a database.

This is an easy one: within a CouchDB database, each document must have a unique _id field. If you require unique values in a database, just assign them to a document’s _id field and CouchDB will enforce uniqueness for you.

There’s one caveat, though: in the distributed case, when you are running more than one CouchDB node that accepts write requests, uniqueness can be guaranteed only per node or outside of CouchDB. CouchDB will allow two identical IDs to be written to two different nodes. On replication, CouchDB will detect a conflict and flag the document accordingly.