1.10. Troubleshooting an Installation

1.10.1. First Install

If your CouchDB doesn’t start after you’ve just installed, check the following things:

  • On UNIX-like systems, this is usually this is a permissions issue. Ensure that you’ve followed the User Registration and Security chown/chmod commands. This problem is indicated by the presence of the keyword eacces somewhere in the error output from CouchDB itself.
  • Some Linux distributions split up Erlang into multiple packages. For your distribution, check that you really installed all the required Erlang modules. This varies from platform to platform, so you’ll just have to work it out for yourself. For example, on recent versions of Ubuntu/Debian, the erlang package includes all Erlang modules.
  • Confirm that Erlang itself starts up with crypto (SSL) support:
  1. ## what version of erlang are you running? Ensure it is supported
  2. erl -noshell -eval 'io:put_chars(erlang:system_info(otp_release)).' -s erlang halt
  3. ## are the erlang crypto (SSL) libraries working?
  4. erl -noshell -eval 'case application:load(crypto) of ok -> io:put_chars("yay_crypto\n") ; _ -> exit(no_crypto) end.' -s init stop
  • Next, identify where your Erlang CouchDB libraries are installed. This will typically be the lib/ subdirectory of the release that you have installed.
  • Use this to start up Erlang with the CouchDB libraries in its path:
  1. erl -env ERL_LIBS $ERL_LIBS:/path/to/couchdb/lib -couch_ini -s crypto
  • In that Erlang shell, let’s check that the key libraries are running. The %% lines are comments, so you can skip them:
  1. %% test SSL support. If this fails, ensure you have the OTP erlang-crypto library installed
  2. crypto:md5_init().
  4. %% test Snappy compression. If this fails, check your CouchDB configure script output or alternatively
  5. %% if your distro comes with erlang-snappy make sure you're using only the CouchDB supplied version
  6. snappy:compress("gogogogogogogogogogogogogogo").
  8. %% test the CouchDB JSON encoder. CouchDB uses different encoders in each release, this one matches
  9. %% what is used in 2.0.x.
  10. jiffy:decode(jiffy:encode(<<"[1,2,3,4,5]">>)).
  12. %% this is how you quit the erlang shell.
  13. q().
  • The output should resemble this, or an error will be thrown:
  1. Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
  3. Eshell V6.2  (abort with ^G)
  4. 1> crypto:md5_init().
  5. <<1,35,69,103,137,171,205,239,254,220,186,152,118,84,50,
  6.   16,0,0,0,0,0,0,0,0,0,0,0,0,0,...>>
  7. 2> snappy:compress("gogogogogogogogogogogogogogo").
  8. {ok,<<28,4,103,111,102,2,0>>}
  9. 3> jiffy:decode(jiffy:encode(<<"[1,2,3,4,5]">>)).
  10. <<"[1,2,3,4,5]">>
  11. 4> q().
  • At this point the only remaining dependency is your system’s Unicode support library, ICU, and the Spidermonkey Javascript VM from Mozilla. Make sure that your LD_LIBRARY_PATH or equivalent for non-Linux systems (DYLD_LIBRARY_PATH on macOS) makes these available to CouchDB. Linux example running as normal user:
  1.   LD_LIBRARY_PATH=/usr/local/lib:/usr/local/spidermonkey/lib couchdb
  3. Linux example running as couchdb user:
  1. echo LD_LIBRARY_PATH=/usr/local/lib:/usr/local/spidermonkey/lib couchdb | sudo -u couchdb sh
  • If you receive an error message including the key word eaddrinuse, such as this:
  1.   Failure to start Mochiweb: eaddrinuse
  3. edit your ``etc/default.ini`` or ``etc/local.ini`` file and change the
  4. ``[chttpd] port = 5984`` line to an available port.
  • If you receive an error including the string:
  1. … OS Process Error … {os_process_error,{exit_status,127}}

then it is likely your SpiderMonkey JavaScript VM installation is not correct. Please recheck your build dependencies and try again.

  • If you receive an error including the string:
  1. … OS Process Error … {os_process_error,{exit_status,139}}

this is caused by the fact that SELinux blocks access to certain areas of the file system. You must re-configure SELinux, or you can fully disable SELinux using the command:

  1. setenforce 0
  • If you are still not able to get CouchDB to start at this point, keep reading.

1.10.2. Quick Build

Having problems getting CouchDB to run for the first time? Follow this simple procedure and report back to the user mailing list or IRC with the output of each step. Please put the output of these steps into a paste service (such as https://paste.ee/) rather than including the output of your entire run in IRC or the mailing list directly.

  1. Note down the name and version of your operating system and your processor architecture.
  2. Note down the installed versions of CouchDB’s dependencies.
  3. Follow the checkout instructions to get a fresh copy of CouchDB’s trunk.
  4. Configure from the couchdb directory:
  1. ./configure
  1. Build the release:
  1. make release
  1. Run the couchdb command and log the output:
  1. cd rel/couchdb
  2. bin/couchdb
  1. Use your system’s kernel trace tool and log the output of the above command.
    1. For example, linux systems should use strace:
  1. strace bin/couchdb 2> strace.out
  1. Report back to the mailing list (or IRC) with the output of each step.

1.10.3. Upgrading

Are you upgrading from CouchDB 1.x? Install CouchDB into a fresh directory. CouchDB’s directory layout has changed and may be confused by libraries present from previous releases.

1.10.4. Runtime Errors

1.10.4.1. Erlang stack trace contains system_limit, open_port, or emfile

Modern Erlang has a default limit of 65536 ports (8196 on Windows), where each open file handle, tcp connection, and linked-in driver uses one port. OSes have different soft and hard limits on the number of open handles per process, often as low as 1024 or 4096 files. You’ve probably exceeded this.

There are two settings that need changing to increase this value. Consult your OS documentation for how to increase the limit for your process. Under Linux and systemd, this setting can be adjusted via systemctl edit couchdb and adding the lines:

  1. [Service]
  2. LimitNOFILE=65536

to the file in the editor.

To increase this value higher than 65536, you must also add the Erlang +Q parameter to your etc/vm.args file by adding the line:

  1. +Q 102400

The old ERL_MAX_PORTS environment variable is ignored by the version of Erlang supplied with CouchDB.

1.10.4.2. Lots of memory being used on startup

Is your CouchDB using a lot of memory (several hundred MB) on startup? This one seems to especially affect Dreamhost installs. It’s really an issue with the Erlang VM pre-allocating data structures when ulimit is very large or unlimited. A detailed discussion can be found on the erlang-questions list, but the short answer is that you should decrease ulimit -n or lower the vm.args parameter +Q to something reasonable like 1024.

1.10.4.3. function raised exception (Cannot encode ‘undefined’ value as JSON)

If you see this in the CouchDB error logs, the JavaScript code you are using for either a map or reduce function is referencing an object member that is not defined in at least one document in your database. Consider this document:

  1. {
  2.   "_id":"XYZ123",
  3.   "_rev":"1BB2BB",
  4.   "field":"value"
  5. }

and this map function:

  1. function(doc) {
  2.   emit(doc.name, doc.address);
  3. }

This will fail on the above document, as it does not contain a name or address member. Instead, use guarding to make sure the function only accesses members when they exist in a document:

  1. function(doc) {
  2.   if(doc.name && doc.address) {
  3.     emit(doc.name, doc.address);
  4.   }
  5. }

While the above guard will work in most cases, it’s worth bearing JavaScript’s understanding of ‘false’ values in mind. Testing against a property with a value of 0 (zero), '' (empty String), false or null will return false. If this is undesired, a guard of the form if (doc.foo !== undefined) should do the trick.

This error can also be caused if a reduce function does not return a value. For example, this reduce function will cause an error:

  1. function(key, values) {
  2.   sum(values);
  3. }

The function needs to return a value:

  1. function(key, values) {
  2.   return sum(values);
  3. }

1.10.4.4. erlang stack trace contains bad_utf8_character_code

CouchDB 1.1.1 and later contain stricter handling of UTF8 encoding. If you are replicating from older versions to newer versions, then this error may occur during replication.

A number of work-arounds exist; the simplest is to do an in-place upgrade of the relevant CouchDB and then compact prior to replicating.

Alternatively, if the number of documents impacted is small, use filtered replication to exclude only those documents.

1.10.4.5. FIPS mode

Operating systems can be configured to disallow the use of OpenSSL MD5 hash functions in order to prevent use of MD5 for cryptographic purposes. CouchDB makes use of MD5 hashes for verifying the integrity of data (and not for cryptography) and will not run without the ability to use MD5 hashes.

The message below indicates that the operating system is running in “FIPS mode,” which, among other restrictions, does not allow the use of OpenSSL’s MD5 functions:

  1. md5_dgst.c(82): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  2. [os_mon] memory supervisor port (memsup): Erlang has closed
  3. [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
  4. Aborted

A workaround for this is provided with the --erlang-md5 compile flag. Use of the flag results in CouchDB substituting the OpenSSL MD5 function calls with equivalent calls to Erlang’s built-in library erlang:md5. NOTE: there may be a performance penalty associated with this workaround.

Because CouchDB does not make use of MD5 hashes for cryptographic purposes, this workaround does not defeat the purpose of “FIPS mode,” provided that the system owner is aware of and consents to its use.

1.10.4.6. Debugging startup

If you’ve compiled from scratch and are having problems getting CouchDB to even start up, you may want to see more detail. Start by enabling logging at the debug level:

  1. [log]
  2. level = debug

You can then pass the -init_debug +W i +v +V -emu_args flags in the ERL_FLAGS environment variable to turn on additional debugging information that CouchDB developers can use to help you.

Then, reach out to the CouchDB development team using the links provided on the CouchDB home page for assistance.

1.10.5. macOS Known Issues

1.10.5.1. undefined error, exit_status 134

Sometimes the Verify Installation fails with an undefined error. This could be due to a missing dependency with Mac. In the logs, you will find couchdb exit_status,134.

Installing the missing nspr via brew install nspr resolves the issue. (see: https://github.com/apache/couchdb/issues/979)