Starting Manticore in Linux

When Manticore Search is installed using DEB or RPM packages, the searchd process can be run and managed by operating system’s init system. Most Linux versions now use systemd, while older releases use SysV init.

If you are not sure about the type of the init system your platform use, run:

  1. ps --no-headers -o comm 1

Starting and stopping using systemd

After the installation the Manticore Search service is not started automatically. To start Manticore run the following command:

  1. sudo systemctl start manticore

To stop Manticore run the following command:

  1. sudo systemctl stop manticore

The Manticore service is set to run at boot. You can check it by running:

  1. sudo systemctl is-enabled manticore

If you want to disable Manticore starting at boot time run:

  1. sudo systemctl disable manticore

To make Manticore start at boot, run:

  1. sudo systemctl enable manticore

In some newer operating systems it can fail with error “Failed to enable unit: Unit … is transient or generated.”. In this case you can remove the generator and try again. It’s unlikely you need the generator ever again after Manticore is installed.

In Debian-based operating systems run:

  1. sudo rm /lib/systemd/system-generators/manticore-generator
  2. sudo systemctl daemon-reload

In CentOS and RHEL run:

  1. sudo rm /usr/lib/systemd/system-generators/manticore-search-generator
  2. sudo systemctl daemon-reload

If you still need the generator again just install the Manticore packages again and it will recover the generator file.

searchd process logs startup information in systemd journal. If systemd logging is enabled you can view the logged information with the following command:

  1. sudo journalctl --unit manticore

Custom startup flags using systemd

systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS allows you to specify custom startup flags Manticore Search daemon should be started with. See full list here.

For example, to start Manticore with debug logging level you can run:

  1. systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS='--logdebug'
  2. systemctl restart manticore

To undo it run:

  1. systemctl set-environment _ADDITIONAL_SEARCHD_PARAMS=''
  2. systemctl restart manticore

Note, systemd environment variables get reset on server reboot.

Starting and stopping using service

Manticore can be started and stopped using service commands:

  1. sudo service manticore start
  2. sudo service manticore stop

To enable the sysV service at boot on RedHat systems run:

  1. chkconfig manticore on

To enable the sysV service at boot on Debian systems (including Ubuntu) run:

  1. update-rc.d manticore defaults

Please note that searchd is started by the init system under manticore user and all files created by the server will be owned by this user. If searchd is started under ,for example, root user, the permissions of files will be changed which may lead to issues when running again searchd as service.

Starting Manticore manually

You can also start Manticore Search by calling searchd (Manticore Search server binary) directly:

  1. searchd [OPTIONS]

Note that without setting a path to the configuration file searchd will try to find it in several locations, depending on the operation system.

searchd command line options

The options available to searchd in all operation systems are:

  • --help (-h for short) lists all of the parameters that can be called in your particular build of searchd.
  • --version (-v for short) shows Manticore Search version information.
  • --config <file> (-c <file> for short) tells searchd to use the given file as its configuration.
  • --stop is used to asynchronously stop searchd, using the details of the PID file as specified in the Manticore configuration file, so you may also need to confirm to searchd which configuration file to use with the --config option. Example:
  1. $ searchd --config /etc/manticoresearch/manticore.conf --stop
  • --stopwait is used to synchronously stop searchd. --stop essentially tells the running instance to exit (by sending it a SIGTERM) and then immediately returns. --stopwait will also attempt to wait until the running searchd instance actually finishes the shutdown (eg. saves all the pending attribute changes) and exits. Example:
  1. $ searchd --config /etc/manticoresearch/manticore.conf --stopwait

Possible exit codes are as follows:

  • 0 on success

  • 1 if connection to running searchd server failed

  • 2 if server reported an error during shutdown

  • 3 if server crashed during shutdown

  • --status command is used to query running searchd instance status using the connection details from the (optionally) provided configuration file. It will try to connect to running instance using the first found UNIX socket or TCP port from the configuration file. On success it will query for a number of status and performance counter values and print them. You can also use SHOW STATUS command to access the very same counters via SQL protocol. Examples:

    1. $ searchd --status
    2. $ searchd --config /etc/manticoresearch/manticore.conf --status

  • --pidfile is used to explicitly force using a PID file (where the searchd process identification number is stored) despite any other debugging options that say otherwise (for instance, --console). This is a debugging option.

    1. $ searchd --console --pidfile

  • --console is used to force searchd into console mode; typically Manticore runs as a conventional server application and logs information into log files (as specified in configuration file). Sometimes though, when debugging issues in the configuration or the server itself or trying to diagnose hard-to-track-down problems it may be easier to force it to dump information directly to the console/command line from which it is being called. Running in console mode also means that the process will not be forked (so searches are done in sequence) and logs will not be written to. (It should be noted that console mode is not the intended method for running searchd.) You can invoke so:

    1. $ searchd --config /etc/manticoresearch/manticore.conf --console

  • --logdebug, --logreplication, --logdebugv, and --logdebugvv options enable additional debug output in the server log. They differ by the logging verboseness level. These are debugging options, they pollute the log a lot, and thus they should not be normally enabled. (The normal use case for these is to enable them temporarily on request, to assist with some particularly complicated debugging session.)

  • --iostats is used in conjunction with the logging options (the query_log will need to have been activated in manticore.conf) to provide more detailed information on a per-query basis as to the input/output operations carried out in the course of that query, with a slight performance hit and a little bit bigger logs. The IO statistics don’t include information about IO operations for attributes, as these are loaded with mmap. To enable it you can start searchd so:

    1. $ searchd --config /etc/manticoresearch/manticore.conf --iostats

  • --cpustats is used to provide actual CPU time report (in addition to wall time) in both query log file (for every given query) and status report (aggregated). It depends on clock_gettime() Linux system call or falls back to less precise call on certain systems. You might start searchd thus:

    1. $ searchd --config /etc/manticoresearch/manticore.conf --cpustats

  • --port portnumber (-p for short) is used to specify the port that Manticore should listen on to accept binary protocol requests, usually for debugging purposes. This will usually default to 9312, but sometimes you need to run it on a different port. Specifying it on the command line will override anything specified in the configuration file. The valid range is 0 to 65535, but ports numbered 1024 and below usually require a privileged account in order to run. An example of usage:

    1.   $ searchd --port 9313

  • --listen ( address ":" port | port | path ) [ ":" protocol ] (or -l for short) Works as --port, but allow you to specify not only the port, but full path, as IP address and port, or Unix-domain socket path, that searchd will listen on. In other words, you can specify either an IP address (or hostname) and port number or just a port number or Unix socket path. If you specify port number, but not the address searchd will listen on all network interfaces. Unix path is identified by a leading slash. As the last param you can also specify a protocol handler (listener) to be used for connections on this socket. Supported protocol values are ‘sphinx’ and ‘mysql’ (MySQL protocol used since 4.1).

  • --force-preread forbids the server to serve any incoming connection until prereading of table files completes. By default, at startup the server accepts connections while table files are lazy loaded into memory. This opens extends the behavior and makes it wait until the files are loaded.

  • --index (--table) <table> (or -i (-t) <table> for short) forces this instance of searchd to only serve the specified table. Like --port, above, this is usually for debugging purposes; more long-term changes would generally be applied to the configuration file itself.

  • --strip-path strips the path names from all the file names referenced from the table (stopwords, wordforms, exceptions, etc). This is useful for picking up tables built on another machine with possibly different path layouts.

  • --replay-flags=<OPTIONS> switch can be used to specify a list of extra binary log replay options. The supported options are:

    • accept-desc-timestamp, ignore descending transaction timestamps and replay such transactions anyway (the default behavior is to exit with an error).
    • ignore-open-errors, ignore missing binlog files (the default behavior is to exit with an error).
    • ignore-trx-errors, ignore any transaction errors and skip current binlog file (the default behavior is to exit with an error).

    • ignore-all-errors, ignore any errors described above (the default behavior is to exit with an error). Example:

      1. $ searchd --replay-flags=accept-desc-timestamp

  • --coredump is used to enable saving a core file or a minidump of the server on crash. Disabled by default to speed up of server restart on crash. This is useful for debugging purposes.

    1. $ searchd --config /etc/manticoresearch/manticore.conf --coredump

  • --new-cluster bootstraps a replication cluster and makes the server a reference node with cluster restart protection. On Linux you can also run manticore_new_cluster. It will start Manticore in --new-cluster mode via systemd.

  • --new-cluster-force bootstraps a replication cluster and makes the server a reference node bypassing cluster restart protection. On Linux you can also run manticore_new_cluster --force. It will start Manticore in --new-cluster-force mode via systemd.

There are some options for searchd that are specific to Windows platforms, concerning handling as a service, and are only available in Windows binaries.

Note that in Windows searchd will default to --console mode, unless you install it as a service.

  • --install installs searchd as a service into the Microsoft Management Console (Control Panel / Administrative Tools / Services). Any other parameters specified on the command line, where --install is specified will also become part of the command line on future starts of the service. For example, as a part of calling searchd, you will likely also need to specify the configuration file with --config, and you would do that as well as specifying --install. Once called, the usual start/stop facilities will become available via the management console, so any methods you could use for starting, stopping and restarting services would also apply to searchd. Example:

    1. C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install
    2.  --config C:\Manticore\manticore.conf

If you want to have the I/O stats every time you start searchd, you need to specify its option on the same line as the --install command thus:

  1. C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install
  2.    --config C:\Manticore\manticore.conf --iostats

  • --delete removes the service from the Microsoft Management Console and other places where services are registered, after previously installed with --install. Note, this does not uninstall the software or delete the tables. It means the service will not be called from the services systems, and will not be started on the machine’s next start. If currently running as a service, the current instance will not be terminated (until the next reboot, or until searchd is called with --stop). If the service was installed with a custom name (with --servicename), the same name will need to be specified with --servicename when calling to uninstall. Example:

    1. C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --delete

  • --servicename <name> applies the given name to searchd when installing or deleting the service, as would appear in the Management Console; this will default to searchd, but if being deployed on servers where multiple administrators may log into the system, or a system with multiple searchd instances, a more descriptive name may be applicable. Note that unless combined with --install or --delete, this option does not do anything. Example:

    1. C:\WINDOWS\system32> C:\Manticore\bin\searchd.exe --install
    2.  --config C:\Manticore\manticore.conf --servicename ManticoreSearch

  • --ntservice is the option that is passed by the Management Console to searchd to invoke it as a service on Windows platforms. It would not normally be necessary to call this directly; this would normally be called by Windows when the service would be started, although if you wanted to call this as a regular service from the command-line (as the complement to --console) you could do so in theory.

  • --safetrace forces searchd to only use system backtrace() call in crash reports. In certain (rare) scenarios, this might be a “safer” way to get that report. This is a debugging option.
  • --nodetach switch (Linux only) tells searchd not to detach into background. This will also cause log entry to be printed out to console. Query processing operates as usual. This is a debugging option and might also be useful when you run Manticore in a docker container to capture its output.

Signals

searchd supports a number of signals:

  • SIGTERM - Initiates a clean shutdown. New queries will not be handled, but queries that are already started will not be forcibly interrupted.
  • SIGHUP - Initiates tables rotation. Depending on the value of seamless_rotate setting, new queries might be shortly stalled; clients will receive temporary errors.
  • SIGUSR1 - Forces reopen of searchd log and query log files, letting you implement log file rotation.

Environment variables

  • MANTICORE_TRACK_DAEMON_SHUTDOWN=1 enables detailed logging while searchd is shutting down. It’s useful in case of some shutdown problems:
    • when Manticore shutdowns too long
    • or when it freezes at shutdown

Starting and using Manticore in Docker

The image is based on current release of Manticore package.

The default configuration includes a sample Real-Time table and listens on the default ports:

  • 9306 for connections from a MySQL client
  • 9308 for connections via HTTP
  • 9312 for connections via a binary protocol (e.g. in case you run a cluster)

The image comes with libraries for easy indexing data from MySQL, PostgreSQL XML and CSV files.

How to run Manticore Search Docker image

Quick usage

The below is the simplest way to start Manticore in a container and log in to it via mysql client:

  1. docker run -e EXTRA=1 --name manticore --rm -d manticoresearch/manticore && echo "Waiting for Manticore docker to start. Consider mapping the data_dir to make it start faster next time" && until docker logs manticore 2>&1 | grep -q "accepting connections"; do sleep 1; echo -n .; done && echo && docker exec -it manticore mysql && docker stop manticore

Note that upon exiting the MySQL client, the Manticore container will be stopped and removed, resulting in no saved data. For information on using Manticore in a production environment, please see below.

The image comes with a sample table which can be loaded like this:

  1. mysql> source /sandbox.sql

Also the mysql client has in history several sample queries that you can run on the above table, just use Up/Down keys in the client to see and run them.

Production use

Ports and mounting points

For data persistence the folder /var/lib/manticore/ should be mounted to local storage or other desired storage engine.

Configuration file inside the instance is located at /etc/manticoresearch/manticore.conf. For custom settings, this file should be mounted to own configuration file.

The ports are 9306/9308/9312 for SQL/HTTP/Binary, expose them depending on how you are going to use Manticore. For example:

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/data:/var/lib/manticore -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

or

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data:/var/lib/manticore/ -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

Make sure to remove 127.0.0.1: if you want the ports to be available for external hosts.

Manticore Columnar Library and Manticore Buddy

The Manticore Search Docker image doesn’t come with the Manticore Columnar Library pre-installed, which is necessary if you require columnar storage and secondary indexes. However, it can easily be enabled during runtime by setting the environment variable EXTRA=1. For example, docker run -e EXTRA=1 ... manticoresearch/manticore. This will download and install the library in the data directory (which is typically mapped as a volume in production environments) and it won’t be re-downloaded unless the Manticore Search version is changed.

Using EXTRA=1 also activates Manticore Buddy, which is used for processing certain commands. For more information, refer to the changelog.

If you only need the MCL, you can use the environment variable MCL=1.

Docker-compose

In many cases you might want to use Manticore together with other images specified in a docker-compose YAML file. Here is the minimal recommended specification for Manticore Search in docker-compose.yml:

  1. version: '2.2'
  3. services:
  4.   manticore:
  5.     container_name: manticore
  6.     image: manticoresearch/manticore
  7.     environment:
  8.       - EXTRA=1
  9.     restart: always
  10.     ports:
  11.       - 127.0.0.1:9306:9306
  12.       - 127.0.0.1:9308:9308
  13.     ulimits:
  14.       nproc: 65535
  15.       nofile:
  16.          soft: 65535
  17.          hard: 65535
  18.       memlock:
  19.         soft: -1
  20.         hard: -1
  21.     volumes:
  22.       - ./data:/var/lib/manticore
  23. #      - ./manticore.conf:/etc/manticoresearch/manticore.conf # uncomment if you use a custom config

Besides using the exposed ports 9306 and 9308 you can log into the instance by running docker-compose exec manticore mysql.

HTTP protocol

HTTP protocol is exposed on port 9308. You can map the port locally and connect with curl:

  1. docker run -e EXTRA=1 --name manticore -p 9308:9308 -d manticoresearch/manticore

Create a table:

  • JSON

JSON

  1. POST /cli -d 'CREATE TABLE testrt ( title text, content text, gid integer)'

Insert a document:

  • JSON

JSON

  1. POST /insert
  2. -d'{"index":"testrt","id":1,"doc":{"title":"Hello","content":"world","gid":1}}'

Perform a simple search:

  • JSON

JSON

  1. POST /search -d '{"index":"testrt","query":{"match":{"*":"hello world"}}}'

Logging

By default, the server is set to send it’s logging to /dev/stdout, which can be viewed from the host with:

  1. docker logs manticore

The query log can be diverted to Docker log by passing variable QUERY_LOG_TO_STDOUT=true.

Multi-node cluster with replication

Here is a simple docker-compose.yml for defining a two node cluster:

  1. version: '2.2'
  3. services:
  5.   manticore-1:
  6.     image: manticoresearch/manticore
  7.     environment:
  8.       - EXTRA=1
  9.     restart: always
  10.     ulimits:
  11.       nproc: 65535
  12.       nofile:
  13.          soft: 65535
  14.          hard: 65535
  15.       memlock:
  16.         soft: -1
  17.         hard: -1
  18.     networks:
  19.       - manticore
  20.   manticore-2:
  21.     image: manticoresearch/manticore
  22.     environment:
  23.       - EXTRA=1
  24.     restart: always
  25.     ulimits:
  26.       nproc: 65535
  27.       nofile:
  28.         soft: 65535
  29.         hard: 65535
  30.       memlock:
  31.         soft: -1
  32.         hard: -1
  33.     networks:
  34.       - manticore
  35. networks:
  36.   manticore:
  37.     driver: bridge

  • Start it: docker-compose up

  • Create a cluster:

    1. $ docker-compose exec manticore-1 mysql
    3. mysql> CREATE TABLE testrt ( title text, content text, gid integer);
    5. mysql> CREATE CLUSTER posts;
    6. Query OK, 0 rows affected (0.24 sec)
    8. mysql> ALTER CLUSTER posts ADD testrt;
    9. Query OK, 0 rows affected (0.07 sec)
    11. MySQL [(none)]> exit
    12. Bye

  • Join to the the cluster on the 2nd instance

    1. $ docker-compose exec manticore-2 mysql
    3. mysql> JOIN CLUSTER posts AT 'manticore-1:9312';
    4. mysql> INSERT INTO posts:testrt(title,content,gid)  VALUES('hello','world',1);
    5. Query OK, 1 row affected (0.00 sec)
    7. MySQL [(none)]> exit
    8. Bye

  • If you now go back to the first instance you’ll see the new record:

    1. $ docker-compose exec manticore-1 mysql
    3. MySQL [(none)]> select * from testrt;
    4. +---------------------+------+-------+---------+
    5. | id                  | gid  | title | content |
    6. +---------------------+------+-------+---------+
    7. | 3891565839006040065 |    1 | hello | world   |
    8. +---------------------+------+-------+---------+
    9. 1 row in set (0.00 sec)
    11. MySQL [(none)]> exit
    12. Bye

Memory locking and limits

It’s recommended to overwrite the default ulimits of docker for the Manticore instance:

  1.  --ulimit nofile=65536:65536

For best performance, table components can be “mlocked” into memory. When Manticore is run under Docker, the instance requires additional privileges to allow memory locking. The following options must be added when running the instance:

  1.   --cap-add=IPC_LOCK --ulimit memlock=-1:-1

Configuring Manticore Search with Docker

If you want to run Manticore with your custom config containing tables definition you will need to mount the configuration to the instance:

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data/:/var/lib/manticore -p 127.0.0.1:9306:9306 -d manticoresearch/manticore

Take into account that Manticore search inside the container is run under user manticore. Performing operations with table files (like creating or rotating plain tables) should be also done under manticore. Otherwise the files will be created under root and the search daemon won’t have rights to open them. For example here is how you can rotate all tables:

  1. docker exec -it manticore gosu manticore indexer --all --rotate

You can also set individual searchd and common configuration settings using Docker environment variables.

The settings must be prefixed with their section name, example for in case of mysql_version_string the variable must be named searchd_mysql_version_string:

  1. docker run -e EXTRA=1 --name manticore  -p 127.0.0.1:9306:9306  -e searchd_mysql_version_string='5.5.0' -d manticoresearch/manticore

In case of listen directive, you can pass using Docker variable searchd_listen new listening interfaces in addition to the default ones. Multiple interfaces can be declared separated by semi-colon (“|”). For listening only on network address, the $ip (retrieved internally from hostname -i) can be used as address alias.

For example -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip' will add an additional SQL interface on port 9307, an SQL VIP listener on port 5443 running only on the instance’s IP and an HTTP listener on port 9316, beside the defaults on 9306 and 9308, respectively.

  1. $ docker run -e EXTRA=1 --rm -p 1188:9307  -e searchd_mysql_version_string='5.5.0' -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip'  manticore
  2. [Mon Aug 17 07:31:58.719 2020] [1] using config file '/etc/manticoresearch/manticore.conf' (9130 chars)...
  3. listening on all interfaces for http, port=9316
  4. listening on all interfaces for mysql, port=9307
  5. listening on 172.17.0.17:5443 for VIP mysql
  6. listening on all interfaces for mysql, port=9306
  7. listening on UNIX socket /var/run/mysqld/mysqld.sock
  8. listening on 172.17.0.17:9312 for sphinx
  9. listening on all interfaces for http, port=9308
  10. prereading 0 indexes
  11. prereaded 0 indexes in 0.000 sec
  12. accepting connections