mongoreplay
Synopsis
New in version 3.4.
Availability
Available for Linux and macOS.
mongoreplay
is a traffic capture and replay tool for MongoDBthat you can use to inspect and record commands sent to a MongoDBinstance, and then replay those commands back onto another host at alater time.
mongoreplay
can help you preview how your MongoDB deploymentwill perform a production workload under a different environment,such as with a different storage engine, on different hardware, orwith a different operating system configuration.mongoreplay
can also help reproduce and investigate an issue byrecording and replaying the operations that trigger the issue.Finally, mongoreplay
serves as a more flexible version ofthe legacy mongosniff
tool to help you investigate database activity.
Run mongoreplay
from the system command line, not the mongo
shell.
Availability
The mongoreplay
tool is part of the MongoDB tools package. Consult theinstallation guide for your platform forinstructions on how to install the tools package as part of yourMongoDB installation.
The tools package is also available from theMongoDB Download Center,either as a separate tools
download, or contained within theTGZ
or ZIP
downloads, depending on platform. The mongoreplay
tool is not available on Windows.
Tip
If downloading the TGZ
or ZIP
files from the DownloadCenter, you may want to update your PATH
environmentvariable to include the directory where you installed these tools.See the installation guidefor your platform for more information.
Required Access
mongoreplay
requires access to the network interface thatthe record
or monitor
commands willlisten on. You may need to run mongoreplay
with root privilegesto access the network device.
mongoreplay
will not work with MongoDB instances using an SSL connection.
Warning
Only use root privileges when connecting to trusted sources.
If you are using play
to connect to a MongoDB deploymentthat enforces access control, you mustconnect as a user with the required privileges to execute therecorded operations. Include the user’s credentials in the—host
MongoDB connection string.
Options
—verbosity
,
-v
- Increases the amount of internal reporting returned on standard outputor in log files. Increase the verbosity with the
-v
form byincluding the option multiple times, (e.g.-vvvvv
.)
—debug
,
-d
- Increases the amount of detail about mongoreplay operationsand errors recordedin log files. Increase the debugging detail with the
-d
form byincluding the option multiple times, (e.g.-ddd
.)
Commands
mongoreplay
includes the following commands to record,play back, and monitor MongoDB network traffic.
mongoreplay record
record
produces a playback file based onnetwork traffic. record
supports collecting networktraffic directly or canaccept a pcap fileto produce the playback file.The playback file contains a list of all requests sent to themongod
instance during the recording as well as allresponses transmitted to the client during the recording. The playbackfile also records metadata for each request, such as the connectionidentifier and timestamp.
The following prototype uses mongoreplay
to record dataon the loopback network interface and creates a playback filelocated at ~/recordings/playback
.
- mongoreplay record -i eth0 -e "port 27017" -p ~/recordings/playback
Similarly, the following prototype uses mongoreplay
to produce a playback file from an existing pcap file:
- mongoreplay record -f traffic.pcap -p ~/recordings/playback
record
supports the following options:
-f
<path>
- Specifies the path to a pcap file that
record
should read toproduce a playback file.
Use -f
as an alternative to capturing network traffic using-i
. You must specify either -f
or -i
. If you includeboth options, mongoreplay record produces an error.
—expr
<filter expression>
,
-e
<filter expression>
- An expression in Berkeley Packet Filter (BPF) syntax to apply to incoming traffic torecord. Required if you are capturing traffic from a network interface using
-i
.
For example, to capture traffic from a MongoDB instance running onport 27017, you would specify -e 'port 27017'
.
-i
<interface>
- Specifies the network interface that
record
should listen on tocapture network traffic.
Use with -e
.
Use -i
as an alternative to reading an existing pcap file with-i
. You must specify either -f
or -i
. If you includeboth options, mongoreplay record produces an error.
—gzip
<boolean>
- If specified,
record
compresses the playback file with gzip.
The produced playback file is a BSON file.
See
Use record for examples of usingmongoreplay
with the record
command.
mongoreplay play
Note
Starting in MongoDB 4.0, mongoreplay play
supports a newMONGOREPLAY_HOST
environment variable that specifies theMongoDB connection string. The new environmentvairable can be used instead of the command-line —host
option.
For example, the following uses mongoreplay play
to replay the~/recordings/playback
playback file to the mongod
instance running on192.168.0.4:27018
:
- Using the
—host
option:
- mongoreplay play -p ~/recordings/playback --report ~/reports/replay_stats.json --host mongodb://192.168.0.4:27018
- Using the
MONGOREPLAY_HOST
environment variable (Available starting in MongoDB 4.0):
- export MONGOREPLAY_HOST="mongodb://192.168.0.4:27018"
- mongoreplay play -p ~/recordings/playback --report ~/reports/replay_stats.json
play Options
play
supports the following options:
Specifies the output format for the collected statistics.
json
: outputs stat information as jsonformat
: uses the formatting specified in the—format
optionto produce the output file.none
: does not provide any output
—report
<path>
- Specifies the path to which to write an execution report.Use
—collect
to specify the output format for the report.
If you do not specify —report
, play
writes to STDOUT.
—no-truncate
- If specified, disables truncation of large reply payload data in the
play
log output.
—format
- Default:
%F{blue}%t%f %F{cyan}(Connection: %o:%i)%f %F{yellow}%l%f%F{red}%T %c%f %F{white}%n%f%F{green}%Q{Request:}%f%q%F{green}%R{Response:}%f%r)
Specifies the format for terminal output.You can specify arguments immediately after the format ‘verbs’ by wrappingthem in curly braces, as in %Q{<arg>}
.
If you specify —format
, also specify format
as the value for the—collect
option.
—format
supports the following verbs:
%n
: namespace%l
: latency%t
: time. You may optionally specify the date layout using theGo Programming Language’s time formatting. GousesMon Jan 2 15:04:05 MST 2006
as its reference time. Youmust specify the time format using the reference time. Thus, if youwanted to print the date in formatyyyy-mm-dd hh:mm
, you wouldspecify%t{2006-01-02 15:04}
. Refer to the Go time formattingdocumentation for more information.%T
: op time%c
: command%o
: number of connections%i
: request ID%q
: request. You may optinally specified a dot-delimited fieldwithin the JSON structure, as in,%q{command_args.documents}
.%r
: response. You may optinally specified a dot-delimited fieldwithin the JSON structure, as in,%q{command_args.documents}
.%Q{<arg>}
: display<arg>
on presence of request data%R{<arg>}
: display<arg>
on presence of response dataIn addition,—format
supports the following start/end ANSI escape sequences:%B
/%b
: bold%U
/%u
: underline%S
/%s
: standout%F
/%f
: text color (required arg – word or number, 8-color)%K
/%k
: background color (required arg – same as %F/%f)
—no-colors
- When set, removes colors from the
default format
.
If the playback file was created using the —gzip
option, you must also specify —gzip
when running play
.
Specifies a multiplier to adjust playback speed. —speed 1.0
processes the playback file in real time; —speed 0.5
at halfspeed; —speed 3.0
at triple speed.
The specified speed is a target speed. If mongoreplay play encountersa bottleneck, playback may be slower than the specified multiplier.
Specifies a MongoDB connection stringfor the MongoDB deployment to which toplay back the captured network traffic.
By default, play
attempts to connect to a mongod
instance running on the localhost onport number 27017
.
Note
Starting in MongoDB 4.0, mongoreplay play
supports a newMONGOREPLAY_HOST
environment variable that specifies theconnection string for the MongoDB deployment. The new environmentvairable can be used instead of the command-line —host
option.
If —host
command-line option isspecified, the —host
value overridesthe environment variable.
Specifies the number of times to play the playback file.
Specifies the maximum time, in seconds, to queue operations in advanceof transmitting them.
—no-preprocess
- When set,
play
does not preprocess the input file to pre-mapdata such as MongoDB cursor IDs.
—gzip
<boolean>
- If specified,
play
decompresses the playback file with gzip.
See
Use play for examples of usingmongoreplay
with the play
command.
mongoreplay monitor
monitor
inspects live or pre-recordedMongoDB network traffic.
The following prototype uses mongoreplay
to produce aJSON report based on the playback.bson
playback file in the ~/recordings
directory:
- mongoreplay monitor --collect json --report ~/reports/monitor-report.json -p ~/recordings/playback.bson
monitor
supports the following options:
Specifies the output format for the collected statistics.
json
: outputs stat information as jsonformat
: uses the formatting specified in the—format
optionto produce the output file.none
: does not provide any output
—report
<path>
- Specifies the path to which to write an execution report.Use
—collect
to specify the output format for the report.
If you do not specify —report
, monitor
writes to STDOUT.
—no-truncate
- If specified, disables truncation of large reply payload data in the
monitor
log output.
—format
- Default:
%F{blue}%t%f %F{cyan}(Connection: %o:%i)%f %F{yellow}%l%f%F{red}%T %c%f %F{white}%n%f%F{green}%Q{Request:}%f%q%F{green}%R{Response:}%f%r)
Specifies the format for terminal output.You can specify arguments immediately after the format ‘verbs’ by wrappingthem in curly braces, as in %Q{<arg>}
.
If you specify —format
, also specify format
as the value for the—collect
option.
—format
supports the following verbs:
%n
: namespace%l
: latency%t
: time. You may optionally specify the date layout using theGo Programming Language’s time formatting. GousesMon Jan 2 15:04:05 MST 2006
as its reference time. Youmust specify the time format using the reference time. Thus, if youwanted to print the date in formatyyyy-mm-dd hh:mm
, you wouldspecify%t{2006-01-02 15:04}
. Refer to the Go time formattingdocumentation for more information.%T
: op time%c
: command%o
: number of connections%i
: request ID%q
: request. You may optinally specified a dot-delimited fieldwithin the JSON structure, as in,%q{command_args.documents}
.%r
: response. You may optinally specified a dot-delimited fieldwithin the JSON structure, as in,%q{command_args.documents}
.%Q{<arg>}
: display<arg>
on presence of request data%R{<arg>}
: display<arg>
on presence of response dataIn addition,—format
supports the following start/end ANSI escape sequences:%B
/%b
: bold%U
/%u
: underline%S
/%s
: standout%F
/%f
: text color (required arg – word or number, 8-color)%K
/%k
: background color (required arg – same as %F/%f)
—no-colors
- When set, removes colors from the
default format
.
-f
<path>
- Specifies the path to a pcap file that
monitor
should read toproduce a playback file.
Use -f
as an alternative to capturing network traffic using-i
. You must specify either -f
or -i
. If you includeboth options, mongoreplay monitor produces an error.
—expr
<filter expression>
,
-e
<filter expression>
- An expression in Berkeley Packet Filter (BPF) syntax to apply to incoming traffic torecord. Required if you are capturing traffic from a network interface using
-i
.
For example, to capture traffic from a MongoDB instance running onport 27017, you would specify -e 'port 27017'
.
-i
<interface>
- Specifies the network interface that
monitor
should listen on tocapture network traffic.
Use with -e
.
Use -i
as an alternative to reading an existing pcap file with-i
. You must specify either -f
or -i
. If you includeboth options, mongoreplay monitor produces an error.
—paired
- When specified,
monitor
outputs one line for each request/reply pair record.
—gzip
<boolean>
- If specified,
monitor
decompresses the playback file with gzip.
See
Use monitor for examples of usingmongoreplay
with the monitor
command.
mongoreplay Report Format
monitor
and play
can producereports based on a playback file when run with the —report
option.
Sample Record
The following is an example record from a JSON-formattedmonitor
report:
- {
- "order" : 21,
- "op" : "op_msg",
- "command" : "aggregate",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "aggregate" : "foo",
- "cursor" : {},
- "lsid" : {
- "id" : {
- "$binary" : "eBZNIaAbRTiAoWkaNZ0T8Q==",
- "$type" : "04"
- }
- },
- "pipeline" : [
- { "$match" : { "borough" : "Queens" } },
- { "$group" : { "_id" : "$cuisine", "count" : { "$sum": 1 } } }
- ]
- },
- "payloadType" : 0
- }
- ]
- },
- "connection_num" : 0,
- "seen" : "2018-11-15T14:07:07.136794-05:00",
- "request_id" : 25
- }
Fields
mongoreplay
reports can include the following fields:
order
- A monotonically increasing key indicating the order in which theoperations were recorded and played back. This can be used toreconstruct the ordering of the series of operations executed on aconnection, since the order in which they appear in the report filemay not match the playback order.
op
- The type of operation represented by the request: i.e. “query”,“insert”, “command”, “getmore”.
command
- The name of the database command performed, such as
isMaster
orgetLastError
. This field is left blank for operations that are notcommands, such as queries and inserts.
ns
- The namespace on which the request was executed.
request_data
The payload of the operation.
- Query operations:
request_data
contains the actualquery that was issued. - Insert operations:
request_data
contains the documentsbeing inserted. - Update operations:
request_data
contains the queryselector and the update modifier.
- Query operations:
reply_data
- The payload of the reply to the request.
nreturned
- The number of documents returned as a result of the operation.
played_at
- The time at which the
play
command executedthe operation.
play_at
- The time at which the operation was supposed to be executed bythe
play
command.
playbacklag_us
- The difference in microseconds in time between
played_at
andplay_at
. Higher values generally indicate that thetarget server is not able to keep up with the rate at which requestsneed to be executed according to the playback file.
connection_num
- A key that identifies the connection on which the request wasexecuted. All requests/replies that executed on the same connectionhave the same value for
connection_num
.
The connectionnum
valuedoes _not match the connection ID logged on the server side.
latency_us
- The time difference in microseconds between when the request wassent by the client and when a response from the server was received.
errors
- Lists any errors returned from the server.
msg
- Lists the error message returned from the server.
seen
- The time at which the operation was originally captured.
request_id
- The ID of the MongoDB operation. The
request_id
for a requestoperation is the same as theresponse_id
for the correspondingreply.
Output Formatting with —format
monitor
and play
output to either theterminal or, when run with —report
(i.e. monitor —report
or play —report
), to a file.
Use the —collect
(monitor —collect
or play —collect
)option to specify the format of the output:
—collect json
produces JSON output,—collect format
outputs the data based on the formattingspecified by the—format
option formonitor
andplay
.
Examples
Use record
Capture TCP data with record
To create a recording of traffic, use the record
command. Thefollowing operation records traffic through port 27017
on theeth0
network interface and writes the captured traffic to~/recordings/recording.bson
:
- mongoreplay record -i eth0 -e "port 27017" -p ~/recordings/recording.bson
The produced playback file contains everything needed to re-executethe workload on another system.
Record a Playback File from Existing pcap Data
If you do not wish to use mongoreplay
to capture traffic, youcan capture traffic using a utility such as tcpdump
and then create amongoreplay
recording from the static pcap file.
Warning
Only use root privileges when connecting to a trusted source.
The following operation uses tcpdump
to capture traffic throughport 27017
on the eth0
network interface and writes the captureddata to a pcap file called traffic.pcap
:
- sudo tcpdump -B $((100*1024)) -i eth0 -n "port 27017" -w traffic.pcap
To create the mongoreplay
playback file, you can userecord
with the -f
option to specify the pcap file from which to create the playback file,as in the following:
- mongoreplay record -f traffic.pcap -p ~/recordings/playback.bson
The produced playback file contains everything needed to re-executethe workload on another system.
Use play
Execute a Playback File Against a Target Host
play
takes a playback file and executes therecorded operations against the mongodb://example.com:27018
host.Since the mongod
enforces authentication, theconnection string specified to—host
also includes the username, password, and authenticationdatabase so that mongoreplay
can write to the database.
- mongoreplay play -p ~/recordings/recording.bson --host mongodb://username:password@example.com:27018/admin
By default, ~bin.play
executes the playbackfile at the rate of the recording. —speed
lets you modifythe playback speed. For example, the following operation executesthe operations in recording.bson
at twice the recording speed:
- mongoreplay play -p ~/recordings/recording.bson --speed=2.0 --host mongodb://username:password@example.com:27018/admin
Log Metrics About Execution Performance during Playback
play
can produce a report with detailed metricsabout the performance of each operation executed during playback.
The following example executes playback against themongodb://example.com:27017
host and produces a JSON report written to~/reports/playback-report.json
- mongoreplay play -p ~/recordings/recording.bson --report ~/reports/playback-report.json --collect json --host mongodb://example.com:27017
The play
report contents would resemble:
- {
- "order" : 0,
- "op" : "op_msg",
- "command" : "isMaster",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "forShell" : 1,
- "isMaster" : 1
- },
- "payloadType" : 0
- }
- ]
- },
- "reply_data" : {
- "sections" : [
- {
- "payload" : {
- "ismaster" : true,
- "localTime" : {
- "$date" : "2018-11-15T21:35:01.843Z"
- },
- "logicalSessionTimeoutMinutes" : 30,
- "maxBsonObjectSize" : 16777216,
- "maxMessageSizeBytes" : 48000000,
- "maxWireVersion" : 7,
- "maxWriteBatchSize" : 100000,
- "minWireVersion" : 0,
- "ok" : 1,
- "readOnly" : false
- },
- "payloadType" : 0
- }
- ]
- },
- "played_at" : "2018-11-15T16:35:01.84304-05:00",
- "play_at" : "2018-11-15T16:35:01.842903-05:00",
- "playbacklag_us" : 137,
- "connection_num" : 1,
- "latency_us" : 151,
- "seen" : "2018-11-15T21:30:33.379675Z",
- "request_id" : 21
- }
- {
- "order" : 2,
- "op" : "op_msg",
- "command" : "listCollections",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "authorizedCollections" : true,
- "filter" : {
- },
- "listCollections" : 1,
- "lsid" : {
- "id" : {
- "$binary" : "esmcqhiXSIWSpKGb4sOekw==",
- "$type" : "04"
- }
- },
- "nameOnly" : true
- },
- "payloadType" : 0
- }
- ]
- },
- "reply_data" : {
- "sections" : [
- {
- "payload" : {
- "cursor" : {
- "firstBatch" : [ ],
- "id" : {
- "$numberLong" : "0"
- },
- "ns" : "test.$cmd.listCollections"
- },
- "ok" : 1
- },
- "payloadType" : 0
- }
- ]
- },
- "played_at" : "2018-11-15T16:35:08.715002-05:00",
- "play_at" : "2018-11-15T16:35:08.713539-05:00",
- "playbacklag_us" : 1463,
- "connection_num" : 1,
- "latency_us" : 331,
- "seen" : "2018-11-15T21:30:40.250311Z",
- "request_id" : 22
- }
- {
- "order" : 4,
- "op" : "op_msg",
- "command" : "isMaster",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "forShell" : 1,
- "isMaster" : 1
- },
- "payloadType" : 0
- }
- ]
- },
- "reply_data" : {
- "sections" : [
- {
- "payload" : {
- "ismaster" : true,
- "localTime" : {
- "$date" : "2018-11-15T21:35:08.715Z"
- },
- "logicalSessionTimeoutMinutes" : 30,
- "maxBsonObjectSize" : 16777216,
- "maxMessageSizeBytes" : 48000000,
- "maxWireVersion" : 7,
- "maxWriteBatchSize" : 100000,
- "minWireVersion" : 0,
- "ok" : 1,
- "readOnly" : false
- },
- "payloadType" : 0
- }
- ]
- },
- "played_at" : "2018-11-15T16:35:08.715554-05:00",
- "play_at" : "2018-11-15T16:35:08.71471-05:00",
- "playbacklag_us" : 844,
- "connection_num" : 1,
- "latency_us" : 208,
- "seen" : "2018-11-15T21:30:40.251482Z",
- "request_id" : 23
- }
Refer to mongoreplay Report Format for a description of each field.
Use monitor
Inspect Recorded Operations
monitor
can create a report based on thecontents of a playback file. monitor
’s report includesall operations and some metadata about each operation’s execution.
The following operation uses monitor
to create a JSONreport based on the contents of the recording.bson
playback filelocated in the ~/recordings
directory and write the report to~/reports/monitoring-report.json
:
- mongoreplay monitor -p ~/recordings/recording.bson --report ~/reports/monitoring-report.json --collect json
The report contents would resemble:
- {
- "order" : 0,
- "op" : "op_msg",
- "command" : "isMaster",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "forShell" : 1,
- "isMaster" : 1
- },
- "payloadType" : 0
- }
- ]
- },
- "connection_num" : 0,
- "seen" : "2018-11-15T21:30:33.379675Z",
- "request_id" : 21
- }
- {
- "order" : 1,
- "op" : "op_msg",
- "command" : "aggregate",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "aggregate" : "restaurants",
- "cursor" : {
- },
- "lsid" : {
- "id" : {
- "$binary" : "esmcqhiXSIWSpKGb4sOekw==",
- "$type" : "04"
- }
- },
- "pipeline" : [
- {
- "$match" : {
- "borough" : "Manhattan"
- }
- },
- {
- "$group" : {
- "_id" : "$cuisine"
- }
- }
- ]
- },
- "payloadType" : 0
- }
- ]
- },
- "connection_num" : 0,
- "seen" : "2018-11-15T16:28:52.870007-05:00",
- "request_id" : 13
- }
- {
- "order" : 1,
- "op" : "op_msg",
- "command" : "reply",
- "reply_data" : {
- "sections" : [
- {
- "payload" : {
- "ismaster" : true,
- "localTime" : {
- "$date" : "2018-11-15T21:30:33.379Z"
- },
- "logicalSessionTimeoutMinutes" : 30,
- "maxBsonObjectSize" : 16777216,
- "maxMessageSizeBytes" : 48000000,
- "maxWireVersion" : 7,
- "maxWriteBatchSize" : 100000,
- "minWireVersion" : 0,
- "ok" : 1,
- "readOnly" : false
- },
- "payloadType" : 0
- }
- ]
- },
- "connection_num" : 0,
- "latency_us" : 174,
- "seen" : "2018-11-15T21:30:33.379849Z",
- "request_id" : 21
- }
- {
- "order" : 2,
- "op" : "op_msg",
- "command" : "listCollections",
- "ns" : "test",
- "request_data" : {
- "sections" : [
- {
- "payload" : {
- "$db" : "test",
- "authorizedCollections" : true,
- "filter" : {
- },
- "listCollections" : 1,
- "lsid" : {
- "id" : {
- "$binary" : "esmcqhiXSIWSpKGb4sOekw==",
- "$type" : "04"
- }
- },
- "nameOnly" : true
- },
- "payloadType" : 0
- }
- ]
- },
- "connection_num" : 0,
- "seen" : "2018-11-15T21:30:40.250311Z",
- "request_id" : 22
- }
Refer to mongoreplay Report Format for a description of each field.
Inspect Live MongoDB Traffic
monitor
can also inspect live traffic and, optionally,create a report based on the observed operations.
To monitor traffic in real time in your terminal, omit the—report
option, as in thefollowing:
- mongoreplay monitor -i eth0 -e 'port 27017' --collect json
Optionally, you can create a report based on the operations observedusing monitor
. The following example creates a JSONreport written to ~/reports/monitor-live.json
based on the trafficthrough port 27017
on the eth0
network interface:
- mongoreplay monitor -i eth0 -e 'port 27017' --report ~/reports/monitor-live.json --collect json