How to setup MongoDB with replication.
Created: Thu 08 September 2011 / Last updated: Wed 26 October 2011
MongoDB is a document storage server. It is very comfortable to work with it because of the flexibility in which one can store, retrieve and update documents in the database while keeping relatively good performances. As with any system, one needs to be aware of the limitations to correctly use it.
A very nice thing of MongoDB is the support of replica sets. A replica set is a basically a couple of servers put together to replicate and serve the data. Simple replication was at the core of the design decisions of MongoDB and as such, it makes the setup really easy.
The logic behind replicat set is to always have at least 3 warm copies of the data at anytime. It does not exempt you from doing backups, but it allows you to have continuity of service in case of a one replica failure. This is good when you want to sleep at night.
I like to create automated scripts to rebuild automatically the services after a failure, you should do the same. As I like Python, I am using fabric.
The setup is done with MongoDB 1.8.3 but is the same with 2.0.1. You can read the upgrade procedure from 1.8 to 2.0 near the end of this note.
The base Debian Squeeze is extremely easy to setup, you can refer to the Ganeti notes to have an idea of how the setup is done, what is important is that:
To save IPv4 in the future, non critical VMs should get only a private IP and use a proxy to access the outside world.
The setup is performed directly from the tarball. It may be seen as strange to use the tarball when Debian and Ubuntu packages are available, put I prefer to be able to run on the version I want and upgrade by simply downloading and copying the new binaries at the right place.
The setup is simple:
mongodb
user under which MongoDB will run;It is maybe one of the simplest database server to install, with a Fabric recipe, it takes me about 5 seconds per host to have MongoDB installed and configured including the download time.
Under Debian, this is simply a call to adduser
:
adduser --system --shell /bin/sh --gecos 'MongoDB user' --group \
--disabled-password --home /home/mongodb mongodb
You must not forget to change the ownership of the directories to the
mongodb
user or MongoDB will not be able to start:
mkdir -p /var/lib/mongodb && chown mongodb.mongodb /var/lib/mongodb
mkdir -p /var/log/mongodb && chown mongodb.mongodb /var/log/mongodb
As I extract the MongoDB archive in /home/mongodb
, I need to symlink
the mongod
binary to have it in the path at the right place:
ln -s /home/mongodb/bin/mongod /usr/bin/mongod
As you noticed when creating the log and data path, by default Debian
does not follow the same conventions as MongoDB, that is, the
databases are stored in /var/lib/mongodb
and not /data/db
. From a
sysadmin point of view, it is easier to follow the Debian way for
everything, so here is my base configuration, please note the
bind_ip
to have MongoDB open only on the private network:
dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
# Fork is done by the init script
fork=false
# Enables periodic logging of CPU utilization and I/O wait
cpu = true
# Disable the HTTP interface (Defaults to localhost:27018).
nohttpinterface = false
rest = true
bind_ip = %s
directoryperdb = true
# in replica set configuration, specify the name of the replica set
replSet = set1
Yes, the configuration is simple, data durability is maintained using a replica set with 3 nodes and without journaling.
The MongoDB init script is simply copied as /etc/init.d/mongodb
and
installed using the new dependency based boot system. I am
using a small variation of the one provided by default. The difference is that I let MongoDB control the
PID file. You can get the file at the end of this document.
# insserv mongodb
You can now start MongoDB on each server and go to the next step, get the configuration of the replica set.
First, you need to be sure that your VMs can in a way or another
resolve the IP addresses of all the servers involved in your replica
set. You can either do that with a DNS entry or less flexible in your
/etc/hosts
file. This is up to you.
The configuration is done by updating the admin
database.
$ ./mongo 192.168.1.120/admin
MongoDB shell version: 1.8.3
connecting to: 192.168.1.120/admin
Notice that in the configuration, I force mongod
to bind only on the
private network, this is why now, I need to give the IP when
connecting or it would default to localhost
.
In the configuration, we define the name of the set as set1
, so the
configuration is really simple, we first create a cfg
object:
> cfg = {
... _id : "set1",
... members : [
... { _id : 0, host : "mset1a.ipr.ceondo.net" },
... { _id : 1, host : "mset1b.ipr.ceondo.net" },
... { _id : 2, host : "mset1c.ipr.ceondo.net" } ] }
And then you initiate with this configuration. You do this only on one server, the configuration will be distributed to the others automatically.
> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
> rs.status()
{
"set" : "set1",
"date" : ISODate("2011-09-08T12:59:04Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "mset1a.ipr.ceondo.net",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" : {
"t" : 1315486715000,
"i" : 1
},
"optimeDate" : ISODate("2011-09-08T12:58:35Z"),
"self" : true
},
{
"_id" : 1,
"name" : "mset1b.ipr.ceondo.net",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 23,
"optime" : {
"t" : 1315486715000,
"i" : 1
},
"optimeDate" : ISODate("2011-09-08T12:58:35Z"),
"lastHeartbeat" : ISODate("2011-09-08T12:59:03Z")
},
{
"_id" : 2,
"name" : "mset1c.ipr.ceondo.net",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 21,
"optime" : {
"t" : 1315486715000,
"i" : 1
},
"optimeDate" : ISODate("2011-09-08T12:58:35Z"),
"lastHeartbeat" : ISODate("2011-09-08T12:59:03Z")
}
],
"ok" : 1
}
set1:PRIMARY>
Enjoy, your replica set is now running. It is time for you to dive into all the documentation regarding backup and monitoring on the MongoDB website.
By default, MongoDB binds on all the interfaces, do not forget to restrict to an IP of your private network.
When note specified, the database storage is compatible from version n to n + 0.2, so the upgrade is mostly painless.
mongod
;mongod
one by one. This will fail the master over a secondary and keep your replica set up.You want
rotation
of your log files to avoid saturation of your /var
partition. These
settings are rotating every week the logs. Because copytruncate
is
used, you may lose some information, but this is not really a problem
as normally, a MongoDB cluster is monitored by connecting to the
server itself and running stats queries.
Just put in /etc/logrotate.d/mongodb
the following:
/var/log/mongodb/*.log {
weekly
rotate 10
copytruncate
delaycompress
compress
notifempty
missingok
}
The log files will compress very well has they contain a lot of redundant information, for example on an idle system, I get thousands of:
Wed Oct 26 11:09:52 [snapshotthread] cpu: elapsed:4000 writelock: 0%
Wed Oct 26 11:09:56 [snapshotthread] cpu: elapsed:4000 writelock: 0%
Wed Oct 26 11:10:00 [snapshotthread] cpu: elapsed:4000 writelock: 0%
Wed Oct 26 11:10:04 [snapshotthread] cpu: elapsed:4000 writelock: 0%
Wed Oct 26 11:10:08 [snapshotthread] cpu: elapsed:4000 writelock: 0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 |
|