Mongodb -- Production notes to self -- How we support 10 million hits a day at 100 ms avg time.

Performance Tuning and Operation gotchas.

-- blockdev (!!!)  ( blockdev --setra 32 /dev/md0 )
single most important parameter. Adjust it to suit your needs

   Why??? => imagine that every time you need to query a document that is not yet on RAM you load a million times more documents in advance. So you freeze your reads while doing that and load this huge amount of data for never to use it. 

   How to adjust it ??? => determine your average document size and the read pattern


A good guess should be a low number if your documents are small and the access pattern is random and larger otherwise.
A good number here will reflect on the resident memory, that in the long run should approach the maximum phisical memory  available. I  have dramatically improved our performance and maximum throughput changing the default read ahead from 8196 to 32 in my 8 disks RAID10 array.

There´s a great post explaining it : http://www.kchodorow.com/blog/2012/05/10/thursday-5-diagnosing-high-readahead/

-- vm.dirty_background_ratio=5
start background paging as early as possible

-- vm.dirty_ratio=90
you should not reach this threshold, for it freezes the database.

-- vm.swapiness=0
I don't want to ever swap, so I set it to 0.

-- vm.page-cluster = 0
the size of the cart on which the vm carries the pages around in powers of 2, =>  0 means 1, 1 means 2 and so on. 

Monitoring: MMS, Serverdensity, page faults and %lock.

-- page faults => how much your mongodb is going to disk.
-- %lock => the percentage of time spent in lock => it represents how long a statement is waiting for something to be read or to be written.


Querying patterns:

If you need to query for a large dataset, try to limit using an index, say "created_at:-1" and query for the most used documents.

In my experience, the load of the documents can cost a lot in a fresh restarted database. So, if you are running a ReplicaSet ( and you must) and you must load a lot of documents at once, try to preload them before promoting it to primary, so the indexes of them all all already loaded into memory before it starts to be hammered.


Aggregation:

If you need aggregation or a rare usage pattern, try to use a set of secondaries with another set of indexes... Or wait until new Aggregation Framework in  2.2 and see if its OK for you.