How to improve performance of update() and save() in MongoDB?
It is very likely that you are hitting a very common bottle neck in MongoDB. Since you are updating documents very frequently by adding strings, there is a good chance that you are running out of space for that document and forcing the database to constantly move that document to a different space in memory\disk by rewriting it at the tail end of the data file.
Adding indexes can only hurt write performance so that will not help improve performance unless you are read heavy.
I would consider changing your application logic to do this:
- Index on the keyword field
- Before inserting anything into the database each time you detect a tweet, query for the document which contains the keyword. If it does not exist, insert a new document but pad the
ids
property by adding a whole bunch of fake strings in the array. Then immediately after inserting it, remove all of the id's from that array. This will cause mongodb to allocate additional room for that entire document so that when you start adding id's to the ids field, it will have plenty of room to grow. - Insert the id of the tweet into the
ids
field
You're on the right path. The query portion of your update needs an index, else it is running a table scan. An indent index on keyword and you'll see update performance increase significantly.