https://github.com/jdzurik/ESWindowsInstaller

It dose basically what you do in an install.

If you build a sparse index on a datetime field (e.g. the timestamp when the item was added) and then $unset this field when the item is processed then sorting the collection will only return the items that haven’t been processed yet and also return them in the correct order.

The only items included in the index will be those in the queue so the index is kept smaller than if you were just using a regular index on a collection which should help performance (by keeping it in RAM).

Normally, this doesn’t affect the tailable cursor used by the server oplog as it’s the only ObjectId producer and for our own queue it doesn’t matter when we’re reading through in natural order. Where it potentially hits is, as you say, when the cursor needs to be re-started and for this the last ObjectId read is used.

What you could do to make sure you don’t lose any messages is create a new ObjectId based on the last one read but set a few seconds previously (whatever the longest time that you think the clocks could be out by). This slight-rewind should ensure that nothing was missed at the expense of re-reading a few messages when the cursor needs to be restarted. If the number of messages was so high that this was an issue then you should probably be looking at other queuing systems anyway (ZeroMQ if you have a really high rate).

The other problem is common with nearly all queuing systems where you *may* get repeated messages. This is caused by the problem above where the system is trying to guarantee that nothing is missed and is why most messaging systems actually guarantee “at least once delivery” – i.e. you may get some messages more than once and you have to handle it. Because of this, it’s good to make the message handling idempotent when possible or have some recent-cache on the receiver to avoid re-processing duplicates.

So, for your options:

1). Use the last id to rewind slightly when creating the new cursor. This is quite efficient and once done you are going through in $natural order so it’s quick and you get all the tailable cursor advantages (blocking on the server with immediate response for new messages etc…)

2). I don’t think this is a viable option other than for very small queues. #1 should be more efficient. It is how the example starts when nothing has been read but you always have to handle the cursor dying and needing to be restarted so you’ll be doing #1 anyway (I don’t think .Skip() works because a capped collection wraps round so keeping track of how much to skip may be impossible).

3). You could use ‘findAndModify’ or ‘findAndRemove’ for this but they were slower than using a capped collection when I tried them.

4). You can actually still have a flag with the capped collection / tailable cursor as long as you are doing an in-place update (i.e. you can’t add a ‘processed’ flag but you could switch an existing processed flag to true). This gives you more certainty in what has and hasn’t been processed in order to re-start the cursor at the expense of having to do an update for every read (which, to be fair, the example above is doing to maintain the last-read id). I don’t think you need an index for this either.

Hope that helps, thanks for the interesting discussion!

Have you had any problems with out of order messages getting lost? I ask because the docs say that ObjectIds should be in “mostly increasing” order, but does not guarantee it. In my application, I have many rows being inserted per second, and I’ve noticed that while the order is generally increasing, it is definitely not consistent. Therefore, every time my process restarts, and the query runs again, I lose some rows and re-process others (since some rows that were previously before the last id are now after and rows that were previously after are not greater than the last id, so they are excluded from the find).

A capped collection with a tailable cursor seems to be the perfect fit for processing lots of rows as they come in, but I can’t seem to come up with a way to do it losslessly, since tailable cursors only support natural order. It would be nice if there was “resume” operation that would let me set up a tailable cursor directly from an id.

My options, as I see them are:

1) Tailable cursor. Advantages: keeps collection at a fixed size, easy/fast to process rows as they come in. Disadvantages: Ids are not necessarily in increasing order so I lose some every time the query is run again (using > last_id as a query param). Cannot order by id since tailable cursor forces $natural sort only.

2) Tailable cursor with no find parameters. Advantages: stable. Disadvantages: Have to scan through whole table to get to last processed id every time query is run. Skip() doesn’t help as it scans the whole table anyway, plus I don’t know how many rows to skip.

3) Regular cursor, periodic batch queries, order by increasing id. Advantages: stable. Disadvantages: requires index for select to be fast, which slows down inserts (I need these as fast as possible).

4) Use a flag on new rows, modifying it after its processed. Advantages: stable. Disadvantages: Now have update calls in addition to inserts and finds. This definitely needs an index.

Anybody have any thoughts or suggestions? Thanks!

It should be possible to create a .NET 3.5 version though if someone wants to fork it or submit a patch.

The nuget package only works with 2010