ByteEasy

I have some code that processes tweets, about 5 million a day, in realtime. They are currently stored in mongodb and also posted on various celery/rabbitmq work queues. The average message size is 5524, so encoding and decoding these messages is an issue. Using the following test code below. Standard Tweet message Encode/Decode with python built-in json package. message size Test Msg Size De-serialize Obj Size Serialize cjosn, bson, ujson Storage Storoge Cost Empty object 41 14.790, 37.565, 0.970 54 6.341, 41.856, 1.249 2050Mb 0.21 {} Empty list 41 15.069, 38.021, 1.005 54 6.675, 41.475, 1.400 2050Mb 0.21 [] Object of objects 843 107.750, 145.440, 25.525 3226 63.555, 828.235, 28.051 42150Mb 4.21 List of lists 563 58.805, 81.950, 16.960 104 43.426, 815.965, 18.311 28150Mb 2.81 Object with only tweet id 93 25.030, 53.360, 2.280 422 23.570, 83.445, 3.295 4650Mb 0.47 Full tweet message 4386 697.221, 867.780, 188.560 12606 360.290, 5847.335, 201.610 219300Mb 21...

Couchdb 0.11.0 2x3 QUAD-Core Intel Xeon 16 GB 667Mhz DDR2 OS X 10.6.4 Inserting NUM BLOCK time Bytes Mb/s records/s 1 1 0.0030 1 0.000 330 10 1 0.0251 100 0.004 398 1000 1 3.2415 1000000 0.308 308 10000 1 34.6263 100000000 2.888 289 1000 10 3.4122 10610000 3.109 2931 100 100 1.5112 10601000 7.015 6617 10 1000 1.9464 10600100 5.446 5138 2 5000 2.3308 10600020 4.548 4290 1 10000 2.0068 10600010 5.282 4983 10 10000 15.7176 106000100 6.744 6362 Average Top: 33423 beam.smp 48.9 05:42.89 13 0 62 151- 39M- 264K 33421 CouchDBX 12.9 00:52.85 6/1 3 124- 322 77M- 29M

Using: Intel 2 Duo T9400 @ 2.53 GHz, 2.99 GB Ram HP Elitebook Hitachi HTS723216L9A360 Couchdb 1.0 Local Host Simple ASCII 1K Payload: { "_id": "00081363", "_rev": "1-1c29ecbf7bc15e7f9226a45594a0605d", "payload": "X"* 1024 } Inserting # of writes Block size time bytes Mb/s Records/s 1 1 0.0050 1 0.000 200 10 1 0.0470 100 0.002 213 1000 1 4.5253 1000000 0.221 221 10000 1 48.3623 100000000 2.068 207 1000 10 4.6867 10610000 2.264 2134 100 100 2.1353 10601000 4.965 4683 10 1000 2.4527 10600100 4.322 4077 2 5000 2.2917 10600020 4.625 4364 1 10000 3.4477 10600010 3.075 2901 Extracting ALL with a js view # of reads time bytes Mb/s Records/s 138485 25.7300 157733320 6.130 5382 Extracting with a js view # of reads Block size time bytes Mb/s Records/s 1 1 0.0000 1189 1189000.000 1000000000 10 1 0.0470 11890 0.253 213 100 1 0.4220 118990 0.282 237 1000 1 4.4990 1...