ByteEasy

I have some code that processes tweets, about 5 million a day, in realtime. They are currently stored in mongodb and also posted on various celery/rabbitmq work queues. The average message size is 5524, so encoding and decoding these messages is an issue. Using the following test code below. Standard Tweet message Encode/Decode with python built-in json package. message size Test Msg Size De-serialize Obj Size Serialize cjosn, bson, ujson Storage Storoge Cost Empty object 41 14.790, 37.565, 0.970 54 6.341, 41.856, 1.249 2050Mb 0.21 {} Empty list 41 15.069, 38.021, 1.005 54 6.675, 41.475, 1.400 2050Mb 0.21 [] Object of objects 843 107.750, 145.440, 25.525 3226 63.555, 828.235, 28.051 42150Mb 4.21 List of lists 563 58.805, 81.950, 16.960 104 43.426, 815.965, 18.311 28150Mb 2.81 Object with only tweet id 93 25.030, 53.360, 2.280 422 23.570, 83.445, 3.295 4650Mb 0.47 Full tweet message 4386 697.221, 867.780, 188.560 12606 360.290, 5847.335, 201.610 219300Mb 21...

Last week  I stuck my head out  in a meeting and declared that XML is verbose and slow to parse and that we should move to something like Google's protocols buffers,  or something readable such as json or YAML, which are  easier to parse etc etc etc! Well is this really true ? The statement seems logical considering how verbose XML can be. Still, after the meeting, some questions stayed in my mind. So I thought I would do some tests. I used  a FIX Globex (CME) swap trade confirmation message to test my theory. Size from Python to Python json cjson 2332 0.222238063812 0.0943419933319 pickle cPickle 1778 0.233518123627 0.128826141357 XML cElementTree 2083 0.407706975937 2.77832698822 json simplejson 2332 3.37723612785 5.11316084862 So this simple test shows that using XML with cElementTree parser  is not so slow, cjson wins in speed and the conclusion must be: Your performance will ultimately depend on your data and the quality of the l...

Cluster GPU Quadruple Extra Large 22 GB memory: 22 GB EC2 Compute Units: 33.5 , GPU: 2 x NVIDIA Tesla “Fermi” M2050 GPUs, 1690 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet cores each: 448 os: CENTOS 64bit Monte Carlo on One Telsa Device Options : 256 Simulation paths CPU GPU Time (ms.) options/sec. Time (ms.) options/sec. 262144 6000 42 3.586 71388 Monte Carlo on Two Telsa Devices Options : 256 split across two Tesla boards Simulation paths CPU GPU Time (ms.) options/sec. Time (ms.) options/sec. 262144 6000 42 3.405 151999 TOTAL Cost: $0.04 including building the environment and sample code from scratch. CUDA Device Query (Runtime API) version (CUDART static linking) There are 2 devices supporting CUDA Device 0: "Tesla M2050" CUDA Driver Version: 3.20 CUDA Runtime Version: 3.10 CUDA Capability Major/Minor version number: 2.0 Total amount of...

MacPro 2x3Ghz Quad-Core Intel Xeon 16GB 667 Mhz DDR2 Drive: ST31500341AS Capacity: 1.5 TB (1,500,301,910,016 bytes) Model: ST31500341AS Revision: SD17 Serial Number: 9VS0A3HN Native Command Queuing: Yes Queue Depth: 32 Removable Media: No Detachable Drive: No BSD Name: disk0 Rotational Rate: 7200 Medium Type: Rotational Bay Name: Bay 1 Partition Map Type: GPT (GUID Partition Table) S.M.A.R.T. status: Verified Volumes: File System: Journaled HFS+ BSD Name: disk0s2 WRITING 12.4665911198 1024000000 82.1395351915 MB/s On first run. READING 1.72446203232 1024000000 593.808376647 MB/s READING 1.66705989838 1024000000 614.255073256 MB/s READING 1.66696095467 1024000000 614.291532824 MB/s Lenovo Think Center Intel Core i5 650 @3.2 Ghz 3.19 GHz, 2GB Ram Hitachi HDS721025CLA382 Windows XP (32 bit) 1st Time READING: 58.547000, 2095736020, 35.7957 MB/s 2nd Time READING: 1.516000, 2095736020, 1382.4115 MB/s obvious...

Macbook Air 1.86 Ghz Intel Core 2 Duo 2 GB 1067 Mhz DDR3 GeForce 9400M Total amount of global memory: 265945088 bytes Number of multiprocessors: 2 Number of cores: 16 Monte Carlo Options : 256 Simulation paths CPU GPU Time (ms.) options/sec. Time (ms.) options/sec. 262144 8000 32.6 245.8979 1041.08 131072 4000 64 127.68 2005 65536 2000 128 63.12 4055.57 I was thinking of building a big GPU box does anyone have any ideas ? I'm thinking of getting: EVGA Classified SR-2 (Super Record 2) 270-WS-W555-A1 LGA 1366 Intel 5520 SATA 6Gb/s USB 3.0 HPTX Intel Motherboard. Adding 48 Gig and then plugging in 4 GeForce GTX 480 ??