So I am asking myself this question. +
So I asked myself this question. hunk ./templates/macidstresstest.st 11 +$!
Like all the world's great businessmen, I then pulled some numbers out of my ass. !$ +
What will make money in the real world is impossible to say until you try it, + but I attempted to come up with some not-completely-ridiculous estimates anyway. + I figure for a typical ad-supported web 2.0 type project, each user is worth \$0.50 cent/year. + So if you have 100,000 users, you are probably not rolling in dough, but hopefully you can at least pay your rent. + +
Therefore, once I had the basic state functionality working I decided to see if I could scale my toy job board + up to 100,000 users. To make things more realistic, each user would have 200 jobs. + So, this would be the equivalent of 20,100,000 data records if we were using an rdbms: + 100,000 users and 20 million jobs. + +
I failed to achieve this goal. + +
Or, more optimistically, + I am declaring defeat for the time being so I can put a version 5 release up on hackage + and share what I learned with a wider audience. In a future release, I still intend to show how you can scale + a HAppS app up to large numbers of users! :) + +
The most important lesson I learned is that putting all application state into macid won't work if you + have a lot of data -- not even if you have one of those heavy iron + boxen from amazon web services, or for that matter a whole virtual data center full of them. + 16GB of RAM gets used up surprisingly fast if you are using ram for everything. + The cloud is no substitute for + thinking your application's transactional requirements through + and coding to them. + +
So, if I make good on the promise I made above about scaling this toy application up to 20,000,000 data records, you + will be getting some future version of this tutorial where macid is being used + in conjunction with disk based storage: either an rdbms or haskell data values serialized to disk, + I haven't decided yet. + Assuming I can get this working, my intuition is that this solution will be + scale up to a lot more than my original goal of 100K users, + and do so in a cloud computing-friendly way, so stay tuned. +
+ +
Another lesson learned is that the macid data structure can have a dramatic effect on application scaleability. + In version 4 of happs-tutorial, I was using sets to store users and a linked list for jobs. Lookup of user/job values + required finding the first value in the set where the appropriate record field matched the desired criteria. + I also had some quirk where modifying a job value resulted in writing the entire user (with all attached jobs) + to state, which was resulting in huge serialization files. (Thanks to LemmiH on the happs group for diagnosing this.) + When I switched to using maps for macid state, + with UserName and JobName values as keys, and was more careful about serialization, I had dramatic performance gains. + Not dramatic enough to get me up to 100,000 users, true -- but initially I was seeing my server slow + to a crawl after inserting a couple hundred + users, and now I can handle 10,000 users comfortably on an 8GB box. I suspect that the macid state I am using now + can be optimized even further -- though at some point, as I said, algorithmic finesse won't help anymore and + part of the state will have to be moved out of RAM and into hard disk. + +
Finally, I created what appears to be the first macid stress test in the public domain. I actually created 3 tests, all of which are activated by clicking a link with an integer argument in the url, that specifies how many users to insert. In all cases, there are 200 jobs inserted for each user, the equivalent of 4200 records in a tradiational rdbms. + +
+ Before running any of these stres tests yourself, I recommend limiting the amount of swap your happs server
+ is allowed to use via the ulimit command. Otherwise stressing HAppS will result in your computer grinding to a crawl.
+ But if you limit your swap, the process just gets killed, which is a lot less annoying.
+ Since every macid action is serialized, you will still see the data your stress test
+ inserted up to the point it was killed when you reestart HAppS. I have
+
+ ulimit -v 262144 # (256M of swap)
+
+ set in .bashrc.
If you click on stresstest/atomicinserts/20 the result is that 20 users + get inserted, each with 200 jobs: about 4000 macid transactions total. + Each job insertion is its own event, so macid winds up having to do a lot of work. + This is useful for seeing how effectively happs deals with getting hammered by insert transactions, + but it's impractical for inserting large amounts of data. + +
If you click on stresstest/onebiginsert/20 + the effect is the same, 20 users and 4000 jobs get inserted, but the insertion is performed with one big + macid transaction. Onebiginsert is faster than atomicinserts for small number of users, but it won't work + with 1000 users, all I got was a churning hard disk that eventually had to be kill -9ed. + From this I conclude that large maps don't play well with serialization. + +
I found that what worked best in practice for inserting large quantities of data + was stresstest/atomicinsertsalljobs/20. + Here, there is one insert per user, but all jobs get inserted at once. + Using this method of insertion, I was able to insert 1000 users at a time on my junky old laptop, + and 4000 users at a time on a heftier server with 8G of ram. On the hefty server I was able to get up to + 20,000 users (4 million jobs) before performance degraded because of the amount of data being kept in ram. + + hunk ./templates/macidstresstest.st 94 +$! hunk ./templates/macidstresstest.st 113 -
In what's written below, ControllerStressTests.hs - has the implementation of the tests. - -
In the macid dummy data chapter, we inserted - two users and two hundreds jobs in a single transaction -- the set up of users got transformed - in a single update. - -
What happens if we push harder? - -
If you click on stresstest/atomicinserts/20 the result is that 20 users - get inserted, each with 200 jobs: about 4000 macid transactions total. - Each insertion is its own event, so macid winds up having to do a lot of work. - -
If you click on stresstest/onebiginsert/20
- the effect is the same, 20 users and 4000 jobs get inserted, but the insertion is performed with one big
- macid transaction. This is faster, but it's less realistic. A web app will typically grow with lots of little
- transactions. And it's only faster for small numbers of users:
-
-
Hm... better than O(n) performance for small numbers of users, worse for larger.
-
one big insert stresstest, 20 users, elapsedtime: 5 secs
-
one big insert stresstest, 200 users, elapsedtime: 12 secs
-
one big insert stresstest, 1000 users, elapsedtime: 446 secs
-
with 200 users it was ok (12 seconds), but with 1000 users seems to be churning ram and still going after 2 minutes
-
Perhaps if you're inserting large amounts of data, better to break things up, eg 1 insert per user,
-
and insert includes inserting dummy jobs
-
-
Then there's stresstest/atomicinsertsalljobs/20. - Here, there is one insert per user, but all jobs get inserted at once. Can I insert 100000 users this way? - - -
- (Note: For the above stress test to work on your local development machine, you have to modify the - onHomeLaptop function in MacidStressTests.hs. - This is a one-off I put in to prevent the stress test from running on my live - server. It's simple, just follow the suggestion in the comment.) -
Result for me (notes taken during the experiment):
-
Okay, started 0:44.... inserting about 1 user every 5 seconds. chugging along... slowly...
- I'm not seming multiple event files getting created, instead one very large event file is growing under _local.
- Perhaps there's an event file for each time happs stop-starts?
- Five minutes later, we've inserted 100 out of 200 users and my laptop is VERY hot.
- There's only 3 files under happs-tutorial_state (we started from pristine, no users no jobs) but
- one of the files is enormous. 15 minutes later the insertion is completed.
-
-
thartman@thartman-laptop:~/happs-tuorial/_local>ls -lth happs-tutorial_state/events-0000000000
-
-rw-r--r-- 1 thartman thartman 356M Oct 5 00:54 happs-tutorial_state/events-0000000000
-
-
Seems like a lot a lot of hard disk for what would amount to about 40,000 data records, if we were - using sql instead of macid. Does that mean that if I had a million users (monster.com) - I would need 356M * 5000 = 178G of hard disk space... and also that much ram? - I think we have terabyte disks now, but I haven't heard of terabyte ram -- maybe in the google cloud? - -
Even granted loads of ram there's another problem, maybe related. - -
If I shut down and restart happs, it's slow to start. - -
time wget http://localhost:5001 shows two minute page retrieval page retrival for the home page, - which doesn't even call macid. When I attempt to re-retrieve that or other pages, it continues to take - minutes -- so it's not a startup issue. Overall my laptop is very sluggish. HAppS seems to have stolen - all my ram. - -
When I repeated the experiment with less ambitious stress tests, eg click /tutorial/strestest/10 - or even just a few (2-3) users at a time, inserts were also very slow, and I experienced the same slowness - starting happs and retrieving the homepage. - -
40,000 some-odd data records doesn't seem all that big to me for a web-app, and the real limit appears to be - even lower than that. - -
Granted, I thought too hard about my data structures, and pretty much used the first thing - that popped into my head just to "get coding". Could I be seeing better performance if I used smarter - data structures? - -
Are there other issues I need to think about? - -
Or is it back to databases if you want to build monster.com? - -
If I'm going to be using happs in real applications, which I would like to do, - getting happs-tutorial to be able to run these types of stress tests without feeling - an overwhelming urge to call - pkill ghc and free up my laptop ram is essential. - hunk ./templates/macidstresstest.st 115 -you wind up with over - 40,000 files in the _local directory. (I think.) - You can view them here, - but it will take a while for the page to load in firefox. Maybe better to cd into local and - do - -
find | wc -l - - - - - hunk ./templates/macidstresstest.st 126 + + hunk ./todo 2 + +Lemmih: +> As a side note, you're filling up the memory with String's. They +> require at least 12 bytes per character. Using a different +> representation would be prudent. +me: Wow, 12 bytes. I'll definitely switch to something else. What would you recommend? Bytestrings? + +-- could bytestring help in terms of display time for tables with a lot of data, eg all jobs?