To do: Wait a second, I thought we were using pbkdf2. We're not. We should be. happstutorial depends on HAppSHelpers, which depends on pbkdf2, which is in a tenuous state. (see cocd review from dominic steinitz, which needs to be acted on.) So make pbkdf2 shipshape. regarding pbkdf2, we could have a chapter called security saying "you'll notice that when you log in, register a new user, or change a password on an existing user, there's a second or so lag, whereas other profile changes are more or less instantaneous." The reason for this is that login security is now based on pbkdf2<>, which is more or less the cryptographic gold standard for password based security. That one second or so lag prevents a dictionary attack on our user passwords, even if the random function that initializes the password salt is compromised. This is obviously overkill for a tutorial, but it gives me a warm feeling to know that web apps based on the tutorial template will have strong security out of the box." email verification on registration Regarding macid stress tests, see lemmih's comments in googlegroup on using a different kind of map that allows more efficient use of memory, more users, more jobs, in stress tests. See how much mileage I can get out of this. In macid migrations, examples, practice what I preach. Don't use types in the examples files. I think this whole chapter is a bit rambling and disjointed due to the reference of experiences on private code. See if this can be made better. See if can have two states for real in StateVersions. Suggest that user implement a third state as an exercise, based on looking at how the first two migrations worked. make depend on hackage package HStringTemplateHelpers (which should be uploaded to hackage) migrate improvements from darcsweb diff Misc diff MiscTemplateStuff, or whatever it's called error handling for new user, new job, in ControllerPostActions, newUserPage, processFormNewJob and corresponding state files (remember to say "job" instead of "repo" in error messages) eg SerializeableUsers.hs writeFileForce can be deleted and use what's in HSH.Helpers AppStateSetBased is a poor name, since it's really a Data.Map now. in processformeditconsultantprofile: -- to do: verify this handles errors, eg try writing to someplace we don't have permission, -- or a filename with spaces, whatever Result: after bytestringification, I can insert 1000 users in 30 seconds on laptop. (I think I could already do this, but maybe not as fast.) Checkpoint file after first 1000 users is 17M. Is this an improvement? After 2000 users, went out of memory when creating checkpoint on laptop. Lemmih had 2119 bytes/insert when still using strings, for 200 users. I have 809 bytes, for 1000 users. for 200 users? Same, 809b. Is the situation any better on linode? I can get up 6000 users, but no huge difference which is to be expected as specs are similar. -- could bytestring help in terms of display time for tables with a lot of data, eg all jobs? strangely, performance is much better on live server, at linode: I can insert in batches of 1000 at a time, and it's fast However, I can't insert 2000, it gets killed by the linode uml hypervisor Also, if I try to view jobs for 15000 users, it gets killed by the hypervisor. Stress Test for atomic inserts, all jobs at once Completed Users 13925 to 14924 have been inserted. Stress test time: 13 secs Users 21 to 1020 have been inserted. Stress test time: 8 secs However, it concs out soon after if I try a 10000 user insert insertuAllJobs, added user: user1127 Killed insertuAllJobs, added user: user20096 atomic inserts, all jobs at once stresstest, 1000 users, elapsedtime: 38 secs insertuAllJobs, added user: user21096 atomic inserts, all jobs at once stresstest, 1000 users, elapsedtime: 15 secs at insertuAllJobs, added user: user21166, seems to be stuck. not inserting more users, but not being killed by the hypervisor either. shit, maybe that's because I inadvertently requested a page that caused a big query action. try agian. How much does more ram cost? Linode has ~3GB ram servers available for $150/month. What about dreamhost? scam offers of 8GB ram vps for $200/month but they're... well... scams Group q: Can HAppS be tweaked to create a checkpoint after every, say, 1000 transactions? In the snip below, nheptane shows how to create a checkpoint after every orderly shutdown. But if you have a long-running process (days) with a lot of data to checkpoint, and your server power goes out, my understanding is that you lose all that data. v5: v5 announcement: Real World HAppS Tutorial Version 5: Macid For Dummies. In this release of the HAppS tutorial I added a lot of material on macid, plus a few other miscelaneous topics. New chapters include: introduction to macid, using macid safely, inserting dummy data using macid, macid updates and queries, speculations on the scalability of macid <,and macid migration?>. Also displaying utf8 correctly, get and post processing, and debugging. Also file uploads, which enable us to have user profile pictures. resolve stress test issues Amazon ec2, 15GB, pay for an hour. See if it makes any difference. see if I can convincingly demo 100,000 users and 2000 transactions/sec. For one thing, users pages now have to paginate (developers, and also developers wanted) notes... after 1000 user insert (one big, but don't think it matters) thartman@thartman-laptop:~/happs-tutorial>ls -lth _local/happs-tutorial_state/ | head -rw-r--r-- 1 thartman thartman 22M Oct 14 14:21 events-0000000101 top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND 23961 thartman 20 0 443m 185m 2132 S 0.0 38.1 1:24.28 257m happs-tutorial PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23961 thartman 20 0 443m 185m 2132 S 0.0 38.1 1:24.28 happs-tutorial google analytics should track happstutorial, fix cookies issue. assuming stress tests passes convincingly, we're going to build for-profit sites with this. update macid-data-safety with fact that seems one macid file gets written per startup/shutdown, after grokking what nheptane has to say on the subject, fix tk places Could point people to job results pagination, for explaining the FromData instancing mechanism. Note that data here means query data, not appdata. tutorial page: reading query data reading macid data (have to justify calling this macid?) leave table pagination for zzz's job posts as an exercise for the reader. whither happs: http://groups.google.am/group/HAppS/browse_thread/thread/8ca61508664d4fb5 (unanswered questions) could move migration q&a in maciddatasafety to new page on migration. HAppS state links: http://groups.google.com/group/HAppS/browse_thread/thread/bcfb81b12b0f9c67/fd9800cba6ba6fa6?lnk=gst&q=love+the+idea#fd9800cba6ba6fa6 (love the idea) http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=60639 (prevayler thread) Explain how to migrate data. (Or maybe just make the link gray and move to v6, this stress testing thing is quite involved.) See if can have less methods created with mkmethods but still have everything work. I'm guessing this would make migration simpler. (Why? Gut?) v6 Clean up main (ControllersBasic has a lot of cruft from when I had a less good understanding of happs than now, the examples may be more confusing than they have to be.) Ask nheptane for help running his HAppS+HSP stuff. Can he just tar up and send me what he has? In return, I will make sure this is cabal installable as a quickstart for anyone who wants to use what he has. Acknowledge Jeremy Shaw (nheptane) in thanks, saying specifically what he helped with. He's got import HSP.Google.Analytics. Can I use that? What does hsp have to offer me if anything? Q: Can the types of "creeping model" problems Terrence TK (ST creator) describes in TK happen in HSP? Try to demonstrate this, and if so argue that this is an advantage of ST over hsp... that ST keeps the view code pure. In thanks: The HAppS devs, obviously: Alex Jacobsen and LemmiH, who answered many questions helpfully on #happs and the googlegroup... did I miss anyone? jobs should definitely be a set, not a list. otherwise you can have two jobs with the same name (maybe part of v5 data cleanup) instance fromData places, double check form fails safely if the user inputs bad data. (try to break the form by behaving like a naughty user) Get rid of handlers as monoids. (In explanation, though can leave them in as code.) Address in-memory (macid transactional query/update) datastore, versus blobs that can be stored on filesystem (IO read), where transactionality is not required, nor guaranteed. menubar: colsize is wrong, search web on correct attribute more idiomatic use of ST for table output ST can play nicely with html tables... right? review email from sclv v7 Make a social accounting system. hstartup-template on hackage? (How about Startup Templates as a for-profit business venture?) hsocial-startup-template? email sclv patches for directoryGroupSafer, setManyAttribsafer A low priority thought: There is aproblem if you are on a page other than home and log in incorrectly -- it switches to error page, whereas it would be better to continue showing existing content and just show the error in the login box. I tried to do this and it turned out to be a lot harder than expected, and after many unproductive hours, I gave up. I think there's an underlying problem here that the fix was not playing to HAppS's strenghts. Partly because we're serving behind apache mod rewrite, and partly because logging in involves a tweak around a HAppS redirect, it's hard intellectually to comprehend what's going on and a lot of time gets lost setting debug traces and generally trying to understand. HAppS's strength is the type system. Being able to think in terms of requests and responses and reason logically. We aren't doing enough of that. HAppS weaknesses: weak community, weak documentation, weak leadership, few working demo apps apart from happs tut, dominance of ruby/python/php.