Re: Documenting a well known "bug" (design flaw) -- blocking db calls in async io context

Wed, 17 Jun 2009 08:34:14 +0000

That's not quite correct. Wikipbx runs every request in its own thread - see twisteddjango module and "grep thread -i". So the blocking DB calls are running in there own threads.

I think that the problem of slowing down on heavy load may be caused by the small default thread pool size. If you can test it with default thread pool size and increased as described in twisted threading docs, that may give interesting numbers.

But decoupling django and twisted is still a better solution here.

Documenting a well known "bug" (design flaw) -- blocking db calls in async io context

Mon, 11 May 2009 23:45:22 +0000

WikiPBX uses blocking database calls in a twisted webserver

This is basically a scalability bug that can have a negative impact when the server runs under a heavy load. The bug will be more apparent if there are long running queries or the database is slow for whatever reason.

The problem is that the webserver is running in a non-blocking io environment (eg, twisted web2), but the code makes blocking database calls in many places. So if a database call takes a long time, the whole server will "lock up" for new requests until the long running database call unblocks.

The best solution is to move away from using twisted and non-blocking io and use a more traditional multi-threaded/multi-process webserver, eg, mod_wsgi running under Apache2 or nginx.