Documenting a well known "bug" (design flaw) -- blocking db calls in async io context
Forum » WikiPBX / Announcements » Documenting a well known "bug" (design flaw) -- blocking db calls in async io context
Started by: tleydentleyden
On: 1242085522|%e %b %Y, %H:%M %Z|agohover
Number of posts: 2
rss icon RSS: New posts
Unfold All | Fold All | +More options
Fold
Documenting a well known "bug" (design flaw) -- blocking db calls in async io context
tleydentleyden 1242085522|%e %b %Y, %H:%M %Z|agohover

WikiPBX uses blocking database calls in a twisted webserver

This is basically a scalability bug that can have a negative impact when the server runs under a heavy load. The bug will be more apparent if there are long running queries or the database is slow for whatever reason.

The problem is that the webserver is running in a non-blocking io environment (eg, twisted web2), but the code makes blocking database calls in many places. So if a database call takes a long time, the whole server will "lock up" for new requests until the long running database call unblocks.

The best solution is to move away from using twisted and non-blocking io and use a more traditional multi-threaded/multi-process webserver, eg, mod_wsgi running under Apache2 or nginx.

Reply  |  Options
Unfold Documenting a well known "bug" (design flaw) -- blocking db calls in async io context by tleydentleyden, 1242085522|%e %b %Y, %H:%M %Z|agohover
Fold
Re: Documenting a well known "bug" (design flaw) -- blocking db calls in async io context
stas_shtinstas_shtin 1245227654|%e %b %Y, %H:%M %Z|agohover

That's not quite correct. Wikipbx runs every request in its own thread - see twisteddjango module and "grep thread -i". So the blocking DB calls are running in there own threads.

I think that the problem of slowing down on heavy load may be caused by the small default thread pool size. If you can test it with default thread pool size and increased as described in twisted threading docs, that may give interesting numbers.

But decoupling django and twisted is still a better solution here.

Reply  |  Options
Unfold Re: Documenting a well known "bug" (design flaw) -- blocking db calls in async io context by stas_shtinstas_shtin, 1245227654|%e %b %Y, %H:%M %Z|agohover
New Post
page_revision: 0, last_edited: 1226259295|%e %b %Y, %H:%M %Z (%O ago)
EditTags History Files Print Site tools+ Options
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License