How do I deal with this race condition in django?

This could be an off-shoot of the same problem as here:

Why doesn't this loop display an updated object count every five seconds?

Basically get_or_create can fail - if you take a look at its source, there you'll see that it's: get, if-problem: save+some_trickery, if-still-problem: get again, if-still-problem: surrender and raise.

This means that if there are two simultaneous threads (or processes) running create_or_update_myobj, both trying to get_or_create the same object, then:

  • first thread tries to get it - but it doesn't yet exist,
  • so, the thread tries to create it, but before the object is created...
  • ...second thread tries to get it - and this obviously fails
  • now, because of the default AUTOCOMMIT=OFF for MySQLdb database connection, and REPEATABLE READ serializable level, both threads have frozen their views of MyObj table.
  • subsequently, first thread creates its object and returns it gracefully, but...
  • ...second thread cannot create anything as it would violate unique constraint
  • what's funny, subsequent get on the second thread doesn't see the object created in the first thread, due to the frozen view of MyObj table

So, if you want to safely get_or_create anything, try something like this:

 @transaction.commit_on_success
 def my_get_or_create(...):
     try:
         obj = MyObj.objects.create(...)
     except IntegrityError:
         transaction.commit()
         obj = MyObj.objects.get(...)
     return obj

Edited on 27/05/2010

There is also a second solution to the problem - using READ COMMITED isolation level, instead of REPEATABLE READ. But it's less tested (at least in MySQL), so there might be more bugs/problems with it - but at least it allows tying views to transactions, without committing in the middle.

Edited on 22/01/2012

Here are some good blog posts (not mine) about MySQL and Django, related to this question:

http://www.no-ack.org/2010/07/mysql-transactions-and-django.html

http://www.no-ack.org/2011/05/broken-transaction-management-in-mysql.html


Your exception handling is masking the error. You should pass a value for state in get_or_create(), or set a default in the model and database.

Tags:

Django

Innodb

Recent Posts