Sunday 15 June 2014

multithreading - Update SQLAlchemy field threadsafe from multiple processes -


I have a database table that describes a job that is processing on a distributed system. Therefore, the whole work is divided into several small parts and each part is processed on a different machine in its own environment.

I want to keep a simple log of how each process is displayed, so each node is connecting with SQLAlchemy to the same work table and its job number till the end of a Unicode text field And adds the situation. >

I hope to see something like this:

  Part 1: Completed & lt; Br / & gt; Part 2: Completed & lt; Br / & gt; Part 3: Error & lt; Br / & gt; Part 4: Completed & lt; Br / & gt; ... etc.   

I'm just doing a code process.log + = "part 1: full \ n" on each node when it finishes processing Does. Then a session.commit ()

I know that part number will fall out of order and it is fine for my purposes, but what is happening instead That I am only getting some entries of the total parts; I think that is because many jobs are ending at the same time and appending + thread is not safe.

I am not good with the session, so I have all the tricks to configure it, I tried to use a NullPool without any profit.

Is there any way to secure the endand operation thread in such a distributed environment?

If you want to update one - I believe - text field in parallel < / Em> It is running internally: Each example reads the original field, the object takes care of immediately ( sessionmaker , hence an example of a data-holding unit for each thread) And adds its part to the version of its field. After this, their own incidents occur. What you get as a result - Completely Thread-Safe - Previous Edition Last Object Written Wins

If you want to all commit your data to the same area with the this data structure, then you have to throw a profit of threading , And synchronize instances of the organization to read and write, so that each thread can wait for the data to be read (for this) until its predecessor arrives.

You should do do this: to convert the advantages of a SQL database to usages in the data structure and each your SQL Update the record.

No comments:

Post a Comment