Batch Processing with NHibernate

July 17th, 2008

I have a project where I need to process every night large amounts of data. Basically I have to fetch a lot of objects and related collections from the DB using NHibernate, operate on them, and then do some updates. This process usually takes about 2-3 hours.

Today I discovered a way to brutally increase the application speed. I noticed that the application started fast, but would gradually become slower. I didn't know why, so I tried to free already used objects and splitting the process into several separate instances. This helped only a little. I'm doing something like this (we use a facade on top of NHibernate, so: different types, but the same concepts/names):

PLAIN TEXT

C#:

public static void Operate()
{
using( IPersistanceSession
session = GetNHibernateSession(MasterSession) ) {
IList list = GetALotOfObjects();
foreach( object obj in list ) {
Operate(session, obj);
}
}
}

This code is very slow. Some time ago, I had to go into NHibernate source code to understand how I could use sessions with ASP.NET, and I noticed that NHibernate uses a lot of internal objects. Well, on this case, I am fetching a lot of objects that have no relation with each other, so, I guessed I could clear the session from time to time... And adding a clear, boosted the speed considerably:

PLAIN TEXT

C#:

public static void Operate()
{
using( IPersistanceSession
session = GetNHibernateSession(MasterSession) ) {
session.Clear();
IList list = GetALotOfObjects();
foreach( object obj in list ) {
session.Clear();
Operate(session, obj);
}
}
}

I really can't explain this as my knowledge of NHibernate internals is minimal, but believe me when I say that Clear made a difference... a 2/3h to 10 minutos difference!

Posted by Pedro Santos

Filed in .net, nhibernate