Running Mono - an Overview

October 29th, 2009

We’ve been using Mono for the production server of Orion’s Belt for a couple of months now. On this article I’d like to share our experiences using Mono. We developed our project fully on Windows with Visual Studio 2005, and at a time we started to consider Mono for a production server. Do note that the Orion’s Belt team has a Windows background, and little experience administrating Linux machines.

Here are the web applications that we are serving with Mono:

Step 1 - Preparing the Server

The game is running on an Ubuntu server. We downloaded the source code and installed it manually. We could have used the packages, but that isn’t versatile enough. Packages aren’t always up to date and we find them hard to manage. For example, how could we have two versions of Mono installed and choose which  to run? How could we always have an up to date version? To fix this problem we chose to install by source, having as a guideline the Parallel Mono Environments article. This was great because we could change to use Mono from SVN or a specific version, just by changing some variables. We wouldn’t want to install a new version, getting problems, and having a bad time to rollback.

We use FastCGI with Nginx to serve the game. Nginx’s really cool, and very easy to configure and manage. We also installed MySQL. Not considering Mono, all the other necessary software was very easy to install and configure, with the help of Google of course. We managed to get the server displaying ASPX pages easily.

Step 2 - Running the Application

The Orion’s Belt project is fairly big, and after some months coding fully on Windows, the move to Linux was peaceful. We had some file case issues, but that was it. We have a NAnt script that creates a deploy package, and we were able to upload it to the mono server and run it. There were some problems at that time, sometimes Mono’s web server would throw compile errors while compiling ASPX pages. But fortunately, at that time, Mono implemented Precompiled ASP.NET web sites in Mono. We incorporated a step on NAnt to precompile the deploy package and everything was faster and we didn’t get those errors anymore.

Other problem we had was the mono web process and the resources it used. Going to 600-900Mb RAM and wasting a lot of CPU, even at idle time. So we started to kill the process from time to time. Sometimes the process would die unexpectedly, so we also started to use supervise, to supervise mono’s process.

There are also some other issues we got along the way: touching Web.config isn’t as stable as it should be. And also when we deployed new versions, mono would not behave properly, it would shutdown or just stop responding. So, we got used to just kill mono when we deployed or needed a reset. It’s very easy, you just kill mono’s process, and supervise will bring it up.

Supporting multi OS makes your code better

It may sound weird, but it’s true. We had a lot of bugs showing up only on mono. For example, we use NHibernate and everything worked fine on Windows, but on Linux sometimes it didn’t. We found out that we needed a flush here and there. On Windows it worked, but on Linux it wasn’t that permissive.

Using Mono brought an interesting mindset to the team. Every time there was a problem, we’d blame it on Mono. But the majority of times, it was our code that wasn’t up to it.

The Linux issues also made us create specific guidelines for file case, forced the use of Path.Combine and related methods. We also tried MoMa, the mono’s problem reporting tool, but we didn’t find it to be that useful on our situation.

Running the Tick

The game’s tick runs every ten minutes. It’s a very heavy process that loads a lot of data from the database, operates on it, and then persists it. This process needs a lot of RAM and CPU to run, and it’s a good performance test. On this specific process we find Mono to be very lacking. If running the process on Mono would take 60 seconds, running the same process on the windows development machine, connecting to the production database, would take 30 seconds. And the development machine is worst than the production server.

However, for the cost of a Windows license, we could buy a great machine just for tick processing. Would it be worth it? We don’t know at the moment.

Conclusion

Preparing the Mono environment was fun and interesting, and the issues we got from porting the code were minimal. We did need some help, and I find the mono list not that friendly to newcomers, but Google provided the help we needed. Even so, it’s not easy for developers without experience administrating Linux machines to prepare mono. There are always some issues here and there, that we’d know how to fix on Windows, but that we loose a lot of time figuring it out on Linux.

Although mono’s behaving nicely most of the times, we don’t find it as stable as a Windows machine. Even so, it’s a great option, that’s for sure. I already have a slicehost account with an ubuntu+mono running all my private ASP.NET sites. It’s cheaper and runs really well.

But for the production server for the game, we aren’t convinced yet if we should continue using mono or not. Maybe we’ll release another server using Windows and then we’ll have a good performance comparison showcase.




Related Posts

12 Responses to “Running Mono - an Overview”

  1. sulfide Says:

    hey thanks for the post, I was looking into mono myself for cheaper solutions and i’ll be very curious to see how your performance comparison goes provided you get another server using windows.

  2. older Says:

    I found that ASP.NET is the least stable and usable part of mono.
    From my experience running custom services with mono-service2 is much better.

  3. moo Says:

    your website hurts my eye balls. i can’t read this.

  4. Reflective Perspective - Chris Alcock » The Morning Brew #466 Says:

    [...] Running Mono - an Overview - Pedro Santos gives an overview of how a recent project which was deployed on Mono worked, talksing about the setup of the server, and some of the issues they ran into with the applications [...]

  5. Jens Says:

    You may want to correct this er.. typo? “For example, how could we have two versions of Mono installed and choose witch to run?” While running two monos may require some magic, I am rather sure that running “witch” (never heard of that package) won’t help :-) (which ?)

  6. Pedro Santos Says:

    Hello Jens, my bad, thank you for reporting it. :)

  7. Luís Miguel Silva Says:

    Interesting! From my tests, when i compiled and ran the same .NET remoting client on Windows and Linux (with Mono), the Linux client was a lot faster (at least to start up :o)).

    You’re either using some slow APIs or Mono has become bloated!

    Best,
    Luis

  8. The Price of Mono | Boycott Novell Says:

    [...] appear in the world of Mono. F-Spot gets slammed for ignoring critical feedback and one blog concludes with: “for the production server for the game, we aren’t convinced yet if we should [...]

  9. Miguel de Icaza Says:

    Hello Pedro,

    I am sorry to hear about the problems that you have had with Mono. I would love if you could provide us with bug reports on the issues that you had (a bug report ideally would have a small test case) so we could provide fixes for these problems, either for you, or for future users of Mono.

    The memory consumption problem is a known issue that happens with Mono’s garbage collector as it is not a compacting collector, which means that some patterns of memory allocation that are no problem with .NET cause trouble with Mono.

    The pattern is the memory fragmentation pattern: you allocate a very large buffer (for example, lets say 1 megabyte), then it gets collected, then you allocate a smaller buffer, say 1024 bytes from this pool and then you request another megabyte allocation.

    In the above scenario you end up with: 1024 bytes, 10megs-1024 bytes block and a new 10-megabyte block. If you repeat this pattern enough times you end up with a situation like this.

    The only solution to this problem is to avoid allocating very large *transient* objects (This is not a problem for permanent objects, as those will be, permanent, or permanent-ish).

    What version of Mono were you using?

    Miguel.

  10. Pedro Santos Says:

    Hello Miguel,

    I already knew the GC’s issue, but thank you for clarifying it. It’s an important issue, because killing mono it’s not a light task. Our application uses a lot of resources and when mono comes up it need to load some thing. The load process can take about 20 seconds, meaning that we don’t want to kill it very often.

    We’re using mono from last March’s SVN:
    Mono JIT compiler version 2.4 (/branches/mono-2-4/mono r130583 Mon Mar 30 23:10:23 WEST 2009)

    It’s old, but when we found this stable version, we sticked to it.

  11. eryeryery Says:

    If it works bad, please stop using it.

    And don’t use aspx!!
    They don’t work good in all browsers.
    In fact, most of the bug reports in Opera are aspx pages bad behaviour!!

  12. Niels Says:

    Hi Pedro,

    We are also experiencing the same issues when using mono with Nginx (using FastCGI).

    For what it’s worth we don’t have the same issues using Apache2 with Mod_mono:
    http://www.mono-project.com/Mod_mono

    Although we would prefer to use Nginx as it uses less memory and doesn’t create a thread for each connection we decided to stick with apache2 for now. We are using the latest 2.4 mono release.

    If you have been able to find a fix for this issue than I would love to hear about it.