October 29th, 2009
We’ve been using Mono for the production server of Orion’s Belt for a couple of months now. On this article I’d like to share our experiences using Mono. We developed our project fully on Windows with Visual Studio 2005, and at a time we started to consider Mono for a production server. Do note that the Orion’s Belt team has a Windows background, and little experience administrating Linux machines.
Here are the web applications that we are serving with Mono:
- orionsbelt.eu - The institutional website, almost all static ASPX, very little dynamic content
- s1.orionsbelt.eu - A game server, the most complex web application from our suite
- manual.orionsbelt.eu - The manual: +300 aspx pages statically generated per language
- lang.orionsbelt.eu - Translation project, simple ASP.net application
Step 1 - Preparing the Server
The game is running on an Ubuntu server. We downloaded the source code and installed it manually. We could have used the packages, but that isn’t versatile enough. Packages aren’t always up to date and we find them hard to manage. For example, how could we have two versions of Mono installed and choose which to run? How could we always have an up to date version? To fix this problem we chose to install by source, having as a guideline the Parallel Mono Environments article. This was great because we could change to use Mono from SVN or a specific version, just by changing some variables. We wouldn’t want to install a new version, getting problems, and having a bad time to rollback.
We use FastCGI with Nginx to serve the game. Nginx’s really cool, and very easy to configure and manage. We also installed MySQL. Not considering Mono, all the other necessary software was very easy to install and configure, with the help of Google of course. We managed to get the server displaying ASPX pages easily.
Step 2 - Running the Application
The Orion’s Belt project is fairly big, and after some months coding fully on Windows, the move to Linux was peaceful. We had some file case issues, but that was it. We have a NAnt script that creates a deploy package, and we were able to upload it to the mono server and run it. There were some problems at that time, sometimes Mono’s web server would throw compile errors while compiling ASPX pages. But fortunately, at that time, Mono implemented Precompiled ASP.NET web sites in Mono. We incorporated a step on NAnt to precompile the deploy package and everything was faster and we didn’t get those errors anymore.
Other problem we had was the mono web process and the resources it used. Going to 600-900Mb RAM and wasting a lot of CPU, even at idle time. So we started to kill the process from time to time. Sometimes the process would die unexpectedly, so we also started to use supervise, to supervise mono’s process.
There are also some other issues we got along the way: touching Web.config isn’t as stable as it should be. And also when we deployed new versions, mono would not behave properly, it would shutdown or just stop responding. So, we got used to just kill mono when we deployed or needed a reset. It’s very easy, you just kill mono’s process, and supervise will bring it up.
Supporting multi OS makes your code better
It may sound weird, but it’s true. We had a lot of bugs showing up only on mono. For example, we use NHibernate and everything worked fine on Windows, but on Linux sometimes it didn’t. We found out that we needed a flush here and there. On Windows it worked, but on Linux it wasn’t that permissive.
Using Mono brought an interesting mindset to the team. Every time there was a problem, we’d blame it on Mono. But the majority of times, it was our code that wasn’t up to it.
The Linux issues also made us create specific guidelines for file case, forced the use of Path.Combine and related methods. We also tried MoMa, the mono’s problem reporting tool, but we didn’t find it to be that useful on our situation.
Running the Tick
The game’s tick runs every ten minutes. It’s a very heavy process that loads a lot of data from the database, operates on it, and then persists it. This process needs a lot of RAM and CPU to run, and it’s a good performance test. On this specific process we find Mono to be very lacking. If running the process on Mono would take 60 seconds, running the same process on the windows development machine, connecting to the production database, would take 30 seconds. And the development machine is worst than the production server.
However, for the cost of a Windows license, we could buy a great machine just for tick processing. Would it be worth it? We don’t know at the moment.
Preparing the Mono environment was fun and interesting, and the issues we got from porting the code were minimal. We did need some help, and I find the mono list not that friendly to newcomers, but Google provided the help we needed. Even so, it’s not easy for developers without experience administrating Linux machines to prepare mono. There are always some issues here and there, that we’d know how to fix on Windows, but that we loose a lot of time figuring it out on Linux.
Although mono’s behaving nicely most of the times, we don’t find it as stable as a Windows machine. Even so, it’s a great option, that’s for sure. I already have a slicehost account with an ubuntu+mono running all my private ASP.NET sites. It’s cheaper and runs really well.
But for the production server for the game, we aren’t convinced yet if we should continue using mono or not. Maybe we’ll release another server using Windows and then we’ll have a good performance comparison showcase.