Read the full story

OpenSSL Love

November 2nd, 2009

Just got to an article that says everything I’d like to say about OpenSSL. It makes me think about the definition of software quality. At first sight, OpenSSL’s code seams a mess, but the point is: it’s one of the most used libraries out there, it works, it’s stable and it’s reliable. But using it is a major PITA.

Cross-platform .NET

November 2nd, 2009

I’ve had a lot of feedback on my running mono article. The majority of feedback was something like: .NET isn’t cross platform, Mono is evil, MS is evil. How can someone say to me that .NET isn’t cross platform when I have an average sized product running out of the box on MS .NET and Mono? I used to participate on a lot of discussions about this a couple of years ago. But then I realized that it was more productive to actually do something instead of discussing these kind of issues.

What makes a Java application cross-platform? Will a Java application be cross-platform if use reference resources like: c:\MyApp\MyApp.ini? And if we have a file named MyApp.ini and get it like getResource(”myapp.ini”)? And if we use specific operating system resources?

The same goes for .NET applications. If the developers are careful, it’s easy to have a cross-platform application. Having a desktop application is hard, that’s true. Microsoft made Windows Forms very Windows specific (using stuff like windows handles), and it’s hard for Mono to make a cross-platform Windows Forms implementation. But there are other alternatives, like GTK#. Even so, I realize that this issue is the least cross-platform of .NET.

But on a web/services scenario, Mono is as cross-platform as you can get.

Running Mono - an Overview

October 29th, 2009

We’ve been using Mono for the production server of Orion’s Belt for a couple of months now. On this article I’d like to share our experiences using Mono. We developed our project fully on Windows with Visual Studio 2005, and at a time we started to consider Mono for a production server. Do note that the Orion’s Belt team has a Windows background, and little experience administrating Linux machines.

Here are the web applications that we are serving with Mono:

Step 1 - Preparing the Server

The game is running on an Ubuntu server. We downloaded the source code and installed it manually. We could have used the packages, but that isn’t versatile enough. Packages aren’t always up to date and we find them hard to manage. For example, how could we have two versions of Mono installed and choose which  to run? How could we always have an up to date version? To fix this problem we chose to install by source, having as a guideline the Parallel Mono Environments article. This was great because we could change to use Mono from SVN or a specific version, just by changing some variables. We wouldn’t want to install a new version, getting problems, and having a bad time to rollback.

We use FastCGI with Nginx to serve the game. Nginx’s really cool, and very easy to configure and manage. We also installed MySQL. Not considering Mono, all the other necessary software was very easy to install and configure, with the help of Google of course. We managed to get the server displaying ASPX pages easily.

Step 2 - Running the Application

The Orion’s Belt project is fairly big, and after some months coding fully on Windows, the move to Linux was peaceful. We had some file case issues, but that was it. We have a NAnt script that creates a deploy package, and we were able to upload it to the mono server and run it. There were some problems at that time, sometimes Mono’s web server would throw compile errors while compiling ASPX pages. But fortunately, at that time, Mono implemented Precompiled ASP.NET web sites in Mono. We incorporated a step on NAnt to precompile the deploy package and everything was faster and we didn’t get those errors anymore.

Other problem we had was the mono web process and the resources it used. Going to 600-900Mb RAM and wasting a lot of CPU, even at idle time. So we started to kill the process from time to time. Sometimes the process would die unexpectedly, so we also started to use supervise, to supervise mono’s process.

There are also some other issues we got along the way: touching Web.config isn’t as stable as it should be. And also when we deployed new versions, mono would not behave properly, it would shutdown or just stop responding. So, we got used to just kill mono when we deployed or needed a reset. It’s very easy, you just kill mono’s process, and supervise will bring it up.

Supporting multi OS makes your code better

It may sound weird, but it’s true. We had a lot of bugs showing up only on mono. For example, we use NHibernate and everything worked fine on Windows, but on Linux sometimes it didn’t. We found out that we needed a flush here and there. On Windows it worked, but on Linux it wasn’t that permissive.

Using Mono brought an interesting mindset to the team. Every time there was a problem, we’d blame it on Mono. But the majority of times, it was our code that wasn’t up to it.

The Linux issues also made us create specific guidelines for file case, forced the use of Path.Combine and related methods. We also tried MoMa, the mono’s problem reporting tool, but we didn’t find it to be that useful on our situation.

Running the Tick

The game’s tick runs every ten minutes. It’s a very heavy process that loads a lot of data from the database, operates on it, and then persists it. This process needs a lot of RAM and CPU to run, and it’s a good performance test. On this specific process we find Mono to be very lacking. If running the process on Mono would take 60 seconds, running the same process on the windows development machine, connecting to the production database, would take 30 seconds. And the development machine is worst than the production server.

However, for the cost of a Windows license, we could buy a great machine just for tick processing. Would it be worth it? We don’t know at the moment.


Preparing the Mono environment was fun and interesting, and the issues we got from porting the code were minimal. We did need some help, and I find the mono list not that friendly to newcomers, but Google provided the help we needed. Even so, it’s not easy for developers without experience administrating Linux machines to prepare mono. There are always some issues here and there, that we’d know how to fix on Windows, but that we loose a lot of time figuring it out on Linux.

Although mono’s behaving nicely most of the times, we don’t find it as stable as a Windows machine. Even so, it’s a great option, that’s for sure. I already have a slicehost account with an ubuntu+mono running all my private ASP.NET sites. It’s cheaper and runs really well.

But for the production server for the game, we aren’t convinced yet if we should continue using mono or not. Maybe we’ll release another server using Windows and then we’ll have a good performance comparison showcase.

Being a programmer is boring…

September 24th, 2009

Don’t get me wrong: I love software engineering. I love developing applications and the challenges that we face when we have a complex problem and we have to model it and make it work. I really like the creative and technological aspects of programming. But talking as someone that has been for three years on some corporate companies, several projects and a few teams, that isn’t really what we do, is it?

From my point of view, here’s the cycle of generic corporate work:

  1. We enter a project, that may be starting or not and we meet the team
  2. We start by analyzing the project’s structure, learning how it works and what it does, during this time we may get some documents to read
  3. Then we get small tasks, like some bug fixes, minor development details and such
  4. Soon we start getting bigger tasks, and in time we’ll feel more confident on the project, we’ll know it well and start making ourselves profitable
  5. After the learning stage we become very valuable for the team, we’ll fix bugs faster, we’ll deploy faster, we’ll implement faster, and we’ll help the junior members of the team
  6. At some point we leave the project… maybe the project is finished, maybe we’re needed elsewhere, and we go back to step number one

Looking at these steps, I’d say that the best ones are 1,2,3 and 4. Learning is very important to keep me motivated. When I’m on a new project, I have to learn new technologies, new concepts, and the structure of the project. I’ll also get to know my team and I’ll learn a lot from them.

Step number 5 is the boring one. When we already know everything related to the project is when we’re most profitable for the company. At that time we mainly get to fix bugs, support the production environment, write documentation and maybe handle customers. We’ll also get new developments and changes, but often these new developments will collide with something already done that wasn’t expecting the new features. This may mean architectural changes that may prove quite annoying to accomplish.

When I’m at step 5, I feel my motivation getting lower every day. And the problem is, I spend the majority of my time at step 5. It comes a time when I get tired and start talking to management to get back to step 1, but management can’t allow that. Now that we’re important on the project, it isn’t easy to take us away, so we need to hang on, for the company’s sake.

Not all programmers dislike step 5. At step 5 developers have won a place on the team and are an important asset for the company: they reached the comfort zone. They may like it there, and sometimes step 5 provides some interesting challenges. Different people get motivated by different things. I’m not a step 5 type of people, but I know a lot that are, that really like what they do and they’re good a it.

This is, of course, my own point of view, and does not reflect the truth for other developers. For me, change is important, and the degree of different things I get to do everyday contributes to my motivation as a developer.

MySQL on full UTF8

May 11th, 2009

On the Orion’s Belt Translation Project we had no problems with encoding, until a player from Croatia told us that Croatian’s characters weren’t being properly persisted. I tried a direct update:

update Lang set Text = 'č,ć,ž,đ,š'

And MySQL complained with an invalid characters error. To fix this I had to change the column encoding to UTF8 (was latin1). But the application still wasn’t  behaving properly. The application was sending to MySQL the following query:

update Lang set Text = 'c,c,z,d,s'

I don’t know who was transforming this (maybe the MySQL connector). To fix this I had to edit the connection string, and add the utf8 charset:

Server=s;User ID=u;Password=p;Database=d;CharSet=utf8

After these steps, everything worked fine.

My head hurts. I'm porting a .NET application to Linux/Mono, and everything was working out fine, except the connectivity to the MySql database.  I don't have experience as an administrator, so I had a machine set up, I installed mono (after so many time I still have to compile stuff on Linux? Come on!), the application worked, but I had the following issues:

  • I could connect to mysql on the production server using
    mysql -u user -p
  • The application couldn't connect to mysql on the production server (using localhost)
  • The application could connect to mysql on another machines
  • The same application on another machines could connect to mysql on the production server

At this time I didn't know if the problem was with mono/application or with mysql configuration. So I run a Python script that connected to localhost and it worked fine... Then I downloaded the MySQL .NET provider sources and started to break some rocks (Portuguese expression).

The provider only said something like:

Unable to connect to any of the specified MySQL hosts.
---> System.Exception:
Exception of type 'System.Exception' was thrown.

Very usefull information as you can see. After digging up, this exception was being thrown like this:

  1. StreamCreator sc = new StreamCreator(Settings.Server, Settings.Port, pipeName);
  2. baseStream = sc.GetStream(Settings.ConnectionTimeout);
  3. if (baseStream == null)
  4.     throw new Exception();

Really classy material. The actual exception was being thrown by:

  1. Socket socket = ...
  2. try {
  3.      socket.EndConnect(ias);
  4. } catch (Exception) {
  5.      socket.Close();
  6.      return null;
  7. }

Oh yeah! What more can I say?

After patching the mysql .net provider with some logging, I had the real exception:

System.Net.Sockets.SocketException: Connection refused

The I did something that I should have done a long time ago:

# telnet localhost 3306
telnet: Unable to connect to remote host: Connection refused

Well, it's proved that it wasn't an application / mono problem. Why did it work with direct mysql on the console and Python? Maybe because they connect using a pipe, instead of the TCP stream that the provider uses, I really don't know.

And wouldn't know how to fix this, but my colleague Paulo Pires came and saved the day. It seams that mysql was configured to use a binding to local named address, and because of that refused connections to localhost.

Twitter API from C#

October 24th, 2008

I did this code to test the API interaction with twitter via .NET. You can see it in action on the Orion's Beltt twitter page. The game logs the result of every battle on that twitter account.

  1. private const string TwitterJsonUrl = "";
  2. private const string TwitterUser = "your_user";
  3. private const string TwitterPass = "your_pass";
  5. private static void SendTwitterMessage( string message )
  6. {
  7.     try {
  8.         HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(TwitterJsonUrl);
  10.         string post = string.Empty;
  11.         using( TextWriter writer = new StringWriter() ) {
  12.             writer.Write("status={0}", HttpUtility.UrlEncode(message));
  13.             post = writer.ToString();
  14.             Console.WriteLine("Post: {0}", post);
  15.         }
  17.         SetRequestParams(request);
  19.         request.Credentials = new NetworkCredential(TwitterUser, TwitterPass);
  21.         using( Stream requestStream = request.GetRequestStream() ) {
  22.             using( StreamWriter writer = new StreamWriter(requestStream) ) {
  23.                 writer.Write(post);
  24.             }
  25.         }
  27.         Console.WriteLine("Length: {0}", request.ContentLength);
  28.         Console.WriteLine("Address: {0}", request.Address);
  30.         WebResponse response = request.GetResponse();
  31.         string content;
  33.         using( Stream responseStream = response.GetResponseStream() ) {
  34.             using( StreamReader reader = new StreamReader(responseStream) ) {
  35.                 content = reader.ReadToEnd();
  36.             }
  37.         }
  39.         Console.WriteLine(content);
  41.     }
  42.     catch( Exception ex )
  43.     {
  44.         Console.WriteLine(ex);
  45.     }
  46. }
  48. private static void SetRequestParams( HttpWebRequest request )
  49. {
  50.     request.Timeout = 500000;
  51.     request.Method = "POST";
  52.     request.ContentType = "application/x-www-form-urlencoded";
  53.     //request.Referer = "";
  54.     request.UserAgent = "Orion's Belt Notifier Bot";
  55. #if USE_PROXY
  56.     request.Proxy = new WebProxy("http://localhost:8080", false);
  57. #endif
  58. }

I was getting a very weird error on my ASP.NET web application. Every time I ran a job the web application would shutdown with an Application_End event called. Being two diferent processes, this was hard to explain. But later on I found out that IIS will recycle applications if it looses a lock on some files.

I was running the job from the Bin/ directory that the uses. I copied the dll/exe content to another directory and since then everything worked fine.

Rule of thumb: never run jobs from the Bin/ directory!

ORM White Paper

October 7th, 2008

The NHibernate community has a very interesting article presenting several ORM concepts on their google groups page.

I found it particularly useful for understanding id generators, what are the available strategies and how to choose the best one. For example, we have a batch job that inserts a lot of objects and it takes about 35 minutes to perform using a "native" generator (we're using MySQL). We changed to a hilo generator and it only took 25 minutes. Quite an improvement.