The mysterious “memory leak”
Recently at my workplace, we started to migrate one of our projects from
Rails 2 to Rails 3.1. Everything seemed to go pretty well. Then one day,
I started noticing my development machine crawling to a halt for no
apparent reason. I could see that the memory was full and swapping. My
initial reaction was that some sort of memory leak was introduced during
the Rails 3 migration, as I also discovered that another colleague was
experiencing the same issue. After going through a quick process of
kill -9 elimination, the culprit turned out to be Passenger
Checking Passenger’s memory profile
Some quick googling turned up the command
tool shows details regarding the Apache/Nginx processes’ memory profiles,
as well as the Passenger processes’ memory profiles. I quickly realized
that passenger uses 6 worker processes by default.
One thing that my team quickly realized during the Rails 3 migration was that Rails 3 has a much larger overhead in development mode than Rails 2. This quickly lead to a substantial amount of the dev machine’s memory being utilized when passenger was run and a few requests were made. In my case, I had also started using gnome-shell, which took up a good chunk of memory, and that compounded with 6 memory-intensive workers meant a machine running out of memory.
I ended up spending some time to find out documentations on how to configure the number of worker instances. Much to my surprise, I was not able to find any information on how to do this for standalone Passenger.
It was time to start digging into Passenger’s internal’s.
Discovering how to configure nginx for standalone Passenger
A quick grep for passenger_max_pool_size under the passenger gem root
directory turned up this file:
In the file, I quickly saw these two hopeful looking lines:
I proceeded to replace the two erb tags with 2 and 1 respectively.
After restarting passenger and running
passenger-memory-stats, I saw
only two passenger workers. Success!!
Looking for a better solution
Modifying the nginx config template worked. However, modifying the gem
directly was not an ideal solution. One option that I had was to fork
the gem and modify the template, but that also did not seem like a clean
solution for this problem. So I started digging a bit further into
passenger. A grep for ‘config.erb’ turned up this line:
lib/phusion_passenger/standalone/command.rb:179: template_filename = File.join(TEMPLATES_DIR, "standalone", "config.erb")
A quick scan through the file revealed this bit of code:
1 2 3 4 5 6 7 8 9 10
This showed that the template was configurable, by looking at
A quick peek into the ConfigFile class showed this initializer:
1 2 3 4 5 6 7 8 9
It looks like Passenger reads in the local config file, and evals the content. Looking at the content of config_file.rb, it contains these two methods:
1 2 3 4 5 6 7 8
Therefore, inside my local config file, I added these two lines:
passenger-memory-stats after restarting Passenger showed that
there were infact 2 workers up and running. Success for real!!
I definitely appreciated Passenger giving the option to configure nginx unintrusively, and it was also nice to realize that there were no memory leaks after all.