The mysterious “memory leak”

Recently at my workplace, we started to migrate one of our projects from Rails 2 to Rails 3.1. Everything seemed to go pretty well. Then one day, I started noticing my development machine crawling to a halt for no apparent reason. I could see that the memory was full and swapping. My initial reaction was that some sort of memory leak was introduced during the Rails 3 migration, as I also discovered that another colleague was experiencing the same issue. After going through a quick process of kill -9 elimination, the culprit turned out to be Passenger Standalone.

Checking Passenger’s memory profile

Some quick googling turned up the command passenger-memory-stats. The tool shows details regarding the Apache/Nginx processes’ memory profiles, as well as the Passenger processes’ memory profiles. I quickly realized that passenger uses 6 worker processes by default.

One thing that my team quickly realized during the Rails 3 migration was that Rails 3 has a much larger overhead in development mode than Rails 2. This quickly lead to a substantial amount of the dev machine’s memory being utilized when passenger was run and a few requests were made. In my case, I had also started using gnome-shell, which took up a good chunk of memory, and that compounded with 6 memory-intensive workers meant a machine running out of memory.

I ended up spending some time to find out documentations on how to configure the number of worker instances. Much to my surprise, I was not able to find any information on how to do this for standalone Passenger.

It was time to start digging into Passenger’s internal’s.

Discovering how to configure nginx for standalone Passenger

A quick grep for passenger_max_pool_size under the passenger gem root directory turned up this file: phusion_passenger/templates/standalone/config.erb. In the file, I quickly saw these two hopeful looking lines:

1
2

passenger_max_pool_size <%= @options[:max_pool_size] %>;
passenger_min_instances <%= @options[:min_instances] %>;

I proceeded to replace the two erb tags with 2 and 1 respectively. After restarting passenger and running passenger-memory-stats, I saw only two passenger workers. Success!!

Looking for a better solution

Modifying the nginx config template worked. However, modifying the gem directly was not an ideal solution. One option that I had was to fork the gem and modify the template, but that also did not seem like a clean solution for this problem. So I started digging a bit further into passenger. A grep for ‘config.erb’ turned up this line: lib/phusion_passenger/standalone/command.rb:179: template_filename = File.join(TEMPLATES_DIR, "standalone", "config.erb")

A quick scan through the file revealed this bit of code:

1
2
3
4
5
6
7
8
9
10

  def parse_options!(command_name, description = nil)
    # cut
    global_config_file = File.join(ENV['HOME'], LOCAL_DIR, "standalone", "config")
    if File.exist?(global_config_file)
      require 'phusion_passenger/standalone/config_file' unless defined?(ConfigFile)
      ConfigFile.new(:global_config, global_config_file).options
      @options.merge!(global_options)
    end
    # cut
  end

This showed that the template was configurable, by looking at ~/.passenger/standalone/config

A quick peek into the ConfigFile class showed this initializer:

1
2
3
4
5
6
7
8
9

  def initialize(context, filename)
    options  = {}
    @context  = context
    @filename = filename
    File.open(filename, 'r') do |f|
      f.flock(File::LOCK_SH)
      instance_eval(f.read, filename)
    end
  end

It looks like Passenger reads in the local config file, and evals the content. Looking at the content of config_file.rb, it contains these two methods:

1
2
3
4
5
6
7
8

  def max_pool_size(number)
    allowed_contexts(:max_pool_size, :global_config)
    @options[:max_pool_size] = number
  end

  def min_instances(number)
    @options[:min_instances] = number
  end

Therefore, inside my local config file, I added these two lines:

1
2

max_pool_size 2
min_instances 1

Running passenger-memory-stats after restarting Passenger showed that there were infact 2 workers up and running. Success for real!!

I definitely appreciated Passenger giving the option to configure nginx unintrusively, and it was also nice to realize that there were no memory leaks after all.

How I got to learn vim registers

One of the things that bugged me the most when I was learning Vim was copy and pasting.

For example, I would copy a line with: yy. Then I would go to the section I wanted to put the yanked line, and of course, I would see a line that I had to delete first, so without thinking, I would type: dd. I would then quickly realize that I had lost the line that I copied, go up to the copied line, yank the line again, go down to the section again, and finally paste it.

Another case where copy and pasting bugged me was copying to/from the system clipboard. For example, I had no idea how to copy and paste between different Vim windows. I also had no idea how to copy from external pages into Vim other than highlighting the text from the external page, going into insert mode in Vim, and middle-clicking the mouse.

After feeling this limitation countless times, I finally decided to learn how the Vim registers worked, and how I could copy and paste at will.

Vim registers

Vim has something called registers, where various strings are stored, depending on the context and what actions you have taken.

For example, the ” register is Vim’s default register, so when you yank or delete lines, they go into that register by default. The % register holds the relative path of the current file. You can view the contents of these registers by typing :reg

You can perform actions with these registers by typing “, followed by the register name, then the action you would like to perform.

For example, if you would like to paste the current file’s relative path twice, you would type "%2p. You can achieve the effect of the usual p with ""p.

The null register

At this point, you’re probably wondering how this helps with the first issue I had with Vim, which is how I can avoid overwriting the content that I yanked in Vim. This is solved by using the _ register, which is Vim’s null register.

Let’s say you’ve copied a line with yy, and you have a line that you want to delete without losing the line you’ve yanked. You can do that by typing "_dd. Want to delete lines 38 to 43 without overwriting the yanked line? No problem with :38,43d_. Alternatively, you can select those lines in visual mode, and type :d_.

Copy and pasting to and from the system register

To solve the second issue I had, I used two other special Vim registers, namely the * and + registers. The * register is where content that you highlight in linux operating systems go to, and the + register is where content that you copy/cut with Ctrl+C or Ctrl+X go to.

Now given the knowledge acquired from using the null register, you can safely guess that "+p would paste whatever you’ve copied using Ctrl+C. Copying into the clipboard is as simple(?) as "+yy, which you can test by pasting into somewhere else with Ctrl+P. If you copy with "*yy instead, you can paste it by middle-clicking the mouse, just as you would do with any content you’ve highlighted in linux.

Re-mapping the default yank, delete, and paste behaviours in Vim

I personally prefer using the Vim registers directly, so that I don’t forget how to use them, but there are definitely a lot of people who would prefer to have copy and pasting to behave like any other editors out there.

In that case, you could consider re-mapping the default Vim behaviours by adding something like the following into your ~/.vimrc:

Re-map d to delete into the null register

nnoremap d "_d
vnoremap d "_d
nnoremap D "_D
vnoremap D "_D

Re-map c to not change the default register’s content

nnoremap c "_c
vnoremap c "_c
nnoremap C "_C
vnoremap C "_C

Re-map y to copy into the system clipboard

nnoremap y "+y
vnoremap y "+y
nnoremaP Y "+Y
vnoremaP Y "+Y

Re-map p to paste from the system clipboard

nnoremap p "+p
vnoremap p "+p
nnoremaP P "+P
vnoremaP P "+P