Slaying Changelog's compilation beast

Recently, I read a pair of articles outlining how to improve compilation performance in Elixir apps. Since I don’t yet have the luxury of working with Elixir professionally, it was interesting knowledge but not immediately actionable.

Then I watched Jerod Santo’s foray into Phoenix Live View.

Quickly, it became clear that the Changelog repo was suffering from several cross-module compilation dependencies. With the Phoenix server running, changes to a single file would require recompilation of 220 files. On a fresh Phoenix project, you change one file and it is the only one that needs to be recompiled. So, what had gone wrong?

First, some background

It’s important to understand the options you have for reusing modular code in Elixir. I’ll focus on the two options relevant to the problem at hand.

• import allows you to call functions from a module without prepending the module’s name. So, you can write say_hi() instead of Changelog.Greeter.say_hi().
• alias allows you to call functions from a module using only the last part of a module’s name. Hence, you can write Greeter.say_hi() instead of Changelog.Greeter.say_hi().
  You can also alias a module with an entirely different name, ie alias Changelog.Greeter, as: Mom allows you to call Mom.say_hi().

There are a couple of catches with using imports. First, they make your code more ambiguous. If you are reading dozens or hundreds of lines of code, it’s not clear where the imported functions are coming from. Second and more importantly, imports can create cross-module compilation dependencies.

If Changelog.Greeter has been imported to other modules, they will all need to be recompiled when Changelog.Greeter is modified. Watching Jerod’s Phoenix jam session, it seemed apparent that the repo was a little too reliant on imports.

It doesn’t have to be this way

At this point, it may sound like I have a full grasp of the inner workings of Elixir’s compiler. This is the benefit of hindsight, and I still can’t claim to understand it completely. Prior to opening an issue and saying It doesn’t have to be this way, I had largely avoided deeply researching what happens when the compiler runs. It was a topic for a more educated and skilled future-self to explore when the time was right.

Casting aside my trepidation, I bravely opened an issue offering to slay the compilation monster. After aknowlegement in the issue and a shout-out from Jerod, it was time to put the articles to use.

Finding the right solution

While I initially attempted to manually search and replace imports (converting them to aliases) this was a futile approach. Fortunately, Elixir contributor and splendid dude Wojtek Mach chimed in, pointing to his second article addressing these compilation complications. Most importantly, he had created a script to programatically convert module imports to aliases.

Running the script on the Changelog source code, I ran into a problem.

In Elixir, you can pipe calls from one function to another. It’s one of the tools provided by the language to make our human lives a little easier. Typically you will see functions piped to each other on separate lines, but it sometimes makes sense to pipe a few calls on a single line. This was tripping up the import2alias script, leading to mangled and broken code.

import2alias uses a compilation tracer to collect information about where a module’s functions are being called throughout the source code. The module name, function name, line number, column, and arity are provided to the tracer. Then, the script collects these results and prepends the user-defined alias to each of the function calls for a provided module.

To illustrate the problem I found with the script, imagine Changelog.Greeter has two functions, say_hi() and offer_help(). You might have another module that calls say_hi() then offer_help(), which looks like this using the pipe feature:

say_hi() |> offer_help()

When the script prepends the alias to say_hi, offer_help is pushed to a new column since it’s on the same line:

Greeter.say_hi() |> offer_help()

import2alias only collects the column info before any changes are made, so when it goes to prepend an alias to offer_help, it starts on the wrong column and we get mangled code:

Greeter.say_Greeter.offer_helphi() |> offer_help()

Fortunately, Wojtek was already aware of this limitation, and pointed to a discussion about adding a new Elixir feature to modify code in a more reliable way. In order to replace the Changelog imports, I modified the script to perform a simple string replace instead of relying on which column the function name started. This worked well enough, only requiring two or three manual fixes where a string was found multiple places in a line.

For example, the app.html.eex template has a description meta tag that contained the string ‘description’ twice. After running the script with the String.replace modification, the alias was prepended to the meta tag’s name attribute, and it had to be fixed.

# Error from the String.replace/3 approach
<meta name="Description.description" content="<%= Description.description(assigns) %>">

# The corrected name attribute
<meta name="description" content="<%= Description.description(assigns) %>">

#results

After converting a few imported modules to use aliases, the application no longer needs to recompile 220 modules; now it only recompiles 5. There is still room for improvement, but this marks a 98% reduction in cross-module compilation dependencies.

With this improvement in place, I took the liberty to add Phoenix’s live reloader back to the application. When you are hacking away on a Phoenix application with the live reloader enabled, the application will automatically recompile watched files (web views and assets), and trigger a browser refresh. No more toggling between your editor and the browser just to refresh the page.

This sums up where you should look first if you are running into this recompilation pain during development. It’s not uncommon or unwise to use or import modules from external packages. (It is usually recommended by the package’s documentation.) However, importing your own custom modules will likely lead to longer recompilation times as your application grows.

My rule of thumb would be to use imports of your own code very judiciously. - Wojtek Mach

If you are starting a new Elixir/Phoenix application or improving an existing one, it’s best to alias your own modules when you don’t want to write out the full module name.

The Future

Elixir 1.11 will make it possible for import2alias to modify .eex files which currently do not provide column information to the tracer function. With this and the proposed Code.format_quoted/2, the script will be able to safely and accurately convert all imported function calls with aliased calls, even if they’re piped on the same line.

Open source is a beautiful thing.

---

Editor’s Note: This post now has a companion episode of our Backstage podcast where Owen joins Jerod to discuss the entire process in detail. Listen below 👇