Slaying Changelog's compilation beast
How I reduced cross-module compilation dependencies by 98%
Recently, I read a pair of articles outlining how to improve compilation performance in Elixir apps. Since I don’t yet have the luxury of working with Elixir professionally, it was interesting knowledge but not immediately actionable.
Then I watched Jerod Santo’s foray into Phoenix Live View.
Quickly, it became clear that the Changelog repo was suffering from several cross-module compilation dependencies. With the Phoenix server running, changes to a single file would require recompilation of 220 files. On a fresh Phoenix project, you change one file and it is the only one that needs to be recompiled. So, what had gone wrong?
First, some background
It’s important to understand the options you have for reusing modular code in Elixir. I’ll focus on the two options relevant to the problem at hand.
importallows you to call functions from a module without prepending the module’s name. So, you can write
aliasallows you to call functions from a module using only the last part of a module’s name. Hence, you can write
You can also alias a module with an entirely different name, ie
alias Changelog.Greeter, as: Momallows you to call
There are a couple of catches with using imports. First, they make your code more ambiguous. If you are reading dozens or hundreds of lines of code, it’s not clear where the imported functions are coming from. Second and more importantly, imports can create cross-module compilation dependencies.
Changelog.Greeter has been imported to other modules, they will all need to be recompiled when
Changelog.Greeter is modified. Watching Jerod’s Phoenix jam session, it seemed apparent that the repo was a little too reliant on imports.
It doesn’t have to be this way
At this point, it may sound like I have a full grasp of the inner workings of Elixir’s compiler. This is the benefit of hindsight, and I still can’t claim to understand it completely. Prior to opening an issue and saying It doesn’t have to be this way, I had largely avoided deeply researching what happens when the compiler runs. It was a topic for a more educated and skilled future-self to explore when the time was right.
Casting aside my trepidation, I bravely opened an issue offering to slay the compilation monster. After aknowlegement in the issue and a shout-out from Jerod, it was time to put the articles to use.
Finding the right solution
While I initially attempted to manually search and replace imports (converting them to aliases) this was a futile approach. Fortunately, Elixir contributor and splendid dude Wojtek Mach chimed in, pointing to his second article addressing these compilation complications. Most importantly, he had created a script to programatically convert module imports to aliases.
Running the script on the Changelog source code, I ran into a problem.
In Elixir, you can pipe calls from one function to another. It’s one of the tools provided by the language to make our human lives a little easier. Typically you will see functions piped to each other on separate lines, but it sometimes makes sense to pipe a few calls on a single line. This was tripping up the
import2alias script, leading to mangled and broken code.
import2alias uses a compilation tracer to collect information about where a module’s functions are being called throughout the source code. The module name, function name, line number, column, and arity are provided to the tracer. Then, the script collects these results and prepends the user-defined alias to each of the function calls for a provided module.
To illustrate the problem I found with the script, imagine
Changelog.Greeter has two functions,
offer_help(). You might have another module that calls
offer_help(), which looks like this using the pipe feature:
say_hi() |> offer_help()
When the script prepends the alias to
offer_help is pushed to a new column since it’s on the same line:
Greeter.say_hi() |> offer_help()
import2alias only collects the column info before any changes are made, so when it goes to prepend an alias to
offer_help, it starts on the wrong column and we get mangled code:
Greeter.say_Greeter.offer_helphi() |> offer_help()
Fortunately, Wojtek was already aware of this limitation, and pointed to a discussion about adding a new Elixir feature to modify code in a more reliable way. In order to replace the Changelog imports, I modified the script to perform a simple string replace instead of relying on which column the function name started. This worked well enough, only requiring two or three manual fixes where a string was found multiple places in a line.
For example, the
app.html.eex template has a description meta tag that contained the string ‘description’ twice. After running the script with the
String.replace modification, the alias was prepended to the meta tag’s name attribute, and it had to be fixed.
# Error from the String.replace/3 approach <meta name="Description.description" content="<%= Description.description(assigns) %>"> # The corrected name attribute <meta name="description" content="<%= Description.description(assigns) %>">
After converting a few imported modules to use aliases, the application no longer needs to recompile 220 modules; now it only recompiles 5. There is still room for improvement, but this marks a 98% reduction in cross-module compilation dependencies.
With this improvement in place, I took the liberty to add Phoenix’s live reloader back to the application. When you are hacking away on a Phoenix application with the live reloader enabled, the application will automatically recompile watched files (web views and assets), and trigger a browser refresh. No more toggling between your editor and the browser just to refresh the page.
This sums up where you should look first if you are running into this recompilation pain during development. It’s not uncommon or unwise to
import modules from external packages. (It is usually recommended by the package’s documentation.) However, importing your own custom modules will likely lead to longer recompilation times as your application grows.
My rule of thumb would be to use imports of your own code very judiciously. - Wojtek Mach
If you are starting a new Elixir/Phoenix application or improving an existing one, it’s best to alias your own modules when you don’t want to write out the full module name.
Elixir 1.11 will make it possible for
import2alias to modify
.eex files which currently do not provide column information to the tracer function. With this and the proposed
Code.format_quoted/2, the script will be able to safely and accurately convert all imported function calls with aliased calls, even if they’re piped on the same line.
Open source is a beautiful thing.
Editor’s Note: This post now has a companion episode of our Backstage podcast where Owen joins Jerod to discuss the entire process in detail. Listen below 👇
Sign in or Join to comment or subscribe