Switching Back to ExUnit

I want to talk about testing in Elixir this week. It’s extremely light on fractals, but I’ll still include it in my series about my fractal program since I’m going to pull examples from that code. Also, that’s the project where I gave ESpec a try.

RSpec!

When I first started in Ruby on Rails (back in 2007), I used Test::Unit. The tutorials and books all used it, and RSpec wasn’t well known then.

The shoulda gem caught my attention a year or two later, specifically the part that’s now the shoulda-context gem.¹ I didn’t like the test_ prefix I needed to use all the time with Test::Unit; should with a string was a much nicer DSL. And I love describe and context blocks.

Then I discovered RSpec. I started using it, and I liked it. Then I loved it. And I still do.

ESpec

When I first started to learn Elixir, I started with ExUnit. It came with Elixir, and all the tutorials used it, so it was a natural choice.

After a few months, I looked around for Elixir testing libraries that more closely resembled RSpec. I was looking for three things (or so I thought):

• describe and context blocks
• let and subject bindings (with overriding behavior in nested blocks)
• An expressive assertion syntax.

I found ESpec. I’m willing to wager it’s the best transliteration of RSpec into ESpec that’s possible in Elixir.

Here’s a sampling of tests for my Complex module:

defmodule ComplexSpec do
  use ESpec, async: true

  import Complex, only: :macros

  describe ".parse" do
    it "parses" do
      expect(Complex.parse("1.1+2.2i")).to eq(cmplx(1.1, 2.2))
    end
    it "parses negative numbers" do
      expect(Complex.parse("-1.1+-2.2i")).to eq(cmplx(-1.1, -2.2))
    end
  end

  let :z0, do: cmplx(5.0, 12.0)
  let :z1, do: cmplx(3.0,  2.0)

  describe ".add" do
    it "adds" do
      expect(Complex.add(z0, z1)).to eq(cmplx(8.0, 14.0))
    end
  end

  # ...
end

The RSpec version wouldn’t look much different. ESpec has describe and context blocks for test groups; it for a test example; expect... to... for an assertion.

I converted my ExUnit tests into ESpec tests last year in June. Over the months since then (and particularly in the last two or three), I’ve become disastified with ESpec.

ESpec versus ExUnit 1.3.0

Elixir 1.3.0 was released a few weeks ago. (See also “What’s coming in Elixir 1.3” by Daniel Perez.) It added a couple of new features to ExUnit that puts it on par with ESpec, and I’ve soured on the features unique to ESpec.

Test groups

The latest version of Elixir (version 1.3) and ExUnit now give me one level of describe/context nesting. I tend to nest more deeply than that, but I’m told that’s all I’ll need. So far I don’t necessarily disagree with that. Yet.

ESpec and ExUnit tie.

Context variables

One of the reasons I like let so much in RSpec is their scoping. A let in an inner context will shadow a let in an outer context, no matter where the let-variable is used, at any nesting level.

This scoping doesn’t work in ESpec. I suspect it has something to do with the order in which macros and functions are defined and used, with the way contexts are translated, and with the lack of easy polymorphism in Elixir.

More importantly, it just didn’t seem like the lets were really necessary in my tests for my fractals program. I used the new named setups in ExUnit, and they do a perfectly find job for setting up variables in the testing context.

ESpec and ExUnit tie for now. I think these named setups are more powerful than I’m imagining.

Object-oriented ESpec

The only thing that ESpec gave me over ExUnit was the expressive syntax. And that’s where my satisfaction really grew over time.

expect(Complex.parse("2.0+3.0i")).to eq(Complex.new(2.0, 3.0))

That code is both Ruby and Elixir code. I wouldn’t have to change a single character on that line to switch between RSpec and ESpec.² Does that seem right to you? I mean, some simple expressions like 2+3 should look the same in both languages, maybe even some function calls, but complicated expressions relying on specific language features calling a method?

Calling a method, .to, is so fundamentally an object-oriented operation, and you have to read it that way even in Elixir. Elixir is built on the function-programming paradigm, on macros, and on pattern matching. .to does not fit in with any of those fundamental language features.

Now, obviously, the code works in Elixir, so ESpec must be pulling a fast one (if I can be a bit melodramatic). But why force this OO paradigm into Elixir?

ESpec does provide an alternate syntax:

expect(Complex.parse("2.0+3.0i")) |> to(eq(Complex.new(2.0, 3.0)))

Throwing |> into the syntax just seems gratuitous. “Hey! Toss in this key Elixir feature that everyone loves, and then everyone will think it’s an Elixir idiom!” And how are we supposed to read |> in this expression? At least the .to syntax was fluent. The |> is just punctuation ; in the middle of a sentence. (Again, forgive the melodrama.)

Compare this with an ExUnit assertion:

assert Complex.parse("2.0+3.0i") == Complex.new(2.0, 3.0)

Much cleaner. And it uses macros and patterning matching—two fundamental tools of Elixir.

ESpec failed me on its syntax and semantics. It’s too object oriented.

Diffing

ExUnit also adding diffing to its testing feedback. This was the last straw.

I really love the messages I get from RSpec when a test fails. It usually points me to the problem without having to look at the test first. One tool that RSpec uses in those message is diffing. I usually see it search results: “Here’s what you expected. Here’s the actual. Here’s what’s missing. Here’s what’s extra.” I decided within the last year or two that failure messages are possibly the most important feature of a testing library.

Testing feedback is lacking in ESpec, and ExUnit improved their (already pretty good) feedback.

The translation

Translating from ESpec to ExUnit went just fine. Search and replace was helpful.

I had only a few problems keeping to just one level of describe/context.

As I mentioned above, I didn’t miss let because of the named setups. In my test for the OutputWorker, I have this call to setup at the top level:

setup do
  {:ok, output_pid} = StringIO.open("")
  [output_pid: output_pid]
end

output_pid is the process for capturing the raw output. In two different describes I have these subsequent calls to setup:

setup [:chunk_count_1, :subject]
# ...
setup [:chunk_count_3, :subject]

The atoms here refer to function defined in the module. They return a keyword list to add to the context:

defp chunk_count_1(_context), do: [chunk_count: 1]

I found these named setups very useful and very readable. Since the context is passed into the functions, these functions have the ability to transform existing data in the context. This provides me with the overriding abilities that I have in RSpec, just in a very Elixir way: transform the data.

Was it worth it?

Yes. ESpec seems just a little too beholden to its RSpec roots, and I just don’t think it works well in Elixir. I got a bit melodramatic, and I don’t mean to take anything away from what ESpec has done. It’s a pretty impressive library, and some things will improve over time (like the feedback message). I’m more frustrated because I don’t know what ESpec could do to make me like it again.

ExUnit now has some features that make it a very effective testing tool for me.

Footnotes

elixir testing