Do you think running tests are too slow? Cannot figure out how to do Test Driven Development (TDD) in Julia? Here is a guide.

Erik Engheim

Jan 23·6 min read

A lot of programming provide you with ready made solutions where there is one particular way of doing things. In the Julia community the approach is more around supply flexible building blocks from which you can construct whatever solution you want.

Read more: Julia v1.5 Testing: How to Organize Tests.

The advantage of this approach is that you can easily tailor a solution to your particular needs. The downside is that to someone fresh to Julia, it may not be immediately obvious what blocks to combine. One example of this is the Julia testing framework. It is flexible and can easily be tailored to your needs. But that also means it is not immediately obvious how you do many common workflows.

In this story I aim to show you patterns to follow for different use cases such as:

• A thorough, correct and complete test run of a package. What you may want to do after downloading a new package you are uncertain about, or the kind of test you run at the end of day of coding.
• Regular testing that you do with some frequency while developing.
• Rapid iteration testing for when you run tests really frequently. E.g. because you follow TDD (Test Driven Development). In this case individual tests you are adding need to run fast, because they are run a lot.

Infrequent and Correct Testing

Say you are developing a package named Foobar which you want to test.

Minor digression. Anyone wondering about the name Foobar? It is just a nonsense word used by us programmers. Whenever you see the words foo, bar, baz and foobar you should know these are just placeholder names. You are meant to fill in something more useful.

Anyway this is the steps a Julia beginner may take to run their first package tests:

$ cd Foobar
$ julia
julia> ]
(@v1.5) pkg>
(@v1.5) pkg> activate .
(Foobar) pkg>
(Foobar) pkg> test
    Testing Foobar
Status `/private/var/folders/qb/zc__4nlj4_z4w24t_trvx1cm0000gn/T/jl_K7IPCe/Project.toml`
  [9a3f8284] Random
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test
Test Summary: | Pass  Total
All Tests     |   57     57
    Testing Foobar tests passed

You could make a bunch of code changes and then you could easily run your tests again with:

(Foobar) pkg> test

Yet despite the convenience and sort of obvious way of running tests, this way of launching the tests is not really the way you should normally run your tests.

The purpose of this form of testing is to be able to do a thorough test of a whole package on occasion. E.g. after you downloaded a new package. This form of testing is not aimed at rapid iteration but correctness. It checks all dependent packages and make sure they are up to date before running the tests. But if you do numerous small code changes and you want to make sure you haven’t broken any tests while doing this, then you don’t want to do pkg> test as that will significantly slow you down.

Regular Testing

In your daily workflow, you will do more regular running of all the tests. Then the pkg> test approach is suboptimal as it is pointless to check state of dependent packages multiple times a day. They are unlikely to change. In this case the sensible way to test is:

shell> julia test/runtests.jl
Test Summary: | Pass  Total
All Tests     |   57     57

In this case you don’t have to pay for the overhead of going online and checking state of dependent packages. That means the tests run much faster.

However you may still find that it is still slow and you are kind of wasting effort by running lots of tests in parts of the code you are not even touching. You can solve this by placing groups of tests into individual files. Say your test/runtests.jl file looks like this:

@testset "All Foobar tests" begin

include("foo-tests.jl")
include("bar-tests.jl")
include("baz-tests.jl")

end

Then you can simply temporarily comment out include for the tests you don't want to run. If you find this primitive one could make a system to pick what files to run at command line:

tests = ["foo", "bar", "baz"]
if !isempty(ARGS)
    tests = ARGS  # Set list to same as command line args
end

@testset "All Foobar tests" begin

for t in tests
    include("$t-tests.jl")
end

end

Here we are utilizing the fact that ARGS is an array in Julia which contains command line arguments.

Thus you could write this on he command line to only run the foo and baz test collections:

$ julia test/runtests.jl foo baz

However there is still an overhead in having to spin up the Julia JIT every time you want to perform these tests.

Testing Environment

An important side note: When you run tests this way, you are actually using the v1.5 Julia environment and not the one setup for your tests. But you can use the --project switch to specify the environment to use:

$ julia --project=test test/runtests.jl

Rapid Iteration Testing

To be able to do anything rapidly in Julia you want to avoid having the Julia JIT compile the same stuff over and over again. Hence you want to stay within your fully initialized REPL environment for as long as possible.

julia> include("test/runtests.jl")
Test Summary: | Pass  Total
All Tests     |   57     57

This has the benefit that the Julia JIT will compile stuff which does not need to be compiled the second time you run the tests. It will get cached within the environment you are already in.

Single Tests

However to move even faster we want to be able to easily run just one single test. To make this practical we need to make some practical changes to our code.

Everything you need to run the code inside an arbitrary unit test has to be put into one file test/setup.jl. This is just a convention. You can call it anything you like.

This should contain stuff like using Test, and include any kind of helper functions you may use in your tests. Place the file wherever you want, but I recommend the test directory.

Your test/runtests.jl should then look like:

include("setup.jl")

@testset "All Foobar tests" begin

include("foo-tests.jl")
include("bar-tests.jl")
include("baz-tests.jl")

end

Inside each test file you would typically want tests written akin to this:

@testset "Without operands" begin
    @test disassemble(901) == "INP"
    @test disassemble(902) == "OUT"
    @test disassemble(000) == "HLT"                                     
end
    
@testset "With operands" begin
    @test disassemble(105) == "ADD 5"
    @test disassemble(112) == "ADD 12"
    @test disassemble(243) == "SUB 43"
    @test disassemble(399) == "STA 99"
    @test disassemble(510) == "LDA 10"                                       
end

To get started running tests the first think you do in your REPL is:

julia> include("test/setup.jl")

Next you have two options. The first one works with any setup. It means copying the test you want to run and paste it into the REPL.

Alternatively is you use VSCode you can place your cursor inside the test you want to run and hit Alt-Enter. This will select the outer most code block and evaluate it. That will typically mean one test set. However as you can see in the example below, where I have nested tests, it causes code in the outermost test set to run.

Keep this in mind when deciding how to structure your test inside files. Alt-Enter is tied to how code is structured inside a source code file. It does not care if you have created nesting by using includes.