Understanding how Julia Packages are able to only pick binaries for your particular architecture.

Erik Engheim

Jan 29·6 min read

This is a followup to my Julia Artifacts for Dummies, which only dealt with non-architecture specific resources.

This story is to help understand how Julia’s JLL packages work. Julia packages that wrap a library written in C/C++ depend on these.

SDL Example

One example of this would be the Simple Direct Media Layer (SDL) which is often for simple 2D pixel games as it abstracts away how you write pixels into a pixel buffer on different platforms. The Julia package is called SimpleDirectMediaLayer.jl. You would install it with:

(@v1.5) pkg> add SimpleDirectMediaLayer

But what we are interested in here is looking at how it wraps the binaries made in C/C++ code. If you take a peek inside its Project.toml file on Github you will find:

name = "SimpleDirectMediaLayer"
uuid = "98e33af6-2ee5-5afd-9e75-cbc738b767c4"
repo = "https://github.com/jonathanBieler/SimpleDirectMediaLayer.jl.git"
version = "0.2.1"

[deps]
ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
SDL2_image_jll = "41f3cfe1-fd11-58ad-8074-b9730c29438b"
SDL2_jll = "ab825dc5-c88e-5901-9575-1e5e20358fcf"
SDL2_mixer_jll = "7e88e5ca-120e-58e7-9660-b2187fd466c0"
SDL2_ttf_jll = "ca9d4746-8a27-5104-babe-80f83b336947"

Do you see it has dependencies with _jll suffixes? These are the ones actually providing the binaries. They are just normal Julia packages which contain a Artifacts.toml file in addition to a Project.toml file. Lets look at the artifacts file inside SDL2_jll.jl (I shortened it. We don't need to see everyting):

[[SDL2]]
arch = "aarch64"
git-tree-sha1 = "34bab89e489684e2be54265098661e07cef18caf"
libc = "glibc"
os = "linux"

    [[SDL2.download]]
    sha256 = "2ff482d1ff6ec7643b7bdc14a7fe821bfcb0959005fc328b21c11569d09fc36a"
    url = "https://github.com/JuliaBinaryWrappers/SDL2_jll.jl/releases/download/SDL2-v2.0.12+1/SDL2.v2.0.12.aarch64-linux-gnu.tar.gz"
[[SDL2]]
arch = "aarch64"
git-tree-sha1 = "2029f4ace28f81e0f34fc6018ebe7e226abbdda9"
libc = "musl"
os = "linux"

    [[SDL2.download]]
    sha256 = "b78715ef97afdb4317693d672f72f17c5c88a26f644db512221efdd5389ccefc"
    url = "https://github.com/JuliaBinaryWrappers/SDL2_jll.jl/releases/download/SDL2-v2.0.12+1/SDL2.v2.0.12.aarch64-linux-musl.tar.gz"

If you look at this on Github you will see that it lists all possible architectures supported.

Normally these JLL packages are created by Julia’s BinaryBuilder package. But we are not going to do that. Instead we are going to look under the hood to understand better how this system works.

A TOML Crash Course

To understand how this system works we are going to use a simple package called ArtifactsPlay.jl which I made specifically to demonstrate how the Artifacts system works.

It might be useful to know some detail about the TOML format. In TOML. Regular arrays can be written as:

xs   = [ 1, 2, 3 ]
strs = [ "Commas", "are", "delimiters" ]

A table is kind of similar to a dictionary. It contains data as key-value pairs.

[table-1]
key1 = "some string"
key2 = 123

Sometimes however you need arrays of tables. You can think of this as arrays of dictionaries. In the TOML format this is done with double square brackets. This creates an array called foobar where each element is a dictionary with the keys arch, git-tree-sha1 and os.

[[foobar]]
arch = "x86_64"
git-tree-sha1 = "5fc005bfb4a5436fcde1cc097be47a3f323b8a14"
os = "macos"

[[foobar]]
arch = "x86_64"
git-tree-sha1 = "9a8dfae78dac5de25e62705aa4017b399e267160"
libc = "glibc"
os = "linux"

[[foobar]]
arch = "x86_64"
git-tree-sha1 = "70d6addecf4f1e67ac4470ee9839bcb07e19d773"
os = "windows"

The way this works is that when accessing an artifact, Julia will look through this array to find the table containing the architecture and operating system you are on. Since I am on an intel based Mac I get this result:

julia> artifact"foobar"
"~/.julia/artifacts/5fc005bfb4a5436fcde1cc097be47a3f323b8a14"

If you tried this now however, you would get the following error message:

julia> artifact"foobar"
ERROR: Cannot locate artifact 'foobar' in '~/Development/Julia/ArtifactsPlay/Artifacts.toml'

That is because the Artifact.toml is intentionally empty. I am going to show you how to populate it with entries for different operating systems.

Creating a Artifacts.toml for Different Binaries

Since this is done for educational purposes I will not be using actual binaries but simple text files. This is what the directory structure in the ArtifactsPlay package looks like:

ArtifactsPlay
├── README.md
├── Artifacts.toml
├── create-artifacts.jl
├── src
│   └── ArtifactsPlay.jl
├── lindata
│   ├── msg.txt
│   └── os.txt
├── macdata
│   ├── msg.txt
│   └── os.txt
└── windata
    ├── msg.txt
    └── os.txt

Each of the directories lindata, macdata and windata contains data specific to the given operating system. But I have just done this in a silly fashion:

$ cat lindata/msg.txt
Hello Linux user!

$ cat macdata/msg.txt
Hello macOS user!

$ cat windata/os.txt
Windows

We are going to turn this data into artifacts which only get loaded for the operating system you are on. To do this we will use the create-artifacts.jl script. Most packages with artifacts will come with a script like this which was used by the package creator to produce the initial Artifacts.toml file.

Let me walk through how this script works. It is not written in a very efficient manner, for educational purposes.

Since we may run this several times as we develop as package, we may want to put in some guards to avoid recreating entries which already exist. That is what this first part does. It checks if the foobar artifact already is defined in the Artifacts.toml file.

using Pkg.Artifacts
using Pkg.BinaryPlatforms

hash = artifact_hash("foobar", "Artifacts.toml")

if !isnothing(hash) && artifact_exists(hash)
    return
end

In our case it will not, so we create the macOS version of this resource. create_artifact creates a temporary directory, into which we copy the macOS specific files. create_artifact will then do a git style hash of the content of this directory after we have done all the copying. That is why this is done with a do-block.

mac_hash = create_artifact() do dir
   cp("macdata/msg.txt", joinpath(dir, "msg.txt"))
   cp("macdata/os.txt", joinpath(dir, "os.txt"))
end

The mac_hash returned is this created hash. The temporary directory under ~/.julia/artifacts where we copied our files will get renamed to this hash. This is what gives us content-addressable artifacts.

Making Tarballs

However somebody who downloads a package with artifacts will not run this script and will thus not have anything stored in this location. That is why we need the next steps which is to make a tarball of the artifact directory.

mac_tar_hash = archive_artifact(mac_hash, "foobar-mac.tar.gz")

The line above makes a tarball (.tar.gz file) out of the content at the stored at the artifacts directory associated with the mac_hash. This tar file is stored under the name foobar-mac.tar.gz in the current directory. Thus you will see it pop up in the package directory.

Record Artifact in Artifacts.toml

Now we will do something which is unrealistic. We record the download location of our our mac artifact as a local location. For a realistic case this would be an URL on some remote server, e.g. using Github releases. But for demonstration purposes a local location is handy.

mac_path = joinpath(pwd(), "foobar-mac.tar.gz")

bind_artifact!("Artifacts.toml", "foobar", mac_hash,
    platform=MacOS(:x86_64),
    download_info=[("file:" * mac_path, mac_tar_hash)])

The mac_path ends up being the current location. To form a local URL we need a file: prefix. E.g. you would write something like file:/home/steve/foobar-mac.tar.gz. Notice that doanload_info takes pairs. That is because you can provide many download locations for your artifact in case one of them is down or fails.

platform is set to MacOS(:x86_64) which causes us to get the entry following keys in our .toml file:

arch = "x86_64"
os = "macos"

Similar code is repeated for every architecture. E.g. for Linux we add entries to the Artifacts.toml file like this:

lin_tar_hash = archive_artifact(lin_hash, "foobar-lin.tar.gz")
lin_path = joinpath(pwd(), "foobar-lin.tar.gz")

bind_artifact!("Artifacts.toml", "foobar", lin_hash,
    platform=Linux(:x86_64),
    download_info=[("file:" * lin_path, lin_tar_hash)])

This leaves us with a Artifacts.toml file which looks like this:

[[foobar]]
arch = "x86_64"
git-tree-sha1 = "5fc005bfb4a5436fcde1cc097be47a3f323b8a14"
os = "macos"

[[foobar.download]]
sha256 = "6fedc42acebe6c82825b8a4a53008d899affe45db582f89028876134d70ba61f"
url = "file:/home/steve/ArtifactsPlay/foobar-mac.tar.gz"
    
[[foobar]]
arch = "x86_64"
git-tree-sha1 = "9a8dfae78dac5de25e62705aa4017b399e267160"
libc = "glibc"
os = "linux"

[[foobar.download]]
sha256 = "107e0faba3ba11b7adc7e2cc648cfe1ab0d19e4eacf7698d1f1f1d446f921561"
url = "file:/home/steve/ArtifactsPlay/foobar-lin.tar.gz"
    
[[foobar]]
arch = "x86_64"
git-tree-sha1 = "70d6addecf4f1e67ac4470ee9839bcb07e19d773"
os = "windows"

[[foobar.download]]
sha256 = "3faa9e386c1b0407b669d9ad7a553eeb70d91ee1bb8f56cf2a97281d15a851b4"
url = "file:/home/steve/ArtifactsPlay/foobar-win.tar.gz"

Accessing Artifacts

Once the artifacts file is done we can simulate what it would be like for somebody installing this package by removing the artifacts directories that got made by the script. I just copy the hashes straight out of the Artifacts.toml file:

julia> mac_hash = "5fc005bfb4a5436fcde1cc097be47a3f323b8a14";

julia> lin_hash = "9a8dfae78dac5de25e62705aa4017b399e267160";

julia> win_hash = "70d6addecf4f1e67ac4470ee9839bcb07e19d773";

Then we can lookup the path to these directories and delete them:

julia> using Pkg.Artifacts

julia> rm(artifact_path(Base.SHA1(mac_hash)), recursive=true)
julia> rm(artifact_path(Base.SHA1(lin_hash)), recursive=true)
julia> rm(artifact_path(Base.SHA1(win_hash)), recursive=true)

In the ArtifactsPlay package there is a greet function defined like this:

function greet()
    path = joinpath(artifact"foobar", "msg.txt")
    println(read(path, String))
end

If you run this function you will see the following output:

julia> greet()
Downloading artifact: foobar
##O=#  #
Downloading artifact: foobar
Hello macOS user!

What happened here was that the artifact system checked the Artifacts.toml file to find the hash for foobar based on the operating system and architecture I am on. It then use this hash to check if is already installed. Since we delete that directory it will not be. Hence it downloads the .tar.gz file and unpacks it into ~/.julia/artifacts/ under the right hash. Thus we can later find the file msg.txt there. If this was on Linux it would have contained a Linux specific greeting instead.