Comparing how Julia and Go deal with dependencies to other code

Erik Engheim

Feb 13·10 min read

Julia and Go are a fun exercise in how terminology can get really confusing. In Julia and Go, the package and module concepts have basically been swapped.

The Go documentation defines a module as:

A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.

This is pretty much the reverse of Julia. In Julia a package is what get distributed and version controlled. A package however may contain multiple modules.

In Julia modules exist at the language level providing a namespace for your types and functions. In Go packages serve that same purpose. To be honest I think Go is the language that kind of screwed up in this case. Go began by rolling the idea of a namespace for code and the versioning and distribution of that code all into one concept called package.

Long term this was not a practical solution because Go did not offer a good way of specifying which particular version of a package your code depended on.

In Julia I can write the code:

using Greetings
message = Greetings.hello("Gladys")
println(message)

When I run this code, Julia will look at the current environment to figure out what actual package provides the Greetings module which contains the hello function.

In Julia this is determined by looking at a file inside our current package named Project.toml. It says what our package is named and what other packages it depends on.

name = "Hello"
uuid = "f4029ba5-1bab-490d-8bbe-d6d1c8a1f0a8"
authors = ["Erik Engheim <erik.engheim@mac.com>"]
version = "0.1.0"

[deps]
Greetings = "1ef576d1-6215-4f11-828a-3bafa7859891"

For the purpose of figuring out dependencies our Hello package is not actually called Hello. That is what Julia's module system will refer to it as. Rather a unique universal identifier (UUID) is used. These are randomly generated and long enough to in practice have very low chance of being the same UUID as a package created by somebody else.

Likewise we specify that from a language level we depend on a package which will be referred to using the module name Greetings. But what uniquely identifies the package is the UUID 1ef576d1-6215-4f11-828a-3bafa7859891. The benefit of this is that two or more people can make packages with the same name. There could be dozens of packages named Greetings and it doesn't matter because their UUID would be different.

Thus in your code when you write using Greetings it could in principle refer to any of 10 different packages. It is the Project.toml file which tells us which package specifically is offering this module.

The next problem is of course where does this package come from. For this Julia uses the Manifest.toml file which get generated and updated by querying a repository. This file contains entries like this:

[[Greetings]]
path = "../Greetings"
uuid = "1ef576d1-6215-4f11-828a-3bafa7859891"
version = "0.1.0"

This says which version of BinaryProvider package we are using. The Project.toml would only have specified the UUID. This gives a git-tree-sha1 hash. That is a hash generated from the content of the directory containing the package. This in a way serves as a unique identifier for a particular version of a package. Thus as you modify the code for a package, the UUID will stay the same but the git tree hash will keep changing. Ideally you also update the version string to match your code changes.

For completeness, this is what the Julia Greetings package looks like:

module Greetings

using Printf

export hello

function hello(name::AbstractString)
    message = @sprintf("Hi, %s. Welcome!", name)
    return message
end

end # module

The code is not really idiomatic Julia code but has been modeled on the Go modules example: Call your code from another module.

The Julia package is organized on disk like this:

shell> tree Greetings/
Greetings/
├── Manifest.toml         # Where you find deps
├── Project.toml          # Direct dependencies
└── src
    └── Greetings.jl

Go Modules and Package

Let us compare this with how Go modules and packages work. First let me show you the Greetings package in Go, as given in the Go module intro.

package greetings

import "fmt"

// Hello returns a greeting for the named person.
func Hello(name string) string {
    // Return a greeting that embeds the name in a message.
    message := fmt.Sprintf("Hi, %v. Welcome!", name)
    return message
}

The module directory looks like this. Again a gentle reminder that a Go module corresponds to Julia package:

greetings/
├── go.mod      # Defines module
└── greet.go    # Our source code

The go.mod serves much the same purpose as Project.toml in Julia. It describes our module:

module example.com/greetings

go 1.13

Now let us look at how this module is used from another module hello that we create. I have made this directory structure:

.
├── greetings
│   ├── go.mod
│   └── greet.go
└── hello
    ├── go.mod
    └── hello.go

In the hello module I have put a source code file hello.go with the code:

package main

import (
    "fmt"

    "example.com/greetings"
)

func main() {
    // Get a greeting message and print it.
    message := greetings.Hello("Gladys")
    fmt.Println(message)
}

So in Go a package is a language level thing. This code is made part of the main package, but it uses the greetings package which has a function named Hello. For Go applications the main function must be defined and part of the main package. Julia has no concept of an application.

Because the concept of modules game late in Go, this code conflates the concept package as namespace and organizational unit in your code with the external dependency management which says something about where the module providing the package is located. Here we see it being implied that the greetings package is hosted at example.com.

However since it is hosted locally we edit the go.mod file to look like this:

module hello

go 1.13

replace example.com/greetings => ../greetings

As you can see, this is very similar to how Julia points to local directory for the Greetings package. A key difference is that Go does not operate with UUIDs.

[[Greetings]]
path = "../Greetings"
uuid = "1ef576d1-6215-4f11-828a-3bafa7859891"
version = "0.1.0"

Also a key difference is that in Julia there is a clear separation between what you conceptually depend on written into Project.toml and the specifics of where that dependency is located which is found in Manifest.toml. In Go both of these concepts are merged into the go.mod file.

If you actually run:

$ go build

In the hello package directory, Go will lookup the dependent package and update the go.mod file reflect that.

module hello

go 1.13

replace example.com/greetings => ../greetings

require example.com/greetings v0.0.0-00010101000000-000000000000

This wires what version we depend on. In Julia this kind of version information is put in the Manifest.toml file. That is the file Julia will usually make automatic changes to. Project.toml is intended to be a lot more stable.

Comparison of Go and Julia Module and Package Commands

Both Julia and Go have tools to automatically deal with modules and packages.

Julia being a dynamic language with a REPL, has a special package mode for dealing with packages. With Go we use command line tools. The first thing to know is how to create a package or module.

Initialize

In Julia a package is created with a command while in package mode in the REPL:

pkg> generate Greetings

This will create source code and the Project.toml file. In Go we use the following command:

$ go mod init example.com/greetings

This only creates the go.mod file. You will have to create the source code yourself. Go does not have an entry point like Julia. That is because the name spacing is done differently. In Go you say what package a file belongs to by putting e.g. package greetings in the header. In Julia a module is defined in one place like this:

module Greetings
include("foo.jl")
include("bar.jl")
end

This means both the file foo.jl and bar.jl are made part of the Greetings module. Unlike Go, Julia files don't know what module they are part of.

Again I am sorry for the massive confusion writing about this given that modules in Julia are packages in Go. Thus if you are Go programmer and you see I refer to Julia modules, then know that in your mind you should think packages. And likewise if you are a Julia developer, keep in mind that when I say a Go module, you should think about a package.

Testing

Given that you have actually written tests, you would run tests for a package/module in Go with:

$ go test

This implied that you are inside the relevant module in Go. You can do similar in Julia. But from the package manager perspective, you are not “inside” the package unless you write:

pkg> activate .
(Greetings) pkg>

This will allow you to run the tests for the Greetings package if you have made them:

(Greetings) pkg> test

If you don’t want to go into the Greetings package directory you could just specify directly which package to test. However this requires that the package is in your current Julia environment.

pkg> test Greetings

List Dependencies

In Go we use the go list command to get the dependencies of a module. Here you can see hello being listed as well. The reason is that the application as such depends on the hello module. Remember that the entry point of an application is always the main package.

$ go list -m all
hello
example.com/greetings v0.0.0-00010101000000-000000000000 => ../greetings

In Julia we have the notion of a current package, or more accurately the currently active environment. A package represents an environment. So we go into the package directory, start Julia and make that package the active one. Then we can ask for status to see what dependencies the currently active package has.

$ cd Hello/
$ julia
julia> ]
 pkg> activate .
 Activating environment at `~/Development/Hello/Project.toml`

(Hello) pkg> status
Project Hello v0.1.0
Status `~/Development/Hello/Project.toml`
  [1ef576d1] Greetings v0.1.0 `../Greetings`

Adding Dependencies

Both Go and Julia allows you to add package dependencies with commands. This download the sampler package and adds it as a dependency:

$ go get rsc.io/sampler@v1.3.1

$ cat go.mod
module hello

go 1.13

replace example.com/greetings => ../greetings

require (
    example.com/greetings v0.0.0-00010101000000-000000000000
    rsc.io/sampler v1.3.1 // indirect
)

In Julia we have to be in package mode and have the package we want to add to be the active one:

(Hello) pkg> add Tar@0.1.0

This adds the Tar package for creating archives. This also specifies that I am not getting the latest Tar package but specifically the 0.1.0 version. You can see Tar added as a direct dependency to the Hello package:

shell> cat Project.toml
name = "Hello"
uuid = "f4029ba5-1bab-490d-8bbe-d6d1c8a1f0a8"
authors = ["Erik Engheim <erik.engheim@mac.com>"]
version = "0.1.0"

[deps]
Greetings = "1ef576d1-6215-4f11-828a-3bafa7859891"
Tar = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

But this will also update the Manifest.toml file with all the indirect dependencies as well as info about what specific version to get.

shell> cat Manifest.toml
# This file is machine-generated - editing it directly is not advised

[[Greetings]]
deps = ["Printf"]
path = "../Greetings"
uuid = "1ef576d1-6215-4f11-828a-3bafa7859891"
version = "0.1.0"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[Tar]]
git-tree-sha1 = "5a5ba451fb83875247e22216609b25b581cddaad"
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
version = "0.1.0"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

Go actually has a similar file, although the responsibilities between the are not quite divided the same as in Julia. In Go, you see version info in the go.mod file. That is only found the Manifest.toml file. However Go has a go.sum file which gets created when you add modules much like Julia will automatically create Manifest.toml for you when stuff gets added.

$ cat go.sum
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/sampler v1.3.1 h1:F0c3J2nQCdk9ODsNhU3sElnvPIxM/xV1c/qZuAeZmac=
rsc.io/sampler v1.3.1/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

The go.sum file records the checksums of modules, so that when somebody who installs your project tries to download dependencies, Go can verify for them that the dependent modules are the same physically on disk as the ones you used. This is similar to Julia, expect in the Manifest.toml file we specifically use the git tree hash algorithm on the directory to give a unique value for the contents of a the source code directory. Exactly how go.sum checksums are computed I don't know. Feel free to drop me a line and enlighten me.

Read more: Understanding go.sum and go.mod file in Go (Golang).

Tidy Dependencies

In this case Go has a feature, which is hard to do in a dynamically typed language like Julia, but which is pretty neat. Check this out:

$ go mod tidy
$ cat go.mod
module hello

go 1.13

replace example.com/greetings => ../greetings

require example.com/greetings v0.0.0-00010101000000-000000000000$ cat go.sum
$

With go mod tidy Go will go through our source code and find out what we are actually using. It finds out that the sampler module I previously added with go get is not actually used. Hence it removes it from the go.mod file. It also removes any direct or indirect dependencies recorded in the go.sum file as well, which turns it empty.

Info About What Package Provides

In Go you could get info about the types and functions in a package:

$ go doc rsc.io/quote/v3
package quote // import "rsc.io/quote"

Package quote collects pithy sayings.

func Concurrency() string
func GlassV3() string
func GoV3() string
func HelloV3() string
func OptV3() string
[plutonium hello] $

Worth nothing that this will actually case a package to be downloaded and added. So it is not exactly a lookup docs on arbitrary-stuff-online tool.

The Julia equivalent is more like the help system in the REPL. Once a module is loaded into the REPL environment you can ask for help about the module, specific types or functions.

Conclusion

Package systems do a lot of the same things and face the same challenges. Hence we see similar kinds of solutions used in different package systems. However as we saw in this example we can still get confused because terminology can be swapped around and how things are conceptually structured can be slightly different.

E.g. we saw how Project.toml corresponds loosely to go.mod and Manifest.toml corresponds to go.sum but they are not completely analogies.

If you are a Go user and you are still confused about the whole module concept, here is another go and trying to get the key point across: Previously Go code pointed directly to Github repos URLs. This caused problems such as:

The repo might add breaking changes to the code. The solution back then was to use a different URL for a different version.

The new Go module system separate what should be made separate. It separates the name of a package you use in your code for import from the specific physical package used to satisfy that name. E.g. if you do import foobar then in principle 20 different packages online could have that name and satisfy that package. However it is your go.mod file together with go.sum which gets to specify exactly which Github repo and exactly which version satisfies the foobar package name in the source code.

In this regard we move the dependency management out of the source code.