Exploring Julia REPL Internals

One of my favorite parts of Julia is the enormous flexibility of the Julia REPL environment. Out of the box when you install Julia you get a beautiful REPL which has great command completion, history matching and code editing.

One of the things I miss the most when I am in REPL environments for other programming languages is the Chameleon behavior of the Julia REPL. It has the ability to switch to different modes with different behavior.

In this story we are going to explore how this works under the hood by playing around with the built in modes.

But first let us have a birds eye view of the different parts that make up the REPL system.

LineEditREPL and Prompt

The Julia REPL itself is represented by an instance of the type LineEditREPL. This object manage six different objects representing a different mode. Each mode is represented by an instance of the Prompt type.

You can add more Prompt objects if you like, but out of the box, Julia comes with six different modes in this order:

  1. Julia mode (backspace)
  2. Shell mode (;)
  3. Help mode (?)
  4. History (Ctrl+r)
  5. Prefix History
  6. Package mode (])

The LineEditREPL object gets hold of your keystrokes, but rely on the Prompt objects to define how it should react to these keystrokes.

This happens in several different ways. For instance when Julia has evaluated some expression, it wants to show the prompt again, to indicate it is ready to receive new input.

Below is an example of me switching between different modes, to show you how the prompt changes.

julia> println("hello")
hello

(v1.3) pkg> status
Status `~/.julia/environments/v1.3/Project.toml`
[336ed68f] CSV v0.5.26
[a93c6f00] DataFrames v0.20.2
[31a5f54b] Debugger v0.6.4
[91a5bcdd] Plots v0.29.6
[ce6b1742] RDatasets v0.6.6
[295af30f] Revise v2.5.4
[2913bbd2] StatsBase v0.32.2
[f3b207a7] StatsPlots v0.14.1
[e88e6eb3] Zygote v0.4.9

help?> 2 + 3
+(x, y...)

Addition operator. x+y+z+... calls this function with all arguments, i.e. +(x, y, z, ...).

Examples
≡≡≡≡≡≡≡≡≡≡

julia> 1 + 20 + 4
25

julia> +(1, 20, 4)
25

shell> ls
Manifest.toml README.md docs src
Project.toml data examples test

In normal Julia mode the prompt is julia> and in e.g. shell mode it is shell>.

The Prompt object representing the mode tells LineEditREPL what the prompt should be and what color should be used.

The Prompt object handle keystrokes through several different fields:

  • keymap_dict a Dict{Char, Any} type which maps a character to an action. This is used to e.g. switch modes.
  • on_enter a function that checks whether input was valid and if Julia can continue to process it.
  • on_done at this point we assume we got valid input. This is a function which process the text you have typed until you hit enter.
  • complete handle completions. E.g. when you hit tab.

So the Julia REPL is just normal Julia code which is kind of neat. This means you can actually run it yourself like any other code. If you put this code into a Julia file and run that file you will see a normal Julia REPL.

import REPL
term = REPL.Terminals.TTYTerminal("dumb", stdin, stdout, stderr)
repl = REPL.LineEditREPL(term, true)
REPL.run_repl(repl)

You can try this by putting it on a file named say myrepl.jl and then at the terminal write:

$ julia myrepl.jl

Let us Experiment with the Julia REPL Modes

Ok enough of the theory. Let us actually try to modify the behavior of the REPL.

First we need to get hold of the LineEditREPL object representing our REPL. julia> repl = Base.active_repl;

And before we can effectively work with the types it is useful to import the REPL module and REPL.LineEdit submodule, since REPL defines the LineEditREPL type and REPL.LineEdit defines Prompt.

julia> using REPL; using REPL.LineEdit

Let us get some prompt object to play with:

julia> juliaprompt = repl.interface.modes[1]
"Prompt(\"julia> \",...)"

julia> shellprompt = repl.interface.modes[2]
"Prompt(\"shell> \",...)"

The text for the prompt is gotten from the prompt field. We can both read and set that. E.g. how about getting a Python style >>> prompt?

julia> juliaprompt.prompt
"julia> "

julia> shellprompt.prompt
"shell> "

>>>juliaprompt.prompt = ">>> "
">>> "

>>> print("look a python prompt!")
look a python prompt!
>>>

We can also change the color of the prompt. Colors in terminals is based on treating particular characters sequences as colors. This is referred to as ANSI escape codes. If this seems odd, let us take a little detour about terminals since this is what this is all about.

The graphical terminals most are used to using today was when Unix came around actual type writers connecting to a mainframe computer. There was no computer screen. The paper on your type writer or teletyper to be accurate, as your screen. You typed stuff on the type writer which got sent to the computer. The computer responded by sending text back. Your teletyper would then automatically type out on paper the response you got.

To do more than just type letters sent over, manufacturers started adding he ability to interpret certain characters as control codes. A standard for this got developed which is called ANSI escape codes, because they start with an escape character written as '\e' in Julia. Typically a '[' follows.

So when some people got electronic terminals and they wanted to display the following text as red, the would send over the ANSI escape code "\e[31m". If they wanted it blue they would send "\e[34m".

Now it may seem silly that we are still using this archaic system with modern computers. The reason is that Unix systems began creating virtual teletype devices so old programs could continue working. Programs such as ls and cat do in principle think they are sending text to a teletype device. In reality of course they send it to a graphical window running which emulates old Unix style terminals.

Fortunately Julia makes it easy to get hold of these escape codes. If I want the character for blue I write Base.text_colors[:blue]. You can try out yourself how this works:

import Base: text_colors
>>> print(text_colors[:blue], "Hello ", text_colors[:red], text_colors[:underline], "World")
Hello World

When you do this in a terminal you would see “Hello” in blue and “World” as underlined red text.

We can use this knowledge to change the color of our prompt, and even the color of the expressions we write for Julia to evaluate.

>>> juliaprompt.prompt_prefix = text_colors[:magenta]
"\e[35m"

>>> juliaprompt.prompt_suffix = text_colors[:cyan]
"\e[36m"

>>> println("hello")
hello

If yo try this out you will see the prompt >>> is colored in magenta and println("hello") gets cyan color.

Let us play with the keymap dictionaries to see how it works.

>>> juliaprompt.keymap_dict[';']
#49 (generic function with 1 method)


>>> juliaprompt.keymap_dict['$'] = juliaprompt.keymap_dict[';']
#49 (generic function with 1 method)

>>> $
shell> ls
Manifest.toml README.md docs src
Project.toml data examples test

What this shows is that a function is stored on ';'. We know already that semicolon causes transition to shell mode. Or at least you know if you have used Julia for some time.

Just for kicks we store this same transition function on the '$' character. Thus now you can change to shell mode also by using a dollar symbol. Not very useful but good for demonstration purposes.

When exploring this API which is largely undocumented it helps having some helper functions. You may not know what kind of functions the keymap expects. How do you find that out?

We just make our own function that can take any argument, but which dumps out the type of every argument provided.

>>> function funcdump(args...)
for arg in args
println(typeof(arg))
end
end
funcdump (generic function with 1 method)

>>> juliaprompt.keymap_dict['$'] = funcdump
funcdump (generic function with 1 method)

>>> $
>>> REPL.LineEdit.MIState
LineEditREPL
String
>>>

So now we discovered that the function that the keymap wants for its values should have the signature

func(state::REPL.LineEdit.MIState, repl::LineEditREPL, char::String)

While I got the number of arguments and types this way, it does not tell us exactly how they are used. That can be tedious to figure out on your own. What I did was to look at other software adding REPL modes such as OhMyREPL.jl and ReplMaker.jl.

Let us build a better function than funcdump to explore how this function works more in detail. To not mess up out terminal we are going to send debug information to another location. You can do this many different ways. E.g. you can create a log or simply use a socket connection as I will do here. I use the Unix NetCat command nc to startup a simple server listening on port 1234 for connection.

$ nc -l 1234

In Julia we import the Sockets module and test it out.

julia> using Sockets

julia> socket = connect(1234)
TCPSocket(RawFD(0x00000014) open, 0 bytes waiting)

julia> println(socket, "Did you get the message?")

In the Terminal window running NetCat we can then see the message Did you get the message? pop up. In the REPL we can define a better debug function.

using REPL
using REPL.LineEdit

function debugmode(state::REPL.LineEdit.MIState, repl::LineEditREPL, char::AbstractString)
iobuffer = LineEdit.buffer(state)

# write character typed into line buffer
LineEdit.edit_insert(iobuffer, char)

# change color of character to magenta
printstyled(char, color=:magenta)

# move to start of iostream to read what the user
# has typed thus far
seekstart(iobuffer)
line = read(iobuffer, String)
println(socket, "line: ", line)

# Show what character user typed
println(socket, "character: ", char)
end

Now let us assign this function to a key, e.g. '9'. If you start the REPL fresh you need to get hold of the REPL object and standard mode object.

julia> repl = Base.active_repl;

julia> juliamode = repl.interface.modes[1]
"Prompt(\"julia> \",...)"

julia> juliamode.keymap_dict['9'] = debugmode

Now we can try out how that works, by typing something with the letter '9'.

julia> foo bar number 9
ERROR: syntax: extra token "bar" after end of expression

julia> 9999
9999

If you look at what our NetCat windows produce you can see it shows:

character: 9
line: foo bar number 9
character: 9
line: 9
character: 9
line: 99
character: 9
line: 999
character: 9
line: 9999
character: 9

This is a way of verifying that the function is using arguments as we expected.

What are the takeaways from this? You can notice e.g. that the buffer of what you have written and what the user sees is not quite the same thing. LineEdit.edit_insert(iobuffer, char) will put stuff in the line buffer but it will not display it.

Although if you wrote LineEdit.edit_insert(state, char) instead it would display it as well.

This gives us some flexibility in that we can e.g. colorize the text the user sees using ANSI Escape codes, but we don’t need to put those codes in the line buffer that Julia evaluates.

Let us check out the on_done field and how we can process input. In this case we will modify it on the shell mode otherwise we easily risk screwing up our Julia prompt.

>>> shellprompt.on_done = funcdump
funcdump (generic function with 1 method)

shell> hello world
REPL.LineEdit.MIState
Base.GenericIOBuffer{Array{UInt8,1}}
Bool

Exactly what all these arguments are for may not be obvious. Fortunately there is a higher order function which will produce functions useable for on_done that we can use:

respond(parser::Function, repl::LineEditREPL, prompt::Prompt)

This is a non-exported function in REPL so you have to write REPL.respond(parser, repl, prompt).

Let us try it out:

>>> shellprompt.on_done = REPL.respond(split, repl, juliaprompt)

shell> hello how are you doing?
5-element Array{SubString{String},1}:
"hello"
"how"
"are"
"you"
"doing?"

>>> shellprompt.on_done = REPL.respond(split, repl, shellprompt)

shell> foo bar qux
3-element Array{SubString{String},1}:
"foo"
"bar"
"qux"

shell> egg spam bacon
3-element Array{SubString{String},1}:
"egg"
"spam"
"bacon"

Did you see how it works? split takes a string and splits it on whitespace by default producing an array. That is what we see as result here, rather than shell command being interpreted.

Second thing to observe is that the prompt argument tells Julia which mode to switch to, after parsing the input. In the first case I jumped back to the Julia mode after each input. In the second case I remain in shell mode until I hit backspace.

Thus far we only used the keymap_dict to register characters/keys which would do a color change. But what if we want to use a particular key to facilitate a mode transition?

If you look at how that normally works in Julia, a certain key causes a mode transition but only when it is typed at the beginning of the line.

We can figure out where the cursor is by looking at the position in the iobuffer representing the current line. If position(iobuffer) == 0 is true, then we know the cursor is at the beginning of the line. All IO buffer type of streams have the notion of a position. You can also use this when reading e.g. a file.

using REPL
using REPL.LineEdit

repl = Base.active_repl;
juliamode = repl.interface.modes[1]
shellprompt = repl.interface.modes[2]
newprompt = shellprompt

function trigger(state::LineEdit.MIState, repl::LineEditREPL, char::AbstractString)
iobuffer = LineEdit.buffer(state)
if position(iobuffer) == 0
LineEdit.transition(state, newprompt) do
# Type of LineEdit.PromptState
prompt_state = LineEdit.state(state, newprompt)
prompt_state.input_buffer = copy(iobuffer)
end
else
LineEdit.edit_insert(state, char)
end
end

# Trigger mode transition to shell when a '6' is written
# at the beginning of a line
juliamode.keymap_dict['6'] = trigger

The actual transition to another mode is done with the LineEdit.transition function. It wants the state of the current mode as well as an instance of a Prompt object representing a mode to transition to.

LineEdit.transition(f::Function, state::LineEdit.MIState, newprompt::REPL.Prompt)

Upon transition to this new mode a function f is called which allows further customization.

You don’t have to do anything here. What we do in the code example above is the following:

prompt_state = LineEdit.state(state, newprompt)
prompt_state.input_buffer = copy(iobuffer)

That copies the content of the current line buffer to the new mode. That is is practical. Imagine you type a function call, and then you want to switch to help mode to get help on this function call.

By doing this, the function call is still on the line even after you switched to help mode.

Also notice that we don’t do any LineEdit.edit_insert(state, char) when switching mode. The effect is that the character is swallowed and never seen. Which would be desirable in most cases.

What is useful about having a way to do define a transition to another mode is that we can create our own custom mode if we want to.

To make it easy for us, we can just copy all the fields from the shell prompt, except the new name of course.

newprompt = REPL.Prompt("[hello] ")

for name in fieldnames(REPL.Prompt)
if name != :prompt
setfield!(newprompt, name, getfield(shellprompt, name))
end
end

push!(repl.interface.modes, newprompt);

The only thing awkward with this solution is that you will have to evaluate the definition of the trigger function because the definition of newprompt has changed. Also make sure you assign the new trigger function to a key.

juliamode.keymap_dict['6'] = trigger

If you hit 6 now, you will get the [hello] prompt:

julia> 6

[hello] ls
Manifest.toml README.md docs src
Project.toml

Now that you got the basics working, you can experiment further by replacing things like the on_done and on_enter fields of newprompt.

Further Exploration

If you are more curious, I highly advice checking out the ReplMaker.jl package. It contains a small amount of code interacting with the REPL modes which you can study, and it offers a more polished interface towards adding your own REPL modes to Julia.

Written by

Geek dad, living in Oslo, Norway with passion for UX, Julia programming, science, teaching, reading and writing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store