Open in app

Sign in

Write

Sign in

Working with the HTTP Protocol and Forms in Julia

Using the HTTP, URIs, Sockets, JSON and FileIO packages

Erik Engheim
10 min readAug 7, 2021
Photo by panumas nikhomkhai from Pexels

The Julia HTTP package is great for using the HTTP protocol to either make simple web servers, connect to web servers or use deal with REST APIs. However the package is not that well documented. Many things in the documentation such as best practices for testing REST APIs is not made clear.

Here I will cover:

  • Mocking requests. Useful for testing your web service.
  • Creating headers for your requests.
  • How routing works — How a URL gets tied to a particular handler.
  • Dealing with the payload. E.g. how to put images into the body of a message.
  • Working with IP addresses and port numbers.
  • Parsing the URI of a request.
  • Sending data from user using HTML forms

Overview

There are a lot of different parts which come together and it is easy to miss the big picture if you are looking at the example code provided with HTTP.jl.

The HTTP requests a client sends to an HTTP server is represented by the HTTP.Request object, while responses sent from server are represented by the HTTP.Response type.

The HTTP.handle function binds request and response objects together:

HTTP.handle(hand::HTTP.Handler, req::HTTP.Request) -> HTTP.Request

You can use different types of handlers, but normally we use the HTTP.Router subtype, as this can route a request to different handling functions.

So let us summarize:

  • HTTP.Requeset — Sent to HTTP.handle function which passes it to a handler.
  • HTTP.Reponse — Object returned from handle function, or more specifically from the various handler function registered with the router.
  • HTTP.Router — Used with the HTTP.handle function to route a request to handler function with signature f(req::HTTP.Request).

This helps convey how the request and response objects are related. They are both of type HTTP.Message.

Type hierarchy for requests and responses.
Type hierarchy for requests and responses.

Constructing Requests

Normally you don’t manually construct request objects, because you also want to send a request to a remote server.

julia> resp = HTTP.request("GET", "http://httpbin.org/ip")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: application/json
{
  "origin": "62.16.239.222"
}
"""

Not that the HTTP.Reponse object returned is given as HTTP.Messages.Response. That is because all request and response objects are defined in the Messages submodule. HTTP.Reponse is simply an alias for HTTP.Messages.Response.

As you can see this sends and actual request over the network. But what if you want to test your handler function instead? You could spin up a server with this code:

using HTTP
using Sockets  # Provides IPv4 type and IP number literal.

# Define handler functions
foo(req::HTTP.Request) = HTTP.Response(200, "Called foo")
bar(req::HTTP.Request) = HTTP.Response(200, "Called bar")

# Setup routing to handler functions
const router = HTTP.Router()
HTTP.@register(router, "GET", "/foo", foo)
HTTP.@register(router, "GET", "/bar", bar)

# Setup asyncronous task to perform listening to requests.
# Avoid blocking REPL
HTTP.serve(router, ip"127.0.0.1", 8080)

You could test this by using a web browser and go to address http://localhost:8080/bar or we can make a request from Julia. You will have to open another terminal window or tab as HTTP.serve will block.

julia> resp = HTTP.request("GET", "http://localhost:8080/bar")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Transfer-Encoding: chunked

Called bar"""

Mocking Requests

If you don’t want to spin up a server, we can mock the requests instead. So we make a request object without actually sending it anywhere.

julia> req = HTTP.Request(
   "GET",      # Could be GET,  POST, UPDATE etc
   "http://localhost:8080/foo",      # URL
   ["Content-Type" => "text/plain"], # Header  fields
   "hello"     # Payload/body
)
HTTP.Messages.Request:
"""
GET http://localhost HTTP/1.1
Content-Type: text/plain

hello"""

By calling HTTP.handle directly we route to the handler functions without actually running a server looking for network communications:

julia> resp = HTTP.handle(router, req)
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK

Called foo"""

Access Content in Requests and Responses

When we get a request in our handler or receive a response, there is various data of interest which we would like to access. E.g. what are the headers, the payload, the URI etc.

For this we use the HTTP.method, HTTP.headers, HTTP.uri and HTTP.payload functions.

HTTP.Request

Let us make a HTTP.Request object to demonstrate:

julia> req = HTTP.Request(
   "GET", 
   "http://foo/bar?qux=123", 
   ["Content-Type" => "text/html"], 
   "hello")
HTTP.Messages.Request:
"""
GET http://foo/bar?qux=123 HTTP/1.1
Content-Type: text/html

hello"""

Let us retrieve the different parts again (There are methods for this but they have been deprecated so assume direct field access is preferred).

julia> req.method
"GET"

julia> reg.target
"http://foo/bar?qux=123"

julia> reg.headers
1-element Vector{Pair{SubString{String}, SubString{String}}}:
 "Content-Type" => "text/html"

However this may not be practical in dealing with headers, so there are higher level functions to make it easier:

julia> req["Content-Type"]
"text/html"

julia> HTTP.header(req, "Content-Type")
"text/html"

The payload is stored as bytes, and must be converted to a string:

julia> HTTP.payload(req)
5-element Vector{UInt8}:
 0x68
 0x65
 0x6c
 0x6c
 0x6f

julia> String(HTTP.payload(req))
"hello"

HTTP.Response

Accessing response data uses similar methods. One thing to note is that functions to create response objects don’t follow exactly the same pattern as for HTTP.Request. The body or payload is a named argument.

julia> resp = HTTP.Response(
   200, # status code, meaning success.
   ["Content-Type" => "text/plain"], 
   body = "hello")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: text/plain

hello"""

Apropos: List of HTTP status codes.

Working with URIs

This can be a bit confusing if you are new to Julia. A lot of functionality is often split over several libraries. HTTP itself does not handle URIs but the URIs package.

julia> using URIs

julia> uri = URI(HTTP.uri(req))
URI("http://foo/bar?qux=123")

julia> uri.scheme
"http"

julia> uri.host
"foo"

julia> uri.path
"/bar"

julia> uri.query
"qux=123"

julia> queryparams(uri)
Dict{String, String} with 1 entry:
  "qux" => "123"

IP Address and Port Numbers

To work with IP addresses we need the use the Sockets library. Again not functionality found in the HTTP library.

We can do this in a variety of ways. One way is to use the parse function which deals with many types in Julia. E.g. you can use it to parse a string and create an Integer, Boolean etc.

julia> parse(Int, "42")
42

julia> parse(Bool, "true")
true

julia> parse(Float64, "42")
42.0

This regularity or consistency in Julia is worth being aware when you contemplate doing something with a new library. Sockets provide the IPv4 type and it turns out you can parse to this one as well:

julia> using Sockets

julia> parse(IPAddr, "127.0.0.1")
ip"127.0.0.1"

The result of the parsing may give you a hint at an alternative way of creating these IP addresses. That is to use the ip"" style string literals.

julia> addr = ip"127.0.0.1"
ip"127.0.0.1"

julia> typeof(addr)
IPv4

But why prefer this solution? String literals are macros. So ip"" is actually the @ip_str macro. And macros are evaluated at parse time, not runtime. That means if you put parse(IPAddr, "127.0.0.1") inside a function, it will get parsed every time the function is called.ip"127.0.0.1" will only get parsed once in contrast.

Working with JSON Data

Let us look at how we push and pull JSON data out of requests and responses. First we use JSON.json(dict) to convert a Julia dictionary dict to a JSON string and stuff it in a HTTP.Response.

julia> using JSON
julia> size = Dict(
               "width" => 20,
               "height" => 30);
               
julia> resp = HTTP.Response(
               200,
               ["Content-Type" => "application/json"],
               body=JSON.json(size)
           )
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: application/json

{"height":30,"width":20}"""

Now imagine the server sent this response and the client has to pull out the JSON data. How would we do that?

julia> body = HTTP.payload(resp)
24-element Vector{UInt8}:
 0x7b
 0x22
 ⋮
 0x32
 0x30
 0x7d

julia> io = IOBuffer(body);

julia> JSON.parse(io)
Dict{String, Any} with 2 entries:
  "height" => 30
  "width"  => 20

Although you don’t need to supply an IO object to JSON.parse, you can also just give a JSON string.

julia> JSON.parse("{\"height\":30,\"width\":20}")
Dict{String, Any} with 2 entries:
  "height" => 30
  "width"  => 20

You got to be aware of some gotchas however, which is that String takes ownership of the input byte data, so it is gone after you turn the bytes into a string.

julia> s = String(HTTP.payload(resp))
"{\"height\":30,\"width\":20}"

julia> s = String(HTTP.payload(resp))
""

To avoid this you would have to use copy:

julia> String(copy(HTTP.payload(resp)))
"{\"height\":30,\"width\":20}"

julia> String(copy(HTTP.payload(resp)))
"{\"height\":30,\"width\":20}"

julia> String(copy(HTTP.payload(resp)))
"{\"height\":30,\"width\":20}"

Working With Images

How do you put an image into an HTTP request? That is what we will cover here.

Putting an image into a message doesn’t actually require much handling. You just need the bytes of the image stuff into the HTTP.Request object.

julia> blob = read("image.png")
69126-element Vector{UInt8}:
 0x89
 0x50
    ⋮
 0x6f
 0x35
 0x63

The read function just treats what it reads as a binary blob. It doesn't care what it actually represents. So the key is really just to get the Content-Type set correctly.

req = HTTP.Request(
            "POST",
            "http://foobar/image",
            ["Content-Type" => "image/png"],
            blob)

So imagine we sent this request to the server, and it needs to pull the image out. First we get hold of our blob of binary data:

julia> blob = HTTP.payload(req)
69126-element Vector{UInt8}:
 0x89
 0x50
    ⋮
 0x6f
 0x35
 0x63

Julia has a generic system for loading any type of file offered through the FileIO package. Others can extend the interfaces offered by FileIO to support the loading of a larger selection of file types.

A key function is FileIO.load, but this only support loading directly from file or from an IO or Stream object. You cannot load from a binary blob. Thus we need to wrap the blob in an IO object. IOBuffer to the rescue!

julia> io = IOBuffer(blob);

Unfortunately loading from an IO object means the load system has to guess what filetype it is dealing with from the data discovered while reading. This seems unnecessary if we actually know what sort of data we are dealing with.

Thus FileIO has introduced the concept of Stream objects which you can think of as typed IO objects. E.g. this is how I create a stream object for reading .png files.

julia> stream = Stream{format"PNG"}(io)

The loader will know that it is dealing with .png data without guessing.

julia> image = load(stream)

I will consider adding more info about how to work with multipart form data and images if there is interest in it.

Working With Forms

Here there are some gotchas. I misunderstood the use of forms by failing to observe a few important facts:

  1. When you click the submit button, web browser sends a GET request by default, not a POST request as I had assumed.
  2. You must give each input widget a name not an id for data to actually be transported. The id attribute is used to refer to the HTML tag from CSS, not for transmitting data to the server.

If this seems abstract. Let us look at an example. We will create a page that looks like this, where we can send the name and race of some Middle Earth character to the server.

Submit info about Frodo to server
Submit info about Frodo to server

To create this form we write the following HTML code:

<html lang="en">
<body>
  <form>
    <input type="text" name="fname" value="Frodo"/>
    <input type="text" name="lname" value="Baggins"/>

    <select name="race">
      <option value="elf">Elf</option>
      <option value="hobbit">Hobbit</option>
      <option value="orc">Orc</option>
      <option value="man">Man</option>
    </select>
    
    <input type="submit" value="update"/>
  </form>
</body>
</html>

Whether you are using a button, textfield or checkbox you use the input tag. However the type attribute will vary. To be able to transmit the data input to this widget to the server through an HTTP request object, you need to add the name attribute such as name="fname".

The value attribute gives a default value for the textfield, but is otherwise not important. The dropdown box is created with the select and option tags and work a bit different since you need a way of storing all the different possible choices.

To be able to create a request storing all these values you need the UI button with type="submit". Once you click this button a request will be made.

Above you see this expressed as a GET request. After the URL you get parameters listed as:

form.html?firstname=Frodo&lastname=Baggins&race=hobbit

Each key-value pair is separated by a question mark ?. If we want to pass data as a POST request instead we need to modify the form tag.

<html lang="en">
<body>
  <form method="post">
  ⋮
  </form>
</body>
</html>

If you click the update button to submit the form you will no longer see the chosen values in the address bar. So how do you see the posted values? They are not part of the payload. We can however explore them with the various built in developer tools which come with different browsers.

Safari Developer tools showing data posted when sending request to server. POST data highlighted with orange outline.
Safari Developer tools showing data posted when sending request to server. POST data highlighted with orange outline.

Okay, so we got some basic idea of how to setup the HTML page and inspect its behavior using a Web browser. Now let us look at how we can process this info from Julia. It is worth knowing that form data will be sent from the browser as the MIME type application/x-www-form-urlencoded. This is stored with the Content-Type header key.

using HTTP, Sockets, URIs

body = read("form.html", String) # your HTML page

HTTP.serve(ip"127.0.0.1", 8080) do req::HTTP.Request
   @show req["Content-Type"]

   if req["Content-Type"] == "application/x-www-form-urlencoded"
       payload = HTTP.payload(req, String)
       dict = queryparams(payload)
       @show dict
       return HTTP.Response(200, body)
   else
       return HTTP.Response(200, body)
   end
end

If you store your HTML page in form.html, you can use this code snippet above to run a little web server on localhost at port 8080. Just direct your web browser to address: http://http://localhost:8080.

Related Articles

Julialang
Web Development
Rest
Programming
Http Request

--

--

Erik Engheim
Erik Engheim

Written by Erik Engheim

12.8K Followers
98 Following

Geek dad, living in Oslo, Norway with passion for UX, Julia programming, science, teaching, reading and writing.

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams