Explore REST APIs with cURL

Clueless about working with Web Services?

Erik Engheim

Oct 18, 2020·7 min read

Nothing makes me feel more like fish out of water than working with web technologies. This this is an overview as much for your benefit as for my own about how to work with REST APIs primarily using the cURL Unix command. I will also look at how you can do this from some different programming languages.

To avoid that you have to jump around all over the place, I will try to put all the key information right in this article.

What is a REST API?

It is not like a programming API when you are using C/C++, Python etc. Rather it is an API exposed as a set of URLs. That means you could actually interact with a REST API using nothing but your Web browser. REST is just a simple way of interacting with an application running on a server using Web technology such as the HTTP protocol.

You do this primarily by using one of two HTTP requests:

• GET This is what happens when you write a URL in your web browser. A GET request is sent to the web sever and some data is returned, usually a web page in HTML which your web browser shows.
• POST Is used to send data. It also involves specifying a URL but this also involves supplying some data to send.

By request I mean that you send some data to a server over your TCP/IP connection with the intention on getting some specific data back.

You could e.g. visit my home page by writing the URL address http://blog.translusion.com in your web browser, or you could use the Unix curl command to download the HTML page:

$ curl http://blog.translusion.com

If you add the -i switch as well you will also get the HTTP header which looks something like this:

$ curl -i http://blog.translusion.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Last-Modified: Thu, 15 Oct 2020 22:47:13 GMT
Content-Length: 9879
Accept-Ranges: bytes
Age: 146
Connection: keep-alive
X-Served-By: cache-osl6530-OSL

You can actually accomplish the same with a far more “primitive” tool called NetCat invoked with the Unix command nc. With this command you basically just send over raw data. The benefit is that you can better understand exactly how the HTTP protocol works.

$ nc blog.translusion.com 80
GET / HTTP/1.1
Host: blog.translusion.com

What you see below the command isn’t returned. Rather this is what I wrote in the terminal. You hit enter an extra time and you will get the same content back as with curl. Hit Ctrl+D to exit.

It may not be obvious what you should write into nc to carry out a HTTP request. But you can cheat by running nc in server mode and connect to it using curl. Then you can learn what exactly it is that curl does. In a separate window you can run this command to listen for connections on port 1234:

$ nc -l 1234

Then in your original terminal window you can make a request to your local nc server using curl

$ curl -i http://localhost:1234/foo/bar.html

On the fake server side you will then see this output:

$ nc -l 1234
GET /foo/bar.html HTTP/1.1
Host: localhost:1234
User-Agent: curl/7.64.1
Accept: */*

With the HTTP response you get a bunch of info in the header. The most important part is parts like this:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8

It tells you the HTTP protocol version used, whether the request went okay and what sort of data you got back and how it was formatted. Here are some common HTTP status codes which you have probably seen at some point already:

• 200 OK Success
• 400 Bad Request Bad syntax in request
• 401 Unauthorized Failed authentication
• 402 Not found Web page isn't there

Tools of the Trade

Ok so hopefully that gives you a vague an idea of what I am talking about when I say a REST API. We will look at some more examples later. It is useful to be aware of some of the tools you could potentially use to work with a REST API.

• curl fetch websites or any other data using HTTP protocol.
• curlish Like curl but support OAuth authentication. Used by e.g. Facebook.
• jq Like sed for JSON data
• nc NetCat is a sort of “swiss army” knife for TCP/IP. You can use it both as a server or client. You can read and send raw HTTP requests and replies.
• ifconfig lists network interfaces and their IP addresses
• Charles Proxy Lets you see all traffic going between you and the internet. Responses and requests can be recorded and replayed. A very versatile tool.

Reading REST API Documentation

Let us look at how you can use curl to explore a documented REST API. Here is an example from JFrog which I have recently been working on. JFrog offers a number of services, one which is to have a repository for binary files which they refer to as an Artifactory.

File Info

One of their API calls is File Info which is used to fetch info about a file stored in a repository you have created in the JFrog artifactory.

The important part of the documentation gives the following info:

• Usage: GET /api/storage/{repoKey}/{filePath}
• Produces: application/json (application/vnd.org.jfrog.artifactory.storage.FileInfo+json)

Usage tells us that we need to specify this as a GET request and where Produces tells us that the response we get back will be a JSON data.

The curly braces indicate place holders. So {repoKey} is the name of your repository and {filePath} is the path to some file in the repo you want info about.

In my example below I am calling this API by specifying stuff as the name of my repo and the file path as myfile.txt.

$ curl -u erik:qwerty -X GET "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"

If you setup nc as a server and made this connection to a local nc server this curl call would have caused this text to be transferred to the server.

GET /artifactory/api/storage/stuff/myfile.txt HTTPS/1.1
Host: mickeymouse.jfrog.io:80
Authorization: Basic ZXJpazpxd2VydHkK

Notice how adding -u erik:qwerty to specify a login name and a password causes the following field to be added to request:

Authorization: Basic ZXJpazpxd2VydHkK

How do we get that? In fact you can create this string yourself. ZXJpazpxd2VydHkK is actually just a Base64 encoding of erik:qwerty. An encoding is just a way of storing some information in a different format. Base64 involves converting every collection of 6-bits into a visible character from 0 to 63.

We can perform this with the base64 Unix command:

echo "erik:qwerty" | base64
ZXJpazpxd2VydHkK

Making this File Info call will give JSON data as feedback. We typically want to parse this in some way.

{
  "repo" : "stuff",
  "path" : "/myfile.txt",
  "created" : "2020-10-16T12:51:33.916Z",
  "createdBy" : "erik",
  "lastModified" : "2020-10-16T12:51:33.699Z",
  "modifiedBy" : "erik",
  "lastUpdated" : "2020-10-16T12:51:33.917Z",
  "downloadUri" : "https://mickeymouse.jfrog.io/artifactory/stuff/myfile.txt",
  "mimeType" : "text/plain",
  "size" : "42",
  "checksums" : {
    "sha1" : "d0d8980bdbe9622acff4c41614d84b9692dd13be",
    "md5" : "db98b69f101495872bda4805e2803742",
    "sha256" : "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
  },
  "originalChecksums" : {
    "sha256" : "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
  },
  "uri" : "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"
}

We can pull out the MD5 checksum using jq like this:

$ curl -u erik:qwerty -X GET "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt" | jq '.["checksums"]["md5"]'
"db98b69f101495872bda4805e2803742"

Using Julia to work with REST APIs

We need some various packages to use Julia with REST APIs.

(@v1.5) pkg> add HTTP
(@v1.5) pkg> add JSON

With the HTTP package we can perform REST requests:

julia> using HTTP

julia> r = HTTP.request(:GET, "https://erik:qwerty@mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt");

julia> julia> fieldnames(typeof(r))
(:version, :status, :headers, :body, :request)

With the JSON package we can extract data from the payload body of the request response:

julia> using JSON
julia> s = String(r.body);

julia> dict = JSON.parse(s);

julia> dict["checksums"]
Dict{String,Any} with 3 entries:
  "sha256" => "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
  "md5"    => "db98b69f101495872bda4805e2803742"
  "sha1"   => "d0d8980bdbe9622acff4c41614d84b9692dd13be"

julia> dict["checksums"]["md5"]
"db98b69f101495872bda4805e2803742"

Using Python to work with REST APIs

The problem with using Julia or jq to process REST APIs is that neither software is usually included in random Linux installation. Why does that matter? Well often you run this stuff in a docker container. E.g. I am using Bitbucket pipelines which is a way to trigger running of code upon source code commits.

In my case I wanted to send the produced binary code from the build to a JFrog binary repository using REST API calls. That means typically using some standard docker containers. In these containers you will usually have the following installed:

• curl
• python 2.7

But jq, julia, ruby and lots of other potentially useful stuff will not be there. Secondly you don't have a lot of Python packages installed either. Thus what we will explore here isn't to work with REST using some ultimate Python package, but rather with the stuff that comes out of the box with Python.

We will typically have access to the python JSON package. Read example of usage here.

Below is a Python 2.7 solution based on standard installed libraries. We use the urllib2 package to create a URL request.

import urllib2
import base64
import json

url = "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"
login = base64.encodestring("erik:qwerty")[:-1]
authheader =  "Basic %s" % login

req = urllib2.Request(url)
req.add_header("Authorization", authheader)
io = urllib2.urlopen(req)

s = io.read()
dict = json.loads(s)
dict[u'checksums'][u'md5']

This solution needs some comments. For instance why is there a [:-1] for creating the login variable? Because the base64.encodestring adds a newline at the end, which we don't want to include. It is not part of the Base-64 encoding

Also note that with urllib2 you cannot write:

url = "https://erik:qwerty@mickeymouse.jfrog.io"

The package will misunderstand and think you tried to provide a port number rather than providing login name and password. For this reason we had to manually construct the header entry for basic authentication with the code:

login = base64.encodestring("erik:qwerty")[:-1]
authheader =  "Basic %s" % login
req.add_header("Authorization", authheader)

Final Remarks

This is in no way an exhaustive treatment. I have really just covered how to do a moderately complicated REST call in different ways.