Explore REST APIs with cURL
Clueless about working with Web Services?
Nothing makes me feel more like fish out of water than working with web technologies. This this is an overview as much for your benefit as for my own about how to work with REST APIs primarily using the cURL Unix command. I will also look at how you can do this from some different programming languages.
To avoid that you have to jump around all over the place, I will try to put all the key information right in this article.
What is a REST API?
It is not like a programming API when you are using C/C++, Python etc. Rather it is an API exposed as a set of URLs. That means you could actually interact with a REST API using nothing but your Web browser. REST is just a simple way of interacting with an application running on a server using Web technology such as the HTTP protocol.
You do this primarily by using one of two HTTP requests:
GET
This is what happens when you write a URL in your web browser. A GET request is sent to the web sever and some data is returned, usually a web page in HTML which your web browser shows.POST
Is used to send data. It also involves specifying a URL but this also involves supplying some data to send.
By request I mean that you send some data to a server over your TCP/IP connection with the intention on getting some specific data back.
You could e.g. visit my home page by writing the URL address http://blog.translusion.com
in your web browser, or you could use the Unix curl
command to download the HTML page:
$ curl http://blog.translusion.com
If you add the -i
switch as well you will also get the HTTP header which looks something like this:
$ curl -i http://blog.translusion.com
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Last-Modified: Thu, 15 Oct 2020 22:47:13 GMT
Content-Length: 9879
Accept-Ranges: bytes
Age: 146
Connection: keep-alive
X-Served-By: cache-osl6530-OSL
You can actually accomplish the same with a far more “primitive” tool called NetCat invoked with the Unix command nc
. With this command you basically just send over raw data. The benefit is that you can better understand exactly how the HTTP protocol works.
$ nc blog.translusion.com 80
GET / HTTP/1.1
Host: blog.translusion.com
What you see below the command isn’t returned. Rather this is what I wrote in the terminal. You hit enter an extra time and you will get the same content back as with curl
. Hit Ctrl+D
to exit.
It may not be obvious what you should write into nc
to carry out a HTTP request. But you can cheat by running nc
in server mode and connect to it using curl
. Then you can learn what exactly it is that curl
does. In a separate window you can run this command to listen for connections on port 1234:
$ nc -l 1234
Then in your original terminal window you can make a request to your local nc
server using curl
$ curl -i http://localhost:1234/foo/bar.html
On the fake server side you will then see this output:
$ nc -l 1234
GET /foo/bar.html HTTP/1.1
Host: localhost:1234
User-Agent: curl/7.64.1
Accept: */*
With the HTTP response you get a bunch of info in the header. The most important part is parts like this:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
It tells you the HTTP protocol version used, whether the request went okay and what sort of data you got back and how it was formatted. Here are some common HTTP status codes which you have probably seen at some point already:
200 OK
Success400 Bad Request
Bad syntax in request401 Unauthorized
Failed authentication402 Not found
Web page isn't there
Tools of the Trade
Ok so hopefully that gives you a vague an idea of what I am talking about when I say a REST API. We will look at some more examples later. It is useful to be aware of some of the tools you could potentially use to work with a REST API.
- curl fetch websites or any other data using HTTP protocol.
- curlish Like curl but support OAuth authentication. Used by e.g. Facebook.
- jq Like sed for JSON data
- nc NetCat is a sort of “swiss army” knife for TCP/IP. You can use it both as a server or client. You can read and send raw HTTP requests and replies.
- ifconfig lists network interfaces and their IP addresses
- Charles Proxy Lets you see all traffic going between you and the internet. Responses and requests can be recorded and replayed. A very versatile tool.
Reading REST API Documentation
Let us look at how you can use curl
to explore a documented REST API. Here is an example from JFrog which I have recently been working on. JFrog offers a number of services, one which is to have a repository for binary files which they refer to as an Artifactory.
File Info
One of their API calls is File Info which is used to fetch info about a file stored in a repository you have created in the JFrog artifactory.
The important part of the documentation gives the following info:
- Usage:
GET /api/storage/{repoKey}/{filePath}
- Produces:
application/json (application/vnd.org.jfrog.artifactory.storage.FileInfo+json)
Usage tells us that we need to specify this as a GET
request and where Produces tells us that the response we get back will be a JSON data.
The curly braces indicate place holders. So {repoKey}
is the name of your repository and {filePath}
is the path to some file in the repo you want info about.
In my example below I am calling this API by specifying stuff
as the name of my repo and the file path as myfile.txt
.
$ curl -u erik:qwerty -X GET "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"
If you setup nc
as a server and made this connection to a local nc
server this curl
call would have caused this text to be transferred to the server.
GET /artifactory/api/storage/stuff/myfile.txt HTTPS/1.1
Host: mickeymouse.jfrog.io:80
Authorization: Basic ZXJpazpxd2VydHkK
Notice how adding -u erik:qwerty
to specify a login name and a password causes the following field to be added to request:
Authorization: Basic ZXJpazpxd2VydHkK
How do we get that? In fact you can create this string yourself. ZXJpazpxd2VydHkK
is actually just a Base64 encoding of erik:qwerty
. An encoding is just a way of storing some information in a different format. Base64 involves converting every collection of 6-bits into a visible character from 0 to 63.
We can perform this with the base64
Unix command:
echo "erik:qwerty" | base64
ZXJpazpxd2VydHkK
Making this File Info call will give JSON data as feedback. We typically want to parse this in some way.
{
"repo" : "stuff",
"path" : "/myfile.txt",
"created" : "2020-10-16T12:51:33.916Z",
"createdBy" : "erik",
"lastModified" : "2020-10-16T12:51:33.699Z",
"modifiedBy" : "erik",
"lastUpdated" : "2020-10-16T12:51:33.917Z",
"downloadUri" : "https://mickeymouse.jfrog.io/artifactory/stuff/myfile.txt",
"mimeType" : "text/plain",
"size" : "42",
"checksums" : {
"sha1" : "d0d8980bdbe9622acff4c41614d84b9692dd13be",
"md5" : "db98b69f101495872bda4805e2803742",
"sha256" : "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
},
"originalChecksums" : {
"sha256" : "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
},
"uri" : "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"
}
We can pull out the MD5 checksum using jq
like this:
$ curl -u erik:qwerty -X GET "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt" | jq '.["checksums"]["md5"]'
"db98b69f101495872bda4805e2803742"
Using Julia to work with REST APIs
We need some various packages to use Julia with REST APIs.
(@v1.5) pkg> add HTTP
(@v1.5) pkg> add JSON
With the HTTP
package we can perform REST requests:
julia> using HTTP
julia> r = HTTP.request(:GET, "https://erik:qwerty@mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt");
julia> julia> fieldnames(typeof(r))
(:version, :status, :headers, :body, :request)
With the JSON package we can extract data from the payload body of the request response:
julia> using JSON
julia> s = String(r.body);
julia> dict = JSON.parse(s);
julia> dict["checksums"]
Dict{String,Any} with 3 entries:
"sha256" => "3bc6b50993e609908d2946c748b8eee664201c9bbb0a45b9648a6bc9f64c1b15"
"md5" => "db98b69f101495872bda4805e2803742"
"sha1" => "d0d8980bdbe9622acff4c41614d84b9692dd13be"
julia> dict["checksums"]["md5"]
"db98b69f101495872bda4805e2803742"
Using Python to work with REST APIs
The problem with using Julia or jq
to process REST APIs is that neither software is usually included in random Linux installation. Why does that matter? Well often you run this stuff in a docker container. E.g. I am using Bitbucket pipelines which is a way to trigger running of code upon source code commits.
In my case I wanted to send the produced binary code from the build to a JFrog binary repository using REST API calls. That means typically using some standard docker containers. In these containers you will usually have the following installed:
curl
python
2.7
But jq
, julia
, ruby
and lots of other potentially useful stuff will not be there. Secondly you don't have a lot of Python packages installed either. Thus what we will explore here isn't to work with REST using some ultimate Python package, but rather with the stuff that comes out of the box with Python.
We will typically have access to the python JSON package. Read example of usage here.
Below is a Python 2.7 solution based on standard installed libraries. We use the urllib2
package to create a URL request.
import urllib2
import base64
import json
url = "https://mickeymouse.jfrog.io/artifactory/api/storage/stuff/myfile.txt"
login = base64.encodestring("erik:qwerty")[:-1]
authheader = "Basic %s" % login
req = urllib2.Request(url)
req.add_header("Authorization", authheader)
io = urllib2.urlopen(req)
s = io.read()
dict = json.loads(s)
dict[u'checksums'][u'md5']
This solution needs some comments. For instance why is there a [:-1]
for creating the login
variable? Because the base64.encodestring
adds a newline at the end, which we don't want to include. It is not part of the Base-64 encoding
Also note that with urllib2
you cannot write:
url = "https://erik:qwerty@mickeymouse.jfrog.io"
The package will misunderstand and think you tried to provide a port number rather than providing login name and password. For this reason we had to manually construct the header entry for basic authentication with the code:
login = base64.encodestring("erik:qwerty")[:-1]
authheader = "Basic %s" % login
req.add_header("Authorization", authheader)
Final Remarks
This is in no way an exhaustive treatment. I have really just covered how to do a moderately complicated REST call in different ways.