August 16, 2019

Working with REST APIs - Part 1

Kevin Hood

A question we often get asked is: What exactly is a REST API, and how does it work? While APIs have long had their place in basic developer lexicon, their growing importance to all IT professionals is becoming hard to ignore. The growth in the number of APIs increased by its fastest ever rate in 2019 and, as a consequence of this API explosion, there are now over 22,000 publicly available APIs (according to leading online journal ProgrammableWeb). These range from the highly amusing Chuck Norris API, through to the business-critical Splunk, Cherwell, PRTG and ServiceNow APIs.

To aid our IT friends in navigating this new terrain, we have written up a two-part series explaining REST APIs. Part One (this blog) will illustrate with examples exactly what REST APIs are, while Part Two will show you how to put this knowledge into practice by integrating the ServiceNow REST API with SquaredUp.

Part 1: REST APIs explained

In this blog post, we’re going to cover the basic concepts around REST APIs, including:

What is REST?
The HTTP protocol functionality that REST APIs rely on
The JSON data format
REST API authorisation mechanisms

What is a REST API?

First and foremost, an API (or Application Programming Interface) is a way for two applications to talk to each other, and the way this communication happens can be proprietary, conform to a protocol standard, or follow general architectural principles that have become best practice over the years.

This last category is where we find REST APIs. REST is an acronym that stands for REpresentational State Transfer – not that its full name aids our understanding whatsoever, so maybe note it down as something to impress grandma with next time she asks you what you’re working on and let’s move on.

A REST API is normally an API that:

Uses the HTTP protocol as a transport mechanism (and TLS for transport security)
Allows access to data via URLs that map to distinct resources
Lets us manipulate resources using standard HTTP methods
Gives us a result for a given operation using the standard HTTP status codes
Often returns response data in JSON format

Looking at this list, you can see the magic of REST APIs: they’re built on existing, widely adopted technologies and standards. There’s no strange, proprietary transport protocol that we have to beg our security team to open up a port for, and, assuming that our client and server support the latest and greatest TLS version, we can feel confident that our data is safe from man-in-the-middle attacks.

That’s easy enough to understand, but you might be wondering: what are these resources, HTTP methods, status codes and data formats that you speak of?

The Magic of URLs

Before we can talk about resources in the context of REST APIs, we need to talk about URLs. The ability of URLs to address data that is in a hierarchical structure makes them a useful tool for accessing collections of related items, as well as individual items. Resources in REST APIs are either collections of related objects or individual objects, so we can use URLs to tell the API what data we want to work with.

Let’s say that you’ve had a stressful week at work and the thing that is guaranteed to relax you is household inventory management. No? Have you tried it though? You might be pleasantly surprised. It’s second only to buying a label gun and labelling everything, in my experience. Anyway, some weeks we have more milk in the fridge than we could ever drink, and other weeks we’re noisily crunching our way through bone-dry corn flakes. We need an automated system to keep track of what is in the fridge. We decide that we want the system to have a REST API, because that way, any device capable of using HTTP can interact with the system. As previously mentioned, REST APIs allow access to resources, and resources can either be collections of objects or the objects themselves. A collective term for everything that is in our fridge would probably be something like products, so this could be a good top-level collection name in our hierarchy:

http://api.myinventory.local/products

If we call our API with this URL it returns all the products in our fridge. As mentioned previously, REST APIs generally return data in a format known as JSON. JavaScript Object Notation (your grandma’s going to think you’re in the NSA when you drop this little beauty into the conversation), is a data format that’s comparable in function to XML but much less likely to make you want to bash your head against the wall in frustration. The data returned from our call to the products resource would probably look something like this in JSON format:

This is great, but I really only want to know about milk – I don’t care about that awful smelly cheese that’s been in there so long it’s tantalisingly close to sentience. Luckily, we can see in the JSON result data that each product that was returned happens to have a unique ID. Using this ID we can ask for the details of the milk product from our collection of products by adding it to the URL:

http://api.myinventory.local/products/2

The result from this call might look like this:

Awesome. Now we have information that applies to all bottles of milk, like barcodes and the default shelf life. How about finding out whether we actually have any milk bottles in the fridge, for crying out loud? Well, there might be multiple bottles bought at different times with different expiry dates. That’s a collection of milk bottles underneath the milk product. A sub-collection. We can ask our system for details on each bottle of milk in the fridge by referencing this sub-collection (I’ve chosen the name instances for this sub-collection in the example URL below):

http://api.myinventory.local/products/2/instances

The result from this call might be as follows:

Now we have details on each bottle of milk in our fridge and those details might, for instance, tell us when each bottle was bought or any other instance-specific information like what price that bottle of milk was in comparison to the others. If no results were returned, then we might as well just pour the cornflakes directly into our mouths because we’re out of milk.

Some specifics about the JSON format:

The screenshots I’ve used so far have examples of what JSON can contain in terms of value types. We can have:

A sequence of characters like “milk”
Null values
Numbers
Collections (Arrays) of values
Other valid JSON

The keys and values are contained within a pair of curly braces and must be separated from each other by a comma.

If you want to ensure that a value is seen as a number by whichever application reads the JSON, then enter it without double quotes around it. In my examples, I’m treating the barcodes as normal text but the shelf_life value is treated as a number.

The instances key demonstrates a combination of the last two value types in my list. The square brackets in the value for this key indicate that it is a collection (array) of other values. What are the values in this collection? They’re other JSON objects. This is the power of JSON: We’ve got data nested within data, the relationship between these two sets of data is clear to see and crucially we will be able to use the relationship to access the values that we need, no matter how complicated the nesting gets.

Finally, if we look at result at the top of the screenshot we can see that all of this JSON is just one complicated value for the result key, which is another example of nesting one JSON object within another JSON object.

Modifying Data

But this is all a little read-only isn’t it? How is this any good for adding, updating and deleting data? After all, we need to tell the system that we’ve drunk the last bottle of milk, which means deleting an instance of the milk product. If you’ve ever looked in a web server log or opened up the network tab in your browser developer tools then you will have likely spotted capitalized words like GET and POST next to URLs that have been requested by a client. These are HTTP methods (sometimes known as verbs). They are part of everyday life when we use our browsers to open a web page or submit our personal details to those web sites that have suspiciously vague Privacy Policies but promise the world.

Luckily the HTTP standard has HTTP methods that match up perfectly with CRUD operations (Create, Read, Update and Delete) so they’re perfect for the kind of data manipulation that we might use an API for. Most REST APIs use the HTTP methods as follows:

Create new data – POST

Read existing data – GET

Update existing data – PUT

Delete existing data - DELETE

Great, so now we can delete that bottle of milk by specifying the DELETE method in our HTTP call using a URL that identifies the instance of the bottle of milk we want to delete e.g.

http://api.myinventory.local/products/2/instances/1

However, if we ask the system to delete some data, then we’re going to need to know if that request was successful. This is where HTTP status codes come in. Like HTTP methods, these are used by our browsers when we try and open a web page.

Errors – when it doesn’t all go to plan

I think most people are familiar with the HTTP status code 404 – Page Not Found, but there are many other status codes that can be returned to a client as a result of a request that it made. These are handily grouped into separate sections that give a general idea of what happened with the request:

Informational 1XX
Successful 2XX
Redirection 3XX
Client Error 4XX
Server Error 5XX

We can see from this list that a 404 is a Client Error; the client browser requested a page that doesn’t exist, most likely because we were simultaneously trying to watch TV, eat our dinner and correctly type a URL into the address bar with one finger – and we failed. It’s the client’s fault that the request didn’t succeed so the error code is in the 4xx range. If we tried this again but this time typed the URL correctly then the browser would receive the content of the web page we wanted along with a Successful status code in the 2xx range, normally 200. REST APIs leverage these same status codes to let clients know the result of their requests.

If we’re deleting a bottle of milk from our inventory system then we’re not really going to expect any actual data about the instance back because it will have been deleted. We just want to know that the deletion succeeded. If the API provides a response in the 2xx range, then we can be assured that the instance was in fact deleted.

HTTP Headers and URL Parameters

In my inventory system example, we have interacted with data exposed by the API using only the URL in combination with HTTP methods like GET and DELETE. But what if we need to give the server a bit more information about our request so that it’s able to properly process that request? For example, the server may be able to return data in both JSON format and XML format, but we want to specify that it must be returned in JSON format. Well it just so happens that there is existing functionality in the HTTP standard that can be used for sending additional data to the server along with the URL and the method: HTTP Headers. Any headers we specify will become part of the request we send to the API service. We can use the Accept header to specify that we want JSON back from the server rather than horrible XML:

Accept: application/json

What’s the purpose of HTTP headers outside of their usage in APIs? Well, an example might be a website needs to know whether you’re on a mobile device or a desktop PC so that it can serve up the appropriate version of the website you requested. The User-Agent header tells the server what browser type and version you’re using, so if you’re using Safari on an iPhone then the server’s going to give you the mobile version of the web site. Here’s an example of what the User-Agent might look like in this scenario:

User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A356 Safari/604.1

There’s other information that we might need to pass to the server when making API calls. For instance, what if our API exposes a server-side search function? We can’t have a URL endpoint for every possible search term a user might enter, so how are we going to submit our search? We could use HTTP Headers for this, but best practice for APIs is to use another HTTP mechanism instead: URL parameters. You may have noticed a question mark appended to the end of URLs in your address bar. This denotes that what follows will be a set of parameters. For example, q is a parameter and squaredup is the value for this parameter:

In the same way that Google uses URL Parameters to pass the search term from the client to the server, API client applications can determine what data they receive from a service by using URL parameters. Some additional information on when and why parameters are used in preference to headers in REST APIs can be found in this blog post from Kay Ploesser.

Different APIs need different combinations of headers and parameters. The API’s documentation should specify what options are available and whether they should be supplied in the HTTP Request Header or as a parameter in the URL.

Authentication – using the keys to open the door

APIs can be as simple as my household inventory example, where all you need to know is the appropriate URL to call to traverse the collections and objects provided by the API. A good example of this is the Star Wars API, which I encourage you to check out. However, whilst information on every known species in the Star Wars universe could arguably be classed as critical information, it is probably less important to secure than say, information about your servers. APIs that host this sort of information need to secure it using authorization and authentication mechanisms.

In our inventory system deletion example all we did was specify the DELETE method along with a URL that identified the bottle of milk we wanted to delete, but what if you’re so obsessed with security that you don’t want anyone else to mess with your inventory system? You need to tell the system that it’s you and not your spouse/child/very IT literate pet (delete as applicable). There are a variety of authentication mechanisms used by REST APIs, ranging from Basic Authentication through to something called OAuth 2.0.

As I mentioned earlier, HTTP Request Headers allow us to put more information into our API request than just the method and the URL: like credentials for authorization. There are a variety of different authentication methodologies that REST APIs use, but the two main ones are Basic Authentication and OAuth. APIs that support basic authentication allow an application to authenticate by passing credentials in a request header field called Authorization. Often these credentials will be Base64 encoded, so they are obfuscated. However, Base64 encoded data can be easily decoded and there are web sites that will do this for you, so credentials encoded in this way are not secure.

Authentication for grown-ups: OAuth

An alternative mechanism to basic authentication is OAuth. An OAuth request from a client application uses an access token to authenticate itself with the service rather than user credentials. How this access token is acquired by the client application depends on the OAuth grant type that the service hosting the API uses. For example, in the case of the authorization code grant type the client application never sees the user’s credentials, whereas in the password grant type the client application stores the user’s credentials and it uses these credentials to obtain the access token. A great explanation of OAuth grant types and how tokens are obtained can be found on Alex Bilbie’s blog post: A Guide to OAuth 2.0 Grants.

Different APIs use different grant types, and this information will can be found in the publicly available documentation for each API. The good news is that SquaredUp supports the common OAuth grant types, so whichever APIs you use we should be able to work with them. The one caveat to this being that data must be returned in JSON format for SquaredUp to be able to work with it.

Hopefully, that has equipped you with an improved understanding of what REST APIs are and how they work. You are now ready for Part 2 of our series – where you put your new skills into practice and learn to integrate ServiceNow's REST API with SquaredUp!