How To Design a REST API

2014-10-272014-10-21 ~ Ray Sinnema ~ 2 Comments

rest-easy There is a lot of interest in REST APIs these days. Unfortunately, most APIs I see are not very mature.

In this post I’d like to share my approach to designing REST APIs:

Understand the problem domain and application requirements and document them as a state diagram
Discover the resources from the transitions
Name the resources with URIs
Select one or more media types to serialize the various representations identified in the resource model
Assign link relations to each of the transitions
Add documentation as required

Note that this is a variation of the design process described in RESTful Web Services. Now, let’s look at each of the steps in detail.

Step 1: Understand the problem domain and application requirements and document them as a state diagram

Without understanding the domain, it’s impossible to come up with a good design for anything.

We want to capture the domain in a way that makes sense for REST APIs. Since the purpose in life for a REST API is to be consumed by REST clients, it makes sense to document the application domain from the point of view of a REST client.

A REST client starts at some well-known URI (the billboard URI, or the URI of the home resource) and then follows links until its goal is met:
restClientFlow
In other words, a REST client starts at some initial state, and then transitions to other states by following links (i.e. executing HTTP methods against URIs) that are discovered from the previously returned representation.

A natural way to capture this information is in the form of a state diagram. For bonus points, we can turn the requirements into executable specifications using BDD techniques and derive the state diagram from the BDD scenarios.

For each scenario, we should specify the happy path, any applicable alternative paths and also the sad paths (edge, error, and abuse cases). We can do this iteratively, starting with only the happy path and then adding progressively more detail based on alternative and sad paths.

We can first collect all scenarios and build the entire state diagram from there. Alternatively, we can start with a select few scenarios, work through the design process, and then repeat everything with new scenarios.

In other words, we can do waterfall-like Big Analysis/Design Up Front, or work through feature by feature in a more Agile manner.

Either way, we document the requirements using a state diagram and work from there.

Step 2: Discover the resources from the transitions

You can build up the resource model piece-by-piece:

Start with the initial state
Create (or re-use) a resource with a representation that corresponds to this state
For each transition starting from the current state, make sure there is a corresponding method in some resource that implements the transition
Repeat for all transitions in each of the remaining states

Step 3: Name the resources with URIs

Every resource should be identified by a URI. From the client’s perspective, this is an implementation detail, but we still need to do this before we can implement the server.

We should follow best practices for URIs, like keeping them cool.

Step 4: Select one or more media types to serialize the various representations identified in the resource model

When extending an existing design, you should stick with the already selected media types.

For new APIs, we should choose a mature format, like Siren or Mason.

There could be specific circumstances where these are not good choices. In that case, carefully select an alternative.

Step 5: Assign link relations to each of the transitions

A REST client follows transitions in the state diagram by discovering links in representations. This discovery process is made possible by link relations.

Link relations decouple the client from the URIs that the server uses, giving the server the freedom to change its URI structure at will without breaking any clients. Link relations are therefore an important part of any REST API.

We should try to use existing link relations as much as possible. They don’t cover every case, however, so sometimes you need to invent your own.

Step 6: Add documentation as required

In order to help developers build clients that work against your API, you will most likely want to add some documentation that explains certain more subtle points.

Examples are very helpful to illustrate those points.

You may also add instructions for server developers that will implement the API, like what caching to use.

Conclusion

I’ve successfully used this approach on a number of APIs. Next time, I’ll show you with an example how the above process is actually very easy with the right support.

In the meantime, I’d love to hear how you approach REST API design. Please leave a comment below.

How To Control Access To REST APIs

2014-10-202014-10-19 ~ Ray Sinnema ~ Leave a comment

hacker Exposing your data or application through a REST API is a wonderful way to reach a wide audience.

The downside of a wide audience, however, is that it’s not just the good guys who come looking.

Securing REST APIs

Security consists of three factors:

In terms of Microsoft’s STRIDE approach, the security compromises we want to avoid with each of these are Information Disclosure, Tampering, and Denial of Service. The remainder of this post will only focus on Confidentiality and Integrity.

In the context of an HTTP-based API, Information Disclosure is applicable for GET methods and any other methods that return information. Tampering is applicable for PUT, POST, and DELETE.

Threat Modeling REST APIs

A good way to think about security is by looking at all the data flows. That’s why threat modeling usually starts with a Data Flow Diagram (DFD). In the context of a REST API, a close approximation to the DFD is the state diagram. For proper access control, we need to secure all the transitions.

The traditional way to do that, is to specify restrictions at the level of URI and HTTP method. For instance, this is the approach that Spring Security takes. The problem with this approach, however, is that both the method and the URI are implementation choices.

URIs shouldn’t be known to anybody but the API designer/developer; the client will discover them through link relations.

Even the HTTP methods can be hidden until runtime with mature media types like Mason or Siren. This is great for decoupling the client and server, but now we have to specify our security constraints in terms of implementation details! This means only the developers can specify the access control policy.

That, of course, flies in the face of best security practices, where the access control policy is externalized from the code (so it can be reused across applications) and specified by a security officer rather than a developer. So how do we satisfy both requirements?

Authorizing REST APIs

I think the answer lies in the state diagram underlying the REST API. Remember, we want to authorize all transitions. Yes, a transition in an HTTP-based API is implemented using an HTTP method on a URI. But in REST, we shield the URI using a link relation. The link relation is very closely related to the type of action you want to perform.

The same link relation can be used from different states, so the link relation can’t be the whole answer. We also need the state, which is based on the representation returned by the REST server. This representation usually contains a set of properties and a set of links. We’ve got the links covered with the link relations, but we also need the properties.

In XACML terms, the link relation indicates the action to be performed, while the properties correspond to resource attributes.

Add to that the subject attributes obtained through the authentication process, and you have all the ingredients for making an XACML request!

There are two places where such access control checks comes into play. The first is obviously when receiving a request.

You should also check permissions on any links you want to put in the response. The links that the requester is not allowed to follow, should be omitted from the response, so that the client can faithfully present the next choices to the user.

Using XACML For Authorizing REST APIs

I think the above shows that REST and XACML are a natural fit.

All the more reason to check out XACML if you haven’t already, especially XACML’s REST Profile and the forthcoming JSON Profile.

RESTBucks Evolved

2014-06-162014-06-22 ~ Ray Sinnema ~ 3 Comments

restbucks The book REST in Practice: Hypermedia and Systems Architecture uses an imaginary StarBucks-like company as its running example.

I think this is a great example, since most people are familiar with the domain.

The design is also simple enough to follow, yet complex enough to be interesting.

Problem Domain

RESTbucks is about ordering and paying for coffee (or tea) and food. Here is the state diagram for the client:

Create the order
Update the order
Cancel the order
Pay for the order
Wait for the order to be prepared
Take the order

Book Design

The hypermedia design in the book for the service is as follows:

The client POSTs an order to the well-known RESTBucks URI. This returns the order URI in the Location header. The client then GETs the order URI
The client POSTs an updated order to the order URI
The client DELETEs the order URI
The client PUTs a payment to the URI found by looking up a link with relation http://relations.restbucks.com/payment
The client GETs the order URI until the state changes
The client DELETEs the URI found by looking up a link with relation http://relations.restbucks.com/receipt

The book uses the specialized media type application/vnd.restbucks.order+xml for all messages exchanged.

Design Problems

Here are some of the problems that I have with the above approach:

I think the well-known URI for the service (what Mike Amundsen calls the billboard URI) should respond to a GET, so that clients can safely explore it.
This adds an extra message, but it also makes it possible to expand the service with additional functionality. For instance, when menus are added in a later chapter of the book, a second well-known URI is introduced. With a proper home document-like resource in front of the order service, this could have been limited to a new link relation.
I’d rather use PUT for updating an order, since that is an idempotent method. The book states that the representation returned by GET contains links and argues that this implies that (1) PUT messages should also contain those links and (2) that that would be strange since those links are under control of the server.
I disagree with both statements. A server doesn’t necessarily have to make the formats for GET and PUT exactly the same. Even if it did, some parts, like the links, could be optional. Furthermore, there is no reason the server couldn’t accept and ignore the links.
The DELETE is fine.
An alternative is to use PUT with a status of canceled, since we already have a status property anyway. That opens up some more possibilities, like re-instating a canceled order, but also introduces issues like garbage collection.
I don’t think PUT is the correct method. Can the service really guarantee under all circumstances that my credit card won’t get charged twice if I repeat the payment?
More importantly, this design assumes that payments are always for the whole order. That may seem logical at first, but once the book introduces vouchers that logic evaporates. If I have a voucher for a free coffee, then I still have to pay for anything to eat or for a second coffee.
I’d create a collection of payments that the client should POST to. I’d also use the standard payment link relation defined in RFC 5988.
This is fine.
This makes no sense to me: taking the order is not the same as deleting the receipt. I need the receipt when I’m on a business trip, so I can get reimbursed!
I’d rather PUT a new order with status taken.

Service Evolution

Suppose you’ve implemented your own service based on the design in the book.

Further suppose that after reading the above, you want to change your service.

How can you do that without breaking any clients that may be out there?

After all, isn’t that what proponents tout as one of the advantages of a RESTful approach?

Well, yes and no. The media type defined in the book is at level 3a, and so will allow you to change URIs. However, the use of HTTP methods is defined out-of-band and you can’t easily change that.

Now imagine that the book would have used a full hypermedia type (level 3b) instead. In that case, the HTTP method used would be part of the message. The client would have to discover it, and the server could easily change it without fear of breaking clients.

Of course, this comes at the cost of having to build more logic into the client. That’s why I think it’s a good idea to use an existing full hypermedia type like Mason, Siren, or UBER. Such generic media types are much more likely to come with libraries that will handle this sort of work for the client.

REST 101 For Developers

2013-09-162013-09-15 ~ Ray Sinnema ~ 1 Comment

rest-easy

Local Code Execution

Functions in high-level languages like C are compiled into procedures in assembly. They add a level of indirection that frees us from having to think about memory addresses.

Methods and polymorphism in object-oriented languages like Java add another level of indirection that frees us from having to think about the specific variant of a set of similar functions.

Despite these indirections, methods are basically still procedure calls, telling the computer to switch execution flow from one memory location to another. All of this happens in the same process running on the same computer.

Remote Code Execution

This is fundamentally different from switching execution to another process or another computer. Especially the latter is very different, as the other computer may not even have the same operating system through which programs access memory.

It is therefore no surprise that mechanisms of remote code execution that try to hide this difference as much as possible, like RMI or SOAP, have largely failed. Such technologies employ what is known as Remote Procedure Calls (RPCs).

One reason we must distinguish between local and remote procedure calls is that RPCs are a lot slower.

For most practical applications, this changes the nature of the calls you make: you’ll want to make less remote calls that are more coarsely grained.

Another reason is more organizational than technical in nature.

When the code you’re calling lives in another process on another computer, chances are that the other process is written and deployed by someone else. For the two pieces of code to cooperate well, some form of coordination is required. That’s the price we pay for coupling.

Coordinating Change With Interfaces

We can also see this problem in a single process, for instance when code is deployed in different jar files. If you upgrade a third party jar file that your code depends on, you may need to change your code to keep everything working.

Such coordination is annoying. It would be much nicer if we could simply deploy the latest security patch of that jar without having to worry about breaking our code. Fortunately, we can if we’re careful.

Interfaces in languages like Java separate the public and private parts of code.

The public part is what clients depend on, so you must evolve interfaces in careful ways to avoid breaking clients.

The private part, in contrast, can be changed at will.

From Interfaces to Services

In OSGi, interfaces are the basis for what are called micro-services. By publishing services in a registry, we can remove the need for clients to know what object implements a given interface. In other words, clients can discover the identity of the object that provides the service. The service registry becomes our entry point for accessing functionality.

There is a reason these interfaces are referred to as micro-services: they are miniature versions of the services that make up a Service Oriented Architecture (SOA).

A straightforward extrapolation of micro-services to “SOA services” leads to RPC-style implementations, for instance with SOAP. However, we’ve established earlier that RPCs are not the best way to invoke remote code.

Enter REST.

RESTful Services

Representational State Transfer (REST) is an architectural style that brings the advantages of the Web to the world of programs.

There is no denying the scalability of the Web, so this is an interesting angle.

Instead of explaining REST as it’s usually done by exploring its architectural constraints, let’s compare it to micro-services.

A well-designed RESTful service has a single entry point, like the micro-services registry. This entry point may take the form of a home resource.

We access the home resource like any other resource: through a representation. A representation is a series of bytes that we need to interpret. The rules for this interpretation are given by the media type.

Most RESTful services these days serve representations based on JSON or XML. The media type of a resource compares to the interface of an object.

Some interfaces contain methods that give us access to other interfaces. Similarly, a representation of a resource may contain hyperlinks to other resources.

Code-Based vs Data-Based Services

The difference between REST and SOAP is now becoming apparent.

In SOAP, like in micro-services, the interface is made up of methods. In other words, it’s code based.

In REST, on the other hand, the interface is made up of code and data. We’ve already seen the data: the representation described by the media type. The code is the uniform interface, which means that it’s the same (uniform) for all resources.

In practice, the uniform interface consists of the HTTP methods GET, POST, PUT, and DELETE.

Since the uniform interface is fixed for all resources, the real juice in any RESTful service is not in the code, but in the data: the media type.

Just as there are rules for evolving a Java interface, there are rules for evolving a media type, for example for XML-based media types. (From this it follows that you can’t use XML Schema validation for XML-based media types.)

Uniform Resource Identifiers

So far I haven’t mentioned Uniform Resource Identifiers (URIs). The documentation of many so-called RESTful services may give you the impression that they are important.

However, since URIs identify resources, their equivalent in micro-services are the identities of the objects implementing the interfaces.

Hopefully this shows that clients shouldn’t care about URIs. Only the URI of the home resource is important.

The representation of the home resource contains links to other resources. The meaning of those links is indicated by link relations.

Through its understanding of link relations, a client can decide which links it wants to follow and discover their URIs from the representation.

Versions of Services

As much as possible, we should follow the rules for evolving media types and not introduce any breaking changes.

However, sometimes that might be unavoidable. We should then create a new version of the service.

Since URIs are not part of the public interface of a RESTful API, they are not the right vehicle for relaying version information. The correct way to indicate major (i.e. non-compatible) versions of an API can be derived by comparison with micro-services.

Whenever a service introduces a breaking change, it should change its interface. In a RESTful API, this means changing the media type. The client can then use content negotiation to request a media type it understands.

What Do You Think?

Literature explaining how to design and document code-based interfaces is readily available.

This is not the case for data-based interfaces like media types.

With RESTful services becoming ever more popular, that is a gap that needs filling. I’ll get back to this topic in the future.

How do you design your services? How do you document them? Please share your ideas in the comments.

Secure Cloud Development

Musings on the Art and Craft of Creating Secure Software in the Cloud Era

link relation