REST Maturity

rest-maturity2In 2008, Leonard Richardson published his Maturity Heuristic that classified web services into three levels based on their use of URI, HTTP, and hypermedia.

Back then, most web services were stuck at either level 1 or 2. Unfortunately, not a whole lot has improved since then in that respect: so-called pragmatic REST is still the norm.

BTW, I really dislike the term “pragmatic REST”. It’s a cheap rhetoric trick to put opponents (“dogmatists”) on the defensive.

More importantly, it creates semantic diffusion: pragmatic REST is not actually REST according to the definition, so please don’t call it that way or else we’re going to have a hard time understanding each other. The term REST hardly means anything anymore these days.

Anyway, there is some light at the end of the tunnel: more services are now at level 3, where they serve hypermedia. A good example by a big name is Amazon’s AppStream API.

The difference between plain media types, like image/jpeg, and hypermedia types, like text/html, is of course the “hyper” part. Links allow a client to discover functionality without being coupled to the server’s URI structure.

JSONBTW, application/json is not a hypermedia type, since JSON doesn’t define links.

We can, of course, use a convention on top of JSON, for instance that there should be a links property with a certain structure to describes the links, like Spring HATEOAS does.

The problem with conventions is that they are out-of-band communication, and a client has no way of knowing for sure whether that convention is followed when it sees a Content-Type of application/json. It’s therefore much better to use a media type that turns the convention into a rule, like HAL does.

Speaking of out-of-band communication, the amount of it steadily decreases as we move up the levels. This is a very good thing, as it reduces the amount of coupling between clients and servers.

Level 3 isn’t really the end station, however. Even with a hypermedia format like HAL there is still a lot of out-of-band communication.

HALHAL doesn’t tell you which HTTP method to use on a particular link, for instance.

The client can only know because a human has programmed it with that knowledge, based on some human-readable description that was published somewhere.

Imagine that the human Web would work this way. We wouldn’t be able to use the same browser to shop at Amazon and read up at Wikipedia and do all those other things we take for granted. Instead, we would need an Amazon Browser, a Wikipedia Browser, etc. This is what we do with APIs today!

Moving further into the direction of less out-of-band communication requires more than just links. Links only specify the URI part and we also need the HTTP and media type parts inside our representations. We might call this level 3b, Full Hypermedia.

Siren gives you this. Uber even goes a step further and also abstracts the protocol, so that you can use it with, say, CoAP rather than HTTP.

These newer hypermedia types allow for the use of a generic client that can handle any REST API that serves that hypermedia type, just like a web browser can be used against anything that serves HTML. An example of such an effort is the HAL browser (even though HAL is stuck at level 3a).

However, even with the inclusion of protocol, media type, and method in the representation, we still need some out-of-band communication.

The HAL browser can navigate any API that serves HAL, but it doesn’t understand the responses it gets. Therefore it can’t navigate links on its own to reach a certain goal. For true machine-to-machine (M2M) communication, we still need more.

ALPSIf we ever get the whole semantic web sorted out, this might one day be the final answer, but I’m not holding my breath.

In the meantime we’ll have to settle for partial answers.

One piece of the puzzle could be to define application semantics using profiles, for instance in the ALPS format. We might call this level 4, Semantic Profile.

We’d still need a human to read out-of-band communication and build a special-purpose client for M2M scenarios. But this client could handle all services in the application domain it is programmed to understand, not just one.

Also, the human could be helped a lot by a generic API browser that fetches ALPS profiles to explain the API.

All this is currently far from a reality. But we can all work towards this vision by choosing generic, full-featured hypermedia types like Siren or Uber for our APIs and by documenting our application semantics using profiles in ALPS.

If you need more convincing then please read RESTful Web APIs, which Leonard Richardson co-wrote with Uber and ALPS creator Mike Amundsen. This is easily the best book on REST on the market today.

Securing HTTP-based APIs With Signatures

CloudSecurityI work at EMC on a platform on top of which SaaS solutions can be built.

This platform has a RESTful HTTP-based API, just like a growing number of other applications.

With development frameworks like JAX-RS, it’s relatively easy to build such APIs.

It is not, however, easy to build them right.

Issues With Building HTTP-based APIs

The problem isn’t so much in getting the functionality out there. We know how to develop software and the available REST/HTTP frameworks and libraries make it easy to expose the functionality.

That’s only half the story, however. There are many more -ilities to consider.

rest-easyThe REST architectural style addresses some of those, like scalability and evolvability.

Many HTTP-based APIs today claim to be RESTful, but in fact are not. This means that they are not reaping all of the benefits that REST can bring.

I’ll be talking more about how to help developers meet all the constraints of the REST architectural style in future posts.

Today I want to focus on another non-functional aspect of APIs: security.

Security of HTTP-based APIs

In security, we care about the CIA-triad: Confidentiality, Integrity, and availability.

Availability of web services is not dramatically different from that of web applications, which is relatively well understood. We have our clusters, load balancers, and what not, and usually we are in good shape.

Confidentiality and integrity, on the other hand, both require proper authentication, and here matters get more interesting.

Authentication of HTTP-based APIs

authenticationFor authentication in an HTTP world, it makes sense to look at HTTP Authentication.

This RFC describes Basic and Digest authentication. Both have their weaknesses, which is why you see many APIs use alternatives.

Luckily, these alternatives can use the same basic machinery defined in the RFC. This machinery includes status code 401 Unauthorized, and the WWW-Authenticate, Authentication-Info, and Authorization headers. Note that the Authorization header is unfortunately misnamed, since it’s used for authentication, not authorization.

The final piece of the puzzle is the custom authentication scheme. For example, Amazon S3 authentication uses the AWS custom scheme.

Authentication of HTTP-based APIs Using Signatures

The AWS scheme relies on signatures. Other services, like EMC Atmos, use the same approach.

It is therefore good to see that a new IETF draft has been proposed to standardize the use of signatures in HTTP-based APIs.

Standardization enables the construction of frameworks and libraries, which will drive down the cost of implementing authentication and will make it easier to build more secure APIs.

What do you think?

what-do-you-thinkIf you’re in the HTTP API building and/or consuming business –and who isn’t these days– then please go ahead and read the draft and provide feedback.

I’m also interested in your experiences with building or consuming secure HTTP APIs. Please leave a comment on this post.

The Differences Between Test-First Programming and Test-Driven Development

Red, Green, RefactorThere seems to be some confusion between Test-First Programming and Test-Driven Development (TDD).

This post explains that merely writing the tests before the code doesn’t necessarily make it TDD.

Similarities Between Test-First Programming and Test-Driven Development

It’s not hard to see why people would confuse the two, since they have many things in common.

My classification of tests distinguishes six dimensions: who, what, when, where, why, and how.

Test-First programming and Test-Driven Development score the same in five of those six dimensions: they are both automated (how) functional (what) programmer (who) tests at the unit level (where) written before the code (when).

The only difference is in why they are written.

Differences Between Test-First Programming and Test-Driven Development

Test-First Programming mandates that tests be written before the code, so that the code will always be testable. This is more efficient than having to change already written code to make it testable.

Test-First Programming doesn’t say anything about other activities in the development cycle, like requirements analysis and design.

This is a big difference with Test-Driven Development (TDD), since in TDD, the tests drive the design. Let’s take a detailed look at the TDD process of Red/Green/Refactor, to find out exactly how that differs from Test-First Programming.

Red

Unit test failureIn the first TDD phase we write a test. Since there is no code yet to make the test pass, this test will fail.

Unit testing frameworks like JUnit will show the result in red to indicate failure.

In both Test-First Programming and Test-Driven Development, we use this phase to record a requirement as a test.

TDD, however, goes a step further: we also explicitly design the client API. Test-First Programming is silent on how and when we should do that.

Green

In the next phase, we write code to make the test pass. Unit testing frameworks show passing tests in green.

In Test-Driven Development, we always write the simplest possible code that makes the test pass. This allows us to keep our options open and evolve the design.

JUnit passing testWe may evolve our code using simple transformations to increase the complexity of the code enough to satisfy the requirements that are expressed in the tests.

Test-First Programming is silent on what sort of code you write in this phase and how you do it, as long as the test will pass.

Refactor

In the final TDD phase, the code is refactored to improve the design of the implementation.

This phase is completely absent in Test-First Programming.

Summary of Differences

So we’ve uncovered two differences that distinguish Test-First Programming from Test-Driven Development:

  1. Test-Driven Development uses the Red phase to design the client API. Test-First Programming is silent on when and how you arrive at a good client API.
  2. Test-Driven Development splits the coding phase into two compared to Test-First Programming. In the first sub-phase (Green), the focus is on meeting the requirements. In the second sub-phase (Refactor), the focus is on creating a good design.

I think there is a lot of value in the second point. Many developers focus too much on getting the requirements implemented and forget to clean up their code. The result is an accumulation of technical debt that will slow development down over time.

TDD also splits the design activity into two. First we design the external face of the code, i.e. the API. Then we design the internal organization of the code.

This is a useful distinction as well, because the heuristics you would use to tell a good API from a bad one are different from those for good internal design.

Try Before You Buy

KataAll in all I think Test-Driven Development provides sufficient value over Test-First Programming to give it a try.

All new things are hard, however, so be sure to practice TDD before you start applying it in the wild.

There are numerous katas that can help you with that, like the Roman Numerals Kata.