REST Messages And Data Transfer Objects

In Patterns of Enterprise Application Architecture, Martin Fowler defines a Data Transfer Object (DTO) as

An object that carries data between processes in order to reduce the number of method calls.

Note that a Data Transfer Object is not the same as a Data Access Object (DAO), although they have some similarities. A Data Access Object is used to hide details from the underlying persistence layer.

REST Messages Are Serialized DTOs

message-transferIn a RESTful architecture, the messages sent across the wire are serializations of DTOs.

This means all the best practices around DTOs are important to follow when building RESTful systems.

For instance, Fowler writes

…encapsulate the serialization mechanism for transferring data over the wire. By encapsulating the serialization like this, the DTOs keep this logic out of the rest of the code and also provide a clear point to change serialization should you wish.

In other words, you should follow the DRY principle and have exactly one place where you convert your internal DTO to a message that is sent over the wire.

In JAX-RS, that one place should be in an entity provider. In Spring, the mechanism to use is the message converter. Note that both frameworks have support for several often-used serialization formats.

Following this advice not only makes it easier to change media types (e.g. from plain JSON or HAL to a more mature media type like Siren, Mason, or UBER). It also makes it easy to support multiple media types.

mediaThis in turn enables you to switch media types without breaking clients.

You can continue to serve old clients with the old media type, while new clients can take advantage of the new media type.

Introducing new media types is one way to evolve your REST API when you must make backwards incompatible changes.

DTOs Are Not Domain Objects

Domain objects implement the ubiquitous language used by subject matter experts and thus are discovered. DTOs, on the other hand, are designed to meet certain non-functional characteristics, like performance, and are subject to trade-offs.

This means the two have very different reasons to change and, following the Single Responsibility Principle, should be separate objects. Blindly serializing domain objects should thus be considered an anti-pattern.

That doesn’t mean you must blindly add DTOs, either. It’s perfectly fine to start with exposing domain objects, e.g. using Spring Data REST, and introducing DTOs as needed. As always, premature optimization is the root of all evil, and you should decide based on measurements.

The point is to keep the difference in mind. Don’t change your domain objects to get better performance, but rather introduce DTOs.

DTOs Have No Behavior

big-dataA DTO should not have any behavior; it’s purpose in life is to transfer data between remote systems.

This is very different from domain objects.

There are two basic approaches for dealing with the data in a DTO.

The first is to make them immutable objects, where all the input is provided in the constructor and the data can only be read.

This doesn’t work well for large objects, and doesn’t play nice with serialization frameworks.

The better approach is to make all the properties writable. Since a DTO must not have logic, this is one of the few occasions where you can safely make the fields public and omit the getters and setters.

Of course, that means some other part of the code is responsible for filling the DTO with combinations of properties that together make sense.

Conversely, you should validate DTOs that come back in from the client.

Managers and Individual Contributors in Code

manager-individualContributorI’m lucky to work for a company that makes it possible to have a good technical career, so that I don’t have to become a manager just to get more money.

I like being an individual contributor because it gives me a sense of accomplishment when I get stuff done myself.

Anyway, I believe the distinction between manager and individual contributor is useful when writing code as well.

Let me show that using the example of Conway’s Game of Life.

Baby Steps

At code retreats, you’ll find people who start working on the problem from the bottom (cell) and those who start at the top (universe, world, grid, game, or whatever they call it).

Interestingly, both groups tend to run into a similar problem.

baby-stepsStarting at the bottom makes it easy to quickly implement the rules in a cell. But then the problem becomes how to connect the cells into a universe.

This often looks like too big of a step to properly test, e.g. with a new green test every two minutes.

Starting at the top makes sure we are not doing anything unnecessary or building any parts that would not connect properly. But here too it’s difficult to keep chipping away in baby steps.

The Single Responsibility Principle

To evolve a whole universe, at least these things need to happen:

  1. Iterate over all cells
  2. Determine each cell’s neighbors
  3. Count the number of alive neighbors
  4. Determine the cell’s next state based on its current state and the number of alive neighbors

Depending on some implementation choices (like keeping the universe immutable), there may be more work.

Yet the Single Responsibility Principle tells us that a class should have only one reason to change. You may count items #2 and #3 as one responsibility (or #3 and #4), but clearly there is more than one involved.

Manager Classes and Individual Contributor Classes and How to Test Them

Now remember the manager/individual contributor (IC) distinction. Managers manage ICs; that’s their whole job. We should count managing other classes as a responsibility too.

That would mean that a class either implements some logic (IC), or coordinates with other classes (manager), but not both.

mockI think this also ties into the state-based versus interaction-based testing debate: we should use state-based testing for IC classes and interaction-based testing (using mocks) for manager classes.

That would explain why both code retreat groups seem to get stuck: they have to switch from state-based to interaction-based testing and it seems that many developers don’t have a lot of experience with the latter.

Test-Driving Manager Classes

So let’s try interaction-based testing of a manager class. I’ll keep with the example of the Game of Life.

I’d like to begin with an architectural choice: I want the universe to be immutable, so that it would be easy to introduce multi-threading later. As the game evolves, I’ll have to create a new universe for every generation:

public class GameTest {
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone()).thenReturn(nextGeneration);

    Game game = new Game(universe);
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
      game.universe());
  }
}

I’m using Mockito to define how the Game interacts with the Universe. Mocking works best with interfaces, so Universe is an interface.

That means the game doesn’t know the type of universe and therefore can’t create a new instance of it. The universe provides a factory method clone() for this purpose.

If the universe is to be immutable, then its state should be provided in its constructor. This constructor is called by clone(), so that method needs the new state as its input.

What does that state look like? At this point we don’t know anything about the universe other than that it contains cells. Let’s keep it simple and provide the cells as input:

public class GameTest {
  @SuppressWarnings("unchecked")
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone(any(Iterable.class)))
        .thenReturn(nextGeneration);

    Game game = new Game(universe);
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
        game.universe());
  }
}

OK, so who is going to supply those cells to clone()? The game is a manager, so it should only manage, not provide logic. The universe is an IC that is responsible for the cells and their interconnections, so it already has a responsibility.

So it seems like we need a new type. The new type should be managed by the game, since that’s the only manager class we have. Thus we should add a new test in GameTest for this interaction.

The new type should be responsible for determining the new state of the universe from the old one. This is where the game’s rules come into play, so let’s call the new type Rules.

Let’s start with testing that the game manages the rules:

public class GameTest {
  @Test
  public void consultsRulesWhenEvolving() {
    Rules rules = mock(Rules.class);

    Game game = new Game(null, rules);
    assertSame("Rules", rules, game.rules());
  }
}

Now we want to test that evolve() consults the rules. Since the game is a manager, it can only request output from a collaborator and provide that as input to another collaborator. It shouldn’t contain any logic itself, since that would make it an IC.

The only other collaborator that the game currently has is the universe and that seems the correct candidate to provide the cells. But we need more than the cells to implement the rules: we also need to number of alive neighbors of each cell.

The universe is clearly the correct place to determine a cell’s neighbors. Should it also count the number of alive ones?

I guess it could, but I don’t particularly like that: the rules for Life use the number of alive neighbors, but it’s not hard to imagine a different set of rules that also depend on the number of dead neighbors. So I feel that the rules should do the counting.

This means that the input to the rules is a cell and its neighbors. Since Java doesn’t allow returning two pieces of information, we need to combine them. Let’s call the combination a neighborhood:

public class NeighborhoodTest {
  @Test
  public void holdsACellAndItsNeighbors() {
    Cell cell = mock(Cell.class, "cell");
    List<Cell> neighbors = Arrays.asList(
        mock(Cell.class, "neighbor1"), 
        mock(Cell.class, "neighbor2"));

    Neighborhood neighborhood = new Neighborhood(cell, 
        neighbors);

    assertSame("Cell", cell, neighborhood.cell());
    assertEquals("Neighbors", neighbors, 
        neighborhood.neighbors());
  }
}

Now we can make the universe return a neighborhood for each of the cells it contains and verify that those neighborhoods are used as input for the rules:

public class GameTest {
  @SuppressWarnings("unchecked")
  @Test
  public void clonesImmutableUniverseWhenEvolving() {
    Universe universe = mock(Universe.class, "old");
    Universe nextGeneration = mock(Universe.class, "new");
    when(universe.clone(any(Iterable.class)))
        .thenReturn(nextGeneration);
    when(universe.neighborhoods()).thenReturn(
        new ArrayList<Neighborhood>());

    Game game = new Game(universe, mock(Rules.class));
    assertSame("Initial", universe, game.universe());

    game.evolve();
    assertSame("Next generation", nextGeneration, 
        game.universe());
  }

  @Test
  public void consultsRulesWhenEvolving() {
    Universe universe = mock(Universe.class);
    Neighborhood neighborhood1 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    Neighborhood neighborhood2 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    when(universe.neighborhoods()).thenReturn(
        Arrays.asList(neighborhood1, neighborhood2));
    Rules rules = mock(Rules.class);

    Game game = new Game(universe, rules);
    assertSame("Rules", rules, game.rules());

    game.evolve();
    verify(rules).nextGeneration(neighborhood1);
    verify(rules).nextGeneration(neighborhood2);
  }
}

The next step is to make sure that the output from the rules is used to construct the new universe:

public class GameTest {
  @Test
  public void consultsRulesWhenEvolving() {
    Universe universe = mock(Universe.class);
    Neighborhood neighborhood1 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    Neighborhood neighborhood2 = new Neighborhood(
        mock(Cell.class), 
        Arrays.asList(mock(Cell.class)));
    when(universe.neighborhoods()).thenReturn(
        Arrays.asList(neighborhood1, neighborhood2));
    Rules rules = mock(Rules.class);
    Cell cell1 = mock(Cell.class, "cell1");
    Cell cell2 = mock(Cell.class, "cell2");
    when(rules.nextGeneration(neighborhood1))
        .thenReturn(cell1);
    when(rules.nextGeneration(neighborhood2))
        .thenReturn(cell2);

    Game game = new Game(universe, rules);
    assertSame("Rules", rules, game.rules());

    game.evolve();
    verify(universe).clone(eq(
        Arrays.asList(cell1, cell2)));
  }
}

At this point we’re done with the Game class. All that is left to do, is create implementations for the three interfaces we introduced: Universe, Cell, and Rules. Each of these is an IC class, and thus fairly straightforward to test-drive using state-based testing.

Conclusion

I find that the distinction between manager and individual contributor classes helps me in deciding what the next test should be.

What do you think? Could this be part of the missing piece for a proper theory of Test-Driven Development?