REST Messages And Data Transfer Objects

In Patterns of Enterprise Application Architecture, Martin Fowler defines a Data Transfer Object (DTO) as

An object that carries data between processes in order to reduce the number of method calls.

Note that a Data Transfer Object is not the same as a Data Access Object (DAO), although they have some similarities. A Data Access Object is used to hide details from the underlying persistence layer.

REST Messages Are Serialized DTOs

message-transferIn a RESTful architecture, the messages sent across the wire are serializations of DTOs.

This means all the best practices around DTOs are important to follow when building RESTful systems.

For instance, Fowler writes

…encapsulate the serialization mechanism for transferring data over the wire. By encapsulating the serialization like this, the DTOs keep this logic out of the rest of the code and also provide a clear point to change serialization should you wish.

In other words, you should follow the DRY principle and have exactly one place where you convert your internal DTO to a message that is sent over the wire.

In JAX-RS, that one place should be in an entity provider. In Spring, the mechanism to use is the message converter. Note that both frameworks have support for several often-used serialization formats.

Following this advice not only makes it easier to change media types (e.g. from plain JSON or HAL to a more mature media type like Siren, Mason, or UBER). It also makes it easy to support multiple media types.

mediaThis in turn enables you to switch media types without breaking clients.

You can continue to serve old clients with the old media type, while new clients can take advantage of the new media type.

Introducing new media types is one way to evolve your REST API when you must make backwards incompatible changes.

DTOs Are Not Domain Objects

Domain objects implement the ubiquitous language used by subject matter experts and thus are discovered. DTOs, on the other hand, are designed to meet certain non-functional characteristics, like performance, and are subject to trade-offs.

This means the two have very different reasons to change and, following the Single Responsibility Principle, should be separate objects. Blindly serializing domain objects should thus be considered an anti-pattern.

That doesn’t mean you must blindly add DTOs, either. It’s perfectly fine to start with exposing domain objects, e.g. using Spring Data REST, and introducing DTOs as needed. As always, premature optimization is the root of all evil, and you should decide based on measurements.

The point is to keep the difference in mind. Don’t change your domain objects to get better performance, but rather introduce DTOs.

DTOs Have No Behavior

big-dataA DTO should not have any behavior; it’s purpose in life is to transfer data between remote systems.

This is very different from domain objects.

There are two basic approaches for dealing with the data in a DTO.

The first is to make them immutable objects, where all the input is provided in the constructor and the data can only be read.

This doesn’t work well for large objects, and doesn’t play nice with serialization frameworks.

The better approach is to make all the properties writable. Since a DTO must not have logic, this is one of the few occasions where you can safely make the fields public and omit the getters and setters.

Of course, that means some other part of the code is responsible for filling the DTO with combinations of properties that together make sense.

Conversely, you should validate DTOs that come back in from the client.


The Nature of Software Development

I’ve been developing software for a while now. Over fifteen years professionally, and quite some time before that too. One would think that when you’ve been doing something for a long time, you would develop a good understanding of what it is that you do. Well, maybe. But every once in a while I come across something that deepens my insight, that takes it to the next level.

I recently came across an article (from 1985!) that does just that, by providing a theory of what software development really is. If you have anything to do with developing software, please read the article. It’s a bit dry and terse at times, but please persevere.

The article promotes the view that, in essence, software development is a theory building activity. This means that it is first and foremost about building a mental model in the developer’s mind about how the world works and how the software being developed handles and supports that world.

This contrasts with the more dominant manufacturing view that sees software development as an activity where some artifacts are to be produced, like code, tests, and documentation. That’s not to say that these artifacts are not produced, since obviously they are, but rather that that’s not the real issue. If we want to be able to develop software that works well, and is maintainable, we’re better off focusing on building the right theories. The right artifacts will then follow.

This view has huge implications for how we organize software development. Below, I will discuss some things that we need to do differently from what you might expect based on the manufacturing view. The funny thing is that we’ve known that for some time now, but for different reasons.

We need strong customer collaboration to build good theories
Mental models can never be externalized completely, some subtleties always get lost when we try that. That’s why requirements as documents don’t work. It’s not just that they will change (Agile), or that hand-offs are a waste (Lean), no, they fundamentally cannot work! So we need the customer around to clarify the subtle details, and we need to go see how the end users work in their own environment to build the best possible theories.

Since this is expensive, and since a customer would not like having to explain the same concept over and over again to different developers, it makes sense to try and capture some of the domain knowledge, even though we know we can never capture every subtle detail. This is where automated acceptance tests, for example as produced in Behavior Driven Development (BDD), can come in handy.

Software developers should share the same theory
It’s of paramount importance that all the developers on the team share the same theory, or else they will develop things that make no sense to their team members and that will not integrate well with their team members’ work. Having the same theory is helped by using the same terminology. Developers should be careful not to let the meaning of the terms drift away from how the end users use them.

The need for a shared theory means that strong code ownership is a bad idea. Weak ownership could work, but we’d better take measures to prevent silos from forming. Code reviews are one candidate for this. The best approach, however, is collective code ownership, especially when combined with pair programming.

Note that eXtreme Programming (XP) makes the shared theory explicit through it’s system metaphor concept. This is the XP practice that I’ve always struggled most with, but now it’s finally starting to make sense to me.

Theory building is an essential skill for software developers
If software development is first and foremost about building theories, then software developers better be good at it. So it makes sense to teach them this. I’m not yet aware of how to best do this, but one obvious place to start is the scientific method, since building theories is precisely what scientist do (even though some dispute the scientific method is actually the way scientists build theories). This reminds me of the relationship between the scientific method and Test-Driven Development (TDD).

Another promising approach is Domain Driven Design, since that places the domain model at the center of software development. Please let me know if you have more ideas on this subject.

Software developers are not interchangeable resources
If the most crucial part of software development is the building of a theory, than you can’t just simply replace one developer with another, since the new girl needs to start building her theory from scratch and that takes time. This is another reason why you can’t add people to a late project without making it later still, as (I wish!) we all know.

Also, developers with a better initial theory are better suited to work on a new project, since they require less theory building. That’s why it makes sense to use developers with pre-existing domain knowledge, if possible. Conversely, that’s also why it makes sense for a developer to specialize in a given domain.

We should expect designs to undergo significant changes
Taking a hint from science, we see that theories sometimes go through radical changes. Now, if the software design is a manifestation of the theory in the developers’ mind, then we can expect the design to undergo radical changes as well. This argues against too much design up front, and for incremental design.

Maintenance should be done by the original developers
Some organizations hand off software to a maintenance team once its initial development is done. From the perspective of software development as theory building, that doesn’t make a whole lot of sense. The maintenance team doesn’t have the theories that the original developers built, and will likely make modifications that don’t fit that theory and are therefore not optimal.

If you do insist on separate maintenance teams, then the maintainers should be phased into the team before the initial stage ends, so they have access to people that already have well formed theories and can learn from them.

Software developers should not be shared between teams
For productivity reasons, software developers shouldn’t divide their attention between multiple projects. But the theory building view of software development gives another perspective. It’s hard enough to build one theory at a time, especially if one is also learning other stuff, like a new technology. Developers really shouldn’t need to learn too much in parallel, or they may feel like their head is going to explode.