Major Tech Projects

Under construction

Last time we saw how a target architecture can guide you in your day to day technical decision making. But what do you do when your target architecture is considerably different from your current architecture?

Ways of working differ considerably between organizations, but most tech companies nowadays use some form of agile process (or at least claim to), where a Product Owner prioritizes a backlog of smallish items for the team to work on. That’s the context we’ll assume for this post.

The question then becomes how you convince a business-oriented Product Owner to prioritize technical improvement work over value-added work, such as new features. This question becomes even more pressing when the amount of tech work is not small, but rather an entire project in itself. This type of work doesn’t fit all that well into the agile model of continuous delivery of value through small increments. So what to do?

First of all, it’s always good if you can split the work into smaller pieces. Those are easier to squeeze in and, more importantly, carry less risk. But even if you manage to do that for a big tech overhaul, you’ll end up with a long list of tech items that somehow need to find their way to the top of the backlog.

You could try to persuade the Product Owner in a separate discussion about each and every one of these tech items. However, since Product Owners are judged by how much value they deliver sooner rather than later, you’re going to be fighting an uphill battle. That doesn’t mean it can’t work in your context, of course, but in general the chances aren’t great when the number of tech items grows.

Alternatively, you could treat this as technical debt, assuming you have a well-functioning process for handling that. That’s quite an assumption, by the way. Some teams use their slack time to dig themselves out of holes, but most teams are not fortunate enough to have slack time. (As an industry, we’re slow learners. Mythical man-month, anyone?) Many teams I’ve seen that keep track of tech debt just keep growing it year over year and I hate to think about the teams that don’t even keep track of it.

Under construction

Part of the problem with tech debt is that the metaphor is broken. Financial people will tell you that some amount of debt is good and that you shouldn’t get rid of all of it. That’s just not how developers see tech debt. I’ve also yet to see a team that regularly and consistently pays off tech debt, like one would make monthly payments to pay off, say, a mortgage.

Idea Flow argues that we should abandon the technical debt concept in favor of risk and that we should let the team be guided by both a Product Owner, for business value, and Technical Risk Manager, for keeping risk within an acceptable range. It’s an interesting concept that sounds like it may work, but the problem will be in estimating the risk. I haven’t seen this done in practice yet, so if you have experience with this, please leave a comment below.

A somewhat similar, but less refined, approach is to allocate two budgets for the team: one for doing value-added work and one for doing technical work. You could assign a 3:1 ratio to those budgets, for instance, which means 75% of developer time would be spent on adding business value and 25% on technical work items. Then if you have a year where you need to do some major technical work, you could ask for a temporary change in this ratio.

Yet another approach is the dreaded rewrite: you stop all value-added work for some time and do only technical work. This allows the team to make quick progress in the technical area, but is generally not liked all that much by the business. You can get away with this only every so often, as it tends to take a huge bite out of your political capital. But at some point it will become your only option if you don’t find a better way first.

What do you think? What ways to schedule time for major technical projects have worked for you? Please leave a comment below.

Data Flow Diagrams and Threat Models

Last time we looked at some generic diagrams from the C4 model, which are useful for most teams. This time we’re going to explore a more specific type of diagram that can be a tremendous help with security.

Data Flow Diagrams

A Data Flow Diagram (DFD), as the name indicates, shows the flow of data through the system. It depicts external entities, processes, data stores, and data flows. Larger systems usually have composite processes, which expand into their own DFD.

Here’s a simple example of a data flow diagram:

Create a Data Flow Diagram from a Container Diagram

If you already have a container diagram of your system, then it’s easy to create a DFD from it:

  1. Convert containers into processes by drawing them as circles. You may or may not want to number them. Complicate containers with many processes running in them may be modeled as composite process that expand into their own sub-DFD.
  2. Convert external systems into external entities by drawing them as rectangles.
  3. Convert databases and queues into data stores by drawing them as horizontal parallel bars.
  4. Modify the directions of all the arrows as needed. In a container diagram, the direction of an arrow usually indicates who initiates a request. In a data flow diagram, the direction of an arrow indicates the direction in which data flows. This could be both ways.
    Some people use arrows to indicate data flows in container diagrams as well. In that case, you don’t have anything to do in this step.

Now, why would you go through the trouble to convert a container diagram into a DFD? There are several uses for DFDs, but in this post I want to focus on using them for threat modeling.

Threat Models

In threat modeling, we look for design flaws with security implications. These are different from implementation bugs, which makes them hard to detect using code-based techniques such as Static Application Security Testing (SAST) or security code reviews. But the flip side of that is that you can do threat modeling even before any code is written. And you should!

Threat modeling falls under Threat Assessment in the Software Assurance Maturity Model.

There are many ways to do build threat models, but the one I’ve found easiest to understand for developers with limited security knowledge (which is the vast majority), is to use STRIDE, an acronym for the types of security threats that exist:

  • Spoofing
  • Tampering
  • Repudiation
  • Information disclosure
  • Denial of service
  • Elevation of privilege

Using DFDs to Build Threat Models

Not all of the STRIDE threats are applicable to all elements of our system, and this is where the DFD comes in handy, since each DFD element maps nicely unto some subset of the STRIDE threats:

So now you have a structured process for reviewing your architecture from a security perspective:

  1. Create a DFD from your container diagram (or from scratch if you don’t have one)
  2. Identify the threats using STRIDE
  3. Score the threats using the Common Vulnerability Scoring System (CVSS)
  4. Manage the risks posed by the identified threats:

I’ve found that using CVSS scores for threats makes it easier to decide how to manage them. You can set a security risk appetite for your system, for example accept threats in the None through Medium levels and work to avoid or mitigate High or Critical levels.

CVSS scores also go a long way towards making the threat assessment objective, which makes it easier to convince people that work should be done to reduce the security risk.

Costs and Benefits of Threat Models

If all of this seems like a lot of work, especially for a big system, that’s because it is. Sorry.

It’s not quite as bad as it may seem, however, because DFD elements can usually be grouped since they all behave the same from a security perspective. So then you only have to score and manage the group as a whole rather than all the elements in it individually. But still, threat modeling is a considerable investment.

When I’ve done this activity with developers, I see that it always provides a lot of value. We usually find one or more security threats that we really need to manage better. And most of the time participants get a much better understanding of their system, which helps them in their non-security work as well.

What do you think? Is threat modeling using data flow diagrams and STRIDE something you’re willing to give a shot, or do you prefer a method that requires less work (but offers less protection), like abuse cases? Please leave a comment.

Target Architecture

In the last two posts, we looked at generic architecture diagrams and security-specific diagrams. These diagrams reflect the current architecture of a system. This time we will look at using diagrams to depict a desired future architecture, or target architecture.

The point of a target architecture is to paint a picture of the desired state that will act as the North Star (or Southern Cross, depending on your hemisphere), in the sense that it guides all the little decisions we make every day when working on our system. This doesn’t mean we’ll always be moving straight towards the target, but we do want to make sure we don’t stray too far of the path.

However, a target architecture is not a static thing.

Assuming you don’t have a crystal ball, it will be hard to predict how circumstances will develop over the years. So think of a target architecture more as a moving target than as something set in stone. We need to keep an open mind at all times, and always react appropriately to changes, and one such reaction could be to update the target architecture as we learn more about our system, the customers it supports, the team that builds it, the evolution of the technologies it uses, etc.

Moving towards a target architecture usually doesn’t happen in a straight line.

So how do we come up with a target architecture in the first place?

One starting point is to look at the diagram that reflects your current state and identify things that give you trouble. Few systems are optimal, so you’re likely to find some things that you would do differently if you had a chance to start all over.

Another source of ideas comes from future requirements, i.e. requests on your backlog. The fact that the current architecture supports the current requirements doesn’t necessarily mean that it will be able to support all future requirements. Sometimes we need to create some architectural runway.

Once you’ve identified an area for improvement, you need to make a decision about whether it’s actually worth fixing. Some things are very costly or risky to fix compared to their benefit.

Next, write up your decision in the form of an Architecture Decision Record (ADR). The structured format of an ADR, which lists the positive and negative consequences of the decision, will help with getting everyone on the same page. It will also give you a nice historical record, which is especially important for any newcomers as the team evolves over time.

Every ADR records a decision. Each such decision constrains the solution space of the system in its own way. As the number of decisions grows, the resulting viable part of the solution space shrinks. The target architecture is a picture of a single point in this viable part of the solution space. Note that there could be other points that are viable as well, i.e. alternative target architectures.

Obviously, the solution space is multi-dimensional. But if we project it in two dimensions, we can visualize the relationship between target architecture and ADRs as follows:

Architecturally significant decisions constrain the viable solution space in which a target architecture lives.

If you find your team arguing about two different options in the viable solution space and you want to resolve this ambiguity, then hopefully the picture above will suggest the solution: you’re missing an ADR that shrinks the viable solution space to exclude one of the options.

What do you think? Do you find a target architecture a useful tool to guide your team’s day to day technical decisions? Do you use ADRs to document why the target architecture looks the way it does? Please leave a comment below.

Architecture Diagrams

Hi, my name is Ray, and I’m a software architect.

Example container diagram

According to my old boss Jeroen van Rotterdam, this means that I draw boxes and lines. In practice, it’s only a small part of what I do, but I do think it’s an important part.

Some people may wonder why, in the age of working software over comprehensive documentation, one would still spend time on creating pretty pictures. Didn’t we leave all this up front, non-code fluff behind? Well, I do value working software more than comprehensive documentation, but that doesn’t mean there is no value in documentation.

I work with several “central” teams that provide shared services to “local” teams. In such discussions, it’s quite often useful to be able to point to a high-level architecture of the system. Something like the context diagram of the C4 model. Such context diagrams were also of great value during the due diligence phase of the recently announced sale of our division. Drawing a context diagram takes very little time, provides clear benefits, and requires little maintenance, so it’s really a no-brainer.

What about other types of diagrams?

The C4 model has several other diagrams. Which ones you need depends on your context. If all your code goes into a single war that you deploy to an application server, for instance, there is little point in creating a container diagram, but a component diagram may be useful. If, on the other hand, your system consists of quite a few microservices, then a container diagram that shows how they are connected may be very valuable, but a component diagram would probably be overkill.

If you work in a larger organization, with lots of teams building lots of systems, then having a diagram standard in place like the C4 model is a great way to reduce the time needed to explain a system to others and to prevent misunderstandings.

You will probably want to “personalize” that standard to make it more expressive for your particular context. For instance, we use color coding of elements on container diagrams to indicate their status: red for things we are ready to remove, orange for things we want to migrate away from, yellow for things we want to review, green for things we’re happy with, and blue for things we’re planning to build in the future.

Such an adaptation of the C4 standard doesn’t reduce the value of the standard if most or all of you communication is within your organization where everyone uses the same adaptation. If, however, you routinely use diagrams in communication with outsiders, you may want to minimize your adaptations, or at the very least provide a legend with your diagrams.

Also use your diagram standard on whiteboards.

The standard should be used everywhere you draw visual representations of your systems, like in whiteboard sessions and in Architecture Decision Records. Consistent use of the standard turns it into a ubiquitous language for visually representing your systems, eliminating ambiguities.

If you have a standard that all your diagrams must adhere to, then you can build tooling around that. For instance, you could write a tool that generates a diagram from a simple YAML file that describes your system. This way, you bring the diagram closer to the code, improve consistency, and reduce the maintenance burden. I’ll come back to tooling in a later post.

There are other types of diagrams besides those in the C4 model that are useful in certain situations. For instance, sequence diagrams are great for showing complex interactions between systems. I’ll talk about some other diagram types in future posts.

Do you use C4 model diagrams? Are they worth the investment? Please leave a comment.

Update: C4 Model author Simon Brown got some answers to the above questions on Twitter:

TDD is like working out

We all know that exercise is good for us, members of the species Homo Sapiens Sitonourasses. And yet most of us don’t do enough of it. The same is true for Test-Driven Development (TDD). But the similarity doesn’t end there.

With regular exercise, you can grow your muscles. The way this works is not linear, however. It follows an upward saw-tooth pattern: first you damage your muscles (training), then they heal (recovery) and eventually grow stronger than before (supercompensation), then you damage them again, etc:

muscle-workout

TDD follows the Damage-Heal-Grow cycle as well.

First we damage the system by writing a test that fails. Where before we could gloat in knowing all was well because our test suite said so, now we have to admit that there is still something wrong with our code. For some, this realization may hurt as much as their sore muscles after working out.

Luckily, we heal the system quickly by writing only the minimal amount of code required to make the test pass. With everything back to green in minutes or even seconds, we have every right to feel good again. TDD is perfect for short-attention spans.

Finally we grow the system by improving the design. The system can now handle everything we’ve ever thrown at it and more, because we generalized concepts and gave them a proper place in the code.

Damaging and healing happen on two levels in TDD.

First there is the syntactic level. You write a test that calls code that doesn’t exist yet, so the code doesn’t even compile. The healing that follows is to make the code compile, even though it doesn’t yet pass the test.

Only after this syntactic healing do you change the code to pass the test. The latter is more of a semantic type of healing.

The distinction between syntactic and semantic healing has implications for how we work.

There are only so many ways that a program can be syntactically broken, and in many cases, a sophisticated enough IDE can help heal it. For example, when you write a test that refers to a class that doesn’t yet exist, Eclipse offers a Quick Fix to create the class for you.

Semantic healing, on the other hand, is more difficult. The transformations of the Transformation Priority Premise can be seen as standard building blocks, and at least some of them can be automated. But that’s still a long way from the IDE generating the code that will make the failing test pass.

get-your-code-in-shape-practice-tddI haven’t seen many TDD practitioners do the equivalent of walking around the beach showing off their rock-hard abs, and that’s probably a good thing.

But just as we appreciate how a strong, muscular friend can easily handle any piece of furniture when he helps us move, so do product owners like it that we can always deliver any feature in a short amount of time.

Unfortunately, it just doesn’t score us any dates; for that we really do need to hit the gym.

Celebrate Learning in Software Development

Every event is either a cause for celebration or an opportunity to learn.

celebrateI don’t remember where I came across this quote, but it has stuck with me. I like how it turns every experience into something positive.

Sometimes I need to remind myself of it, however, especially when there are a lot of, well, learning opportunities in a row.

One recent case was when I had started a long running performance test overnight. The next morning when I came back to it, there was no useful information at all. None whatsoever.

What had happened?

scdfOur system is a fast data solution built on Spring Cloud Data Flow (SCDF). SCDF allows you to compose stream processing solutions out of data microservices built with Spring Boot.

The performance test spun up a local cluster, ingested a lot of data, and spun the cluster down, all the while capturing performance metrics.

(This is early stages performance testing, so it doesn’t necessarily need to run on a production-like remote cluster.)

Part of the shutdown procedure was to destroy the SCDF stream. The stream destroy command to the SCDF shell is supposed to terminate the apps that make up the stream. It did in our functional tests.

But somehow it hadn’t this time. After the performance test ran, the supporting services were terminated, but the stream apps kept running. And that was the problem. These apps continued to try to connect to the supporting services, failed to do that, and wrote those failures to the log files. The log files had overflown and the old ones had been removed, in an effort to save disk space.

All that was left, were log files filled with nothing but connection failures. All the useful information was gone. While I was grateful that I still had space on my disk left, it was definitely not a cause for celebration.

So then what could we learn from this event?

Obviously we need to fix the stream shutdown procedure.

kubernetesCome to think of it, we had already learned that lesson. The code to shut down our Kubernetes cluster doesn’t use stream destroy, but simply deletes all the replication controllers and pods that SCDF creates.

We did it that way, because the alternative proved unreliable. And yet we had failed to update the equivalent code for a local cluster. In other words, we had previously missed an opportunity to learn!

Determined not that make that mistake again, we tried to look beyond fixing the local cluster shutdown code.

One option is to not delete old logs, so we wouldn’t have lost the useful information. However, that almost certainly would have led to a full disk and a world of hurt. So maybe, just maybe, we shouldn’t go there.

Another idea is to not log the connection failures that filled up the log files. Silently ignoring problems isn’t exactly a brilliant strategy either, however. If we don’t log problems, we have nothing to monitor and alert on.

release-itA better idea is to reduce the number of connection attempts in the face of repeated failures. Actually, resiliency features like circuit breakers were already in the backlog, since the need for it was firmly drilled into us by the likes of Nygard.

We just hadn’t worked on that story yet, because we didn’t have much experience in this area and needed to do some homework.

So why not spend a little bit of time to do that research now? It’s not like we could work on analyzing the performance test results.

It turns out that this kind of stuff is very easy to accomplish with the FailSafe library:

private final CircuitBreaker circuitBreaker = new CircuitBreaker()
    .withFailureThreshold(3, 10)
    .withSuccessThreshold(3)
    .withDelay(1, TimeUnit.SECONDS);
private final SyncFailsafe<Object> safeService = Failsafe
    .with(circuitBreaker)
    .withFallback(() -> DEFAULT_VALUE);

@PostConstruct
public void init() {
  circuitBreaker.onOpen(() -> LOG.warn("Circuit breaker opened"));
  circuitBreaker.onClose(() -> LOG.warn("Circuit breaker closed"));
}

private Object getValue() {
  return safeService.get(() -> remoteService.getValue());
}

learnI always feel better after learning something new. Taking every opportunity to learn keeps my job interesting and makes it easier to deal with the inevitable problems that come my way.

Instead of being overwhelmed with negativity, the positive experience of improving my skills keeps me motivated to keep going.

What else could we have learned from this incident? What have you learned recently? Please leave a comment below.

Functional FizzBuzz Kata in Java

A while ago I solved the FizzBuzz kata using Java 8 streams and lambdas. While the end result was functional, the intermediate steps were not. Surely I can do better.

As always, let’s start with a failing test:

+ package remonsinnema.blog.fizzbuzz;
+
+ import static org.junit.Assert.assertEquals;
+
+ import org.junit.Test;
+
+
+ public class WhenFunctionallyFuzzingAndBuzzing {
+
+   private final FizzBuzzer fizzBuzzer = new FizzBuzzer();
+
+   @Test
+   public void shouldReplaceMultiplesOfThreeWithFizzAndMultiplesOfFiveWithBuzz() {
+     assertEquals(“1”, “1”, fizzBuzzer.apply(1));
+   }
+
+ }
+ package remonsinnema.blog.fizzbuzz;
+
+ import java.util.function.Function;
+
+
+ public class FizzBuzzer implements Function<Integer, String> {
+
+   @Override
+   public String apply(Integer n) {
+     return null;
+   }
+
+ }

Note that I start off on a functional course right away, using Java’s Function.

I fake the implementation to make the test pass:

  public class FizzBuzzer implements Function<Integer, String> {
    @Override
    public String apply(Integer n) {
–     return null;
+     return “1”;
    }
  }

And refactor the test to remove duplication:

  public class WhenFunctionallyFuzzingAndBuzzing {
    @Test
    public void shouldReplaceMultiplesOfThreeWithFizzAndMultiplesOfFiveWithBuzz() {
–     assertEquals(“1”, “1”, fizzBuzzer.apply(1));
+     assertFizzBuzz(“1”, 1);
+   }
+
+   private void assertFizzBuzz(String expected, int value) {
+     assertEquals(Integer.toString(value), expected, fizzBuzzer.apply(value));
    }
  }

Then I add another test to generalize the implementation:

  public class WhenFunctionallyFuzzingAndBuzzing {
    @Test
    public void shouldReplaceMultiplesOfThreeWithFizzAndMultiplesOfFiveWithBuzz() {
      assertFizzBuzz(“1”, 1);
+     assertFizzBuzz(“2”, 2);
    }
    private void assertFizzBuzz(String expected, int value) {
  public class FizzBuzzer implements Function<Integer, String> {
    @Override
    public String apply(Integer n) {
–     return “1”;
+     return Integer.toString(n);
    }
  }

OK, pretty standard stuff so far. Next I need to replace 3 with “Fizz”:

  public class WhenFunctionallyFuzzingAndBuzzing {
    public void shouldReplaceMultiplesOfThreeWithFizzAndMultiplesOfFiveWithBuzz() {
      assertFizzBuzz(“1”, 1);
      assertFizzBuzz(“2”, 2);
+     assertFizzBuzz(“Fizz”, 3);
    }
  nbsp;
    private void assertFizzBuzz(String expected, int value) {
  public class FizzBuzzer implements Function<Integer, String> {
    @Override
    public String apply(Integer n) {
–     return Integer.toString(n);
+     return numberReplacerFor(n).apply(n);
+   }
+
+   private Function<Integer, String> numberReplacerFor(Integer n) {
+     return n == 3
+         ? i -> “Fizz”
+         : i -> Integer.toString(i);
    }
  }

Here I recognize that I need to apply one of two functions, depending on the input. This code works, but needs some cleaning up. First, as a stepping stone, I extract the lambdas into fields:

  import java.util.function.Function;
  public class FizzBuzzer implements Function<Integer, String> {
+   private final Function<Integer, String> replaceNumberWithStringRepresentation
+       = n -> Integer.toString(n);
+   private final Function<Integer, String> replaceNumberWithFizz
+       = n -> “Fizz”;
+
    @Override
    public String apply(Integer n) {
      return numberReplacerFor(n).apply(n);
    private Function<Integer, String> numberReplacerFor(Integer n) {
      return n == 3
–         ? i -> “Fizz”
–         : i -> Integer.toString(i);
+         ? replaceNumberWithFizz
+         : replaceNumberWithStringRepresentation;
    }
  }

Next I emphasize that “3” and “Fizz” go together by extracting a class:

  public class FizzBuzzer implements Function<Integer, String> {
    private final Function<Integer, String> replaceNumberWithStringRepresentation
        = n -> Integer.toString(n);
–   private final Function<Integer, String> replaceNumberWithFizz
–       = n -> “Fizz”;
+   private final Fizzer replaceNumberWithFizz = new Fizzer();
    @Override
    public String apply(Integer n) {
    }
    private Function<Integer, String> numberReplacerFor(Integer n) {
–     return n == 3
+     return replaceNumberWithFizz.test(n)
          ? replaceNumberWithFizz
          : replaceNumberWithStringRepresentation;
    }
+ package remonsinnema.blog.fizzbuzz;
+
+ import java.util.function.Function;
+ import java.util.function.Predicate;
+
+
+ public class Fizzer implements Function<Integer, String>, Predicate<Integer> {
+
+   @Override
+   public boolean test(Integer n) {
+     return n == 3;
+   }
+
+   @Override
+   public String apply(Integer n) {
+     return “Fizz”;
+   }
+
+ }

Here I’m using the standard Java Predicate functional interface.

To add “Buzz”, I need to generalize the code from a single if (hidden as the ternary operator) to a loop:

  public class WhenFunctionallyFuzzingAndBuzzing {
      assertFizzBuzz(“1”, 1);
      assertFizzBuzz(“2”, 2);
      assertFizzBuzz(“Fizz”, 3);
+     assertFizzBuzz(“4”, 4);
+     assertFizzBuzz(“Buzz”, 5);
    }
    private void assertFizzBuzz(String expected, int value) {
  package remonsinnema.blog.fizzbuzz;
+ import java.util.Arrays;
+ import java.util.Collection;
  import java.util.function.Function;
    private final Function<Integer, String> replaceNumberWithStringRepresentation
        = n -> Integer.toString(n);
–   private final Fizzer replaceNumberWithFizz = new Fizzer();
+   private final Collection<ReplaceNumberWithFixedText> replacers = Arrays.asList(
+       new ReplaceNumberWithFixedText(3, “Fizz”),
+       new ReplaceNumberWithFixedText(5, “Buzz”)
+   );
    @Override
    public String apply(Integer n) {
    }
    private Function<Integer, String> numberReplacerFor(Integer n) {
–     return replaceNumberWithFizz.test(n)
–         ? replaceNumberWithFizz
–         : replaceNumberWithStringRepresentation;
+     for (ReplaceNumberWithFixedText replacer : replacers) {
+       if (replacer.test(n)) {
+         return replacer;
+       }
+     }
+     return replaceNumberWithStringRepresentation;
    }
  }
– package remonsinnema.blog.fizzbuzz;
– import java.util.function.Function;
– import java.util.function.Predicate;
– public class Fizzer implements Function<Integer, String>, Predicate<Integer> {
–   @Override
–   public boolean test(Integer n) {
–     return n == 3;
–   }
–   @Override
–   public String apply(Integer n) {
–     return “Fizz”;
–   }
– }
+ package remonsinnema.blog.fizzbuzz;
+
+ import java.util.function.Function;
+ import java.util.function.Predicate;
+
+
+ public class ReplaceNumberWithFixedText implements Function<Integer, String>,
+     Predicate<Integer> {
+
+   private final int target;
+   private final String replacement;
+
+   public ReplaceNumberWithFixedText(int target, String replacement) {
+     this.target = target;
+     this.replacement = replacement;
+   }
+
+   @Override
+   public boolean test(Integer n) {
+     return n == target;
+   }
+
+   @Override
+   public String apply(Integer n) {
+     return replacement;
+   }
+
+ }

Oops, old habits… That should be a stream rather than a loop:

  import java.util.function.Function;
  public class FizzBuzzer implements Function<Integer, String> {
–   private final Function<Integer, String> replaceNumberWithStringRepresentation
+   private final Function<Integer, String> defaultReplacer
        = n -> Integer.toString(n);
    private final Collection<ReplaceNumberWithFixedText> replacers = Arrays.asList(
        new ReplaceNumberWithFixedText(3, “Fizz”),
    }
    private Function<Integer, String> numberReplacerFor(Integer n) {
–     for (ReplaceNumberWithFixedText replacer : replacers) {
–       if (replacer.test(n)) {
–         return replacer;
–       }
–     }
–     return replaceNumberWithStringRepresentation;
+     return replacers.stream()
+         .filter(replacer -> replacer.test(n))
+         .map(replacer -> (Function<Integer, String>) replacer)
+         .findFirst()
+         .orElse(defaultReplacer);
    }
  }

Much better. The next test is for multiples:

  public class WhenFunctionallyFuzzingAndBuzzing {
      assertFizzBuzz(“Fizz”, 3);
      assertFizzBuzz(“4”, 4);
      assertFizzBuzz(“Buzz”, 5);
+     assertFizzBuzz(“Fizz”, 6);
    }
    private void assertFizzBuzz(String expected, int value) {
  public class FizzBuzzer implements Function<Integer, String> {
    private final Function<Integer, String> defaultReplacer
        = n -> Integer.toString(n);
–   private final Collection<ReplaceNumberWithFixedText> replacers = Arrays.asList(
–       new ReplaceNumberWithFixedText(3, “Fizz”),
–       new ReplaceNumberWithFixedText(5, “Buzz”)
+   private final Collection<ReplaceMultipleWithFixedText> replacers = Arrays.asList(
+       new ReplaceMultipleWithFixedText(3, “Fizz”),
+       new ReplaceMultipleWithFixedText(5, “Buzz”)
    );
    @Override
+ package remonsinnema.blog.fizzbuzz;
+
+ import java.util.function.Function;
+ import java.util.function.Predicate;
+
+
+ public class ReplaceNumberWithFixedText implements Function<Integer, String>,
+     Predicate<Integer> {
+
+   private final int target;
+   private final String replacement;
+
+   public ReplaceNumberWithFixedText(int target, String replacement) {
+     this.target = target;
+     this.replacement = replacement;
+   }
+
+   @Override
+   public boolean test(Integer n) {
+     return n % target == 0;
+   }
+
+   @Override
+   public String apply(Integer n) {
+     return replacement;
+   }
+
+ }
– package remonsinnema.blog.fizzbuzz;
– import java.util.function.Function;
– import java.util.function.Predicate;
– public class ReplaceNumberWithFixedText implements Function<Integer, String>, Predicate<Integer> {
–   private final int target;
–   private final String replacement;
–   public ReplaceNumberWithFixedText(int target, String replacement) {
–     this.target = target;
–     this.replacement = replacement;
–   }
–   @Override
–   public boolean test(Integer n) {
–     return n == target;
–   }
–   @Override
–   public String apply(Integer n) {
–     return replacement;
–   }
– }

The last test is to combine Fizz and Buzz:

  public class WhenFunctionallyFuzzingAndBuzzing {
      assertFizzBuzz(“4”, 4);
      assertFizzBuzz(“Buzz”, 5);
      assertFizzBuzz(“Fizz”, 6);
+     assertFizzBuzz(“7”, 7);
+     assertFizzBuzz(“8”, 8);
+     assertFizzBuzz(“Fizz”, 9);
+     assertFizzBuzz(“Buzz”, 10);
+     assertFizzBuzz(“11”, 11);
+     assertFizzBuzz(“Fizz”, 12);
+     assertFizzBuzz(“13”, 13);
+     assertFizzBuzz(“14”, 14);
+     assertFizzBuzz(“FizzBuzz”, 15);
    }
    private void assertFizzBuzz(String expected, int value) {
  package remonsinnema.blog.fizzbuzz;
  import java.util.Arrays;
  import java.util.Collection;
  import java.util.function.Function;
+ import java.util.stream.Collectors;
+ import java.util.stream.Stream;
  public class FizzBuzzer implements Function<Integer, String> {
    @Override
    public String apply(Integer n) {
–     return numberReplacerFor(n).apply(n);
+     return numberReplacersFor(n)
+         .map(function -> function.apply(n))
+         .collect(Collectors.joining());
    }
–   private Function<Integer, String> numberReplacerFor(Integer n) {
–     return replacers.stream()
+   private Stream<Function<Integer, String>> numberReplacersFor(Integer n) {
+     return Stream.of(replacers.stream()
          .filter(replacer -> replacer.test(n))
          .map(replacer -> (Function<Integer, String>) replacer)
          .findFirst()
–         .orElse(defaultReplacer);
+         .orElse(defaultReplacer));
    }
  }

I generalized the single Function into a Stream of Functions, to which I apply the Map-Reduce pattern. I could have spelled out the Reduce part using something like .reduce("", (a, b) -> a + b), but I think Collectors.joining() is more expressive.

This doesn’t pass the test yet, since I return a stream of a single function. The fix is a little bit tricky, because I need to know whether any applicable replacer functions were found, and you can’t do that without terminating the stream. So I need to create a new stream using StreamSupport:

  package remonsinnema.blog.fizzbuzz;
  import java.util.Arrays;
  import java.util.Collection;
+ import java.util.Iterator;
+ import java.util.Spliterators;
  import java.util.function.Function;
  import java.util.stream.Collectors;
  import java.util.stream.Stream;
+ import java.util.stream.StreamSupport;
  public class FizzBuzzer implements Function<Integer, String> {
    }
    private Stream<Function<Integer, String>> numberReplacersFor(Integer n) {
–     return Stream.of(replacers.stream()
+     Iterator<Function<Integer, String>> result = replacers.stream()
          .filter(replacer -> replacer.test(n))
          .map(replacer -> (Function<Integer, String>) replacer)
–         .findFirst()
–         .orElse(defaultReplacer));
+         .iterator();
+     return result.hasNext()
+         ? StreamSupport.stream(Spliterators.spliteratorUnknownSize(result, 0), false)
+         : Stream.of(defaultReplacer);
    }
  }

And that’s it. The full code is on GitHub.

I learned two lessons from this little exercise:

  1. Java comes with a whole bunch of functional interfaces, like Function and Predicate, that are easily combined with streams to solve a variety of problems.
  2. The standard if → while transformation becomes if → stream in the functional world.

 

How to manage dependencies in a Gradle multi-project build

gradleI’ve been a fan of the Gradle build tool from quite early on. Its potential was clear even before the 1.0 version, when changes were regularly breaking. Today, upgrading rarely cause surprises. The tool has become mature and performs well.

Gradle includes a powerful dependency management system that can work with Maven and Ivy repositories as well as local file system dependencies.

During my work with Gradle I’ve come to rely on a pattern for managing dependencies in a multi-project build that I want to share. This pattern consists of two key practices:

  1. Centralize dependency declarations in build.gradle
  2. Centralize dependency version declarations in gradle.properties

Both practices are examples of applying software development best practices like DRY to the code that makes up the Gradle build. Let’s look at them in some more detail.

Centralize dependency declarations

In the root project’s build.gradle file, declare a new configuration for each dependency used in the entire project. In each sub-project that uses the dependency, declare that the compile (or testCompile, etc) configuration extends the configuration for the dependency:

subprojects {
  configurations {
    commonsIo
  }

  dependencies {
    commonsIo 'commons-io:commons-io:2.5'
  }
}
configurations {
  compile.extendsFrom commonsIo
}

By putting all dependency declarations in a single place, we know where to look and we prevent multiple sub-projects from declaring the same dependency with different versions.

Furthermore, the sub-projects are now more declarative, specifying only what logical components they depend on, rather than all the details of how a component is built up from individual jar files. When there is a one-to-one correspondence, as in the commons IO example, that’s not such a big deal, but the difference is pronounced when working with components that are made up of multiple jars, like the Spring framework or Jetty.

Centralize dependency version declarations

The next step is to replace all the version numbers from the root project’s build.gradle file by properties defined in the root project’s gradle.properties:

dependencies {
  commonsIo "commons-io:commons-io:$commonsIoVersion"
}
commonsIoVersion=2.5

This practice allows you to reuse the version numbers for related dependencies. For instance, if you’re using the Spring framework, you may want to declare dependencies on spring-mvc and spring-jdbc with the same version number.

There is an additional advantage of this approach. Upgrading a dependency means updating gradle.properties, while adding a new dependency means updating build.gradle. This makes it easy to gauge from a commit feed what types of changes could have been made and thus to determine whether a closer inspection is warranted.

You can take this a step further and put the configurations and dependencies blocks in a separate file, e.g. dependencies.gradle.

And beyond…

Having all the dependencies declared in a single location is a stepping stone to more advanced supply chain management practices.

The centrally declared configurations give a good overview of all the components that you use in your product, the so-called Bill of Materials (BOM). You can use the above technique, or use the Gradle BOM plugin.

The BOM makes it easier to use a tool like OWASP DependencyCheck to check for publicly disclosed vulnerabilities in the dependencies that you use. At EMC, about 80% of vulnerabilities reported against our products are caused by issues in 3rd party components, so it makes sense to keep a security eye on dependencies.

A solid BOM also makes it easier to review licenses and their compliance requirements. If you can’t afford a tool like BlackDuck Protex, you can write something less advanced yourself with modest effort.

FizzBuzz Kata With Java Streams

black-beltAfter only a couple of weeks of Judo practice, my son got bored. He complained that he wasn’t learning anything, because he kept doing the same thing over and over.

It’s not just young children that confuse learning and doing new things. For instance, how many software developers go through the trouble of deliberate practice by performing katas or attending dojos?

It may seem silly to repeat exercises that you’ve already done many times, but it’s not. It’s the only way to become a black belt in your field. And remember that mastery is one of the three intrinsic motivators (the others being autonomy and purpose).

Practicing means slowing down and moving focus from outcome to process. It’s best to use simple exercises that you can complete in a limited amount of time, so you can do the same exercise multiple times.

I’ve found that I virtually always learn something new when I practice. That’s not because I’ve forgotten how to solve the problem since last time, but because I’ve learned new things since then and thus see the world through new eyes.

For example, since Java 8 came out I’ve been trying to use the new stream classes to help move to a more functional style of programming. This has changed the way I look at old problems, like FizzBuzz.

Let’s see this in action. Of course, I start by adding a test:

+ package remonsinnema.blog.fizzbuzz;
+
+ import static org.junit.Assert.assertEquals;
+
+ import org.junit.Test;
+
+
+ public class WhenFizzingAndBuzzing {
+
+   private final FizzBuzz fizzbuzz = new FizzBuzz();
+
+   @Test
+   public void shouldReplaceWithFizzAndBuzz() {
+     assertEquals(“1”, “1”, fizzbuzz.get(1));
+   }
+
+ }

This test uses the When…Should form of unit testing that helps focus on behavior rather than implementation details. I let Eclipse generate the code required to make this compile:

+ package remonsinnema.blog.fizzbuzz;
+
+
+ public class FizzBuzz {
+
+   public String get(int i) {
+     return null;
+   }
+
+ }

The simplest code that makes the test pass is to fake it:

  package remonsinnema.blog.fizzbuzz;
  public class FizzBuzz {
    public String get(int i) {
–     return null;
+     return “1”;
    }
  }

Now that the test passes, it’s time for refactoring. I remove duplication from the test:

  public class WhenFizzingAndBuzzing {
    @Test
    public void shouldReplaceWithFizzAndBuzz() {
–     assertEquals(“1”, “1”, fizzbuzz.get(1));
+     assertFizzBuzz(“1”, 1);
+   }
+
+   private void assertFizzBuzz(String expected, int n) {
+     assertEquals(Integer.toString(n), expected, fizzbuzz.get(n));
    }
  }

Next I add a test to force the real implementation:

  public class WhenFizzingAndBuzzing {
    @Test
    public void shouldReplaceWithFizzAndBuzz() {
      assertFizzBuzz(“1”, 1);
+     assertFizzBuzz(“2”, 2);
    }
    private void assertFizzBuzz(String expected, int n) {
  package remonsinnema.blog.fizzbuzz;
  public class FizzBuzz {
–   public String get(int i) {
–     return “1”;
+   public String get(int n) {
+     return Integer.toString(n);
    }
  }

OK, now let’s get real with a test for Fizz:

  public class WhenFizzingAndBuzzing {
    public void shouldReplaceWithFizzAndBuzz() {
      assertFizzBuzz(“1”, 1);
      assertFizzBuzz(“2”, 2);
+     assertFizzBuzz(“Fizz”, 3);
    }
    private void assertFizzBuzz(String expected, int n) {
  package remonsinnema.blog.fizzbuzz;
  public class FizzBuzz {
    public String get(int n) {
+     if (n == 3) {
+       return “Fizz”;
+     }
      return Integer.toString(n);
    }

Similar for Buzz:

  public class WhenFizzingAndBuzzing {
      assertFizzBuzz(“Fizz”, 3);
+     assertFizzBuzz(“4”, 4);
+     assertFizzBuzz(“Buzz”, 5);
    }
    private void assertFizzBuzz(String expected, int n) {
  public class FizzBuzz {
      if (n == 3) {
        return “Fizz”;
      }
+     if (n == 5) {
+       return “Buzz”;
+     }
      return Integer.toString(n);
    }

Here I just copied and pasted the if statement to get it working quickly. We shouldn’t stop there, of course, but get rid of the dirty stuff. In this case, that’s duplication.

First, let’s update the code to make the duplication more apparent:

  package remonsinnema.blog.fizzbuzz;
  public class FizzBuzz {
    public String get(int n) {
–     if (n == 3) {
–       return “Fizz”;
+     MultipleReplacer replacer = new MultipleReplacer(3, “Fizz”);
+     if (n == replacer.getValue()) {
+       return replacer.getText();
      }
–     if (n == 5) {
–       return “Buzz”;
+     replacer = new MultipleReplacer(5, “Buzz”);
+     if (n == replacer.getValue()) {
+       return replacer.getText();
      }
      return Integer.toString(n);
    }
+ package remonsinnema.blog.fizzbuzz;
+
+
+ public class MultipleReplacer {
+
+   private final int value;
+   private final String text;
+
+   public MultipleReplacer(int value, String text) {
+     this.value = value;
+     this.text = text;
+   }
+
+   public int getValue() {
+     return value;
+   }
+
+   public String getText() {
+     return text;
+   }
+
+ }

I just created a new value object to hold the two values that I had to change after the copy/paste.

Now that the duplication is clearer, it’s easy to remove:

  package remonsinnema.blog.fizzbuzz;
+ import java.util.Arrays;
+ import java.util.Collection;
+
  public class FizzBuzz {
+   private final Collection<MultipleReplacer> replacers = Arrays.asList(
+       new MultipleReplacer(3, “Fizz”), new MultipleReplacer(5, “Buzz”));
+
    public String get(int n) {
–     MultipleReplacer replacer = new MultipleReplacer(3, “Fizz”);
–     if (n == replacer.getValue()) {
–       return replacer.getText();
–     }
–     replacer = new MultipleReplacer(5, “Buzz”);
–     if (n == replacer.getValue()) {
–       return replacer.getText();
+     for (MultipleReplacer replacer : replacers) {
+       if (n == replacer.getValue()) {
+         return replacer.getText();
+       }
      }
      return Integer.toString(n);
    }

I’m not done cleaning up, however. The current code suffers from feature envy, which I resolve by moving behavior into the value object:

  package remonsinnema.blog.fizzbuzz;
  import java.util.Arrays;
  import java.util.Collection;
+ import java.util.Optional;
  public class FizzBuzz {
    public String get(int n) {
      for (MultipleReplacer replacer : replacers) {
–       if (n == replacer.getValue()) {
–         return replacer.getText();
+       Optional<String> result = replacer.textFor(n);
+       if (result.isPresent()) {
+         return result.get();
        }
      }
      return Integer.toString(n);
  package remonsinnema.blog.fizzbuzz;
+ import java.util.Optional;
+
  public class MultipleReplacer {
      this.text = text;
    }
–   public int getValue() {
–     return value;
–   }
–   public String getText() {
–     return text;
+   public Optional<String> textFor(int n) {
+     if (n == value) {
+       return Optional.of(text);
+     }
+     return Optional.empty();
    }
  }

Now that I’m done refactoring, I can continue with multiples:

  public class WhenFizzingAndBuzzing {
      assertFizzBuzz(“Fizz”, 3);
      assertFizzBuzz(“4”, 4);
      assertFizzBuzz(“Buzz”, 5);
+     assertFizzBuzz(“Fizz”, 6);
    }
    private void assertFizzBuzz(String expected, int n) {
  public class MultipleReplacer {
    }
    public Optional<String> textFor(int n) {
–     if (n == value) {
+     if (n % value == 0) {
        return Optional.of(text);
      }
      return Optional.empty();

The final test is for simultaneous “Fizz” and “Buzz”:

  public class WhenFizzingAndBuzzing {
      assertFizzBuzz(“4”, 4);
      assertFizzBuzz(“Buzz”, 5);
      assertFizzBuzz(“Fizz”, 6);
+     assertFizzBuzz(“7”, 7);
+     assertFizzBuzz(“8”, 8);
+     assertFizzBuzz(“Fizz”, 9);
+     assertFizzBuzz(“Buzz”, 10);
+     assertFizzBuzz(“11”, 11);
+     assertFizzBuzz(“Fizz”, 12);
+     assertFizzBuzz(“13”, 13);
+     assertFizzBuzz(“14”, 14);
+     assertFizzBuzz(“FizzBuzz”, 15);
    }
    private void assertFizzBuzz(String expected, int n) {
  public class FizzBuzz {
        new MultipleReplacer(3, “Fizz”), new MultipleReplacer(5, “Buzz”));
    public String get(int n) {
+     StringBuilder result = new StringBuilder();
      for (MultipleReplacer replacer : replacers) {
–       Optional<String> result = replacer.textFor(n);
–       if (result.isPresent()) {
–         return result.get();
+       Optional<String> replacement = replacer.textFor(n);
+       if (replacement.isPresent()) {
+         result.append(replacement.get());
        }
      }
+     if (result.length() > 0) {
+       return result.toString();
+     }
      return Integer.toString(n);
    }

This code is rather complex, but this is where streams come to the rescue:

  public class FizzBuzz {
        new MultipleReplacer(3, “Fizz”), new MultipleReplacer(5, “Buzz”));
    public String get(int n) {
–     StringBuilder result = new StringBuilder();
–     for (MultipleReplacer replacer : replacers) {
–       Optional<String> replacement = replacer.textFor(n);
–       if (replacement.isPresent()) {
–         result.append(replacement.get());
–       }
–     }
–     if (result.length() > 0) {
–       return result.toString();
–     }
–     return Integer.toString(n);
+     return replacers.stream()
+         .map(replacer -> replacer.textFor(n))
+         .filter(Optional::isPresent)
+         .map(optional -> optional.get())
+         .reduce((a, b) -> a + b)
+         .orElse(Integer.toString(n));
    }
  }

Note how the for and if statements disappear. Rather than spelling out how something needs to be done, we say what we want to achieve.

We can apply the same trick to get rid of the remainingif statement in our ode base:

  public class MultipleReplacer {
    }
    public Optional<String> textFor(int n) {
–     if (n % value == 0) {
–       return Optional.of(text);
–     }
–     return Optional.empty();
+     return Optional.of(text)
+         .filter(ignored -> n % value == 0);
    }
  }

The code is on GitHub.

Software Engineering in 2016

engineeringI write computer programs for a living. Many people therefore refer to me as a software engineer, but that term has always made me uncomfortable.

Mary Shaw from the Software Engineering Institute explains where that unease comes from: our industry doesn’t meet the standards for engineers as set by real engineering fields like structural engineering.

Shaw defines engineering as “Creating cost-effective solutions to practical problems by applying codified knowledge, building things in the service of mankind.” (my italics) While we certainly do this some of the time in some places, I believe we are still quite a ways away from doing it all of the time everywhere.

Examples abound. Code is still being written without automated tests, without peer review, and without continuously integrating it with code of fellow team members. Basic stuff, really.

The State of DevOps 2015 report goes into some more advanced practices. It presents compelling evidence that these practices lead to better business results. High-performing companies in the study deploy code 30x more often and have a 60x higher success rate and 168x lower Mean Time To Recovery (MTTR) than low-performing companies.

Yes, you’ve read that right: lower risk of failure and quicker recovery when things do go wrong!

So why aren’t we all applying this codified knowledge?

DevOps practices, as well as more advanced Agile practices like test-driven development and pair programming, require more discipline and that seems to be lacking in our field. Uncle Bob explains this lack of discipline by pointing to the population demographics in our field.

go-fast-go-wellThere still seems to be a strong belief that “quick and dirty” really is quicker. But in the long term, it almost always ends up just being dirty, and many times actually slower.

Yes, going into a production database and manually updating rows to compensate for a bug in the code can be a quick fix, but it also increases the risk of unanticipated things going wrong. Do it often enough, and something will go wrong, requiring fixes for the fixes, etc.

Following a defined process may seem slower than a quick hack, but it also reduces risk. If you can make the defined process itself quicker, then at some point there is little to gain from deviating from the process.

The high-performing DevOps companies from the aforementioned study prove that this is not a pipe dream. With a good Continuous Delivery process you can make fixing urgent critical bugs a non-event. This is good, because people make poorer decisions under stress.

This finding shouldn’t be news to us. It’s the same lesson that we learned with Continuous Integration. Following a process and doing it more often leads to an optimization of the process and an overall increase in productivity.

Until the majority in our field have learned to apply this and other parts of our codified knowledge, we should probably stop calling ourselves engineers.