How To Process Java Annotations

One of the cool new features of Java 8 is the support for lambda expressions. Lambda expressions lean heavily on the FunctionalInterface annotation.

In this post, we’ll look at annotations and how to process them so you can implement your own cool features.

Annotations

Annotations were added in Java 5. The Java language comes with some predefined annotations, but you can also define custom annotations.

Many frameworks and libraries make good use of custom annotations. JAX-RS, for instance, uses them to turn POJOs into REST resources.

Annotations can be processed at compile time or at runtime (or even both).

At runtime, you can use the reflection API. Each element of the Java language that can be annotated, like class or method, implements the AnnotatedElement interface. Note that an annotation is only available at runtime if it has the RUNTIME RetentionPolicy.

Compile-Time Annotation Processing

Java 5 came with the separate apt tool to process annotations, but since Java 6 this functionality is integrated into the compiler.

You can either call the compiler directly, e.g. from the command line, or indirectly, from your program.

In the former case, you specify the -processor option to javac, or you use the ServiceLoader framework by adding the file META-INF/services/javax.annotation.processing.Processor to your jar. The contents of this file should be a single line containing the fully qualified name of your processor class.

The ServiceLoader approach is especially convenient in an automated build, since all you have to do is put the annotation processor on the classpath during compilation, which build tools like Maven or Gradle will do for you.

Compile-Time Annotation Processing From Within Your Application

You can also use the compile-time tools to process annotations from within your running application.

Rather than calling javac directly, use the more convenient JavaCompiler interface. Either way, you’ll need to run your application with a JDK rather than just a JRE.

The JavaCompiler interface gives you programmatic access to the Java compiler. You can obtain an implementation of this interface using ToolProvider.getSystemJavaCompiler(). This method is sensitive to the JAVA_HOME environment variable.

The getTask() method of JavaCompiler allows you to add your annotation processor instances. This is the only way to control the construction of annotation processors; all other methods of invoking annotation processors require the processor to have a public no-arg constructor.

Annotation Processors

A processor must implement the Processor interface. Usually you will want to extend the AbstractProcessor base class rather than implement the interface from scratch.

Each annotation processor must indicate the types of annotations it is interested in through the getSupportedAnnotationTypes() method. You may return * to process all annotations.

The other important thing is to indicate which Java language version you support. Override the getSupportedSourceVersion() method and return one of the RELEASE_x constants.

With these methods implemented, your annotation processor is ready to get to work. The meat of the processor is in the process() method.

When process() returns true, the annotations processed are claimed by this processor, and will not be offered to other processors. Normally, you should play nice with other processors and return false.

Elements and TypeMirrors

The annotations and the Java elements they are present on are provided to your process() method as Element objects. You may want to process them using the Visitor pattern.

The most interesting types of elements are TypeElement for classes and interfaces (including annotations), ExecutableElement for methods, and VariableElement for fields.

Each Element points to a TypeMirror, which represents a type in the Java programming language. You can use the TypeMirror to walk the class relationships of the annotated code you’re processing, much like you would using reflection on the code running in the JVM.

Processing Rounds

Annotation processing happens in separate stages, called rounds. During each round, a processor gets a chance to process the annotations it is interested in.

The annotations to process and the elements they are present on are available via the RoundEnvironment parameter passed into the process() method.

If annotation processors generate new source or class files during a round, then the compiler will make those available for processing in the next round. This continues until no more new files are generated.

The last round contains no input, and is thus a good opportunity to release any resources the processor may have acquired.

Initializing and Configuring Processors

Annotation processors are initialized with a ProcessingEnvironment. This processing environment allows you to create new source or class files.

It also provides access to configuration in the form of options. Options are key-value pairs that you can supply on the command line to javac using the -A option. For this to work, you must return the options’ keys in the processor’s getSupportedOptions() method.

Finally, the processing environment provides some support routines (e.g. to get the JavaDoc for an element, or to get the direct super types of a type) that come in handy during processing.

Classpath Issues

To get the most accurate information during annotation processing, you must make sure that all imported classes are on the classpath, because classes that refer to types that are not available may have incomplete or altogether missing information.

When processing large numbers of annotated classes, this may lead to a problem on Windows systems where the command line becomes too large (> 8K). Even when you use the JavaCompiler interface, it still calls javac behind the scenes.

The Java compiler has a nice solution to this problem: you can use argument files that contain the arguments to javac. The name of the argument file is then supplied on the command line, preceded by @.

Unfortunately, the JavaCompiler.getTask() method doesn’t support argument files, so you’ll have to use the underlying run() method.

Remember that the getTask() approach is the only one that allows you to construct your annotation processors. If you must use argument files, then you have to use a public no-arg constructor.

If you’re in that situation, and you have multiple annotation processors that need to share a single instance of a class, you can’t pass that instance into the constructor, so you’ll be forced to use something like the Singleton pattern.

Conclusion

Annotations are an exciting technology that have lots of interesting applications. For example, I used them to extract the resources from a REST API into a resource model for further processing, like generating documentation.

I’m very interested to learn what you have used them for. Please leave a comment below.

Advertisements

The Art of the Visitor Pattern (3)

In the previous posts in this series I looked at the basic Visitor design pattern and a couple of variations, like eliminating recursion to make the visitor interruptable. As promised, this time I will make our visitor re-entrant.

Here’s what we have so far:

public class BaseFileSystemVisitor 
    implements FileSystemVisitor {

  public FileSystemNode visit(final FileSystemNode node) {
    int numNodesVisited = 0;
    FileSystemNode current = node;

   while (current != null && numNodesVisited < maxNodes) {
      numNodesVisited++;
      stack.push(current);
      if (current.start(this)) {
        current = current.getNext();
      } else {
        current = current.getNextNoChild();
      }
      while (!stack.isEmpty() 
          && !stack.peek().isAncestorOf(current)) {
        stack.pop().stop(this);
      }
    }
    return current;
  }

First of all, note there is a bug in here: when start returns false, the stop method is called anyway! This is because the folder is still on the stack and therefore will be stopped when it is popped from the stack. Given this analysis, the fix is easy:

  public FileSystemNode visit(final FileSystemNode node) {
    int numNodesVisited = 0;
    FileSystemNode current = node;

    while (current != null && numNodesVisited < maxNodes) {
      numNodesVisited++;
      stack.push(current);
      if (current.start(this)) {
        current = current.getNext();
      } else {
        current = current.getNextNoChild();
        stack.pop();
      }
      while (!stack.isEmpty()
          && !stack.peek().isAncestorOf(current)) {
        stack.pop().stop(this);
      }
    }
    return current;
  }

Now suppose we want to change the order in which children are displayed, e.g. first show all sub-folders, than the others. We could write that as follows:

public class XmlVisitor 
    extends BaseFileSystemVisitor {

  public boolean start(Folder folder) {
    out.println("<folder name='" + folder.getName() 
        + "'>");
  
    Iterator children = folder.getChildren();
    while (children.hasNext()) {
      FileSystemNode child = children.next();
      if (child instanceof Folder) {
        while ((child = visit(child)) != null) { 
          // Repeat, since the visitor may be interrupted
        }
      }
    }
    
    children = folder.getChildren();
    while (children.hasNext()) {
      FileSystemNode child = children.next();
      if (!(child instanceof Folder)) {
        while ((child = visit(child)) != null) { 
          // Repeat, since the visitor may be interrupted
        }
      }
    }

    stop(folder);
    return false;
  }

But this produces the wrong output, and an EmptyStackException on top of that. The problem is that the visit method just iterates until there are no more nodes left. This is wrong for the re-entrant case: it should iterate until the nodes are no longer descendants of the root that is passed into the method:

  public FileSystemNode visit(final FileSystemNode node) {
    final FileSystemNode root;
    FileSystemNode current;
    if (stack.isEmpty() || stack.peek() != node) {
      current = node;
      root = node;
    } else {
      current = (FileSystemNode) stack.pop();
      root = (FileSystemNode) stack.firstElement();
    }
    while (root.isAncestorOf(current) 
        && numNodesVisited < maxNodes) {
      numNodesVisited++;
      stack.push(current);
      if (current.start(this)) {
        current = current.getNext();
      } else {
        current = current.getNextNoChild();
        stack.pop();
      }
      if (root.isAncestorOf(current)) {
        while (!stack.isEmpty() 
            && !stack.peek().isAncestorOf(current)) {
          stack.pop().stop(this);
        }
      }
    }
    
    if (numNodesVisited >= maxNodes) {
      numNodesVisited = 0;
    }

    final FileSystemNode result;
    if (root.isAncestorOf(current)) {
      stack.push(current);
      result = current;
    } else {
      result = null;
    }    
    return result;
  }

Note that numNodesVisited is now a field instead of local variable.

The code has become much more complex, but we are now successfully supporting both interruptability and re-entrancy. Next time, we will make things even more complex by adding support for modification of the data structure being visited.

The Art of the Visitor Pattern (2)

In my previous post, we took a look at the basic Visitor design pattern, and a couple of simple variations. I now continue with a more ambitious variation.

This is where the fun begins.

Variation 3: Eliminate recursion

Suppose we have a huge file system tree and that processing it takes a lot of time. We’d then want to update some progress indicator, or give other threads some time to run. To do that, we’d need to interrupt the processing. The current implementation doesn’t support that, because of the recursion.

Most developers probably know that every recursive implementation can be replaced by an iterative one. In a recursive function, there is an iteration of the same method call on the program’s stack. We can replace that using an explicit stack.

But in order to be able to use iteration, we need to be able to move from one node in the file system tree to the next:

public interface FileSystemNode {

  FileSystemNode getFirstChild();
  FileSystemNode getLastChild();
  
  FileSystemNode getNextSibling();
  FileSystemNode getPrevSibling();
  
  FileSystemNode getNext();
  FileSystemNode getPrev();
  
  FileSystemNode getNextNoChild();

  boolean isAncestorOf(FileSystemNode next);

  // other methods as before ...
}

We can implement that as follows:

public abstract class AbstractFileSystemNode
    implements FileSystemNode {

  public FileSystemNode getFirstChild() {
    return children.isEmpty() ? null : children.get(0);
  }

  public FileSystemNode getLastChild() {
    return children.isEmpty() ? null
        : children.get(children.size() - 1);
  }

  public FileSystemNode getNextSibling() {
    FileSystemNode result = null;
    if (parent != null) {
      final int index = 
          parent.children.indexOf(this) + 1;
      if (index < parent.children.size()) {
        result = parent.children.get(index);
      }
    }
    return result;
  }
  
  public FileSystemNode getPrevSibling() {
    FileSystemNode result = null;
    if (parent != null) {
      final int index = 
          parent.children.indexOf(this) - 1;
      if (index >= 0) {
        result = parent.children.get(index);
      }
    }
    return result;
  }
  
  public FileSystemNode getNext() {
    FileSystemNode result = getFirstChild();
    if (result == null) {
      result = getNextNoChild();
    }
    return result;
  }

  public FileSystemNode getNextNoChild() {
    FileSystemNode result = getNextSibling();
    if (result == null) {
      FileSystemNode next = this;
      while (next.getParent() != null) {
        next = next.getParent();
        if (next.getNextSibling() != null) {
          result = next.getNextSibling();
          break;
        }
      }
    }
    return result;
  }

  public FileSystemNode getPrev() {
    FileSystemNode result = null;
    FileSystemNode prev = getPrevSibling();
    if (prev == null) {
      result = getParent();
    } else {
      while (prev.getLastChild() != null) {
        prev = prev.getLastChild();
      }
      result = prev;
    }
    return result;
  }

  public boolean isAncestorOf(
      final FileSystemNode descendant) {
    boolean result = false;
    FileSystemNode current = descendant;
    while (!result && current != null) {
      result = equals(current);
      current = current.getParent();
    }
    return result;
  }

  // other methods as before...
}

With these navigational methods in place, we can eliminate the recursion:

public class BaseFileSystemVisitor
    implements FileSystemVisitor {

  public void visit(final FileSystemNode node) {
    Stack stack = new Stack();
    FileSystemNode current = node;

    while (current != null) {
      stack.push(current);
      if (current.start(this)) {
        current = current.getNext();
      } else {
        current = current.getNextNoChild();
      }
      while (!stack.isEmpty() 



          && !stack.peek().isAncestorOf(current)) {
        stack.pop().stop(this);
      }
    }
  }

  // other methods as before...

}

Note: I use generics here (e.g. Stack<FileSystemNode>), but am not showing it, since that messes up the formatting in the code viewer thingie.

The current solution has some advantages over the recursive one, like not suffering from stack overflows with deeply nested trees, but my goal was to be able to traverse the tree in multiple calls. To make that possible, we need to be able to specify how many nodes can be visited in a single call:

public interface FileSystemVisitor {
  FileSystemNode visit(FileSystemNode node);
  void setMaxNodes(int maxNodes);
  
  // ...
}

The visit() method returns where it stopped, so you can supply that information with your next call. In our implementation, we need to make the stack a field instead of a local variable, so that its contents is preserved between calls:

public class BaseFileSystemVisitor
    implements FileSystemVisitor {

  private int maxNodes = Integer.MAX_VALUE;
  private final Stack stack = new Stack();
  
  public void setMaxNodes(int maxNodes) {
    this.maxNodes = maxNodes;
  }

  public FileSystemNode visit(final FileSystemNode node) {
    int numNodesVisited = 0;
    FileSystemNode current = node;

    while (current != null
        && numNodesVisited < maxNodes) {
      numNodesVisited++;
      stack.push(current);
      if (current.start(this)) {
        current = current.getNext();
      } else {
        current = current.getNextNoChild();
      }
      while (!stack.isEmpty()
          && !stack.peek().isAncestorOf(current)) {
        stack.pop().stop(this);
      }
    }
    return current;
  }

  // other methods as before...
}

And voila, we’re done. Next time we will look at making this code re-entrant, so that we can start visiting a node while we’re visiting another node.

The Art of the Visitor Pattern (1)

The visitor design pattern is one of those relatively simple patterns that you can use in many situations. Some situations are more complex than others, though, and may require some modifications. I will first present the basic pattern, and then some variations:

  1. process before and after a node
  2. skip parts of the data structure
  3. eliminate recursion
  4. support re-entrancy
  5. manipulate the data structure during visiting

This first post in a series will set the stage by discussing the basic pattern and the first two variations.

The basic pattern

Consider the following example. We have a data structure that represents a file system, with folders, files, links, devices, etc. This structure is basically a tree:

public interface FileSystemNode {

  FileSystemNode getParent();
  void setParent(FileSystemNode node);
	
  Iterator getChildren();
  void add(FileSystemNode child);
	
  void accept(FileSystemVisitor visitor);

  String getName();

}

Note: I’m using generics here but am not showing it, since it messes up the formatting.

Each of the implementors of this interface is listed in the FileSystemVisitor, so that it can handle them differently:

public interface FileSystemVisitor {

  void visit(Folder folder);
  void visit(File file);
  void visit(Link link);
  void visit(Device device);
  // ...

}

The gist of the visitor pattern is a double dispatch where the visitor calls the accept() method of the FileSystemNode:

public class BaseFileSystemVisitor 
    implements FileSystemVisitor {

  public void visit(final FileSystemNode node) {
    node.accept(this);
	  
    Iterator children = node.getChildren();
    while (children.hasNext()) {
      visit(children.next());
    }
  }

  public void visit(Folder folder) {
    // For descendants to implement
  }

  public void visit(File file) {
    // For descendants to implement
  }

  public void visit(Link link) {
    // For descendants to implement
  }

  public void visit(Device device) {
    // For descendants to implement
  }

  // ...
	
}

…and the FileSystemNode calls the visit() method of the visitor:

public class FolderImpl extends AbstractFileSystemNode 
    implements Folder {

  public void accept(final FileSystemVisitor visitor) {
    visitor.visit(this);
  }

  // Folder implementation...

}

An implementation for listing files (like the Unix ls command) could be implemented as follows:

public class ListVisitor extends BaseFileSystemVisitor {

  private final PrintStream out;

  public ListVisitor() {
    this(System.out);  
  }
  
  public ListVisitor(final PrintStream out) {
    super();
    this.out = out;
  }

  @Override
  public void visit(final Device device) {
    out.println(device.getType() + ": " 
        + device.getName());
  }

  @Override
  public void visit(final File file) {
    out.println(file.getName());
  }

  @Override
  public void visit(final Folder folder) {
    out.println(folder.getName() + "/");
  }

  @Override
  public void visit(final Link link) {
    out.println(link.getName() + " ->" 
        + link.getTarget());
  }
  
  // ...
  
}

Variation 1: Process before and after a node

Now suppose we don’t want a simple list like the one provided by ListVisitor, but we want to export to XML. For that to work, we need to open and close the containing folder tag, e.g.:

<folder name='remon'>
  <file name='quotes.txt'/>
  <device name='wlan' type='usb'/>
  <link name='java'
      target='/usr/lib/jvm/java-6-sun/bin/java'/>
</folder>

To implement this, we replace the visit() methods with start() and stop() methods:

public interface FileSystemVisitor {
  void start(Folder folder);
  void stop(Folder folder);

  void start(File file);
  void stop(File file);


  void start(Link link);
  void stop(Link link);

  void start(Device device);
  void stop(Device device);

  // ...
}

public class BaseFileSystemVisitor 
    implements FileSystemVisitor {

  public void visit(final FileSystemNode node) {
    node.start(this);
	  
    Iterator children = node.getChildren();
    while (children.hasNext()) {
      visit(children.next());
    }
	  
    node.stop(this);      
  }

  // ...
}

This allows us to implement the XmlVisitor:

public class XmlVisitor extends BaseFileSystemVisitor {

  private final PrintStream out;

  public XmlVisitor() {
    this(System.out);  
  }
  
  public XmlVisitor(final PrintStream out) {
    this.out = out;
  }

  @Override
  public void start(Device device) {
    out.println("<device type='" + device.getType() 
        + "' name='" + device.getName() + "'/>");
  }

  @Override
  public void start(File file) {
    out.println("<file name='" + file.getName() 
        + "'/>");
  }

  @Override
  public void start(Folder folder) {
    out.println("<folder name='" + folder.getName() 
        + "'>");
  }

  @Override
  public void stop(Folder folder) {
    out.println("</folder>");
  }
  
  @Override
  public void start(Link link) {
    out.println("<link name='" + link.getName() 
        + "' target='" + link.getTarget() + "'/>");
  }

}

Variation 2: Skip parts of the data structure

Now suppose we want to skip processing a sub-tree. For instance, we may want to skip folders and files whose names start with a dot. We can achieve this by having the start() method return a boolean that indicates whether the sub-tree starting at the node is to be processed:

public interface FileSystemVisitor {
  boolean start(Folder folder);
  void stop(Folder folder);

  boolean start(File file);
  void stop(File file);

  boolean start(Link link);
  void stop(Link link);

  boolean start(Device device);
  void stop(Device device);
  
  // ...
}

public class BaseFileSystemVisitor
    implements FileSystemVisitor {

  public void visit(final FileSystemNode node) {
    if (node.start(this)) {
      Iterator children = node.getChildren();
      while (children.hasNext()) {
        visit(children.next());
      }
  	  
      node.stop(this);
    }
  }

  public boolean start(Folder folder) {
    // For descendants to implement
    return true;
  }

  // ...
}

public class FolderImpl extends AbstractFileSystemNode 
    implements Folder {

  public FolderImpl(final String name) {
    super(name);
  }

  public boolean start(final FileSystemVisitor visitor) {
    return visitor.start(this);
  }

  public void stop(final FileSystemVisitor visitor) {
    visitor.stop(this);
  }

}

public class XmlVisitor extends BaseFileSystemVisitor {
  // ...

  @Override
  public boolean start(Folder folder) {
    boolean result = !folder.getName().startsWith(".");
    if (result) {
      out.println("<folder name='" + folder.getName() 
          + "'>");
    }
    return result;
  }

  // ...
}

Next time we’ll get serious with the visitor pattern by eliminating the recursion, so stay tuned!