SCM

How To Develop Software Using Only SaaS

~ Ray Sinnema ~ Leave a comment

cloud-codeThe world is fast moving to Software-as-a-Service (SaaS) and we developers are busy learning how to build SaaS applications.

We can now finally do that using nothing but SaaS applications ourselves.

The Developer’s Toolbox

As developers, we don’t ask for much.

An Integrated Development Environment (IDE) lets us do our main task: writing code. A Source Code Management (SCM) system stores our Heartbreaking Work of Staggering Genius. A Continuous Integration (CI) server pulls our code through hoops that prove it is ready for use. And finally a Platform-as-a-Service (PaaS) or other deployment environment runs our applications.

We are used to running all of these on premises. IDEs like Eclipse or IntelliJ run on our local machines. SCMs like Git or Subversion run on some company server, as does our Jenkins/Hudson or TeamCity CI server. Finally, we deploy to a Paas like CloudFoundry, or to a custom server.

Most of those tools already run in the cloud. For those that don’t, we can easily find good alternatives. Let’s take a look at some of the candidates.

Integrated Development Environments

I’ve written about Cloud9 before. It’s mainly focused on web languages like JavaScript. For Java, Codenvy seems a better choice. For both, you can run the hosted offering, or deploy it in your own data center.

Neither can match a local IDE experience yet, but the gap is closing. On the other hand, they offer some functionality you won’t easily find in locally installed IDEs, like remote pair programming.

Source Code Management

githubGit has taken over the world, and the SaaS version of it, GitHub, is following suit.

Some people even think that your GitHub profile is your resume.

Again, you can use the hosted version (with public or private repositories), or install GitHub in your data center.

Both Cloud9 and Codenvy work seamlessly with GitHub repositories.

Continuous Integration

Jenkins/Hudson is the leader in this space, and CloudBees offers a SaaS version. Other products include Bamboo, Travis CI and CodeShip. Some of these are free for open source projects. Again, there are hosted and on premises versions.

The CI tools support GitHub through public SSH keys for access and commit hooks for starting jobs.

Platform-as-a-Service

After GitHub, these are probably the most familiar to you: Pivotal CloudFoundry, Heroku, Google App Engine, and Azure. CloudFoundry is backed by many big organizations (including the company I work for, EMC) and seems to be emerging as the leader.

cloudfoundrySome cloud IDEs let you push to a PaaS directly, but I don’t think that’s the right way to do it.

You should commit to your SCM and let CI pick up your changes.

Your CI jobs should be responsible for pushing to the PaaS. Your CI may have a custom integration to your PaaS, or you may have to use something like the CloudFoundry command-line interface to push your changes.

Conclusion

It seems that our entire tool chain is now available as a service, although the IDEs still leave us wanting a bit. Most of these tools are available as open source and can be deployed in your own data center.

Looks like we’re making some progress towards a Frictionless Development Environment!

What SaaS applications are you using for software development? Please leave a comment below.

Advertisement

How To Remove Friction From Your Version Control Experience

~ Ray Sinnema ~ 2 Comments

ErrorLast week, I spend several days fixing a bug that only surfaced in a distributed environment.

I felt pressure to fix it quickly, because our continuous integration build was red, and we treat that as a “stop the line” event.

Then I came across a post from Tomasz Nurkiewicz who claims that breaking the build is not a crime.

Tomasz argues that a better way to organize software development is to make sure that breaking changes don’t affect your team mates. I agree.

Broken Builds Create Friction

Breaking changes from your co-workers are a form of friction, since they take away time and focus from your job. Tomasz’ setup has less friction than ours.

But I feel we can do better still. In a perfect Frictionless Development Environment (FDE), all friction is removed. So what would that look like with regard to version control?

With current version control systems, there is lots of friction. I complained about Perforce before because of that.

Git is much better, but even then there are steps that have to be performed that take away focus from the real goal you’re trying to achieve: solving the customer’s problem using software.

For instance, you still have to create a new topic branch to work on. And you have to merge it with the main development line. In a perfect world, we wouldn’t have to do that.

Frictionless Version Control

version-controlSo how would a Frictionless Development Environment do version control for us?

Knowing when to create a branch is easy.

All work happens on a topic branch, so every time you start to work on something, the FDE could create a new branch.

The problem is knowing when to merge. But even this is not as hard as it seems.

You’re done with your current work item (user story or whatever you want to call it) when it’s coded, all the tests pass, and the code is clean.

So how would the FDE know when you’re done thinking of new tests for the story?

Well, if you practice Behavior-Driven Development (BDD), you start out with defining the behavior of the story in automated tests. So the story is functionally complete when there is a BDD test for it, and all scenarios in that test pass.

Now we’re left with figuring out when the code is clean. Most teams have a process for deciding this too. For instance, code is clean when static code analysis tools like PMD, CheckStyle, and FindBugs give no warnings.

Some people will argue that we need a minimum amount of code coverage from our tests as well. Or that the code needs to be reviewed by a co-worker. Or that Fortify must not find security vulnerabilities. That’s fine.

pipelineThe basic point is that we can formally define a pipeline of processes that we want to run automatically.

At each stage of the pipeline can we reject the work. Only when all stages complete successfully, are we done.

And then the FDE can simply merge the branch with the main line, and delete it. Zero friction from version control.

What do you think?

Would you like to lubricate your version control experience? Do you think an automated branching strategy as outlined above would work?

How to Create Extensible Java Applications

~ Ray Sinnema ~ 4 Comments

Extension pointsMany applications benefit from being open to extension. This post describes two ways to implement such extensibility in Java.

Extensible Applications

Extensible applications are applications whose functionality can be extended without having to recompile them and sometimes even without having to restart them. This may happen by simply adding a jar to the classpath, or by a more involved installation procedure.

One example of an extensible application is the Eclipse IDE. It allows extensions, called plug-ins, to be installed so that new functionality becomes available. For instance, you could install a Source Code Management (SCM) plug-in to work with your favorite SCM.

As another example, imagine an implementation of the XACML specification for authorization. The “X” in XACML stands for “eXtensible” and the specification defines a number of extension points, like attribute and category IDs, combining algorithms, functions, and Policy Information Points. A good XACML implementation will allow you to extend the product by providing a module that implements the extension point.

Service Provider Interface

Oracle’s solution for creating extensible applications is the Service Provider Interface (SPI).

In this approach, an extension point is defined by an interface:

package com.company.application;

public interface MyService {
  // ...
}

You can find all extensions for such an extension point by using the ServiceLoader class:

public class Client {

  public void useService() {
    Iterator<MyService> services = ServiceLoader.load(
        MyService.class).iterator();
    while (services.hasNext()) {
      MyService service = services.next();
      // ... use service ...
  }

}

An extension for this extension point can be any class that implements that interface:

package com.company.application.impl;

public class MyServiceImpl implements MyService {
  // ...
}

The implementation class must be publicly available and have a public no-arg constructor. However, that’s not enough for the ServiceLoader class to find it.

You must also create a file named after the fully qualified name of the extension point interface in META-INF/services. In our example, that would be:

META-INF/services/com.company.application.Myservice

This file must be UTF-8 encoded, or ServiceLoader will not be able to read it. Each line of this file should contain the fully qualified name of one extension implementing the extension point, for instance:

com.company.application.impl.MyServiceImpl

OSGi Services

Service registryThe SPI approach described above only works when the extension point files are on the classpath.

In an OSGi environment, this is not the case. Luckily, OSGi has its own solution to the extensibility problem: OSGi services.

With Declarative Services, OSGi services are easy to implement, especially when using the annotations of Apache Felix Service Component Runtime (SCR):

@Service
@Component
public class MyServiceImpl implements MyService {
  // ...
}

With OSGi and SCR, it is also very easy to use a service:

@Component
public class Client {

  @Reference
  private MyService myService;

  protected void bindMyService(MyService bound) {
    myService = bound;
  }

  protected void unbindMyService(MyService bound) {
    if (myService == bound) {
      myService = null;
    }
  }

  public void useService() {
    // ... use myService ...
  }

}

Best of Both Worlds

So which of the two options should you chose? It depends on your situation, of course. When you’re in an OSGi environment, the choice should obviously be OSGi services. If you’re not in an OSGi environment, you can’t use those, so you’re left with SPI.

CakeBut what if you’re writing a framework or library and you don’t know whether your code will be used in an OSGi or classpath based environment?

You will want to serve as many uses of your library as possible, so the best would be to support both models. This can be done if you’re careful.

Note that adding a Declarative Services service component file like OSGI-INF/myServiceComponent.xml to your jar (which is what the SCR annotations end up doing when they are processed) will only work in an OSGi environment, but is harmless outside OSGi.

Likewise, the SPI service file will work in a traditional classpath environment, but is harmless in OSGi.

So the two approaches are actually mutually exclusive and in any given environment, only one of the two approaches will find anything. Therefore, you can write code that uses both approaches. It’s a bit of duplication, but it allows your code to work in both types of environments, so you can have your cake and eat it too.

The verdict on Perforce

~ Ray Sinnema ~ 14 Comments

At work, I’m now forced to use the Perforce version control system, since that’s what our company has standardized upon. I’ve had some bad feelings about that from the start (based on reading about it), but I’ve hold off on publicizing them so I could give Perforce a fair chance. After all, I have been wrong before ;). So now that I’ve worked with it for several months, here’s my verdict.

Speed
The Perforce slogan is Perforce, The Fast Software Configuration Management System. So they’re basically claiming that they are faster than their competitors. How does this claim hold up?

That question is not so easy to answer, since their competitors are not a homogeneous bunch. But let’s look at one category of competitors: the distributed version control systems. The most well-known of these are Git, Bazaar, and Mercurial. Interestingly, Git calls itself The fast version control system and Mercurial’s slogan is Work easier, Work faster.

Distributed version control systems work locally, meaning they don’t need a network connection. Contrast this with Perforce, that needs a network connection for everything. And I mean everything, to a ridiculous level. For instance, even the help command needs a network connection:

$ p4 help
Perforce client error:
        Connect to server failed; check $P4PORT.
        TCP connect to perforce failed.
        perforce: host unknown.

Now, obviously network access is going to slow things down a lot, so it’s difficult to see how Perforce can still beat their competitors on speed. And my experience has been very clear: it doesn’t! In fact, Perforce is very slow. Now obviously, that depends on your network bandwidth, so your mileage may vary.

But it gets worse. My primary interface to Perforce is not the command line client, but the Eclipse integration, P4WSAD. And although Perforce claims that this is the best of both worlds, my opinion is that this is a piece of crap. There, I said it. P4WSAD makes my life as a developer a hell.

Perforce makes all files read-only by default. Only once you’ve checked out a file, will it become writable. And, you’ve guessed it, that requires network access. In practice, this means that everytime I want to change some source file, I have to wait until P4SWAD checks out the file, which can take up to five seconds! This is extremely annoying, because it completely breaks my flow. And if you think one file is bad, try doing a refactoring that affects multiple files… It is reason enough for me to not ever want to work with Perforce again.

BTW, it is interesting to note that none of the aforementioned distributed version control systems appear in Perforce’s comparison with its competitors.

More connection troubles
Now if all this slowness actually bought me some nice features, that could change the story, right? Well, yes, it could. But it doesn’t.

The same cause for slowness, accessing the network for everything, is also limiting what you can do.

For instance, I have a long commute in the train, so I like to work there. And guess what, I don’t have an internet connection there. Not to worry, Perforce has a workaround called offline mode. This basically means that P4WSAD will nag you for confirmation every time you try to change a file.

It also means that it looses track of which files were changed, so that when you get back online, you forget to submit some files and break the build. That has happened to me quite a few times now, because the reconcile feature is not available in P4SWAD. You need to use the Perforce Visual Client (P4V) for that. So now I need to use two tools to get my work done.

Another limitation of P4WSAD is that it will block a refactoring affecting a file that you haven’t already modified since you went offline. This means you have to hunt down all the places where, say, a method to be renamed is used, and force a “checkout” of all those places by changing something in the file. Only then can you do your refactoring. Very annoying.

Transactions
Perforce claims to support transactions, which is a must for a source code control system. We don’t want our automated build to pick up part of a set of changed files and break because of that!

Unfortunately, transactions in Perforce only work when they work. In other words, when an error occurs, it’s very well possible that Perforce will have comitted only a subset of the files in the “transaction”. This is not a really big deal, as it doesn’t happen all that often, but still.

Directories
Perforce is completely file based; it doesn’t track directories. So it’s impossible to add an empty directory to a repository, for instance. Also, when someone removes a directory, Perforce by default will leave empty directories on people’s file systems when they synchronize. There is a setting to fix that, but it’s set to the wrong value by default. I consider this only a minor flaw, but it’s annoying nonetheless.

Conclusion
Would I recommend Perforce to anybody? Not really. I think there are better alternatives out there. Free ones, mind you. So save yourself some money and check out (pun intended) Git, Mercurial, or Bazaar.