Invert Your Mind

I’ve been working on a decent sized C++ project recently and since the application will be used on a Windows server, I wanted to stay close to that platform. I fired up the latest Visual Studio 2008 version hot off the presses with my MSDN subscription and boy is that thing a piece of junk. It is like working in the stone ages. I mean writing C++ is bad enough and then I have to battle with the worse IDE imaginable?

I think not.

So, I grabbed Eclipse and NetBeans and fired both up. Eclipse, is a beast and I wanted to avoid it at all costs if possible. I’ve never liked Eclipse, no matter the version, and they haven’t improved the issues enough for me to really use it for Java development. However, I found the Eclipse C++ integration decent, but still pretty rough. The Eclipse interface is so clunky that it makes it hard to be productive.

Next I grabbed the latest NetBeans (6.5) and fired it up. I have to say that it started pretty darn fast for a Java IDE. It also has a really decent layout and configuration system. The C++ support, although experimental for on the fly error highlighting, is impressive. After tweaking my colors and battling through keymappings I’m actually productive. NetBeans lacks a number of editor features that are required these days, including multiple clipboards, duplicate lines (has this but no keyboard shortcut), join lines, etc. Besides this problems, I like it.

On the flip side, IntelliJ is still great. 8.0 is just out and it adds a number of new great features and more language support. However, I think that IJ is probably getting close to needing a good chunk rewritten. The configuration system is becoming clunky, the projects difficult to manage, and most aspects of the IDE are beginning to fail.

The speed is horrible, the memory consumption off the charts and the productivity increases that we all used IntelliJ for are is slowly being integrated into the other IDEs. I think IJ is still the best, but their margin is slowing being reduced. If the folks over at JetBrains want to stay competitive they are gonna need more speed, more productivity, more simplicity and more power. These things are all hard to get at the same time, but I’m confident that they will find at way.

For now, I’m gonna give NetBeans the full shake-down, log a ton of bugs and wait for 7.0 to see how many they fix. I’d say my list is pretty short at this point and I could envision a full switch to NetBeans with 5-10 enhancements if IntelliJ continues its slow decline.

Only time will tell.

Been working this week on Savant 2.0. My first thought when I started writing Savant 2.0 was to write a complete replacement for both Maven and Ant that used Groovy and allowed for both a plugin model as well as a simple build script approach. This was too much to bite off when you consider all of the other changes to the dependency management part of Savant we made for 2.0.

For JCatapult we created a set of Ant build scripts that could be plugged into any build file and reused. It looks like this:

<project name="foo">
  <import file="${jcatapult.home}/ant/plugins/java/1.0/compile.xml"/>
  ...
</project>

This model meant that the build.xml file had nothing but import statements in it. It made life much simpler when working with numerous projects that were virtually the same.

The part I didn’t like was that I loved this model. I liked it so much I started using them for everything including Java.net Commons, Savant and internal Inversoft projects. This meant that I had a dependency on JCatapult, which for all intents and purposes is a webapp platform. This seemed strange.

I started thinking about moving these files over to Savant. At first I just figured I would migrate the entire set of Ant plugins from JCatapult over and be done. However, once I moved them over it meant that I would need to start pulling in plugins for a variety of projects. Some would come from Savant and others might come from JCatapult or elsewhere. This was due to the fact that some of the JCatapult plugins were very specific to JCatapult projects. I could force developers to download each plugin they needed from various places, install them in a specific directory and then update their build files before they could build any projects. It all started to look very clunky.

Then I remembered (duh!) that Savant is a dependency management tool and it can find, version and download ANYTHING. Why not apply this pattern to Ant build files in the form of plugins?

So, I did just that!

It works great. The way it works is that I wrote a simple Java class that figures everything out and then generates an ant build file. The full process looks like this:

1. New script called svnt invokes a Savant class
2. This new class parses the project.xml file, which defines the plugins the project will be using
3. It downloads any missing plugins to the Savant local cache
4. Since the plugins are JAR files, it explodes the JAR file to ~/.savant/plugins under a specific directory just for that plugin
5. This class then generates a temporary build.xml file that imports all of the plugins the project needs
6. It outputs the location of this temporary build.xml file
7. The svnt script executes Ant passing it the location of this build file (i.e. ant -f /tmp/tmpbuild.xml)

And that’s it. Nothing fancy, just a little sugar on top of Ant to provide downloading of plugins and build files automatically.

The last part of this exercise is to write a bunch of plugins. Since these are just Ant build scripts and we’ve all created hundreds of those so it should be a simple matter of tweaking them ever so slightly to work when imported.

My new projects don’t have a build.xml file (but if they do it is imported to add additional targets or override targets from the plugins) and just contain the Savant project.xml file. This file now defines plugins and it looks like this:

<project name="foo" group="inversoft.org" version="1.0">
  <plugin group="plugins.savant.inversoft.org" name="clean" version="1.0"/>
  <plugin group="plugins.savant.inversoft.org" name="java" version="1.0"/>
  <plugin group="plugins.savant.inversoft.org" name="junit" version="1.0"/>
  <plugin group="plugins.savant.inversoft.org" name="release-svn" version="1.0"/>
  ...
</project>

That’s it. If anyone is interested in trying it out, I should have a release available in the next few days along with some documentation for the 5-10 plugins I will have ready. If you want to try and build all this by hand right now, you’ll need to install JCatapult since Savant uses that until it can be self building.

Russ Teabeault and I were just talking about our recent experiences with Objective-C and developing applications for either the iPhone or OS X. In general, we both agree that Objective-C is necessary, but painful. The language is quasi-dynamic, not very modern and poorly adopted.

Let me clarify that a bit. Objective-C is dynamic but lacks nice features that most modern dynamic languages provide such as closures (although these aren’t strictly tied to dynamic languages by any means). It also uses a fairly non-standard message passing syntax, that is probably because it had to select something that would not conflict with C and C++. It also still uses header files and lacks good namespacing, both of which are obvious signs of antiquity. Plus, we both shared stories of having impossible times finding good open source libraries, tools and framework. The last bit of our conversation was the extreme lackluster of the XCode IDE, which is more like a text editor than a modern development environment.

I mentioned that prior to 10.4, Apple had provided integration of most of the Cocoa libraries with Java and had even written some OS applications in Java. Not surprising, we both thought the shift from Java to Objective-C was a step in the wrong direction. It seemed as though the correct direction would be a better, faster and more integrated VM that would run many languages including Java, much like the CLR offers developers on Windows. Instead, Apple seems to be chugging along the CPU native path and providing bridges between Objective-C and various scripting languages. This means that each new language they want to support on the operating systems requires work to bridge effectively and that work can only be done well by Apple. If they had gone down the VM path, new languages would be supported once the community built support for them on the VM. Therefore, if I wanted to write my next application for OS X in Scala, I could.

This brought us to a discussion of the merits of Android for mobile development. This is where we had different opinions. I think Android will be great and the development community and support behind it will be much larger than the iPhone. He thought that Apple might release a Java VM for the iPhone if Android picks up steam and believes that the iPhone is still the best platform for mobile.

The reason I like the idea of Android (although the G1 might be a crappy implementation without the ease of use and cool features of the iPhone) because it was fundamentally VM based. This meant that the platform was based on Java-byte code rather than the CPU instruction set. It also meant that the platform had modern concepts like garbage collection, memory management, ClassLoading and all the other good parts of the VM. But it also meant that it will be capable of running any Java-byte code regardless of the language it is written in. Therefore, I could write my Android application in Groovy and with the help of a few extra JARs, get it running just as well as plain old Java.

(Update 10/22/08 Groovy doesn’t quite work yet, because of a number of additional classes and JARs it uses. However, the principle is sound because Android is VM based. JRuby might be a better solution than Groovy depending.)

The problem with the iPhone is that it is tied directly to the CPU instruction set and the Objective-C language. This means you take a step backwards 20 years and now have all the overhead of header files, pre-compilers, directives, manual memory management, no-namespacing, and numerous other headaches that VMs have done exceedingly well in fixing.

What I wonder is why Microsoft is the only operating system company to figure out that wiring a VM, capable of everything we except out of a modern VM plus application isolation, into the lowest levels of the OS is a great idea. In fact, this idea is fantastic. This means applications targeted to the VM improve as the VM improves, without any code changes.

The Java VM still has a long way to go to catch up with the CLR as a full platform. JSR 121 was finalized a few years back, which will provide good application isolation within a single Java VM instance. This means that you can start up the Java VM and then run multiple applications inside it without any conflicts or concerns about one application impacting the others (such as an application calling System.exit or running out of memory). The Java VM still needs extremely tight integration with the operating system and needs to be started when the OS boots. It also needs new ways to start applications and manage applications, which JSRs 277 and 294 should help with. I think though that Google has done all of this work for Android. It runs multiple applications in isolation well, it provides a mechanism for applications to start and stop and the Java VM is completely integrated into the operating system of the mobile device.

I think this is the way of the future. I’m just trying to figure out where Apple is going and why they are heading that way.

I might be smokin’ crack, but I think that todays (September 30th, 2008) Java update from Apple finally fixed the command-tab issue. I haven’t verified it with different apps yet, but the ones I use regularly seem to be working as expected when you command-tab to them.

I have recently gone through the exercise of restoring a medium sized SubVersion repository from an older backup and wanted to share my experience with everyone. First, the problem:

The problem

After you restore the older backup, if any work was performed between the last backup and when the repository crashed, the repository will be “older” than what developers have on their boxes. Here is an example:

• Let’s say you have a backup from Monday evening
• The repository crashed sometime Tuesday
• There was work done during the day Monday

Now, let’s say there is a project A that looks like this:

• The last change to the project was at revision 100 just before the repository crashed
• The last change to the project from the backup was 80
• Frank’s computer contains a checkout of the project at revision 85
• Mary’s computer contains a checkout of the project at revision 100

This means that both Frank and Mary’s computers contain newer code than the repository, but not the latest code. Mary’s computer contains newer code than the repository, but might not be a complete snapshot of what version 100 was before the repository was corrupt. The reason why Mary’s computer might not be 100% correct is that Mary might have committed files to the repository but not performed a “svn update” prior to committing.

SubVersion is like CVS in that each file contains a version number. So, you might have a local checkout that contains version 100 for one file and 90 of another file. Therefore, you might be missing an updated version of the file from revision 93 when it was checked in.

Okay, now onto the fix:

The fix

Each developer’s computer must be analyzed before anything new is put into the repository. You must have a complete picture of the entire company, otherwise you might miss some changes. These changes can be merged in by hand from each developers machine, but this could be error prone and lengthy process. It is usually better to script out as much as possible.

Step 1

In order to determine the local “revision” of a project on a developers computer, you will need to look at each file in the checkout. You can run an ’svn stat’ on each file to determine the version number of that file. Write a script to output a file like this for each local checked out project on all developers machines:

src/java/foo.java: 90

The first part is the file and the second part is the revision of that file.

Next, once you have the complete list of revisions for all projects on all developer’s computers in the entire company, you can compare each file with the current revision in the restored repository to determine if the developer has a later version of a file than the repository. This should ignore all files that the developer has modified locally, but not committed.

This comparison will look like this:

Restored repository
src/java/foo.java: 80

Mary's checkout
src/java/foo.java: 90

Mary has a later version of foo.java!

You should script all of these comparisons out. If a developer doesn’t have later revisions than the repository or any locally modified files, they can safely take these steps:

1. Make a backup of the local checkout
2. Delete the local checkout
3. Re-checkout the project
4. Don’t do anything until the restore is complete

Step 2

The next step is to make a list of the revision that each file in the project was lasted changed on in the restored repository. This report will look like this:

build.xml: 10
src/java/main/foo.java: 16293
src/conf/main/logging.properties: 12093

Next, for each project, collect all of the revision reports from the previous step into a single location. These reports look like this:

frank-project-a-report.txt
build.xml: 10
src/java/main/foo.java: 16301
src/conf/main/logging.properties: 16405

mary-project-a-report.txt
build.xml: 10
src/java/main/foo.java: 16410
src/conf/main/logging.properties: 16321

Combine each of the developer reports into a global report that gives you the revision number for each file on every developers computer. Next, use this global developer report to determine whose computer contains the latest version for each file in the project. Based on the examples above, you can see that build.xml didn’t change recently enough to matter since the version in the repository is the same as the version on Frank’s and Mary’s computers. However, Frank made the most recent commit in revision 16405 to logging.properties and Mary made the most recent commit in revision 16410 to foo.java. Both of these files are more current that the repository and therefore need to be re-committed.

Step 3

Finally, setup a staging area that contains a copy of every checked out project from every developers computer who has a later revision than the repository (from step 1). This will be pretty large, but necessary. Based on the results from step 2, copy the latest version of each file over to a clean checkout of the project from the restored repository. Once you have all of the changes copied over for a single project, commit all of those files for that project back to TRUNK.

You should now have a fully restored repository based on the files from various developers computers.