Maven Central

We started by attempting to use Maven Central for our dependencies, but quickly realized that Maven Central is a complete mess. We also realized that it suffers from the “most surprise” pattern as well as “GPL” infestation. During our transition, we started to realize that using Maven Central actually caused our number of JARs to dramatically increase and for a couple of GPL JARs to sneak into our classpath.

In the end, we had to rever to managing our own repository and manage all of the ivy.xml dependency files by hand to ensure that everything was correct.

Transitive Mess

Gradle appears to suffer from the “let’s make it simple” methodology of Maven. In order to actually make life simple and avoid actual work, Maven pulls in the entire universe of your project dependencies for compiling and runtime. This means that even if you never import a class from a transitive dependency, Maven puts it in the classpath. This allows you to use classes from transitive dependencies. If you do this though, you’ll never actually know what JAR file a class comes from since it could come from some deeply nested dependency. It also tightly couples your project to your dependencies and those dependencies transitives. We prefer loose coupling rather than tight, so this is generally bad. We also like fewer dependencies at compile time rather than more. Again, bad news all around.

Flat Structure

Gradle plugins and domain are entirely flat. This makes it difficult to separate concerns and distinguish between Java compiling, testing and runtime. Maven actually got this right by cleaning separating concerns and separating plugins.

Configuration vs. tasks

Gradle build files can quickly become large and complex. It is often hard to distinguish between what is configuration and what are tasks.

Order matters

Order matters in Gradle build files unless you always use the String syntax for all configurations and tasks. If you have one task that depends on another task, the order of declaration matters. This is quite painful to debug and I thought we had finally grown out of single pass compiling. Apparently not.

No Versions

There are many pieces of Gradle that don’t like versions, e.g. plugins. These don’t have versions and just use the latest available. This is usually a bad idea because historical builds can break and you’ll never know what version you’ll be using when you execute a build.

MD5 and SHA1?

Why both?

Hashes not always used

The MD5 and SHA1 hashes aren’t always used. For example, the ivy.xml files don’t appear to use the hashes associated with that file. It really should use the hashes for everything.

UploadArchives doesn’t run the tests

WHAT?!?!?

Summary

Overall, I’m not a big fan of Gradle. It seems to be overly complex and sorta punts on dependency management. Having used Maven, Savant, Ant and Ivy, I would say that Savant 2.0 is still the best build system out there, even though it still isn’t finished.

Please join us for the discussion if you have ideas you want to share.

Here are the details:

10am PDT October 21
##java-modularity on irc.freenode.net

Here is the rough HTML:

<div class="main-nav">
  <ul>
    <li class="level-1">
      <a href="...">Menu 1</a>
      <ul>
        <li><a href="...">Sub Menu 1.1</a></li>
        <li><a href="...">Sub Menu 1.2</a></li>
      </ul>
    </li>
    <li class="level-1">
      <a href="...">Menu 2</a>
      <ul>
        <li><a href="...">Sub Menu 2.1</a></li>
        <li><a href="...">Sub Menu 2.2</a></li>
      </ul>
    </li>
  </ul>

The CSS that was failing looked something like this:

.main-nav {
  background: #e5e6ec; /* Old browsers */
  background: -moz-linear-gradient(top, #ffffff 0%, #e5e6ec 100%);  /* FF3.6+ */
  background: -webkit-gradient(linear, left top, left bottom, color-stop(0%, #fffff), color-stop(100%, #e5e6ec));  /* Chrome,Safari4+ */
  background: -webkit-linear-gradient(top, #ffffff 0%, #e5e6ec 100%);  /* Chrome10+,Safari5.1+ */
  background: -o-linear-gradient(top, #ffffff 0%, #e5e6ec 100%);  /* Opera11.10+ */
  background: -ms-linear-gradient(top, #ffffff 0%, #e5e6ec 100%); /* IE10+; */
  background: linear-gradient(top, #ffffff 0%, #e5e6ec 100%); /* W3C; */
  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr = '#ffff', endColorstr = '#e5e6ec', GradientType = 0); /* IE6-9 */
}

.main-nav ul {
  padding: 4px 35px 2px 35px;
  vertical-align: top;
}

.main-nav ul li.level-1 {
  display: inline-block;
  position: relative;
}

.main-nav ul li.level-1 a {
  display: inline-block;
  padding: 5px 20px 5px 0;
}

.main-nav ul li ul {
  background: #e5e6ec;
  display: none;
  padding: 0 15px 10px 15px;
  position: absolute;
  top: 29px;
}

The issue with IE 8 (and probably IE 9) is that the gradient definition of:

filter: progid:DXImageTransform.Microsoft.gradient(startColorstr = '#ffff', endColorstr = '#e5e6ec', GradientType = 0);

caused the drop-down menus to not show up correctly. After moving them around and using some visibility magic, I found that the drop-downs were actually getting clipped inside the outer UL element. Once I removed the filter it all worked fine.

The main issues with both these databases is that the lack the ability to store values as UTC without any timezone information. That’s right, I don’t want the database to know anything about timezones. See, once the database starts to store timezone information or worse, store dates and times without timezone information but not in UTC, you get into the situation where everything melts down if the timezone of the operating system or the database server changes. You also get into really nasty issues when you migrate data across locations, for example by exporting data from a server in San Francisco and migrating it to a server in London.

If you think about it, databases are primarily used for storing and retrieving data. In 99.9999% of applications, the database is NOT responsible for displaying data. For this reason, the database should not know anything about timezones because these are purely for display purposes. The correct way to store date and time data is at UTC.

So, what’s my solution given the horrible date and time support in databases? Don’t use them and use bigint instead. Store everything using the number of milliseconds since epoch in UTC. Never ever do any timezone math with your dates and times. Handle everything in UTC and only do timezone manipulations just before you display the data to a user.

I pulled this off by changing all of my datetime and timestamp columns to be bigint instead. They now look like this:

create table foo (
  insert_instant bigint not null,
  update_instant bigint not null
);

Additionally, in my application code when I use JDBC I pull the data out as Longs like this:

// using Joda here - if you don't use Joda you should
ResultSet rs = ...;
long insertInstant = rs.getLong(1);
DateTime dt = new DateTime(insertInstant);

Lastly, I wrote a Hibernate UserType that converts from longs to Joda DateTime instances and annotate my entities like this:

@Entity
public class Foo {
  @Column(name = "insert_instant", updatable = false)
  @Type(type = "com.inversoft.cleanspeak.api.domain.hibernate.DateTimeType")
  public DateTime insertInstant;
}

And that’s it. It works perfectly and I never have to worry about any of my data getting screwed up if someone changes the system timezone or migrates data from one server to another. Furthermore, I finally get full support for milliseconds on MySQL and I also can add support to Clean Speak for new databases easily.

• One that stores the date-time without any time zone information
• One that stores the date-time with time zone information

For Postgresql, these are:

• timestamp without time zone
• timestamp with time zone

respectively.

For MySQL, these are:

• datetime
• timestamp

There are a number of things to consider when dealing with date-time values:

• What if the timezone of the server changes?
• What if the server moves physical locations thereby indicating a new time zone?
• What if the timezone of the database server changes (different than the timezone of the server)?
• How does the JDBC driver handle timezones?
• How does the database handle timezones?

To figure all of this out, it is important to understand how the date-time value goes from the application, to the database, out of the database and back to the application (for inserts and later selects). Here is how this works for values without timezone information:

Insert

1. Java creates a java.sql.Timestamp instance. This is stored as UTC
2. Java sends this value to the JDBC driver in UTC
3. The JDBC driver sends the value to the database in UTC
4. The database converts the date-time from UTC to its current timezone setting
5. The database inserts the value into the column in the current timezone (NOT UTC)

Select

1. The database selects the value from the column in the system timezone
2. The database converts the value to UTC
3. The database sends the UTC value to the JDBC driver
4. The JDBC driver creates a new java.sql.Timestamp instance with the UTC value

For this scenario, you will run into major issues if the the server or database timezone changes between Insert #5 and Select #1. In this case, the value will not be correct. The only way to make things work correctly for this setup is to ensure no timezone settings ever change or to set all timezones to UTC for everything.

For the types that store timezone information, here is how the data is passed around:

Insert

1. Java creates a java.sql.Timestamp instance. This is stored as UTC
2. Java sends this value to the JDBC driver in UTC
3. The JDBC driver sends the value to the database in UTC
4. The database calculates the offset between the current timezone and UTC
5. The database inserts the value as UTC into the column along with the offset it calculated

Select

1. The database selects the value from the column in UTC along with the offset
2. The database sends the UTC value to the JDBC driver
3. The JDBC driver creates a new java.sql.Timestamp instance with the UTC value

Since the JDBC driver only handles UTC values, there is no potential for data mangling here since everything is stored in UTC inside the database. Although the database stores the timezone information along with the date-time value in UTC, it isn’t ever used because the JDBC driver doesn’t care what the original timezone was and ignores that value.

The Clean Speak profanity filter handles between 5,000 and 10,000 messages per second. The profanity filter doesn’t hit the database, but the Clean Speak Moderator tool does.

The latest tests of the Moderation tool put it around 1,000-2,000 messages per second.

So, if we use Oracle’s time granularity of 2 minutes, Clean Speak handles anywhere from 60,000-600,000 messages. Perhaps Oracle needs to work on optimizing their OTLP test system a bit.

public class JSONEscape implements TemplateMethodModelEx {
  @Override
  public Object exec(List arguments) throws TemplateModelException {
    if (arguments.size() != 1) {
      throw new TemplateModelException("jsonescape takes a single parameter");
    }

    return JSONBuilder.escape(arguments.get(0).toString());
  }
}

The JSONBuilder is a class that I wrote for creating JSON using the builder pattern. However, the escape method looks like this:

  public static String escape(String str) {
    StringBuilder build = new StringBuilder();
    escape(str, build);
    return build.toString();
  }

  public static void escape(String str, StringBuilder build) {
    char[] ca = str.toCharArray();
    for (char c : ca) {
      switch (c) {
        case '"':
        case '\\':
          build.append('\\');
          build.append(c);
          break;
        case '\r':
          build.append("\\r");
          break;
        case '\n':
          build.append("\\n");
          break;
        case '\t':
          build.append("\\t");
          break;
        case '\b':
          build.append("\\b");
          break;
        case '\f':
          build.append("\\f");
          break;
        case '/':
          build.append("\\/");
          break;
        default:
          if (c < = 0x1F) {
            unicodeEscape(c, build);
          } else {
            build.append(c);
          }
      }
    }
  }

  public static void unicodeEscape(int ch, StringBuilder build) {
    build.append('\\');
    build.append('u');
    build.append(HEX_CHARS[ch >>> 12]);
    build.append(HEX_CHARS[(ch >>> 8) & 0xf]);
    build.append(HEX_CHARS[(ch >>> 4) & 0xf]);
    build.append(HEX_CHARS[ch & 0xf]);
  }

Then I add this method to my model and render the template:

model.put("jsonescape", new JSONEscape());

My templates look like this:

{
  "foo": "${jsonescape(some.value.here)}"
}

------_=_FSter_001_3251309519900466--


I wasn’t sure whether or not JDBC batch or using a single insert statement that inserted multiple rows would be faster. Therefore, I setup a little test to see which was going to work better. First some code:

Here is the code for a single insert statement that inserts multiple rows.

long start = System.currentTimeMillis();
StringBuilder build = new StringBuilder("insert into insert_values (foo, bar, baz) values ");
for (int i = 0; i < 1000; i++) {
  build.append("(?, ?, ?)");
  if (i < 999) {
    build.append(", ");
  } else {
    build.append(";");
  }
}

Connection c = DriverManager.getConnection("jdbc:mysql://localhost:3306/batch_vs_insert_test", "dev", "dev");
c.setAutoCommit(false);
PreparedStatement ps = c.prepareStatement(build.toString());
for (int i = 0; i < 3000; i++) {
  ps.setString(i + 1, "value" + i);
}

int result = ps.executeUpdate();
c.commit();

long end = System.currentTimeMillis();
System.out.println("Insert time was " + (end - start));

ps.close();
c.close();

Here is the code for the JDBC batch:

long start = System.currentTimeMillis();
Connection c = DriverManager.getConnection("jdbc:mysql://localhost:3306/batch_vs_insert_test", "dev", "dev");
c.setAutoCommit(false);
PreparedStatement ps = c.prepareStatement("insert into insert_values (foo, bar, baz) values (?, ?, ?);");
for (int i = 0; i < 3000; i++) {
  ps.setString((i % 3) + 1, "value" + i);
  if ((i + 1) % 3 == 0) {
    ps.addBatch();
  }
}

int[] results = ps.executeBatch();
c.commit();

long end = System.currentTimeMillis();
System.out.println("Batch time was " + (end - start));

ps.close();
c.close();

The average time over 10 iterations where roughly as follows:

JDBC Batch: ~100 milliseconds
Single Insert: ~10 milliseconds

It looks like the single insert statement wins by a large margin.

Here’s my break down:

• CPU (i7 2600K 3.4GHz Sandybridge) $320
• Motherboard (ASUS P8P67) $185
• RAM (8GB DDR3 1333MHz) $83
• Hard Drive (WD 1TB 7,200 RPM SATA 6G) $85
• Video (ATI Radeon 6970M) $300
• Keyboard/Mouse (MS wireless) $80
• Monitor (Dell Ultrasharp 27″ 2560×1440) $1000
• Webcam (Any) $50
• Speakers (Any) $15

The main expense here is definitely the CPU and the monitor. However, for equal comparisons I wanted to add a monitor that was comparable in quality, display, depth, etc. You could obviously go with a cheaper monitor from an off-brand, but I think that’s not a fair comparison.

Here’s the results:

• Apple iMac 27″ with 3.4 GHz CPU and 8GB RAM factory installed: $2,199
• BYO with above specs: $2,118

The difference is about $81.

I poked around a bit at some desktop solutions as well, and even an entry level desktop with the i7 2600K and the Dell ultrasharp monitor is going to start around $2,000.

It looks like my notion about the value of a Mac is accurate. Furthermore, with the fact that the iMac is a single piece of equipment that includes the computer and the monitor and you get what I still feel is the worlds best operating system plus a bunch of free software, it is hard to argue that an iMac isn’t a great value.