#Java #Programming Tidbits

Observations from the world of Traackr programming

  1. Using Apache Commons CLI - Very helpful library for parsing command line arguments passed to your script. You are able to construct an org.apache.commons.cli.Options object and add an option for each acceptable argument to your script (the argument name, whether it accepts a value, a user-friendly description, etc.)
    options.addOption("h", true, "required: server host name:port");

    You can parse the arguments passed to your script into a org.apache.commons.cli.CommandLine object, and quickly validate it against your predefined options. The library can also nicely format your Options object into a user-friendly system message. #propz to @gstathis for finding this!

  2. Using org.slf4j.profiler.Profiler - Nifty little class for performing time profiling in your scripts, and being able to chain or nest different Stopwatches together. Again, #propz to @gstathis for this find.
  3. Java’s jvisualvm Plugins - Pretty cool that jvisualvm offers additional plug-ins to supplement its profiling capabilities. One in particular that came in really handy was the Threads Inspector plugin. Recently, one of our long-standing processes stopped reporting anything in our logs, and it was unclear whether it had failed because no exceptions were thrown. The Threads Inspector plugin allowed us to select one of our managed threads for this process and we could see that if was stuck waiting indefinitely on a CountDownLatch! A recent change had caused one of the child processes to die unexpectedly and the countdown latch was never properly updated (plus, we mistakenly never configured a time-out when waiting on this particular lock). #propz to the random Java developer who created this plugin, wherever you are…

Amazon now offers SSD-backed EC2 instance

Amazon just announced they are now offering SSD-backed instance. This pretty awesome. I have been hearing great feedback on performance from everybody using SSD for IO intensive application. Obviously. We can now finally try it on AWS. Here are the specs they offer for now:

  • 8 virtual cores, clocking in at a total of 35 ECU (EC2 Compute Units).
  • 60.5 GB of RAM.
  • 10 Gigabit Ethernet connectivity
  • 2 TB of local SSD-backed storage

The only downside? It ain’t cheap: $3.10/h


Rajiv's blog: Scaling lessons learned at Dropbox, part 1

Rajiv's blog: Scaling lessons learned at Dropbox, part 1

Twitter Engineering: Caching with Twemcache

Twitter Engineering: Caching with Twemcache

Maven - Separating Unit / Integration Tests

We use Maven as our internal build tool for our Java projects, and most of time it works great. But recently, when trying to separate the execution of our integration tests from our unit tests, I ran into an unexpected hurdle.

Our internal code is a multi-module project, where each module provides its set of unit tests and integration tests for our internal build to execute. If the tests are successful (among other things), then each module is deployed as a snapshot JAR to our internal Nexus repository. Deploying snapshots allows our developers to work on other areas of our project’s code-base without having to build all internal dependencies locally.

Now, we need our internal build to be a) fast and b) reliable as we commit changes frequently throughout the day, which means we want to be building & deploying code frequently throughout the day as well. As you would guess, this is exactly the opposite of what integration tests are: slow (non-millisecond execution times) and unreliable.

While Maven distinguishes between unit tests and integration tests, it has not excluded the latter during its default build cycle. So, when running “mvn clean deploy”, Maven will execute all the build phases underneath it (..compile->test->..->integration-test->..->install->deploy). For us, this means that every time we execute an internal build, each module’s integration tests will be run as a result (no thanks).

So, I wanted to know how I could run our modules’ integration tests separately from our regular build. There were quite a few questions on Stack Overflow around this, but none of the responses were 100% accurate. So, after a bunch of trial-and-error to get Maven to behave how I wanted, I figured I’d share our solution.

Note: Solution assumes that Integration tests conform to the naming convention of *IntegrationTest.java

1) Exclude integration tests from the regular build test phase. Include the following in the build section of your POM:

       <plugin>
         <artifactId>maven-surefire-plugin</artifactId>
         <groupId>org.apache.maven.plugins</groupId>
         <version>2.9</version>
         <configuration>
             <skip>false</skip>
             <useFile>false</useFile>
             <argLine>-Xms1024m -Xmx1024m -XX:MaxPermSize=512m</argLine>
             <excludes>
                 <exclude>**/*IntegrationTest.java</exclude>
             </excludes>
         </configuration>
       </plugin>

Now our regular build on our CI environment can run the standard “mvn clean deploy” fast and efficiently. The only tests that will be run will be unit tests.

2) Create a separate Maven profile that will only execute integration tests (and exclude all unit tests):

<profiles>
    <profile>
        <id>integration-test-builder</id>
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <groupId>org.apache.maven.plugins</groupId>
                    <version>2.9</version>
                    <configuration>
                        <skip>false</skip>
                        <useFile>false</useFile>
                        <argLine>-Xms1024m -Xmx1024m -XX:MaxPermSize=512m</argLine>
                        <excludes>
                            <exclude>none</exclude>
                        </excludes>
                        <includes>
                            <include>**/*IntegrationTest.java</include>
                        </includes>
                    </configuration>
                    <executions>
                        <execution>
                             <id>integration-tests</id>
                              <phase>integration-test</phase>
                            <goals>
                                <goal>test</goal>
                            </goals>
                          </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>   

This profile can be run as such: mvn clean integration-test -P integration-test-builder. We can have a separate build job on our CI environment whose only purpose is to run integration tests once per day using this profile.