Simple math in Unix

I was trying to get some basic stats from our Apache access logs today. For each request we log how long it took to serve the request. The obvious choice would be to download the logs locally and try to load it in Excel to calculate, say, average request duration. But there is a simpler and faster way to do this directly on the server. ’awk’ to the rescue!. You can simply do:

grep “/path/request” access.log | awk ‘BEGIN{k=0;s=0}{k++; s=s+$14}END{print s/k}’

The ’BEGIN’ block will initialize variables. ’k’ for the number of rows found and ’s’ for the sum of requests duration.

The main block adds field ’$14’ to the sum (remember awk will split input lines on spaces by default -you can change that-. Choose the field you are interested in accordingly).

Finally the ’END’ block will print the average.

Easy a pie!

#Scala Starting points

Recently, I’ve been building a lightweight API in Scala that sits on top of our existing Java back-end. It was a good opportunity to dive into the world of Scala but there were a lot of learning pains along the way. Here are some useful tips if you want to get started with Scala (quick note, I’ve been using using Scala v2.9.2 and SBT v.0.12.0)

  1. Take some time to get familiar with the Scala syntax and concepts. provides good resource links depending on what background you are coming from. provides an interactive Scala console for compiling and executing your code snippets, and offers good starting tutorials. You should also take some time to learn how to use the Scala interpreter locally.
  2. Get familiar with Simple Build Tool (SBT). Almost every Scala project / framework is built using SBT, so you need to learn how it works. Luckily, if you have experience with Apache Maven (a Java build tool), then you will see a lot of similarities in the project directory structure that SBT expects. Additionally, SBT utilizes Apache Ivy for its dependency management, which is quite popular and used in other frameworks such as Grails
  3. One more note about SBT, it has great plugin support. For starters, it integrates pretty tightly with Jetty and allows you to launch a Jetty web server from within your project’s SBT console. Provided you have a valid Ivy dependency for Jetty  (e.g. “org.eclipse.jetty” %  “jetty-webapp” % 8.1.0.v20120127 % “container”) and include it in your project’s dependencies, you can launch the Jetty server from your SBT console by typing “container:start” and navigate to localhost:8080 to test your app. If you make code changes, you can either manually reload the jetty server by typing “container:reload /” or have the it reload automatically on code changes by typing “~container:reload /

    If you use Eclipse, there’s also a plugin to generate the necessary Eclipse classpath/project files from your project’s SBT settings (see

  4. If you are using Scala to consume Java services (or vice-versa), then you need to get familiar with the functionality provided in scala.collection.JavaConversions for converting between Java collections/maps/etc. and Scala collections/maps/etc. This is because …
  5. By default in Scala, if you are using Seq, List, Map, etc. then you are using the immutable implementations of those interfaces. If you try to add/insert an item into these objects, then Scala will barf. Instead, you have to use syntax like (myList  ::= “hey”), which means (1) make a copy of myList, (2) append “hey” to the copy, and (3) then assign the copy to the myList variable (actually, that doesn’t seem to be 100% correct, I guess Scala optimizes to avoid copying the whole data structure -…) . Scala does provide some mutable implementations of these interfaces, but it’s unclear to me when you are supposed to use them. Based on some reading, it seems that immutable collections don’t suffer performance penalties when having to copy/add-element(s) due to optimizations and increased efficiency of the GC.
  6. Options - One of the weird things you might see in Scala is an Option, but it’s actually really simple to understand. An Option is just an interface that only has two implementations, Some(Any) or None. Some is just a wrapper around Any* object, where Any is the root of the Scala object hierarchy. And you can use Scala’s case matching to deal with an Option:
    val name: Option[String] = Some("Bryan") // optionally, you could assign None
    name match {
        case Some(value) => { println "Hi " + value }
        case None => { println "No name provided" }
  7. Either - Another weird concept in Scala is Either. The Either object encapsulates either (lolz) a Right object or a Left Object. Convention dictates that Left is used for failure and Right is used for success. What’s funny about that (at least to me) is at a Traackr Hack Day, one of our friends who does a lot of Node.js programming was telling us to construct all our callback methods like “function handler(error, response) {…}”, which I assume is a Node.js convention. Kind of similar, right? The first argument (i.e. the Left) is the error and the second argument (i.e. the Right) is the success response.

    This either code snippet was taken from

    val in = Console.readLine("Type Either a string or an Int: ")
     val result: Either[String,Int] = try {
       } catch {
         case e: Exception =>

     println( result match {
       case Right(x) => "You passed me the Int: " + x + ", which I will increment. " + x + " + 1 = " + (x+1)
       case Left(x) => "You passed me the String: " + x
  8. In Scala, for dealing with any level of concurrent, asynchronous work, definitely utilize Akka ( Akka utilizes an actor system for distributing work (either on the same JVM or remotely), and let’s you define dispatchers for handling thread pool configurations, throughput concerns, etc. for one or more actor systems. It is really quite powerful and simple to work with, so definitely check it out

Oh Smarty, you mock me

{if false}
{else if false}
results in “bye”. Do you want to know why? Because Smarty only supports “elseif”, not “else if”. It treats that as an “else”. No warning, no error. Just “else”. Thanks, Smarty.

#Java #Programming Tidbits

Observations from the world of Traackr programming

  1. Using Apache Commons CLI - Very helpful library for parsing command line arguments passed to your script. You are able to construct an org.apache.commons.cli.Options object and add an option for each acceptable argument to your script (the argument name, whether it accepts a value, a user-friendly description, etc.)
    options.addOption("h", true, "required: server host name:port");

    You can parse the arguments passed to your script into a org.apache.commons.cli.CommandLine object, and quickly validate it against your predefined options. The library can also nicely format your Options object into a user-friendly system message. #propz to @gstathis for finding this!

  2. Using org.slf4j.profiler.Profiler - Nifty little class for performing time profiling in your scripts, and being able to chain or nest different Stopwatches together. Again, #propz to @gstathis for this find.
  3. Java’s jvisualvm Plugins - Pretty cool that jvisualvm offers additional plug-ins to supplement its profiling capabilities. One in particular that came in really handy was the Threads Inspector plugin. Recently, one of our long-standing processes stopped reporting anything in our logs, and it was unclear whether it had failed because no exceptions were thrown. The Threads Inspector plugin allowed us to select one of our managed threads for this process and we could see that if was stuck waiting indefinitely on a CountDownLatch! A recent change had caused one of the child processes to die unexpectedly and the countdown latch was never properly updated (plus, we mistakenly never configured a time-out when waiting on this particular lock). #propz to the random Java developer who created this plugin, wherever you are…

Twitter Engineering: Caching with Twemcache

Twitter Engineering: Caching with Twemcache

Conversion Optimization

Conversion Optimization

Backbone.js v.s. Spine.js

Backbone.js v.s. Spine.js

Generic Bash Java Executor

Simple script to compile and execute a Java application with any number of parameters

  • e.g. ./
  • e.g. ./ “arg1” “arg2” 33


#Remove old class files (suppress warning for DNE)
rm ${args[0]}*.class 2> /dev/null

echo "Compiling Script ${args[0]}"
$JAVAC -classpath "$CLASSPATH" ${args[0]}.java

for (( c=1; c <= $numOfArgs; c++))
   let pos=$c-1

$JAVA $JAVA_HEAP_MAX -classpath "$CLASSPATH" ${args[0]} ${arr[@]}