Simple math in Unix

I was trying to get some basic stats from our Apache access logs today. For each request we log how long it took to serve the request. The obvious choice would be to download the logs locally and try to load it in Excel to calculate, say, average request duration. But there is a simpler and faster way to do this directly on the server. ’awk’ to the rescue!. You can simply do:

grep “/path/request” access.log | awk ‘BEGIN{k=0;s=0}{k++; s=s+$14}END{print s/k}’

The ’BEGIN’ block will initialize variables. ’k’ for the number of rows found and ’s’ for the sum of requests duration.

The main block adds field ’$14’ to the sum (remember awk will split input lines on spaces by default -you can change that-. Choose the field you are interested in accordingly).

Finally the ’END’ block will print the average.

Easy a pie!


#Scala Starting points

Recently, I’ve been building a lightweight API in Scala that sits on top of our existing Java back-end. It was a good opportunity to dive into the world of Scala but there were a lot of learning pains along the way. Here are some useful tips if you want to get started with Scala (quick note, I’ve been using using Scala v2.9.2 and SBT v.0.12.0)

  1. Take some time to get familiar with the Scala syntax and concepts. www.scala-lang.org provides good resource links depending on what background you are coming from. www.simplyscala.com provides an interactive Scala console for compiling and executing your code snippets, and offers good starting tutorials. You should also take some time to learn how to use the Scala interpreter locally.
  2. Get familiar with Simple Build Tool (SBT). Almost every Scala project / framework is built using SBT, so you need to learn how it works. Luckily, if you have experience with Apache Maven (a Java build tool), then you will see a lot of similarities in the project directory structure that SBT expects. Additionally, SBT utilizes Apache Ivy for its dependency management, which is quite popular and used in other frameworks such as Grails
  3. One more note about SBT, it has great plugin support. For starters, it integrates pretty tightly with Jetty and allows you to launch a Jetty web server from within your project’s SBT console. Provided you have a valid Ivy dependency for Jetty  (e.g. “org.eclipse.jetty” %  “jetty-webapp” % 8.1.0.v20120127 % “container”) and include it in your project’s dependencies, you can launch the Jetty server from your SBT console by typing “container:start” and navigate to localhost:8080 to test your app. If you make code changes, you can either manually reload the jetty server by typing “container:reload /” or have the it reload automatically on code changes by typing “~container:reload /

    If you use Eclipse, there’s also a plugin to generate the necessary Eclipse classpath/project files from your project’s SBT settings (see https://github.com/typesafehub/sbteclipse/wiki)

  4. If you are using Scala to consume Java services (or vice-versa), then you need to get familiar with the functionality provided in scala.collection.JavaConversions for converting between Java collections/maps/etc. and Scala collections/maps/etc. This is because …
  5. By default in Scala, if you are using Seq, List, Map, etc. then you are using the immutable implementations of those interfaces. If you try to add/insert an item into these objects, then Scala will barf. Instead, you have to use syntax like (myList  ::= “hey”), which means (1) make a copy of myList, (2) append “hey” to the copy, and (3) then assign the copy to the myList variable (actually, that doesn’t seem to be 100% correct, I guess Scala optimizes to avoid copying the whole data structure - http://stackoverflow.com/questions/10141654…) . Scala does provide some mutable implementations of these interfaces, but it’s unclear to me when you are supposed to use them. Based on some reading, it seems that immutable collections don’t suffer performance penalties when having to copy/add-element(s) due to optimizations and increased efficiency of the GC.
  6. Options - One of the weird things you might see in Scala is an Option, but it’s actually really simple to understand. An Option is just an interface that only has two implementations, Some(Any) or None. Some is just a wrapper around Any* object, where Any is the root of the Scala object hierarchy. And you can use Scala’s case matching to deal with an Option:
    val name: Option[String] = Some("Bryan") // optionally, you could assign None
    name match {
        case Some(value) => { println "Hi " + value }
        case None => { println "No name provided" }
    }
  7. Either - Another weird concept in Scala is Either. The Either object encapsulates either (lolz) a Right object or a Left Object. Convention dictates that Left is used for failure and Right is used for success. What’s funny about that (at least to me) is at a Traackr Hack Day, one of our friends who does a lot of Node.js programming was telling us to construct all our callback methods like “function handler(error, response) {…}”, which I assume is a Node.js convention. Kind of similar, right? The first argument (i.e. the Left) is the error and the second argument (i.e. the Right) is the success response.

    This either code snippet was taken from http://www.scala-lang.org/api/current/scala/Either.html

    val in = Console.readLine("Type Either a string or an Int: ")
     val result: Either[String,Int] = try {
         Right(in.toInt)
       } catch {
         case e: Exception =>
           Left(in)
     }

     println( result match {
       case Right(x) => "You passed me the Int: " + x + ", which I will increment. " + x + " + 1 = " + (x+1)
       case Left(x) => "You passed me the String: " + x
     })
  8. In Scala, for dealing with any level of concurrent, asynchronous work, definitely utilize Akka (http://akka.io/). Akka utilizes an actor system for distributing work (either on the same JVM or remotely), and let’s you define dispatchers for handling thread pool configurations, throughput concerns, etc. for one or more actor systems. It is really quite powerful and simple to work with, so definitely check it out

Oh Smarty, you mock me

{if false}
hi
{else if false}
bye
{/if}
results in “bye”. Do you want to know why? Because Smarty only supports “elseif”, not “else if”. It treats that as an “else”. No warning, no error. Just “else”. Thanks, Smarty.