Photo by Javier Allegue Barros on Unsplash

Build Better Software By Going Farther Together

TL;DR: Growing up in an immigrant community in the New York Metro area, you never think the unique, random, and crazy experiences you have in such a setting could have a direct impact on your career in tech, until it does. And I've learned many lessons, and here's one of them. If you get out of your own way, you along with your team, will accomplish great things.

Growing up, my family was quite plugged into a faith community that comprised mostly of people who were at the time recent immigrants to the United States from Egypt. We were the beginnings of the Coptic Orthodox community in America. Most of the non-liturgical music generated by the community was geared towards the ears and culture of those who immigrated here. I was part of a different generation, born in the USA, but very much Egyptian. It was very difficult to relate to some of the art and music that had been imported and shared with us.

By the time I got to high school, I had different ambitions than my peers. While most kids were out there being kids, I had felt a deep responsibility to help create art that we could connect with. After a few attempts, my work was often dismissed as dissenting, and non-adherent to our traditions. My team of musicians and I stayed persistent, despite doors (sometimes literally) being closed in our faces. While there was an initial rejection by community leaders, our work was getting recognition. The youth of the New York/New Jersey metro area started to know and enjoy our music.

Making A Record

In college I had a new vision: an album. My hope was that it would be an album that would embody the values and essence of our traditions, while connecting them with the creation of something original that our generation could resonate with. I wanted to send a message that even though a ton of art and music was handed to us, that we could be empowered to become creators of art and music, ourselves!

I found an excellent team of like-minded individuals. We sought funding, and eventually partnered with a local church who liked our idea. They offered to bankroll an album, in exchange for inventory. This did mean however, that I did not have complete creative control over the outcome. (dun.dun.dunnnnn!)

While most of our ideas were welcomed, quite a few were met with concern. I was often asked to hold back, edit, or even omit, for the sake of not rocking the boat too much. I had to make a choice, was I going to "compromise" on our vision, or was I going to trust this collaboration with an outside partner? This partner was older, a lot more experienced, and had a perspective that was a lot broader than my own. He knew intimately the ins-and-outs of our community, across multiple generations. He obviously believed in me enough to work with me, but seemed to restrict what we were trying to create for reasons I couldn't understand at the time.

There were moments I really wanted to lead and just create, yet felt like I had to be a team player, and there were a lot of reluctant compromises.

Unexpected Outcomes

We powered through, and the record was produced. The end result surprised me, and was beyond what I could have imagined: two sold-out printings, and an east coast tour that lasted four years. In 2001, I even got a phone call from office of the Ambassador from Egypt to the United States. On behalf of His Excellency, the office invited us to perform at a gathering of dignitaries and officials from all over the world, at an event honoring the music of Egypt. It was pretty incredible and completely unexpected!

But aside from all this "big stuff" that I'm mentioning, it was the people-impact that mattered most to me. We met and received letters from youth all around the world, with stories about how our work had impacted them, or encouraged others to follow suit and create music of their own.

I am convinced that had it been all up to me, and my direction alone, we would not have had the impact we experienced. We would have had total creative freedom, but we may have never reached such broad audience. I was satisfied, although there was less “getting my way” and more “getting out of my own way”.

Ok, great. You may be reading this, and thinking "what on earth does this have to do with building software?" and my answer is: "Everything."

"Don't Throw The $necessary Out With The $unnecessary"

As a hot-headed 21 year old, although I was committed to achieve something for my community, there was a blurred line between the overall desired outcome and the means to achieving that outcome. Art is highly personal. Making the art is as much of a goal as the outcome. If there was a particular song that was going to be cut from the record, or a direction that needed editing, I didn't always handle it gracefully. I nearly quit the project until my mentor told me, "don't throw the baby out with the bathwater". I had no idea what he was talking about. Once it was explained to me, I thought that was the weirdest metaphor ever.

But if you replace baby and bathwater, with the necessary/unnecessary thing of your choosing, the lesson starts to take shape. I was given a choice, if I stayed in this partnership, would the vision still be achieved? As I looked around, and saw how much freedom we actually had, was I willing to throw the whole project out the window because of 1 or 2 cut songs? I eventually decided to trust, and to refocus my attention on the goal, and preserve the relationship through the details. Had I disengaged, or quit, I would have missed out on something huge. The proverbial baby would have been lost.

That piece of advice kept me in the game, but the actual lesson I needed to learn was one that has to do with the importance of "us" over "me". It's the fact that if I'm working on something with a group of people, it's more important for us to be aligned than for each of us to do things their own way.

But, I'm an Artist and I Cannot Be Stifled!

They say that building software is an art. And there are a ton of similarities between those who make art, and those who write software. Unlike the building of a car or a house, where there are clear specifications for how each part comes together, building software retains the imprint of the developer. Even with strict team standards, the developer's personal style finds its way into the code. By looking at a piece of code, I can usually tell who on my team wrote it. It's why we say that we "write" software as opposed to "assemble" or "manufacture" it.

Unless you're one of the unicorns of 1-2 person teams, who get acquired by multi-billion dollar corporations, it usually requires more than 1 or 2 engineers to create something that can see the light of day. Our VP of Engineering has a saying, "Software is a people endeavor". We build software together. "Together" would mean, a group of people who have spent various durations of time (from months to decades) perfecting their craft, each with their own sense of "the right way" to do something.

When Being Skilled Isn't Enough

Looking back at the production of that record, yes I had the musical expertise, but our partner had a much deeper understanding of the community we were serving. As a musician who was mostly American, I wanted the album to be much more progressive than it was shaping up to be. I wanted to take more creative risks, but the brakes kept getting put on. I wasn't taking into consideration what my mentor was prioritizing, and that was the needs of the "customer". The community I was making music for wasn't necessarily interested in the risks I was trying to make nor would it have benefitted them much. He reminded me of the needs of our audience, and his user-centered perspective helped pave a way for this new thing to take root and land on listening ears. Luckily, I had wonderful, and much-wiser team members who reminded me of this.

And we find similar predicaments in engineering teams today. We want to build on the latest and greatest tech. We want to take technical risks or work on things because it is a more cutting-edge way of doing things. Maybe we want to upgrade a major version of a library that would cost several weeks of development on our roadmap. While these may be great ideas in isolation, we may not always be able to tie these changes to features that directly benefit the business. Sometimes, we as engineers need to be edited to keep our work focused on the goals we are trying to achieve.

So yes, I can now appreciate the lessons learned by these fierce negotiations of my past. I see now, that we were able to create something that was familiar enough to be mostly* accepted, however different enough to challenge, inspire, and spark conversations among communities in our diaspora. Smart and talented people can accomplish some amazing things, but only if they're aligned. Make no mistake, getting alignment is challenging.

* Actually, we got banned in one of the dioceses. I received a letter from a well-respected Bishop, telling me that our music was not allowed in any of the churches in his geography. While that may seem like a setback, at the time it reminded me that we didn't keep it too safe. 

Letting Go: Side-Effects May Include...

Getting on the same page as a group, requires individuals to give up a degree of control. This is required when building software as a team. Usually, letting-go is usually met with the acceptance that comes with being a professional. But software engineering often attracts people who put so much of themselves into their work. Because of this, letting-go can be met in with the following emotional responses:

  • Frustration: Often times, including myself, I've witnessed engineers be frustrated when a particular course of action, or even a pull request, is not approved, or requires changes that would move the outcome in a completely different direction. You think to yourself, "would the Sistine Chapel been what it was today had Michaelangelo been ruled by a committee?" Righteous indignation takes over, or maybe it's the blow against one's ego that can happen when work is challenged in its current state. Been there? I know I have.
  • Apathy: Tables aren't flipped, but hands are up in the air. (And I don't mean this celebratory emoji 🙌.) Apathy leads to detaching from both the work and the goal. While the impact of this is not immediate as the previous item, it does make teams vulnerable to morale being slowly chipped away. This will have long-term and debilitating effects.
  • Acceptance: There are others, however, who can remain detached enough from their work, but see it as part of a collective, and will welcome changes and advice, because ultimately, there's a shared trust in the team, and a strong commitment to what the team is trying to achieve.

But Don't Follow Blindly

We have to be aligned to make great things happen. And alignment means letting go. That's not to say that blood doesn't get spilled, or tears won't flow. That should happen with a team of experienced individuals, however, there's a mutual respect and striving for what's best, collectively. And this sort of refinement by an engineer and their peers, can lead to some great outcomes. (I'm not advocating for decisions by consensus, but that's another blog entry.)

As engineers who work on teams, we have to constantly manage an important balance. It's one between what each of us brings based on our individual experiences, convictions, and baggage, with the roles we're assigned, with the goals of the organizations we work for. Now there's absolutely a place to draw some hard lines, and offer non-negotiables, when you see a particular course of action is going to put the big goal at risk. Those should be rare occurrences. Be sure you understand the difference between risks to the goal, vs. risks to the way you want to do things.

But There Is Hope: Some Helpful Tips

Having trouble letting-go? Like my experience with making music, and my experiences in the present: here are some strategies that help me do just that:

  • Focus on the goal: The shared goal you and your team have, should be one that you really believe in. If you're not on board with the goal, you may want to reconsider your employment situation. But let's assume you're still on board with what your team is trying to achieve. Having a larger goal that drives you is extremely important for satisfaction in one's career. That goal has to sit a step beyond how you write code. Commit to a goal and it will  help you entertain other possibilities of achieving it. This will make it possible to let go and try things a different way.
  • Make it about the work: Don't take things personally! It's not about you, and most of the time, your team isn't focused on you, it's about the work. By having the discipline to not take things personally, you allow your team to challenge you, and then it builds trust that allows you to challenge your team. Because collectively you care about the same thing, the work.
  • Get a hobby: Ok, so your team has a norm of doing very strict test-driven development. (I've been on such a team, before.) The engineering lead wants to see the tests written out before a single line of code is written. What a drag, right? You love building software by running and gunning it. So do that! Just don't do it at work. By having interests and outlets outside of what you're doing at work, allows you to get go of things that may be very personal to how you work, because you have other areas in life where you get to do these things. You can let go of the small stuff, so you and your team can work better together to achieve the big stuff!

Parting Words

Decades later, I barely remember the things I argued about while making that album. I value the music we made and the things we achieved so much more than what we had to lose. I tell this to all my fellow engineers out there who find themselves sometimes frustrated.  In the constant negotiation and struggle, we hope to make each other become better engineers and help refine our individual an collective crafts. By staying committed to a bigger picture, we give ourselves a better chance to achieving the things we want to. And it goes back to an old proverb that has come across my path time and time again: alone, we can move quickly, but together, we can go far.

Great Scott! HackWeek 5/1 Recap


It’s 6am on a Thursday morning, and I’m realizing the feature I had built last night to demonstrate Vue.js’s double-binding capabilities has regressed due to some code-reorganization I decided to do last-minute.  I thought I had done a good thing, but I ignored my sensibilities to quit while I was ahead and just do that “one last thing”.

So I put on a pot of coffee, and I spent the next few hours not only fixing what I had broken, (in order to make sure my demo doesn’t fall flat on its face in front of my esteemed colleagues), but I also tried to strategize the best way to communicate the week-long and tearful journey towards carving out a path to refactoring an important yet 3-year old feature on our app, in a way that employs the best that VueJS has to offer without throwing the jQuery baby out with the bathwater, all in 360 seconds. Why? because it’s our Hack Week Demo Day!

Hack Weeks @ Traackr

Once a quarter, we take a collective pause from roadmap-focused development, and carve out an entire week for the engineering team to work on projects of their own. What started out as a monthly hack day, we quickly realized that a single day was not enough for any of us to really flesh out let alone implement our ideas. Experimentation needs time and iteration if it would materialize into something solid; we needed more than a day to experiment, fail, retry, and repeat!  So, we decided to have our hacks less frequently but for a much longer period of time, and thus Quarterly Hackathon was born.

The 5/1 Hackweek Recap

This particular Hack Week arrived shortly after we completed a project that the entire Engineering team had been working on for the good part of 6 months. The project was a set of tremendous improvements and new functionality added to the Analytics Suite of the Traackr App, altogether has been named Integrated Influencer Intelligence (or I3).

The temptation after such a sizable release, is to use this unstructured time to make all those small improvements to what we just released vs. spending the time looking at new problems with fresh eyes. This time around, most of the team spent time on fairy new ideas, with some folks improving on existing ones.  By the time the week ended we a whole collection of demos and proof of concepts ranging from: Devops dashboards, to Slack integrations, to enhancements to security, just to name a few.  But like most Hack Weeks at Traackr, it was a great time to shift gears, recharge, and inspire new ideas not just for our team, but for the company as a whole.

So, without further ado, let’s dig into some of the Traackr team’s Hack Week projects, in the gallery below:

Our 5/1 Hackweek Projects Gallery

The gallery below only describes high level info on the projects.  More in-depth articles about our Hack Week is still to come!

EasyMock Delegation: Testing Twitter... without Twitter


So you’ve decided the pain of maintaining your third-party unit tests have finally outweighed the pain of reworking them into something more predictable, and you’ve entered into the wild world of mock objects. There’s no way around it, really… it was bound to happen. Look, even if your app didn’t talk to a third-party service, you’d still want to prevent any possible variation in the environment in which your test is running, outside of the exact thing you’re testing, making all things absolutely predictable. This way if your tests fail, you’ll know exactly where to look. But why am I telling you this? You are all pros! There are many resources out there for the “hows” and “whys” of mock objects, but if you’re already familiar with the concepts and libraries, this post is for you.

This is a post about how to use EasyMock for mocking Twitter, and how you can use Delegation to allow you to accomplish certain things more directly in dealing with complex implementations you’d otherwise have to do yourself.

Disclaimer: We’re using EasyMock, But you might also wanna check out jMock and Mockito as other options.

The Use Case

So let’s say you’ve built an app that integrates with a thirdparty API such as Twitter, and so you’ll probably want to test your app against specific responses from Twitter to make sure the business logic is performing correctly. For example, let’s just say every time Justin Bieber gets retweeted, your app sets off a fireworks show. Now, your unit tests want to test this behavior and you are using EasyMock to mock whatever home grown REST client you’ve written, returning the mock JSON response you’ve put together. You set your Expectations and Answers, and voila! You’ve mocked Twitter!

But then you realize that popular thirdparty Java clients exist for a reason, and instead of going through the process maintaining every single endpoint that Twitter provides, you decide to switch to something like Twitter4J, which makes integrating with Twitter very easy (it really does). So you figure, “I’ll just Mock Twitter4J”, and it should be easy, right? Wrong. Here’s why:

Twitter4J’s User Timeline endpoint is defined as:

ResponseList<Status> Twitter.getUserTimeline()

And this is the code you expect to write:

ResponseList<Status> list = ... // Figure out how to construct a ResponseList object with the Statuses you want.
Twitter mockTwitter = createMock(Twitter.class);expect(mockTwitter.getUserTimeline(EasyMock.isA(String.class), EasyMock.isA(Paging.class))).andReturn(list);

… but you you find yourself in a rabbit hole. At this point you’ve just painfully realized that to create the ResponseList object with custom JSON is not exactly trivial and here’s why: ResponseList is an interface, and so is Status. Are there implementations? Well, if you follow the Twitter4J source far down enough, you’ll find a series of factories that translate the JSON responses from the HttpResponse objects into implementations of the return types we’re working with. It’s a lot of code to write! If you want to put all the control of the response in your hands, because essentially you’re going to have to recreate what’s being done under the hood for Twitter4J. So, I’ve put together some code so you can easily mock Twitter4J’s getUserTimeline(), in a way that puts control back in your hands that uses some of EasyMock’s tricks, without writing a lot of code. So without further ado:

* Drumroll *

Step 1: Define Your JSON Response

This is the hard part :) Creating your own custom tweets as JSON Strings is such a pain. So do yourself a favor and run a few calls against Twitter, copy/paste, escape all those quotes, and make any necessary changes via grep.. etc. So let’s say in your unit test, you’ll want the first Twitter UserTimeline call to returns 30 statuses from Oprah, the second call to return 20 statuses from Justin Bieber, and the last call returns nothing, your String array looks something like:

public static String OPRAH_RESPONSE  = "[ { ... // 30 statuses from Oprah
public static String JBIEBER_RESPONSE = "[ { ..// 20 statuses from Biebs
public static String EMPTY_RESPONSE = "[]"

String expectedJsonResponses = new String[]{ OPRAH_RESPONSE, JBIEBER_RESPONSE, EMPTY_RESPONSE };

And let’s turn the String array into JSON:

JSONArray array = new JSONArray(twitterRawJSONResponse);

Step 2: Create A Method Which Mocks ResponseList

If you’ve taken a look at the Twitter4J source, you’ll notice that ResultStatus is also an extension of java.util.List and there are a bunch of methods that are already defined by the List interface, and we’re going to use EasyMock to delegate those particular methods by ResponseList to an implementation of List that we can more easily create and work with… an ArrayList. Creating an ArrayList, is a lot less verbose than recreating Twitter4J’s ResponseList. Go ahead and check out the source, it’s rather complex.

This is what EasyMock refers to as Delegation. EasyMock describes delegation as:

“ <Delegation allows mock objects> to delegate the call to a concrete implementation of the mocked interface that will then provide the answer. The pros are that the arguments found in EasyMock.getCurrentArguments() for IAnswer are now passed to the method of the concrete implementation. This is refactoring safe. The cons are that you have to provide an implementation which is kind of doing a mock manually… Which is what you try to avoid by using EasyMock. It can also be painful if the interface has many methods.”

… however in this case, it works so beautifully!  Instead of redefining a behavior from scratch, you can just delegate to a similar object that already has the behavior defined, except your delegate is completely in your control.

Another scenario in which you would want to use Delegation could be around a database cursor, for example MongoDB’s DBCursor.   It implements the Iterator interface.  If you were mocking some database behavior and you had a pre-constructed list of objects you wanted work with, instead of trying to mock the behavior of and manually plot out what each subsequent next() would return (because remember, mock objects is a recording that needs to be replayed in a certain order), you can just delegate the behavior to a more easily constructed type that you can work with.   

Let’s see it in action and create an ArrayList object and populate it with some tweets!

final List<Status> list = new ArrayList<Status>();
if (array.length() > 0) {
  for (int x = 0; x < array.length(); x++) {
    JSONObject obj = (JSONObject) array.get(x);
    Status status = DataObjectFactory.createStatus(obj.toString());

Now we’re going to create a mock of the ResponseList interface and delegate the methods being used by Twitter4J over to our List that we just created:

ResponseList<Status>  responseList = EasyMock.createMock(ResponseList.class);

Now you can use Delegation! Since you’re mocking the ResponseList and you’ll probably want to treat your ResponseList as a regular ArrayList to iterate through your results, all you have to do is simply delegate those method calls to your list object:


Your list contains the custom Status objects you’ve generated from your JSON, and they’ve been loaded into a list, so all you’d need to do is just delegate the work to ArrayList.  This way,  you don’t have to write custom code for every single Twitter scenario you want to test. Simply load the JSON into objects, parse them into Status objects, put them in a collection and delegate ResponseList’s behavior to that collection.

Let’s put all this code into a method called: buildTwitterServiceStatusResponseList(String jsonResponse)

Step 3. Mock Twitter4J

Remember this from above?

String expectedJsonResponses = new String[]{ OPRAH_RESPONSE, JBIEBER_RESPONSE, EMPTY_RESPONSE };

Let’s pass those into the above method and create a mock Twitter4J:

Twitter twitter = EasyMock.createMock(Twitter.class);
IExpectationSetters<ResponseList<Status>> expectation = EasyMock.expect(twitter.getUserTimeline(EasyMock.isA(String.class), EasyMock.isA(Paging.class)));
for (String resp : expectedJsonResponses) {

That’s all there really is to it. I’ve put a gist together where you can see the code in it’s entirety.

Remember: If your code uses other methods for ResponseList, you’ll have to create an expectation for those method calls that you can either Answer, Return, or DelegateTo…  or EasyMock will fail.

In summary, delegation in EasyMock is useful, if you care about the results that are coming back from your mocked service, and constructing a delegate is trivial compared to actually working with the objects themselves.

At TRAACKR our team spends a LOT of time thinking and implementing the ins and outs, as well as the dos and don'ts of interfacing with third party API’s and so we hope to provide engineers with some of the tools of the trade that will hopefully make your lives a little easier.

Geohashing Demystified by xkcd

Geohashing Demystified: xkcd