A pipeline for cleaning up technical debt?

Traackr's customer facing application is built on top of CakePHP and has accumulated its fair share of technical debt over time. One particular piece of debt weighed on my mind. Our application fetches a core set of data from an API and then augments it with additional data our customers have entered. A single method handles augmenting the initial data through a series of transformations. This beast of a method cannot be tested. There are far too many pieces going on to write maintainable tests. It was long overdue for a refactor, and there was a particular pattern and accompanying package I wanted to try out.

A Pipeline for Success?

I decided to use the Pipeline library from The League of Extraordinary Packages for a few reasons. First, the pattern lends itself to writing code that does one thing and one thing very well. Here is a sample pipe:

A pipe is invoked and passed a single parameter, a payload of data. You act upon that payload and then return it for the next pipe to use or to be used as the final output of the entire pipeline. Given you can chain any number of pipes together, your logic can be focused on performing a single task. This in turn leads to being able to write and maintain unit tests for a pipe.

Second, it introduces a level of reuse the beast method could not. Each pipe is a reusable element we can chain together in any fashion. If we only needed to augment the data with a small subset of our customer data, this could easily be done by putting together a different pipeline. For example, here is our entire augmentation pipeline:

Just need to augment the data with notes and projects? Simple:

Small, easy to read classes... Classes that are easy to test and maintain... Easy to reuse items in a flexible way... Yes, please!

How's It Flowin'?

Excitement must be tempered with reality though. Yes, the beast may be bloated, but it does run very fast. Would this pipeline run just as fast? The initial results were... not promising. The augmentation was not horribly slow, but it was noticeably slower. Timing logic revealed the beast ran under half a second on average, while the pipeline was running a full second on average.

Why is this approach slower? I cannot provide concrete answers, but I have a theory. Notice the pipes in our full pipeline take in a number of arguments like $this->InfData or $this->TagUtil. These are CakePHP component and model classes. My best guess is there is some overhead in having these items passed around into the pipe classes. The beast does not suffer from this overhead because the framework has already bootstrapped those classes and made it available.

So... Now what?

While the execution time of the pipeline makes using it impractical, it was worth the effort to go through the exercise. Yes, we may not be able to use these pipe classes, but they and their tests can easily be ported back into CakePHP component methods. We will still get the benefit of small, easy to test, and reusable units of code.

This library may not be used in conjunction with our CakePHP code base, but it is certainly a tool I will reach for outside of that context. More importantly, using it reinforces the mind set of writing small, testable code. And that, more than any library or tool, is the best pipeline towards reducing your technical debt.


Book Review: Agile Retrospectives

Agile Retrospectives: Making Good Teams Great*cough* *cough* Goodness... It's a bit dusty around here. Nearly a year of silence will do that. One of the goals our team had last year was to blog more. This effort failed. But, why did it fail? How do we determine what went wrong and what actions to take to ensure we do better going forward? There happens to be a handy tool in the agile / scrum toolkit known as the "retrospective". A mini retrospective of what happened with our blogging might look like this.

What did we do well? While we did not generate much content, the content we did generate was something the writer was passionate about. What could we have done better? We could have encouraged the team to write about anything. It does not need to be a long, involved post. Just write about something that interested you. How can we improve things going forward? Have someone leading by example and being a resource to cajole ask people to provide more content in addition to theirs.

This is a simple example from my own mind. Things become more complex when trying to ask these questions from an entire team. The most invaluable resource I have found for learning how to lead retrospectives is Agile Retrospectives: Making Good Teams Great. This is typically one of the books I regularly gift to co-workers.

The book is only 142 pages long but contains a wealth of information and tools to use. It is organized into three sections. First, three chapters are dedicated to explaining how to prepare to lead a retrospective, how to tailor it to your team, and the actual process of the retrospective meeting. The next five chapters outline various activities you could use for each step of the meeting. The final two chapters provide guidance on how to take all these ideas and apply them to lead retrospectives.

The best advice I have for consuming this book is to really absorb the first three chapters and final two chapters. Much of the advice there will help immensely in leading a retrospective and many other meetings. Skim the next five chapters; find a few activities your team might connect best with and start using those. Revisit these chapters every so often to find new activities to keep your retrospectives fresh.

Even if you are not in charge of leading retrospectives or other agile meetings, the advice carries over to leading or participating in other meetings. Also, the retrospective process is a useful tool outside of software development. You can have a retrospective about anything -- your recent home renovation project, your car buying experience, your vacation plans. There are few titles I gush over and push onto people. This is one of them, a title I firmly believe should be on every developer's shelf.