I've had some unique experiences interacting-with and participating-in tool development at previous companies that I've worked for, with the quality of those experiences in the broad spectrum from train-wreck to near-satisfactory. From that mental scarring has emerged the weighty goo of an idea, which may be interesting food for thought. [*]
| [*] | Yes, I'm suggesting you digest my mind-goo. |
How it starts
At some point in a company's growth, management notices that there is a lot of sub-par, [†] redundant, and distributed tool development going on. Employees have been manufacturing quick-and-dirty tools in order to perform their jobs more efficiently.
| [†] | For some definition of par. |
Management then ponders the benefit of centralizing that tool development. It seems like an easy sell:
- Help ensure ongoing productivity gains -- mission-critical tools that stop working or contain heinous bugs are a hidden liability
- Foster tool developer expertise on a specialized team
- Eliminate needless repetition by creating shared code bases and consolidating infrastructure resources
Good management will also consider the negative repercussions of turning distributed and independent resources into a shared and centrally managed resource:
- Everybody wants a slice of the tool pie, because tools make our life better, and there always seems to be some stupid crap to automate.
- Everybody wants a piece of the compute resources, because more compute makes parallelizable analyses go faster.
- If you build it, they will come.
How I've seen it work (warning: depressing, hyperbolic)
A group at the company makes a strong enough case to the centralized-tool-management machinery -- a request for tool development is granted.
A series of inevitably painful meetings are scheduled where the customer dictates their requirements, after which the tool team either rejects them or misunderstands/mis-prioritizes them because: a) that's not how it works -- they have to actively gather the requirements, and b) they don't have enough time to do all the silly little things that the customer wants.
Because people are fighting each other to get what they want, everybody forgets that the customers haven't really described the problem domain in any relevant detail.
The tool team developers are happy to go code in peace, without going back for more painful meetings. They create a tool according to their understanding of the requirements during the first iteration.
The customer has no idea how the tool team came up with a product that was nothing like their expectation. They say something overly dramatic like, "it's all wrong," pissing off the tool team, and lose faith in the ability of the tool team to deliver the product they want.
The customer goes back to doing it manually or continue to develop their own tools, expecting that the tool team will fail.
The tool team fails because the customer lost interest in telling them what they actually needed and giving good feedback. It wasn't the tool that anybody was looking for because the process doomed it from the start.
I say that this scenario is depressing because tool teams exist to make life better for everybody -- they enjoy writing software that makes your life easier. Working with a tool team should not be painful. You should want to jump for joy when you start working with them and take them out to beers when you're finished working with them, because they're just that good. I think that, by taking a less traditional approach, you will be able to achieve much better results...
How it should work
A group at the company makes a strong enough case to the centralized-tool-management machinery -- a request for tool development is granted.
A small handful of tool team operatives [‡], probably around two or three people, split off from the rest of the tool team and are placed in the company hierarchy under the team of the customers. They sit the customers' cube farm, go to their meetings to listen (but no laptops!), etc., just like a typical team member would.
The customer team brings the operatives up to speed on the automatable task that must be performed each day through immersion. Depending on the frequency, breadth, and duration of the manual processes, the operatives must perform this manual process somewhere on the scale from weeks to months, until they develop a full understanding of the variety of manual processes that must be performed. [§]
All operatives should be 100% assigned to the manual tasks for this duration, temporarily offloading members of customer team after their ramp-up.
Bam! With an unquestionably solid understanding of the problem domain, the tool team sleeper cells activate. 80% of the manual task load is transitioned off of the operatives so that they can begin development work. Agile-style iterations of 1-2 weeks should be used.
After each iteration there must be a usable product (by definition of an iteration). As a result of this, a percentage of the manual task load is shifted back onto the operatives each iteration, augmenting the original 20%. If the tool is actually developing properly, the operatives will be able to cope with the increased load over time.
As the feature set begins to stabilize or the manual task load approaches zero (because it has all been automated), the product is released to the customers for feedback and a limited amount of future-proofing is considered for final iterations.
Most customer feedback is ignored, but a small and reasonable subset is acted on. If the operatives were able to make do with the full task load plus development, it's probably a lot better than it used to be, and the customer is just getting greedy.
The customer takes the operatives out for beers, since the tool team saved them a crapload of time and accounted for all the issues in the problem domain.
A single operative hangs back with the customer for a few more iterations to eyeball maintenance concerns and maybe do a little more future-proofing while the rest head back to the tool team. The one who hangs back gets some kind of special reward for being a team player.
| [‡] | I'm calling them operatives now, because their roles are different from tool developers, as you'll see. |
| [§] | It is beneficial if a small seed of hatred for the manual task begins to develop, though care should be taken not to allow operatives to be consumed by said hatred. |
Conclusion
In the sleeper cell approach, the operatives have a clear understanding of what's important through first hand knowledge and experience and, consequently, know the ways in which the software has to be flexible. It emulates the way that organic tool development is found in the wild, as described in the introductory paragraph, but puts the task of creating the actual tool in the hands of experienced tool developers (our operatives!).
I think it's also noteworthy that this approach adheres to a reasonable principle: to write a good program to automate a task, you have to know/understand the variety of ways in which you might perform that task by hand, across all the likely variables.
The operatives are forced to live with the fruits of their labor; i.e. a defect like slow load times will be more painful for them, because they have to work with their tool regularly and take on larger workloads on an ongoing basis, before developers can ever get their hands on it.
Notice that there's still the benefit through centralization of tool developers: central contact point for tool needs, cultivating expertise in developers, knowledge of shared code base, understanding of infrastructure and contact points for infrastructural resource needs; however, you avoid the weird customer disconnect that comes with time slicing a traditional tool team.
Tools developers may also find that they enjoy the team that they're working in so much that they request to stay on that team! How awesome of a pitch is that to new hires? "Do you have a strong background in software development? Work closely with established software experts, make connections to people who will love you when you're done awesome-ing their lives, and take a whirlwind tour of the company within one year."