Granularity Optimization

[I've been working on this entry for quite a while, I've just finally decided to publish it.]

When planning a project it is critically important to find an optimal task size. There are several reasons behind this. First and foremost, any estimate is destined too be inaccurate. Additionally, it is impossible to effectively assign more than one resource to a task. Lastly, if tasks are defined to specifically they become brittle and must be remade more frequently. 

With respect to inaccuracy, an estimate made early in a project, when the details are not well understood, or made under pressure to simply publish a schedule is commonly incorrect by as much as 50%, sometimes more. If the project team iteratively re-estimates tasks the accuracy of the estimate might progressively improve as the start-date for the task approaches. However, the nature of estimating tells us that we are simply making the best educated guess we can about the effort required to achieve some goal. None of us are able to predict the future with consistent accuracy and precision. So, if by using an iterative re-estimation technique we can manage a accuracy of +/-10% on our estimate we're doing pretty well. But, if a task is estimated at 30 days (240 hours) and we are wrong by 10% we have at least three things to consider. 

First, if we finish 10% early (27 days) we may create idle time for the person(s) working the task. Idle time can lead to one or more dangerous effects; gold-plating being the most dangerous. As they say, Idle Hands are the Devils Tools. There are also resource utilization considerations, but they are less relevant and, at least in my opinion, irrelevant. 

Second, if we finish 10% late (33 days) we may slip a delivery or otherwise impact future tasks which has a cascading effect on the overall project. This is a more pronounced problem to be sure. I think the consequences should be obvious to the reader.

Third, in either case we over pay for the task in terms of money; for early delivery we pay for idle time (or the Devil's work), for late delivery we pay for the additional work.

In order to make effective assignments of resources ideally we have one task for one worker. This is true in most kinds of work. Those who choose to employ paired programming, for all intents and purposes are using two people in the place of one. Often times large tasks are created and assigned to a team member; e.g. Build sub-system X. But as time passes and sub-system X is not complete, more people may be assigned to do the work; to help. This is not an effective way to bring a task back in line with the schedule. If one team member is in the midst of developing a solution for a task, and suddenly more team members are added to the task, everyone's productivity slows. The original team member must slow down in order to communicate the work he has done, the work that remains, and all the considerations that he may have made to this point. The new team members can't make any progress until all this information has been communicated. Finally, all of the assignees must now communicate with each other about what has been and will be done. 

Worse still, if too many people are assigned to a task you can exceed the maximum partition-ability of the task. Think two workers, one shovel. If an when you exceed the maximum partition-ability of a task, the result is that the 'extra workers' must be idle in order to prevent conflicts. Furthermore, communication about the task must also increase which degrades all worker performance. The next result is that the task takes even longer to complete than originally scheduled.

The final concern about task granularity is over analysis and the creation of too many too detailed tasks. This becomes a problem simply because changes in the design and architecture specifics of a project will cause these tasks to become invalid. They must then be discarded or replaced. If this occurs too frequently the overhead of managing the task list outweighs the advantages of the details. This will impact team performance creating 'start delays' of tasks and also potentially increasing the amount of waste within a project; especially if some of these detailed tasks are taken from a backlog to fill gaps in the utilization of team members. 

Finding the balance between too much detail and not enough is a significant challenge for a project planner; it is one of the distinguishing characteristics of a great planner. Keeping in mind the impact of inaccuracy, the maximum partition-ability of a task,  and the risk of over analysis while planning can help smooth out a task plan while still providing a solid time/effort management mechanism.

Partitionability

Partitionability describes the degree to which something can be functionally decomposed. The partitionability of a process, algorithm, or function is determined by the refinement of a design. The partitionability directly correlates to the number of distinct atomic tasks required to complete the implementation of the process, algorithm, or function.

Partitionability is used when estimating the man-time requirements for implementation. For example, some functionality X requires 22 man-hours to complete. X can be decomposed into 5 distinct component parts (its partitionability is 5). Optimally, X can be completed in 22 / 5 = 4.4 hours by a team of 5 developers. Stated formulaically;

T_o = T_m / P

Where T_o = Optimal Time, T_m = Man-Time, and P = Partitionability

Good, Fast, and Cheap. Pick Two

The designer's Holy Triangle or, as I call it, the Magic Triangle. There is another set of considerations beyond the Magic Triangle. Call it a corollary triangle, I call it 'the technology balance'. The idea is that within a technology project there are three considerations; The use or creation of new and innovative technology, getting the job done (or doing business), and correctness (in the academic sense). I'm aware of other considerations but I feel these three are the the most contentious considerations. 

When technology projects start there is often a lot of conversation about what technologies should be applied. Often, teams choose to use newer or more cutting edge technologies in favor of older more stable technologies. Part of this is facination with the new, part of it is a hedge against those things we don't like about the old. All of these are valid and reasonable determining factors for technology selection. 

Another observation is that technology teams, given enough time, will argue endlessly about the correctness of any given solution. There is always debate about what is 'the right thing' in tool selection, pattern application, hardware configuration, design and planning approaches and just about any conceivable facet of the project. 

Always in a project there is a push to deliver the product on-time and within the budget. We've all heard the complaints about schedule driven decisions, right-to-left scheduling, scope creep and defect management; contributors to concern about delivering on-time.

We go round and round these three topics in every project. Depending upon the focus and dicipline of the team and it's leaders we may not complete a project because of one or all of these considerations. By being aware of these contentions and managing our activities in accordance with the balance that must be maintained we can be more successful with our projects. 

Prejudicial Refactoring

We are all familiar with the concept of refactoring. I think generally we agree that refactoring can be a useful thing when changes are needed in our applications; but what about making changes for change sake. I've seen recently an abuse of the word; every change becomes a refactoring effort. Properly applied, refactoring is an effective way to reduce code-debt or change from one implementation to another or as an effective means of adjusting the underlying architecture of an application as the needs of an IT organization evolve. 

Sometimes though, refactoring is undertaken for no good reason at all. A recent example that I experienced was a 8 month .NET project that had been underway for 4 months. The initial implementation used the CSLA  approach to data persistence. It was determined (somehow) that CSLA was a 'bad thing' and the code was refactored to use nHibernate  and Spring.NET. The justification for this change (given to management) was technology standardization; not a very convincing argument since this was the only .NET application in-house. Other technical arguments were made about these ease of testing provided by nHibernate and Spring.NET, but these were more academic and preference based arguments. The end result was 7 weeks of lost progress by the project team. The cost in time and money was not determined and the impact on the delivery schedule was not accounted for in the project plan. Estimates vary about the dollar impact of this change, but for argument sake lets say it cost 90,000 USD. 

This event was an act of prejudicial refactoring. Prejudicial refactoring is the act of refactoring for the some poorly justified reason. In our example case, 18% of the budget and 25% of the time was expended on an unnecessary change. The objective of any software project is to deliver the software. There are numerous reasons to argue for or against any one technology decision; CSLA v. nHibernate, Oracle v. MySQL etc. Once those decisions have been made and the project is in motion, changes to these underlying technologies must be made with caution. 

Software is tricky business, the unforeseen technology issue, or the ever changing requirements can derail a project from it's budget at a moments notice. Knowingly consuming the budget for an arbitrary reason is grossly irresponsible. Change for the sake of change is a 'bad thing', and therefore refactoring just because we can is also a 'bad thing'. Careful consideration of the reason for change, and the impact of that change are a must in order to responsibly develop software.