## Tuesday, October 09, 2012

### Functional meets Fortran

I decided my life needed a little bit more math, and so to decided to add a few differential equations in some F# code. Math in code means applied math and lots of arrays, but arrays are a pain functional style. So heres my appoach.

I have written many numerial applications, mostly in C++, yet also some in Fortran. I mention Fortran because it has the notion of equivalence, where you can map arrays on to each other.  Equivalence is like C unions for array and there to  reminds us is that arrays do not need to be managed as dynamic memory.

A functional style of programming is a lot about preserving your invariants with immutability, yet immutable vectors are not a solution (yet). Still, numerical code has its own invariants, and therefore some immutability is possible, and should be exploited to write robust code.  In my case, I have vectors and matrices; Their sizes are invariant, and so is the structure of the matrices, they have lots of zeros, they are sparse.  The approach is then to specialize a state monad to maintain size, and structure in an immutable manner, while allocating mutable vectors only within the core of the numerical engine.

Long ago I wrote an matrix assembly algorithm. You can see it as a sparse matrix structure inference code. It was really pretty simple: the final matrix is an assembly of many small sparse matrices, finding the final structure can be seen as solving a set of matrix structure equations, the algorithm defines an order of traversal for all this matrix points and then proceeds to "crawl" across all matrices at once, working out all the local structures. In a functional style of programming, I can use the same newVar like concept you would find in a constraint or inference solver, extend it with an equivalent newEq, to build a monadic environment to support my numerical code. The result is that "application code" just uses variable and defines equations, these are then assembled and only later does the code allocate just a few very large arrays to store the real data associated to the problem

This approach is "easy" because all invariants are built first and then used. Much modern applied math depends on hieararchical and adaptive grids. Hierarchy is probably easy to bring in to a functional style, adaptive grids maybe less obvious.

## Wednesday, October 03, 2012

### Greenfield project development, or "How I learned to love scrum and good product owners"

﻿

There is nothing special to greenfield projects yet teams that can tackle greenfield project have something special! It is not the projects that matter as much as the approach to resolve unknowns when they appear in the projects that matters.  My goal here is to give you a few insights on how do this.

Greenfield projects, or greenfield areas of non-greenfield project, is really about hitting the limits of what you know, it is about meeting unknowns. When that happens, and it does happen decently often, you should not apply a different process, but continue with the one you have, one that already has those “off road tires” that allow you to move in greenfield pastures.  This concept of “one process” fits them all is really the hardest thing to get across to teams that have not really managed to deepen their agility. Yet processes like Scrum works equally well on greenfield projects than on non-greenfield projects.
Best explain thing in action, here is how it works. We know that software development is a combination of different tasks. I’ll simplify it to the following:
• Requirements
• Design
• Code
• Test
• Deliverable
The most efficient way for a team to “develop” is follow the flow of dependencies between these tasks.  With this approach, you start by writing requirements, then you think about the design, then you code, you test and finally you deliver a software package. You can do this the agile way, or the waterfall way.  You can start one phase before ending the previous, but you still follow the dependency order. This works well because although there are cyclic constraints between the different development phases, they are manageable. For example, requirements (stories) are analyzed by the team before the sprint, then during the sprint team members have their independence to pick up work as suites them best. Another example, the team gets together and debates design, when all are aligned each can continue with their code development.

A “non-greenfield” project has "manageable" interdependencies between the different development tasks. By manageable I mean predictable, and by predictable I mean that the team has enough common knowledge that they can organize a process around their unknowns and resolve them together (e.g. as in preparing your next scrum). While, to the contrary, greenfield projects and greenfield areas of a project are, before all else, situations where it is simply harder for the team to efficiently pick up the development tasks. And the number one reason for this breakdown in team productivity is that the tasks no longer are supported by sufficiently clear dependencies, and this results in the team no longer have clear enough objectives when working on the development tasks, and finally too many tasks "just do not go somewhere fast enough"!

You may say: “Ok, so yes sometime we do not know, that is when we can take the time to learn!” But if you do think that, do read on because you are wrong, projects are not the right place to "take time"! Here is the reasoning:
• It is hard to notice what you do not know.
• It is hard to learn what you do not know.
• Having extra time to “observe” and learn will only help you so much
Do not expect to recognize what you do not understand
What you do not understand is what is missing, what is unknown, and unknowns are mostly invisible and therefore hard to notice. That is because they are unknown! (Duh… )

Many of us feel anxious when confronted with unknowns and things we do not understand, yet in a team these feelings are hard to express and hard to use as drivers that can be shared. In fact, in a team negative feelings may distance people, which is the last thing you want. As anxious feelings do not support a team well, we need a surer and easier way to detect the presence of “unknowns” that are affecting development , we want an “unknown detector”.  And we want some automatic way of using this detection tool, so as not “miss” reacting to the team's lack of understanding. And we are in luck, because  nothing is better than an agile iterative process to serve as your unknown detector!  When you fail your development iteration, when your sprint does not deliver, that is when you did not know what you thought you did!  It could be that you quickly understood what you did not know before, for example that a sick team member is the cause of your troubles, but more often, failed deliverable are an indication that you have unknowns and that your team as entered “the greenfield zone”… (add here appropriate music).

Rule #2, for any development, is therefore very important (in scrum language):
Track velocity, review your sprints truthfully
Here is the idea, you are ending your sprint, things have not happened the way you wanted them to happen, you have delivered half what you hoped you could. Obviously something is wrong! The easy action is to do nothing about it, just push the failed tasks into the next sprint and move on. Yet that is absolutely the wrong thing to do! Each failure needs to bring a learning.  The next level of easy action is to talk about the failure in a sprint retrospective and add future action of improvement list. Beep… Failed again. That list will not be consumed fast enough, and no learning will happen.  Now we are ready for rule #4:

I’ll start by saying: “yes, I realize this sound a little bit crazy”.
Here is the reasoning:
One or more unknowns are keeping your team from achieving its highest productivity, specific aspects of the sprint’s tasks have failed.  Your mission, as a team, is to understand what changes can be made to the backlog that will bring your productivity back up AND move your project forwards as much as possible in the right direction. The whole team’s goal is to move around these unknowns that have been blocking you and yet to get as close as possible to them because in many cases that is where the projects added value will be. The added value will be learning but also new deliverables that will make you think differently.

How then to extend your process to tackle these unknowns? The situation is that you have a scrum process, you are normally able to tackle the deeper unknowns in the pre-sprint story grooming and design discussion phases, but now you find yourself starting your sprints neither with good stories, nor good design, nor a clear idea of how you want to develop. Again you are in the greenfield territory. This is when you have team members that make suggestions like: "we should use domain specific languages", or "can we use the GPU for hardware acceleration"? This type of suggestions are signs that your team is “uneasy”, that they are out of their comfort zone and are looking for ways to feel better. As I mentioned above, detecting that subtlety of emotions in a team is not a process, but noting your sprint failure and drop in velocity is. The team needs then to take action, but given their current state of "failure" something different needs to be done. Yet I have already told you what you should do, you should revisit your backlog and rework it. The trick is how to make it systematic! This leads to rule #6:
In case of doubt, the product owner bridges the gaps, not the team
Again, you do not change your scrum process, neither do you change your product owner and his role. Yet the product owner's tactics are now somewhat reversed. Normally, it is the team that tries to meet the expectations of the product owner. In greenfield territory, the tables are turned, it is the product owner's task to meet the team's capabilities. The process is actually pretty simple: the product owner asks the team "can you deliver this" over and over, with different variants, until he has rebuilt the best of possible backlogs. Again, it is not the team that decides that it needs to learn, yet in their answer to the product owner they may say: "yes, we can deliver that, we will need to do some learning, yet we are pretty sure we can deliver what you are asking". This is scrum, so the double checking process is known to all: if the team's independent estimations (sizing) of a task are similar enough (and small enough!), then all have a good confidence that they can deliver that task.

In non-greenfield, the team is charge of delivering what it masters. In greenfield projects, it is the product owner responsibility to maintain a backlog that the team can master. Often this means that the product owner needs to be creative in formulating his desires to meet up with the teams capabilities. Here a few examples how such a process works, to make it obvious I'll imagine the product owner wants to build a car, but for software development, the process is no different:
• Basic features
• Example: Deliver a motor connected to wheels
• Limits, boundaries
• Example: Work out what is the minimum, maximum, and average weight, power, length, height, etc. of the 50 most popular cars on the market
• Out of order
• Example: Design a test track
• Disconnected, but related
• Example: Deliver the driver's seat design, the interior sizing, and the design of the windows
In each of these example, the product owner needed to pull back from his initial goal of getting a full car, each time he/she chooses a new team goal that moves the team in the right direction, delivering something of real value, but while staying in the comfort zone of the team. You may note that the out of order example takes the Test Driven Development approach, but out of order with a good product owner can be much more flexible than TDD, especially when the team has no idea of how to hook in the tests into an nonexistent design. For example, let's say the the team needs to develop a cloud app. They will need to use some new APIs, project's often have given technical constraints, in such a case, the product owner owner can focus on asking for the shell of an app that uses those APIs in order to at least tick off the boxes "code calls API", and "deliver code that works in the cloud". Nevertheless, the product owner should always ask that that the app does something which is line with his business. This is an implicit rule: always deliver some form of business value! Also note that the product owner, and the scrum process, continue to strive to deliver deliverables with enough size so that much of the whole team contributes. Fragmentation of knowledge, and new learning is a high risk in new domains and all should work together to keep the knowledge flowing.