Thursday, December 13, 2012

Fun with higher order monadic construction (F#, part 1)

Higher order X really means that X is used in something bigger. So when I say higher monadic construction, I mean that we use monads in more than the bind function or creating them with return like functions. Note that the intro post to this series, shows how you can lift monads, that is a higher order operation!. Now that we have a stateful maybe monad with error management (see earlier post) we can bring in higher order operators that are similar to the bind, they control sequencing, but in a different way. The maybe monad, stateful or not, has a logical notion, you either do things or not, so it makes sense that start with a focus on boolean operators. Note that these are really operators for computational logic not really boolean logic.
We start with the OR operator. Note how the transactional part of the state goes through the "true" part of the computation, while the error state goes through the "false" part of the code. (This is what I call a trace centric coding style).
let (|||) m1 m2 = SM (fun ((s,err) as sr) ->
    let (optX1, ((s1,err1) as sr1)) = run m1 sr
    match optX1 with
    | Some _ -> (optX1, sr1)
    | None   ->
        let (optX2, (s2,err2)) = run m2 (s, err1)
        match optX2 with
        | Some _ -> (optX2, (s2, err))
        | None   -> (None, (s, err2)))
Now I can show off its behavior by trying to fetch one or an other value from the stateful map (see this post for details), it fails becaue the map only has values associated to "a" and "b": 
let test16 =
    optState {
        let! a = get7 "a"
        let! b = (get7 "x") ||| (get7 "y")
        do! set7 ("c",(a+b))
        let! counter = increment8
        return counter
        }
The interest here is the programming style: we either do one part of the expression or the other. But more importantly that we have defined the OR operator to capture errors as a state. So running (run test16 (testState1,[]) ) the above test will produce:
(null, ({keyValues = map [("a", 1.0); ("b", 2.0)];
         counter = 100;}, ["key "y" not found"; "key "x" not found"]))
Which is nice because it contains both failures of lookup.
To the contrary, the following variation of our test:
let test17 =
    optState {
        let! a = get7 "a"
        let! b = (get7 "x") ||| (get7 "b")
        do! set7 ("c",(a+b))
        let! counter = increment8
        return counter
        }
Fails first with "x" to then succeed with "b" and so return no error:
  (Some 101, ({keyValues = map [("a", 1.0); ("b", 2.0); ("c", 3.0)];
               counter = 101;}, []))
With the OR operator we want an AND operator, defined as follows:

let (&&&) m1 m2 = SM (fun ((s,err) as sr) ->
    let (optX1, sr1) = run m1 sr
    match optX1 with
    | Some x1 ->
        let (optX2, sr2) = run m2 sr1
        match optX2 with
        | Some x2 -> (Some(x1,x2), sr2)
        | None   -> (None, sr2)
    | None   ->
        (None, sr1))
The code is much more simple than the OR because we do not need to take the state appart as we depend on each of the monadic arguments to do that for us.
As we want to get "a" AND get "b" we can write are test code as following:
let test17 =
    optState {
        let! (a, b) = (get7 "a") &&& (get7 "b")
        do! set7 ("c",(a+b))
        let! counter = increment8
        return counter
        }
This puts the focus on the fact that this definition of the AND operator is returning a tuplet of the two successfuly return values.  Now again we can see it manage errors:
let test18 =
    optState {
        let! (a, b) = (get7 "a") &&& (get7 "x")
        do! set7 ("c",(a+b))
        let! counter = increment8
        return counter
        }

When run will produce the expected initial "transactional" state with the error state:
  (null, ({keyValues = map [("a", 1.0); ("b", 2.0)];
           counter = 100;}, ["key "x" not found"]))
The good thing about this version of the AND is that the left and right arguments can be returning different types.  Yet often we have lists of things to do, So having a similar set of boolean like operators to work with lists of monads would be a nice thing.
I'll show you how to do that in my next post (as my daughter's drumming lesson is coming to an end so that will be all for this post)!

Wednesday, December 12, 2012

Error management with state monads

The Maybe, Option in F#, monad returns only Nothing/None when "things go bad".
In real life we want to return details of what when wrong.

The other day someone asked me: how do I choose if something should go in the return part of the state monad, or in the state itself? Last post I showed you how to combine the Maybe and State monad. The natural next step is to extend things further in order to allow some form of error management; This provides more comfort than the  "all or nothing" standard behavior of the Maybe monad;  And, is exactly a case where we can choose either to return error information, or to augment the state to store error state.

Returning the error seems the natural thing to do. The idea is to use an Either like type, which either returns what you want normally, or the error status.  In F#, the Maybe monad is either returning "Some value" or "None",  the idea would be instead to return something like "Ok value" or "Failed error" based on a type declaration like:

type ReturnWithErr<'t,'err> =
| Ok     of  't
| Failed of  'err

 The problem with this, is that it brings you back to programming with return errors. One of the reasons why exception were invented is that it is a painful effort to decide what to do with each of these returned error.

The magic of functional programming is that we can "swivel" between a functional and state view. (I'll need to make a post on this concept, very important). Therefore we can "decide" that our error management is stateful. And as this is a local state that is caried around by the monad (it is "local" enough that it is ok to put things in it like the current error state).

We need to do better than using the basic state monad because it removes failed states, and therefore would remove any "state of error" that we might try to keep. The idea is then to split the state into two parts:
  • the "transactional" state that is lost on failure
  • the "error" state that is kept on failure
Normally the "error" state is only updated in case of error. Although you could imagine having a "not" like operator that clears the error state.

We know enough to write some code. In the previous post, I showed you how the bind operator of the stateful Maybe monad needs to track one state. Now we write a bind works with the two states instead of one. From the orginal state monad perspective there is still only one state but it is a tuplet transactionalState*errorState :

let bindOpt m g =
    SM(fun ((s,err) as sr) ->
    let (rf, ((s2,err2) as sr2)) = run m sr
    match rf with
    | Some rfv ->
        let (rg, ((s3,err3) as sr3)) = run (g rfv) sr2
        match rg with
        | Some _ ->
            (rg, sr3)
        | None ->
            (None, (s,err3))
    | None ->
        (None, (s,err2)))

It pretty much does what I mentioned above: the transactional state follows the "ok" path (states s, s2, and s3), the error state follow the failed path (state err, err2, and err3)

We want a few helper functions:

let getState f = SM (fun ((s,err) as sr) -> (Some(f s), sr))
let mapState f = SM (fun(s,err) -> (Some(), (f s,err)))
let ret x = SM (fun s -> (Some x,s))
let retSome mapErr optX = SM (fun((s,err) as sr) ->
    match optX with
    | Some _ -> (optX, sr)
    | None ->(None, (s, mapErr err)))
let retNone mapErr = SM (fun(s,err) -> (None, (s, mapErr err)))

And now we can wrap our monad in the "do" notation semantic of F#:

type OptState() =
    member inline b.Bind (m, f) =  bindOpt m f
    member inline b.Combine(m1, m2) = bindOpt m1 (fun _ ->m2)
    member inline b.Return x = ret x
    member inline b.ReturnFrom m = m

To demontrate the concept, I assume my error state is just a list of strings. So I need a little helper function to add to this list:



let cons x t = x :: t
And now I can refactor the "get" function (see previous post):
let get7 key =
    optState {
        let! kv = getState getKeyValues
        let! value = retSome (cons (sprintf "key %A not found" key)) (Map.tryFind key kv)
        return value
        }
If the get function fails, the error state will hold a list of string where the failure is reported. In this next example, the get fails because "x" is not part of the state map (again see last post the map only has keys "a" and "b"):
let test15 =
    optState {
        let! a = get7 "a"
        let! b = get7 "x"
        do! set7 ("c",(a+b))
        let! counter = increment8
        return counter
        }
Running this, run test15 (testState1,[]), gives:
(null, ({keyValues = map [("a", 1.0); ("b", 2.0)];
           counter = 100;}, ["key "x" not found"]))
And it has the error message in the error state! One of the key concept of the arrow is that a flow of information gets modified, while the other part stays the same. Augmenting state monads as shown above is really about applying the same idea. Next time I'll talk above higher order monadic constructions. We can again use our simple example for that. 
 

Saturday, December 01, 2012

Introduction to stateful monads

I went to my first Zurich FSharp meeting the other day and made a presentation on how to program with state monads in F#.

The presentation is designed to pull you off the ground, to a higher order programming style, using the state and maybe monad as stepping stones. It finishes with stateful lifting.

Here is a link to a slightly improved version: youtube .

Link to sources are below but I would recommend that if you really do not know this type of programming and want to learn, then you should type it back in and allow yourself some creative liberty. This is usually a much better way to learn, than to just copy code!

You can watch it here if you want:

http://files.meetup.com/2794852/ModularStateMonadsInFSharp.pdf
http://files.meetup.com/2794852/XState1.fs
http://files.meetup.com/2794852/XState2.fs
http://files.meetup.com/2794852/Script.fsx



Monday, November 19, 2012

Why would Sinofsky leave?

I think we all love to be shocked.

I was shocked in Sinofsky's departure from Microsoft; The captain does not leave the boat at the first major port of call in the window 8 itinerary, and neither does he get fired. And yet...

The hardest thing of to do is work in a corner, and Microsoft is and has been in one hell of a corner for more than ten years now. And so has Sinofsky, relative to his position in the hierarchy of Microsoft. The tricky problem is growth!

From Steve Sinofsky's perspective the limit to his growth is his CEO. As such the CEO is a constraint. Looking at this from a game theoretic perspective, this constraint becomes a variable in dual formulation. Yet the last thing the CEO wants is for people to start looking at a system were he is the variable. Therefore a probable interpretation of the events is that this was preemptive protective action taken by the CEO.

Wednesday, November 07, 2012

From Chinese famines to agile

A recent economist reviews a book on the "The Great Chinese Famine 1958-1962" and writes the following:

"The Great Leap Forward was the high point of ignorant Maoist folly. Chairman Mao said in 1957 that China could well overtake the industrial output of Britain within 15 years. People left the fields to build backyard furnaces in which pots and pans were melted down to produce steel. The end product was unusable. As farmers abandoned the land, their commune leaders reported hugely exaggerated grain output to show their ideological fervour. The state took its share on the basis of these inflated figures and villagers were left with little or nothing to eat. When they complained, they were labelled counter-revolutionary and punished severely. As the cadres feasted, the people starved. Mr Yang calculates that about 36m died as a result."
When I read this, I could not help thinking that too many software development teams are starved from doing the right thing because they have promised, or someone promised for them, something they cannot deliver. The result and its dynamics are not very different from the story above, let me paraphrase this as follows:
Someone with authority decides that X will be developed in Y amount of time. No one in development dares to contradict the order and work is started. Given the predefined schedule, no solid analysis can be made (aka harvest sowed) but more importantly little effort is made to make the project "self-consistent" across all its requirements, for example that it is defined to be usable from day one (aka making sure the village stays sustainable). As there is not enough resources "to go around" (aka not enough food for all), there is no point thinking about "fixing" the problems as they accumulate (aka farms are deserted, crops rot, nothing is done to fix this), and so developer will "just write code" with the foremost goal of fulfill the initial request (aka pots and pans will be melted) but as design and cohesive goals are lacking, as well as the developer team's understanding of what they are really trying to deliver (aka farmers do not know how to work steel), the effort to fit things together to cover the scope the project just grows and grows... Upon failing to deliver, and there will be a few failed deliveries, moral will plummet, as will productivity, key people may leave (aka farmers and their family starve to death). In the end mush pain and suffering will happen before something is done. One option is to pull the plug on the project, the other is to go agile with it: throw away the promised time line, build a backlog, see what you can do, at which point you can still decide to pull the plug.

This, maybe bleak, example of failure does lead us to an observation: capitalism does put pressure on management to avoid the scenario above. The problem is that many managers, even under pressure to make a profit, do not know how to do so! They do not understand how projects differ, they know only how to do what they did before. Therefore from time to time, you will see them kicking off projects that ultimately will be disastrous.

Agile does not keep you from making mistakes, but applied well, it will alert you of your follies (or others people follies) before they destroy you. Understanding how different types of projects need different style of leadership is helpful (see blog entry on greenfield projects, and on legacy projects).

Tuesday, October 09, 2012

Functional meets Fortran

I decided my life needed a little bit more math, and so to decided to add a few differential equations in some F# code. Math in code means applied math and lots of arrays, but arrays are a pain functional style. So heres my appoach.

I have written many numerial applications, mostly in C++, yet also some in Fortran. I mention Fortran because it has the notion of equivalence, where you can map arrays on to each other.  Equivalence is like C unions for array and there to  reminds us is that arrays do not need to be managed as dynamic memory.

A functional style of programming is a lot about preserving your invariants with immutability, yet immutable vectors are not a solution (yet). Still, numerical code has its own invariants, and therefore some immutability is possible, and should be exploited to write robust code.  In my case, I have vectors and matrices; Their sizes are invariant, and so is the structure of the matrices, they have lots of zeros, they are sparse.  The approach is then to specialize a state monad to maintain size, and structure in an immutable manner, while allocating mutable vectors only within the core of the numerical engine.

Long ago I wrote an matrix assembly algorithm. You can see it as a sparse matrix structure inference code. It was really pretty simple: the final matrix is an assembly of many small sparse matrices, finding the final structure can be seen as solving a set of matrix structure equations, the algorithm defines an order of traversal for all this matrix points and then proceeds to "crawl" across all matrices at once, working out all the local structures. In a functional style of programming, I can use the same newVar like concept you would find in a constraint or inference solver, extend it with an equivalent newEq, to build a monadic environment to support my numerical code. The result is that "application code" just uses variable and defines equations, these are then assembled and only later does the code allocate just a few very large arrays to store the real data associated to the problem

This approach is "easy" because all invariants are built first and then used. Much modern applied math depends on hieararchical and adaptive grids. Hierarchy is probably easy to bring in to a functional style, adaptive grids maybe less obvious.

Wednesday, October 03, 2012

Greenfield project development, or "How I learned to love scrum and good product owners"



There is nothing special to greenfield projects yet teams that can tackle greenfield project have something special! It is not the projects that matter as much as the approach to resolve unknowns when they appear in the projects that matters.  My goal here is to give you a few insights on how do this.

Greenfield projects, or greenfield areas of non-greenfield project, is really about hitting the limits of what you know, it is about meeting unknowns. When that happens, and it does happen decently often, you should not apply a different process, but continue with the one you have, one that already has those “off road tires” that allow you to move in greenfield pastures.  This concept of “one process” fits them all is really the hardest thing to get across to teams that have not really managed to deepen their agility. Yet processes like Scrum works equally well on greenfield projects than on non-greenfield projects.
 Best explain thing in action, here is how it works. We know that software development is a combination of different tasks. I’ll simplify it to the following:
  • Requirements
  • Design
  • Code
  • Test
  • Deliverable
The most efficient way for a team to “develop” is follow the flow of dependencies between these tasks.  With this approach, you start by writing requirements, then you think about the design, then you code, you test and finally you deliver a software package. You can do this the agile way, or the waterfall way.  You can start one phase before ending the previous, but you still follow the dependency order. This works well because although there are cyclic constraints between the different development phases, they are manageable. For example, requirements (stories) are analyzed by the team before the sprint, then during the sprint team members have their independence to pick up work as suites them best. Another example, the team gets together and debates design, when all are aligned each can continue with their code development.

A “non-greenfield” project has "manageable" interdependencies between the different development tasks. By manageable I mean predictable, and by predictable I mean that the team has enough common knowledge that they can organize a process around their unknowns and resolve them together (e.g. as in preparing your next scrum). While, to the contrary, greenfield projects and greenfield areas of a project are, before all else, situations where it is simply harder for the team to efficiently pick up the development tasks. And the number one reason for this breakdown in team productivity is that the tasks no longer are supported by sufficiently clear dependencies, and this results in the team no longer have clear enough objectives when working on the development tasks, and finally too many tasks "just do not go somewhere fast enough"!

You may say: “Ok, so yes sometime we do not know, that is when we can take the time to learn!” But if you do think that, do read on because you are wrong, projects are not the right place to "take time"! Here is the reasoning:
  • It is hard to notice what you do not know.
  • It is hard to learn what you do not know.
  • Having extra time to “observe” and learn will only help you so much
Let’s start with the first one, which leads me to greenfield project rule #1: 
Do not expect to recognize what you do not understand
What you do not understand is what is missing, what is unknown, and unknowns are mostly invisible and therefore hard to notice. That is because they are unknown! (Duh… )

Many of us feel anxious when confronted with unknowns and things we do not understand, yet in a team these feelings are hard to express and hard to use as drivers that can be shared. In fact, in a team negative feelings may distance people, which is the last thing you want. As anxious feelings do not support a team well, we need a surer and easier way to detect the presence of “unknowns” that are affecting development , we want an “unknown detector”.  And we want some automatic way of using this detection tool, so as not “miss” reacting to the team's lack of understanding. And we are in luck, because  nothing is better than an agile iterative process to serve as your unknown detector!  When you fail your development iteration, when your sprint does not deliver, that is when you did not know what you thought you did!  It could be that you quickly understood what you did not know before, for example that a sick team member is the cause of your troubles, but more often, failed deliverable are an indication that you have unknowns and that your team as entered “the greenfield zone”… (add here appropriate music).

Rule #2, for any development, is therefore very important (in scrum language):
Track velocity, review your sprints truthfully 
Velocity tracking and sprint reviews are your team’s unknown detector. The more newness and unknowns you have the more precious they are in alerting the whole team of unknown dangers. The next question is “what should you do when the knowledge alarm bell rings”?  As I forewarned before, the last thing to do is to “go back to school”. It is not the learning that is wrong, it is what you learn, how quickly you learn, how productively you learn that is most often the problem. Or to put it another way, with every development task you can be expected to learn something, the more you learn the better, as long as you get your task done! In a non-greenfield project, there are still lots of unknowns, but these get resolved because your team has a process to conquer these unknowns. In a greenfield project, unknowns accumulate, and when the team fails a sprint, it is in fact the team’s unknown resolution process that is failing. Trying to fix a process by reading up on things rarely does the job. First because, reading is a personal thing, not a group process, then because learning from reading is simply not going to happen fast enough. To keep your team going you need your learning to be part of the team process, not external to it. This bring us to rule #3:
Drive learning within your process 
Here is the idea, you are ending your sprint, things have not happened the way you wanted them to happen, you have delivered half what you hoped you could. Obviously something is wrong! The easy action is to do nothing about it, just push the failed tasks into the next sprint and move on. Yet that is absolutely the wrong thing to do! Each failure needs to bring a learning.  The next level of easy action is to talk about the failure in a sprint retrospective and add future action of improvement list. Beep… Failed again. That list will not be consumed fast enough, and no learning will happen.  Now we are ready for rule #4: 

Realign your backlog to avoid your unknowns but still touch them
I’ll start by saying: “yes, I realize this sound a little bit crazy”.
Here is the reasoning:
One or more unknowns are keeping your team from achieving its highest productivity, specific aspects of the sprint’s tasks have failed.  Your mission, as a team, is to understand what changes can be made to the backlog that will bring your productivity back up AND move your project forwards as much as possible in the right direction. The whole team’s goal is to move around these unknowns that have been blocking you and yet to get as close as possible to them because in many cases that is where the projects added value will be. The added value will be learning but also new deliverables that will make you think differently.

How then to extend your process to tackle these unknowns? The situation is that you have a scrum process, you are normally able to tackle the deeper unknowns in the pre-sprint story grooming and design discussion phases, but now you find yourself starting your sprints neither with good stories, nor good design, nor a clear idea of how you want to develop. Again you are in the greenfield territory. This is when you have team members that make suggestions like: "we should use domain specific languages", or "can we use the GPU for hardware acceleration"? This type of suggestions are signs that your team is “uneasy”, that they are out of their comfort zone and are looking for ways to feel better. As I mentioned above, detecting that subtlety of emotions in a team is not a process, but noting your sprint failure and drop in velocity is. The team needs then to take action, but given their current state of "failure" something different needs to be done. Yet I have already told you what you should do, you should revisit your backlog and rework it. The trick is how to make it systematic! This leads to rule #6:
In case of doubt, the product owner bridges the gaps, not the team
Again, you do not change your scrum process, neither do you change your product owner and his role. Yet the product owner's tactics are now somewhat reversed. Normally, it is the team that tries to meet the expectations of the product owner. In greenfield territory, the tables are turned, it is the product owner's task to meet the team's capabilities. The process is actually pretty simple: the product owner asks the team "can you deliver this" over and over, with different variants, until he has rebuilt the best of possible backlogs. Again, it is not the team that decides that it needs to learn, yet in their answer to the product owner they may say: "yes, we can deliver that, we will need to do some learning, yet we are pretty sure we can deliver what you are asking". This is scrum, so the double checking process is known to all: if the team's independent estimations (sizing) of a task are similar enough (and small enough!), then all have a good confidence that they can deliver that task.

In non-greenfield, the team is charge of delivering what it masters. In greenfield projects, it is the product owner responsibility to maintain a backlog that the team can master. Often this means that the product owner needs to be creative in formulating his desires to meet up with the teams capabilities. Here a few examples how such a process works, to make it obvious I'll imagine the product owner wants to build a car, but for software development, the process is no different:
  • Basic features
    • Example: Deliver a motor connected to wheels
  • Limits, boundaries
    • Example: Work out what is the minimum, maximum, and average weight, power, length, height, etc. of the 50 most popular cars on the market
  • Out of order
    • Example: Design a test track
  • Disconnected, but related
    • Example: Deliver the driver's seat design, the interior sizing, and the design of the windows
In each of these example, the product owner needed to pull back from his initial goal of getting a full car, each time he/she chooses a new team goal that moves the team in the right direction, delivering something of real value, but while staying in the comfort zone of the team. You may note that the out of order example takes the Test Driven Development approach, but out of order with a good product owner can be much more flexible than TDD, especially when the team has no idea of how to hook in the tests into an nonexistent design. For example, let's say the the team needs to develop a cloud app. They will need to use some new APIs, project's often have given technical constraints, in such a case, the product owner owner can focus on asking for the shell of an app that uses those APIs in order to at least tick off the boxes "code calls API", and "deliver code that works in the cloud". Nevertheless, the product owner should always ask that that the app does something which is line with his business. This is an implicit rule: always deliver some form of business value! Also note that the product owner, and the scrum process, continue to strive to deliver deliverables with enough size so that much of the whole team contributes. Fragmentation of knowledge, and new learning is a high risk in new domains and all should work together to keep the knowledge flowing. 


Monday, September 17, 2012

From old to new: dealing with legacy software

A tricky problem is to deal with software legacy.
I think I have never been anywhere where I did not need to deal with legacy some way or another. In fact my first task on my first job was to improve the performance of "old" design rule checker (DRC) by upgrading its geometric store with a KD-tree. That was during a summer job, and later as employee, I was asked to write a new DRC to replace the old one. (A DRC is used to check geometric rules in layouts of integrated circuits). The new DRC was better than the old one, and part of an automated layout tool for analog circuit, which... ultimatly failed. Commercially the project did break even, but other systems worked better. Looking back, with the hindsight of knowing which system succeeded, this CAD sytem failed to approach the problem the right way, which was incrementaly. For example, the requirements for the DRC were: do what the old software does but better and faster, yet they really should have been "make it incremental": a small change in the input should generate only moderate of work to revalidate the rules.

Legacy is about dealing with code that exists already and that causes you trouble. Typically the trouble is that the legacy software is equivalent to a massive traffic jam where it is no longer economic to work on it. Changes to the software are simply too expensive: they take too long and they are risky. Unlike new software, legacy software is already in use, and this simple fact is the root of all problems: it makes you stupid!  Take the example of that DRC project, my boss had said “do what the old software does but better and faster”,  I wrote a great boolean operations on large geometric figures package; It was fast, it was better than the previous package, but… it was not what was needed!  This is what I call  the “dumbifying” effect of legacy software, and the subject of this post.

When a magician performs a trick, you may know the magician is tricking you but mostly unable to do anything than to “fall for it” and think it looks real. Legacy software is a little bit like a nasty magician, it makes things look more real than they deserve. So for example, if your team tells you they can rewrite a legacy system, your tendency will be to believe them, just because the legacy system exists already and this somehow makes you more confident. If you are given analysis of the legacy system and it is incomplete, you will judge it less harshly than analysis for new yet to be written system. If your team tells you they can only test 10% of the features of your legacy system, you might say that is better than nothing, while a proposal to test “only” 10% coverage of a new system would be treated as a joke.   Legacy software is a little bit like a con artist! It puts down your guards and corrupts your sense of proportion. Worse than that, legacy software makes you feel bad when you think of some of its “legacy properties”, the result is like a Jedi mind shield, legacy software actually blocks you from thinking clearly by making you suffer when you look at it "truthfully "!

In a former job, as a development manager, I was continuously saying “no”, “no”, “no”… to the same question which was “can we rewrite it”. The thing is that rewriting legacy software when you have not yet accepted to look at it “truthfully” is often not going to work. So here are my recommendation on how approach you legacy software:
  • Never believe you can rewrite legacy software from scratch without keeping something. Maybe you are keeping a protocol, maybe you are keeping the business functionalities, maybe you are keeping the algorithm, but you need to keep something.
  • Keep what is most precious in your legacy software. Again, it might be the protocol, the business functionalities, .. but as you need to keep something (see above) you want to make sure that you keep what is most valuable. The more the better!
  • Make sure to deeply analyze what you have identified as valuable and that you want to keep in the legacy software. Analyze it to death, until it hurts!  That is because by default, you will shy away from doing this analysis, and that will cost you!  This is the really the number one rule with legacy software: do not let “it” convince you that you do not need to analyze what you will keep! (See dumbifying and con artist remark above).
  • Also, bring in new resources to provide fresh views on your legacy analysis. Again, by default the legacy will bias you, bring in someone that can look you problem and solution without prior experience with the legacy software.
Sometimes what you want to keep is deeply imbedded into your legacy software, for example as an implied data model or API. Do not hesitate to use “tricks” to extract it out at the lowest cost. For example you can extend your types to produce a trace info with each operation, to then analyze the trace files extract an implied structure.
Once I wanted to extract an implied model in C++, so I wrote a “hand crafted” C++ parser. It took about three days, I started with an equivalent yacc and lex and the first C++ file to parse, and proceeded to write the parser to parse exactly what I needed in the first file, extending the parser by running the parser over and over, line after line. When I got to the second file, I was already moving by a few lines at a time with each iteration, and by the end of the effort I had parsed the 50 or so files that had the data I needed.
Finally, do not forget the business: if you are in it to make a profit, never write code unless it is supporting your business strategy. This is probably the hardest part with legacy software: your business and marketing people usually have no idea how to approach your legacy rework. If you not careful your development will effectively hijack your legacy strategy and possibly lead you into business oblivion.

Wednesday, September 12, 2012

Its is buggy world

I would normally not want to talk about bugs, because they are a fact of life, but today two bug related issues happened. The first one was spending 30m with a developer who was stuck with bug. He had previously been asked to validate a series of assumptions, he had only partially done that, and it looked like we were making only little progress until in a post meeting cafeteria moment he mentioned that he was using yet another feature of an API, something that had never been mentioned before. That was the aha moment, and hopefully fix the bug.

The second bug was in code that yet again I was refactoing. You might remember I have a coroutine like functionality my F# super monad. I had taken that feature out to extend the monad and when I put it back in, one function failed to compile. As this is my typical late evening programming, I was watching TV, web surfing, and watching skateboarding on youtube. And suddenly what had been a routine exercise of coding became a pain and I ended up going to bed at midnight with an unhappy feeling. I'll admit I wondered if I was getting too old for this, no being able to do it! (i.e. programming).

So I started my day ready for action; I have three programming slots in one day: going to work, coming back and late evening. I missed the first slot: I missed my bus and was offered a drive. So it was only this early evening on the way home that I managed to get my laptop out and proceeded to carefully analyse what was happening. Like with all bugs this one was behaving illogically. But then I noted something even more strange: the compiler error was "talking to me"; It was giving the types "human names" even though I had no type anotations (see a previous post on why I try not to explicity declare signatures).  Was this a compiler bug? Sure enough, the function just above my problem had some old type anotations, so I reversed the order of declations and the bug was gone.

All this to say that when you are debugging you really need to apply a very rigid enumerative logic approach, and that yes, compiler bugs still happen, and that finally I am not getting too old for this type of fun.

Thursday, September 06, 2012

Where is my state?

People often associated functional programming with statelessness. The reality is that there is always a notion of state, functional programming is about stateful programming. The ambiguity is probably the result of state appearing in different ways than in a procedural style.

In a procedural style, state is associated to memory and you change it by using the writable property of memory.

In a functional style, state is still in memory  but the “classical” approach is to disallow yourself from changing it by writing again over memory. This is the immutable programming style. It goes with functional programming because it is very difficult to build on immutability without some functional programming concepts (I have tried!).  So if you have state and you cannot write over it, the only thing you can do is to read it, and if you want to change the state, you need to write a new memory location with your new state value (a non-functional  near immutable approach is to use arrays and grow them). The problem for beginners is “where should that new state be”?  It needs  to be accessable, and while you maybe knew how to access the old state, where should you put the new state so as to be able to work with it, instead of the old state?
Recursiveness is then the first  way  to manage state; The idea is that if you have a function F that works with state, then if you create a new version of the state you can simply call F recursively  with the new state, and again and again with each new version of the state.

Recursiveness is always somewhere nearby if you are using immutability; Yet you do not need to expose it to manage state. Again, suppose a function F uses state, have it return its state after use, if the state has not changed, then  return it unchanged, if the state changed, then return its new version. The trick is then to have a loop that calls F with the state, then takes the returned value to call F again, and so on. As this is immutable code, the loop is a small tail recursive function.

State can be big and heavy, and not for the wrong reasons, your application may be a big thing. The last thing you want to do is to copy that whole structure just to make a small change. The simple approach is to reuse parts of your old state to make a new state, changing only what needs to be changed. Yet  as the state is usually a tree like structure, if what is changing is deep in the original state, then it will cost you to create a new state because you will need to rebuild much of the hierarchy of the state. One way around this is to take the old state, encapsulate it in a “change of state” structure and use this as the new state  This approach is what I call a differential approach: the state change is the difference. By stacking differences you build a complete state. Usually the differential approach is combined with the “normal” partial copy approach. The reason is that if pursued forever, the “stack” of state changes become too big and starts both to consume memory and take time to access.

In the end your new state will either leave your function “onwards” into recursion or backwards by return, and often the same algorithm may use both methods.
When beginning with the functional style, state seems to be forever a pain to drag around. But with time patterns appear. First you may notice that not all states have the same lifetime, and they do not all have the same read write properties. It is a good idea to try to separate states that have different access and update properties.  You may then note that the hierarchy of functions aligns itself to with the access patterns of your state. Yet if you tie state to function in an layered like fashion that is classical to procedural programming you realize that what you have is not so much state as parameters. That is, while you are working in an inner function you have no ability to modify the outer “state”, so it is more a parameter like than state like. Nevertheless, with situation where states have a strict hierarchy, this approach is ok.

At some point, carrying around state becomes such a standardized routine that  it makes sense to look for help to go to the next level. And the next level is… monads. Here is the deal, I said before that states will either exit your function through a further recursion or by return.  That means that there is a set of compatible patterns that can be used to assemble little bits of code that work on state, so that they nicely fit and the state zips around as an argument or as a return value. This pattern appears naturally when your work with states: it is the idea that the state is passed as last argument and that the stateful functions always return a state. As you want the function to do something useful, you systematically return a tuplet with a state and a function specific return value. Before understanding monads,  I wrote whole applications like this. What monads allow you to do is to standardize so much this method of call pattern that it becomes possible to strip the exposed code of these “peripheral” state going in and state going out. The result is code that looks stateless but that isn’t.  The value of monadic state management is that you make much less mistakes and you can specialize your monad further to give it special properties. Still, from a stateful perspective the main gain of monadic states is productivity: hiding the state mean you can focus on more important things.

Saturday, August 11, 2012

The case of the mysterious distracting type class

I had a magical insight recently about something that was bugging me almost two years. The question was: why the hell was I so unproductive creating type classes!

So here is the deal: when you develop, you combine invariants and transformation properties of your  different pieces to make something new. If you are someone like me, you live your code, and that means that the relations between the different concepts throughout the code drive me "emotionaly". Yesterday, it occured to me that what is special with type classes is that they are "open".  They are open because their properties are not always obvious from the definitions. Worse, new type classes often have unambiguous properties.

I say open because by contrast if I avoid type classes by explictly defining type specific functions or if I create encapsulating types to replace the functional polymorphism by data variants, then I am in effect "closing" the design. With a "closed" design I have the whole design in my head and I "just know" what is best to do next. With an open design I find myself worrying about peripheral design issues which are mostly not so important,  yet I still worry about then... and that really hurts my productivity.

I will therefore adopt a much stricter approach to creating new type classes. The basic rule is NOT to create new type clases. Yes, I love to think about them, but I can't afford to. Then, when I have mentally built all the invariants with the explicit function calls or encapsulating types, and more importantly when I have finished what I was orginally trying to do, I can imagine refactoring and bringing in new type classes.

Friday, August 10, 2012

monadic folds

I have been rewriting my type inference framework over late evening sessions this week. I would not do it just for the refactoring, I want to be able to do more with it. Although I will not tell you why, and I can tell you how.

This is F# code, and my first version was monadic, modular, but flat. In some sense I was using the monadic abstraction as a comfort zone but without pushing it. Now I have "internalized" the concept and "objectified" it to the point where I am comfortable to program mostly without the "do" notation, composing higher order monadic constructions. And this brings me to my favorite pattern of the moment: the monadic fold.

Here is how it works:
  • the fold pattern is "joining" an operator accross a slice of data. In effect it is a universal "block" but which is "projected" against existing data, see each fold as an new inner hierarchy in your code.
  • the monadic environment supports the "environmental" needs of the operators
Building your specialized library of environments and operators is no easy. Yet the good news is that it easier with time.  The good news is that monads are like "single dimensional objects", and that is to say that if you have packed things in objects and you know how to be selective, then packing single concepts into monads is not too hard.

Friday, August 03, 2012

Easy to make money, hard to keep money, assymetric systems

A market making and brokerage firm, Knight Capital Group, lost all their money (~400M$) when their automated trading system went crazy and it seems repeatedly bought high and sold low, as fast as it could, across many instruments.
My guess is that they turned on a test app on the live market. This would explain why they were not feeding the trades into their portfolios.
An algo to buy the spread makes sense if it is used to act as a counterparty to test other algos such as a market making application.
Then again, a test app normally has hard coded detections of the live market that turn it off. So who knows?!

This event is interesting in a few different ways:
  • What fail-safe code do you want to include in your algo system?
  • What fail-safe development management process do you implement to try to guarantee that those failsafe coding standards are met?
  • Trading is really about keeping your profits, not about making them!
  • Organic systems are assymetric, their behaviors change with the change of direction of flows!
My fail-safe coding rules are the following:
  • Do hard code constants! For example, test applicaiton includes production IP addresses and will refuse to run on these.
  • Too much of a good thing is a bad thing: automatically disable strategies  that have triggered "too many times".
  • Four eyes: have important code reviewed by someone else. 
Fail-safe development rules are much harder to implement:
  • Keep developer that worry about overall consistency. Developers that forget things are expensive.
  • Encourage your developer to expose and maintain a clean stateful design for your algo system.
(to be continued)

Wednesday, July 18, 2012

Managing software development

This blog entry is more than of a little essay than a blog posting, the question I am trying to answer is what are my key learnings in managing software development.
Software development management is about achieving business success, as well as team and individual success, by leveraging skills and process while focusing on productivity and learning. I know this because I have experienced it and applied it as developer, architect, product owner, project management, scrum master, development manager and CTO.
Software development is a self-sustained process: much of what needs to be done depends on what has been done, much of future productivity depends on existing team culture and past investments. It is when starting up greenfield software projects that this “chicken and egg” nature of development is especially noticeable, developers starting in completely new domains are often acutely aware of what they do not know. They may not know, for example, who are their users, what are their requirements, what architecture to use, how they will test their deliverable, or which third party technology to use.  The team itself may not know how to work together, nor how to best invest their efforts to keep productivity high. The sad story is that many development projects fail because they go too far into these “uncharted territories of knowledge. The good story is that applying a good process with enough experienced people will help you stay away from taking unmanageable risk.
Good developers are naturally top of my list of what makes a good development team; Developers bring dexterity of “thinking in code” and also bring broader application and work culture experience with them. You would be surprised how much bad experience a developer can acquire, so beware how you build your team. My experience is that observing a developer working out a solution gives you the best insight of the technical skills they have acquired, as well as their cultural skills; Therefore to hire developers, I prefer case interviews with real coding tasks, while making sure that there is enough people presence and open items to push the candidates to interact and show their soft skills.
Most developers are grown up enough to manage themselves… most of the time. Therefore, although I have run waterfall processes, I really recommend agile development because it gives developers their independence while regularly confronting them with the realities of “the real world”.  A key role to help them in this process of “confronting the facts”, is the requirement manager (e.g. product owner). He/she brings analytical domain expertise but also the product evangelism that aligns the development team around the product features. Getting people to agree is much easier when they already agree about something; Therefore, having your teams agree with your requirement managers makes it much easier for them to agree “together” on how they will design, build and test software.  My experience, as product owner in a high frequency derivatives trading environment, is that the biggest challenge in product management is aligning commercial expectations and development throughput. That brings up the subject of a development’s team productivity and good places to look for it are in architecture, QA  and build management.
Architecture, quality assurance, and build management are magical ingredients to software development: get them right and you rarely need to revisit them, but get them wrong, and they will drag you through hell, especially hell of lost productivity, productivity that often makes the difference between success and failure. One thing these three development processes have in common is that they are not needed to start development. Another is that they are best managed as a “coordinated” solution across development, and not as an assembly of different solutions: you want to have one build management, one architecture, one notion of QA. That does not necessarily mean one “uniform” solution, good solutions often have a granularity of diversity, but there is an optimal balance of overall cohesion versus local choices. And that is what make these particularly tricky to manage, both as a team effort and as part of development management.
There are two key understanding in team management, independently whether it is a top-down style of management or a self-managed team: The first is very tight relation between what can be changed and what should not be changed, the second is the importance of sharing knowledge and beliefs among all team members.  I will not dig deeper in the importance of sharing, there is no notion of team without it, yet first is crucial to understand how to manage cross team activities, especially with regards to productivity.  The reasoning is as follows:
  • To improve productivity it makes sense to change things; It makes sense that you look for a better architecture (e.g. refactor designs), a better build system (e.g. bring in a new tool), a better test coverage (e.g. new tests).
  • Yet making changes makes less sense when near optimum productivity.
The consequence of this is that your number one mission as a development manager is to help your teams track their productivity, stay aligned around their commitments (e.g. encourage discipline and hard work!), and that they maintain a common  and realistic notion of “optimality”. Achieve this, and your development team will know what to change and when to make changes. They will decide if it is worth investing to improve the architecture, the build system or the QA.
Pain point in all of this is achieving that REALISTIC notion of optimality, and getting agreement around a “discount factor” for sustainability, no everything should be 100% sustainable. Give your requirement management the mission to evangelize all with their product oriented view of realism. Getting the right balance in what is sustainable and not really depends on your budget!