Delivery focus versus creative slack

I made a presentation on scrum today in which I first presented a generic agile framework, to then present scrum within that agile framework. Although I had not planned it, I found myself emphasizing the need to focus on delivery in order to make time available, what I call slack time, to be creative and bring your development up to a new level. One can compare agility and running hedge funds: management and product development processes share much with how one maximizes a portfolio. The key learning is that you cannot afford to waste time when you are squeezed by your boss, by your investors, or even by the market! Agility from this view is much about maximizing your productivity while reserving time to ensure growth. That is the notion of focusing on delivery and on your client, and yet still have time (and space) to tackle the hard problems! You need time to reinvent yourself as you progress. That is the notion of making space for slack time. Make sure you accumulate enough of it and then invest it wisely. Your focus needs to be breakthrough, nothing else; keep your "normal" efforts on delivery effort. For the slack time you need to try to be extraordinary because it will change you or your company and may well give you a chance of new futures that would not exist without that special effort.

Bad scrum, good scrum and trust

Years ago I switched my development team to scrum. Or at least what I thought was scrum: we had teams, product owners, we carefully followed the sprint and stand-up process. Then we carefully sized our stories and we carefully worked to produce average work. of decent quality. It was scrum, but now I would call it "bad scrum".
It was bad because we had built ourselves a straight jacket, and none of us dared to step out of line with the risk of blowing the sprint. Arguably, there were attenuating circumstances: the teams were walking on eggs, working on a large C++ program that few us understood. Still I have seen this behavior elsewhere. It is a behavior of risk aversion, sustained by worries of the whole team.

So you might ask, "What is then good scrum"? Good scrum is when you follow the process but stop worrying about it.But most importantly, good scrum is when you have managed to build a liberating trust among all members of the team. All too often developers are suspicious of other people's work, or worse they have the "not invented here syndrome" where they need to make their mark in the code or the design for it to be acceptable. Strangely enough this type of behavior is more destructive in an agile process than a waterfall process! In an agile process like scrum, a lack of trust will amplify the risk aversion of the team, and this will kill your productivity. Remarkably, the notion of trust is rarely mentioned and yet it is fundamental. Take the size of a scrum team; Some people say 5 members, some say 7 or 9, or even 11 in certain domains; And yet you could even have a scrum team size of 21... if you manage to get the whole team to trust each other! A team of 5 often works better than a team of 9, simply because it is easier to build trust among 5 than among 9!

So you might ask me, how to build trust? If you are like me, you have spent years building mathematical models of processes and people, you see trust as a process. But if you are not, then the simple rule is to show others that you trust them. You need to show it with your words but also in your actions. Then, if you are the scrum master, be a trust policeman! Anytime you see someone behaving in a manner that goes against trust, you stop them gently and bring the conversation back to trusting relation. Finally, you eliminate people who really cannot play the game, or if they are "the chosen ones" you ring fence them, which usually means you give them a babysitter to keep them from damaging trust relations (a painful job).

Having said all this, I am sure you have heard of catastrophe theory, which can visually be seen as moving from a continuous regime to another as a single event. Here is the insight: what do you think happens when you over trust your scrum team, and they over trust themselves? They can hit a wall, fall completely apart, and lose their trust (and yours too!). The magic then of a the good manager is to understand how complex the projects you give to your teams can be; And that my friend, is a very difficult task!

Immutable, mutable, higher order designs and monads

One of the great pleasures of programming functionally is to be able to rely on immutable data. With immutability, side effects need to be explicitly defined, either as return values from functions or as encapsulations in monadic states. Immutability allows two important properties: first it increases productivity by diminishing what needs to be cared about (freeing the mind to care about the right things!), then, almost as more importantly, it provides the foundation for higher order programming concepts by preserving an acyclic relationship between functions and data. By higher order, I mean that data is replaced by functions or that functions encapsulated in data; for example, in a context like data differentiation or in the use of monads. By acyclic, I mean that all trace trees are acyclic (trace tree can be seen as dynamic call graph where recursion is not a loop). Let me explain…

It is OK to have cycles in the static call graph. That is because we know how to infer invariants (e.g. like types) from inter-dependent and recursive functions. Likewise, recursive data types are also OK. Nevertheless everything is a bit trickier if we look at things from a data flow dependency point of view. This is the view we need to take if we really want understand how the functions are working on data and producing new data. And the sad reality is that it is generally hard to work out invariants from dependency graphs that have cycles because of mutable data, Nevertheless, there are cyclic dependency graphs that are easier to work with than others. More specifically, the more local, or of limited life span, a part of the dependency graph is, the easier it is to understand it scope and properties.

I mentioned trace trees above because they give me the bridge I need to wrap up this posting. The insight is the following: mutability brings in cycles in dependencies, cycles in dependencies kill invariants that allow higher order code constructions, trace trees are a down to earth way to work with dependencies, monads can seen as redefining call points in the trace tree (nodes), and as a consequence one the safest way to introduce mutability is within the scope of a monad. Again, we know the local trace tree will end up with cycles when we introduce mutability, but if this trace tree is “protected” in a monad, then its impact into the rest of the trace tree is easier to understand because of the intimate relation between monads and trace tree nodes.

You could argue that we do not need mutability. Yet real world I/O is mutable and therefore you you cannot escape working with mutable data. Not to mention that to get performance you may need to go mutable too.

What to recommend to a young quant

I share with you my recommendation to a "young quant" that wrote me asking for leads:

...

A tricky question. As with any industry, there is a certain “commoditization” of derivative technology including quant work, that plus the uncertain future of the US regulations on derivatives, and to add to that a bleak economic outlook in Europe and the US. The grass is not necessarily greener on the other side and many parts of the market are in a squeeze.

Having said that, there are still smart people out there making money and there is still a lot of new development in Asia. My recommendation is not to look for change of activity, but to get closer to successful firms. I know that is easier said than done. Still, one idea is that you are better off accepting a low pay to be closer to success than a higher pay that keeps you out of the fast lane. Again, I know this is not easy. And worse than that, you are in competition with many others.

The best I can do is to give you the following advice: I have looked at your profile and it is too much like “another quant”. Further more, you admit you do not have “...” experience by saying that you have strong interest in it. Therefore my recommendation is to try to stretch your resume in new “original” directions that will make it stand out. Ideally you should back this up with personal work. My gut feeling would be to add more software development experience: if I hire a quant I want him also to be good programmer to get quick turnaround times. You say nothing about your development skills. Spend time to work on these skills if you do not have enough of them. The second concept would be to be more exotic with your choice of quant projects: think of the senior quant that reads your resume and give him something to think about. So go buy a book on approximate dynamic programming or spectral methods in stochastic computation, do a little development at home and add it to your resume.

...