Tuesday, May 31, 2011

Moving to Haskell (from F#)

I am a bit busy these days helping out with architecture and agile development for a large ticker plant and calculation farm. Therefore when I sit down to program (using my slack time!), it needs to deliver bang for the buck. I have my sturdy little Dell X1 laptop, visual studio 2010 and F#; and although I have suffered by F# limitations, I have not been blocked. Yet now I am stuck, my needs are getting seriously heavy and the F# type system cannot cope. One issue is that monadic programming style in F# needs to be defined with the help of a class and this class needs explicit types, and the other issue is that there is no explicit way to define type class, or types with localized polymorphic definitions. In a non monadic framework I have been bootstrapping my polymorphic types on start-up: The trick is to wrap all functions of a module in to one big function whose arguments are the functions that constrain the types as would the parameters to a type class; The big function returns one big tuplet or record with all the functions of the module that have now been “bound” to a type structure (you can still have some unbound types to cover the localized polymorphic needs). The application will then first bind all modules as functions, the “main” then proceeds with the bound functions. Yet now I am stuck as the monadic programming style need the use of explicit type variable and this does not work with the trick above. I could write all my monadic code “raw”, as functions. But that beats the purpose of the abstraction. Therefore…

I am ditching F# as a core language; I’ll generate it when I want .net compatibility with the advantage that it is a functional language and can hope that the compiler will do a good job with it. In search of a replacement, I see only three candidates: ocaml, Scala, and Haskell. I have always liked ocaml, but I feel the support for it diminishing. I see Scala as F#; better in some ways, worse in others, it is only a candidate for those that are forced to stay in a Java and JVM environment. Therefore Haskell it will be by elimination.

Haskell is easy to install on Windows (XP on my old laptop):
  • Install a minimum Cygwin to have the grep function.
  • Install Haskell (HaskellPlatform)
  • Install Leksah (a Haskell IDE)
I’ll tell you what I like and dislike about Haskell in a future post.

Thursday, May 19, 2011

delivery focus versus creative slack

I made a presentation on scrum today in which I first presented a generic agile framework, to then present scrum within that agile framework. Although I had not planned it, I found myself emphasizing the need to focus on delivery in order to make time available, what I call slack time, to be creative and bring your development up to a new level. I made the mistake to compare this to how you run a hedge fund, and although there are mathematical links between the management process and product development process and how to maximize a portfolio, I think that it was lost on my public of software developers. Still the reality is that you are squeezed by your boss, or by your investors, or even by the market and you cannot afford to waste time, therefore you must maximize the effort to sustain your current growth model; That is the notion of focusing on delivery and on your client. And yet you cannot fully keep that focus, you need space to tackle the hard problems, you need time to reinvent yourself; That is the notion of making space for slack time. Make sure you accumulate enough of it and then invest it wisely. Your focus needs to be breakthrough, nothing else; keep your "normal" efforts on delivery effort. For the slack time you need to try to be extraordinary because it will change you or your company and may well give you a chance of new futures that would not exist without that special effort.

Friday, May 13, 2011

Bad scrum, good scrum and trust

Years ago I switched my development team to scrum. Or at least what I thought was scrum: we had teams, product owners, we carefully followed the sprint and stand-up process. Then we carefully sized our stories and we carefully worked to produce average work. of decent quality. It was scrum, but now I would call it "bad scrum".
It was bad because we had built ourselves a straight jacket, and none of us dared to step out of line with the risk of blowing the sprint. Arguably, there were attenuating circumstances: the teams were walking on eggs, working on a large C++ program that few us understood. Still I have seen this behavior elsewhere. It is a behavior of risk aversion, sustained by worries of the whole team.

So you might ask, "What is then good scrum"? Good scrum is when you follow the process but stop worrying about it.But most importantly, good scrum is when you have managed to build a liberating trust among all members of the team. All too often developers are suspicious of other people's work, or worse they have the "not invented here syndrome" where they need to make their mark in the code or the design for it to be acceptable. Strangely enough this type of behavior is more destructive in an agile process than a waterfall process! In an agile process like scrum, a lack of trust will amplify the risk aversion of the team, and this will kill your productivity. Remarkably, the notion of trust is rarely mentioned and yet it is fundamental. Take the size of a scrum team; Some people say 5 members, some say 7 or 9, or even 11 in certain domains; And yet you could even have a scrum team size of 21... if you manage to get the whole team to trust each other! A team of 5 often works better than a team of 9, simply because it is easier to build trust among 5 than among 9!

So you might ask me, how to build trust? If you are a crazy person like me, you have spent years building mathematical models of processes and people, you see trust as a process. But if you are not, then the simple rule is to show others that you trust them. You need to show it with your words but also in your actions. Then, if you are the scrum master, be a trust policeman! Anytime you see someone behaving in a manner that goes against trust, you stop them gently and bring the conversation back to trusting relation. Finally, you eliminate people who really cannot play the game, or if they are "the chosen ones" you ring fence them, which usually means you give them a babysitter to keep them from damaging trust relations (a painful job).

Having said all this, I am sure you have heard of catastrophe theory, which can visually be seen as moving from a continuous regime to another as a single event. Here is the insight: what do you think happens when you over trust your scrum team, and they over trust themselves? They can hit a wall, fall completely apart, and lose their trust (and yours too!). The magic then of a the good manager is to understand how complex the projects you give to your teams can be; And that my friend, is a very difficult task!

Arduino and the joys of electrical engineering

I have purposely stayed away from my kid's Arduino kit in order to let it be “their” toy. But the other day I was forced to give in when I found my daughter controlling a servo motor with the Arduino. Searching the web I see that I can order a 2d gyroscope and 3d accelerator sensor board for about 40$ that would be easy to interface with the Arduino. Now I am dreaming of using this sensor and the Arduino as a fly by wire layer between a remote control receiver and servos, to control a car or airplane

(Note: I lost this post and the last one. So if they appear twice it is because both I and blogger has brought them back)

Immutable, mutable, higher order designs and monads


One of the great pleasures of programming functionally is to be able to rely on immutable data. With immutability, side effects need to be explicitly defined, either as return values from functions or as encapsulations in monadic states. Immutability allows two important properties: first it increases productivity by diminishing what needs to be cared about (freeing the mind to care about the right things!), then, almost as more importantly, it provides the foundation for higher order programming concepts by preserving an acyclic relationship between functions and data. By higher order, I mean that data is replaced by functions or that functions encapsulated in data; for example, in a context like data differentiation or in the use of monads. By acyclic, I mean that all trace trees are acyclic (trace tree can be seen as dynamic call graph where recursion is not a loop). Let me explain…
It is OK to have cycles in the static call graph. That is because we know how to infer invariants (e.g. like types) from inter-dependent and recursive functions. Likewise, recursive data types are also OK. Nevertheless everything is a bit trickier if we look at things from a data flow dependency point of view. This is the view we need to take if we really want understand how the functions are working on data and producing new data. And the sad reality is that it is generally hard to work out invariants from dependency graphs that have cycles because of mutable data, Nevertheless, there are cyclic dependency graphs that are easier to work with than others. More specifically, the more local, or of limited life span, a part of the dependency graph is, the easier it is to understand it scope and properties.
I mentioned trace trees above because they give me the bridge I need to wrap up this posting. The insight is the following: mutability brings in cycles in dependencies, cycles in dependencies kill invariants that allow higher order code constructions, trace trees are a down to earth way to work with dependencies, monads can seen as redefining call points in the trace tree (nodes), and as a consequence one the safest way to introduce mutability is within the scope of a monad. Again, we know the local trace tree will end up with cycles when we introduce mutability, but if this trace tree is “protected” in a monad, then its impact into the rest of the trace tree is easier to understand because of the intimate relation between monads and trace tree nodes.
You could argue that we do not need mutability. Yet real world I/O is mutable and therefore you you cannot escape working with mutable data. Not to mention that to get performance you may need to go mutable too.

Monday, May 02, 2011

What to recommend to a young quant

I share with you my recommendation to a "young quant" that wrote me asking for leads:

...

A tricky question. As with any industry, there is a certain “commoditization” of derivative technology including quant work, that plus the uncertain future of the US regulations on derivatives, and to add to that a bleak economic outlook in Europe and the US. The grass is not necessarily greener on the other side and many parts of the market are in a squeeze.

Having said that, there are still smart people out there making money and there is still a lot of new development in Asia. My recommendation is not to look for change of activity, but to get closer to successful firms. I know that is easier said than done. Still, one idea is that you are better off accepting a low pay to be closer to success than a higher pay that keeps you out of the fast lane. Again, I know this is not easy. And worse than that, you are in competition with many others.

The best I can do is to give you the following advice: I have looked at your profile and it is too much like “another quant”. Further more, you admit you do not have “...” experience by saying that you have strong interest in it. Therefore my recommendation is to try to stretch your resume in new “original” directions that will make it stand out. Ideally you should back this up with personal work. My gut feeling would be to add more software development experience: if I hire a quant I want him also to be good programmer to get quick turnaround times. You say nothing about your development skills. Spend time to work on these skills if you do not have enough of them. The second concept would be to be more exotic with your choice of quant projects: think of the senior quant that reads your resume and give him something to think about. So go buy a book on approximate dynamic programming or spectral methods in stochastic computation, do a little development at home and add it to your resume.

...