Personalize your containers (in F#)

Many years ago I implemented an augmented tree structure in C++. Inspired, I decided to implement the structure in F#. FYI by augmentation I mean storing extra data at each node of the tree that will accelerate the later operations on the tree. In DB terms, augmentations are typically used include secondary key conditions in your tree query. This is what I did:

1. Take a balanced tree implementation off the web. I was thinking finger tree but then though better to start with a simpler implementation. I chose an AVL tree. (I may revert the code to a red and black tree to simplify).
2. Add a node field to store the augmentation (a generic type)
3. Refactor the code by replacing the call of the node constructor by a call to a function that will first build an augmentation and then call the node constructor
4. Pass the generic augmentation "maker" function as argument to the function calls

On this last point I wasted much time. Initialy, I tried to encapsulate the "augmentation" api with abstract methods on one the the types. As all this code is polymorphic the compiler barfed with "would leave scope" and this type of error. After a bit too much time, I reverted to only define API signatures are with functions, no methods. (I understood later that this is mostly true for all languages: stick to functions/lambda constructions because they always scale). Then I wrote a generic select like operation that uses augmentations. And finally a "general" map operation: it uses the augmentations to optimize its search, it only transforms nodes that have the selected property AND it allows nodes to be deleted. There are still a few more things to do to wrap this code up but I again can only admire the speed at which one can write functional style. I'd love to tell you why you want to have these types of data structures but not everything can be for free.

Flash crash 2010 thinking

An old friend asked me if the flash crash had been caused by "spoofing" of the exchanges with very large flows of order entry/order cancel commands. My feeling was that this was not the case and a quick search on google picked up a confirmation in this article: The Microstructure of the ‘Flash Crash’: Flow Toxicity, Liquidity Crashes and the Probability of Informed Trading” This article supports the explanation that the main cause of the flash crash was simply the lack of liquidity because the market makers had become too edgy due to too much new information coming in to the market. It is an excellent article and brings back the notion that some systems are only sustainable with enough "friction", or said differently, by being under-optimal with regards to certain ways to look at them. When a group of people need to work together, they nominate a chief, or at least they give specific roles and ownerships to members of the group. These people get an "unfair" advantage as their ownership will most often provide benefits, even if they is not trying to use them for their own gain. Ideally for the group, processes are in place that limit personal advantages. And these advantages cannot be completely removed as that would take away the leverage that allows the overall process to happen AND BE SUSTAINABLE! There are real challenges with these types of "social process": If people remove all their leadership they end up in chaos, and yet people are often upset with some of the unfair advantages that the leaders have because of their role's functions. Market makers that cannot make a profit on the bid and ask spread must make a profit on "another spread". This can be a volatility spread, or more generally on an "information spread". Information spread is, for example, "I don't know and you don't know", going to "Now I know, and you don't"; Or it could be "I expect parameter P3 to change in my model, you don't". When a market maker can no longer find a profit making spread, he will limit his exposure; especially when this exposure leads to continuous losses caused by others having their own "better" information spread (often at the limit of inside trading). The sad conclusion is that market makers become less of a sure thing when rules that diminish their "advantage" are imposed by the exchanges, the SEC or the ESMA. Mathematically you can see this as moving from a continuous to a non-continuous regime. Call it a "percolation" of the market making process past a critical level. The big problem is that in the non-continuous regime, one where parts of the markets simply stop, as no prices are quoted, is "a nightmare". And many people simply ignore it as "impossible". The truth is that it is much more a possibility now than it ever was and that seems to be what the flash crash was all about.

Types versus architecture

There are three invariants in a software system:

• Code
• Types
• The architecture

One old rule is: more types, less code. This is because the less "sophisticated" type provides less "support" to the code, and therefore you need more code to keep things in control. Another way to put this is that the invariants provided by the use of more types allows the code to be more simple.
For example, long ago I experienced this type of code simplification with the use of the record/struct moving to C and Pascal from early basic and FORTRAN (and assembly). And again with classes and objects moving to C++. And again with variant types and later monads with functional programming.

Another rule is: better architecture, less code. I have repeated this mantra for many years and yet thinking about it, I am not even very sure I have a good definition for architecture. I use to say: 'architecture is what does not change in the system'. Now maybe I should say: 'it's what does not change and what is not a type'. This then begs the question: 'If I can capture more of my system in its types, does that make the architecture more simple'?

I am starting to get the feeling that architecture is a little bit like dynamic typing: you use it when you do not know better. I am tempted to try to introduce a new rule: better typing, less architecture". I will see if I can make that rule stick!