Stick man in haskell

Today some feel good code in Haskell:

man :: Diagram B
man =
        circle 1.2 # center
    ===
        ((rotateBy (-0.1) (vrule 3)) ||| (vrule 3) ||| (rotateBy 0.1 (vrule 3))) # center
    ===
        ((rotateBy (-0.1) (vrule 3)) ||| (rotateBy 0.1 (vrule 3))) # center

  which produces the following svg stick man:

You need to call it like this:

main = mainWith $ man

Flash crash revisited

If you try to boil water in pyrex glass container, you may well see the water's temperature go past boiling point (100c/212F), because the smoothness of the glass is not providing any starting point for the boiling to start. Then if you take a wooden spoon and drop it in to the water... the boiling starts in an explosif manner, triggered by the spoon.

Would it be right to say that the spoon caused the water to boil? Yes.

Would it be right to say that heating the water caused it to boil? Also yes!

Would it be right to say that choosing a glass container caused the water to boil explosively? Yes.

Would it be right to say that dropping the spoon in caused the water to boil explosively? Also yes!

The analogy here is that the wooden spoon is a single trader. Take lots of spoons, the first one that touches that water will cause it to boil.

Would it be right to say that there was one trader that caused the flash crash? Just as much as that first wooden spoon makes that water boil.

Should this trader be prosecuted?

Well they did burn Joan of Arc...

Unbearable truths

I found myself explaining to my daughter that people will go a long way to explain why there is value in the products they have purchased, especialy relative to products they have not aquired.

The hard reality is that to invest oneself is to blind oneself.  What happens is that because personal investments set us up to lose under certain scenarios, we will naturally think of how to avoid these scenarios. The thing is, having worked out how to minize loss, we have usually built a whole reasoning on what can happen, and what will not happen. The last thing that we want then is to be proven wrong. It is this avoidance to be proven wrong (among others) that blocks us from revisiting certain scenarios. Indirectly, what happens is that we set up a subconscious barrier as an effort not to be proven wrong.

This is very much black swan territory: events that do not happen often and surprize us in a bad way. But it is actually more than that, it is about being careful not to accept just any solution. Some solutions will kill you! You take them, you explain to yourself why you took them, and by doing so you have locked yourself into a thinking pattern that may harm you.

We cannot reject all because of their futur risks. We must accept new solutions when we can. Still, there are two ways to avoid disasters. First, listen to your gut feeling: if something feels wrong, then it might be wrong, and then better avoid it as a solution. Secondly, listen to your gut feeling: if something feels odd or out of place, then explain it. If you cannot, then one of your solutions may well be wrong and needs revisiting.

One last remark, work hard! It is the hard work that creats those "gut feelings"!

About making bad decisions

This video by Sydney Finkelstein is excellent. Here is the link:

Think Again: Why Good Leaders Make Bad Decisions and How to Keep it from Happening to You

Mentioned are four red flags in decision-making:

• Are your personal experiences misleading you?
• Is your personal self-interest clouding your thinking?
• Have you made a dangerous pre-judgment that you are locked into?
• Are inappropriate attachments pushing you in the wrong direction?

These are very good!

The tricky part is that to really understand them, you need need to have traveled the multiple sides of the human psyche, which many managers have not.

All original content copyright James Litsios, 2015.

Is order spoofing ok?

Spoofing is the act of generating orders to buy or sell something on an electronic market, but to then immediately cancel these orders in order not to trade. Spoofing is used to generate a burst of activity with the hope that it will cause other algorithmic trading systems to react in a sub-optimal manner, allowing the spoofer to follow through with profit making trading activity. Spoofing happens when you enter orders with no intention of letting those orders trade. Which is different from entering orders in to a market and to then cancel them because you have changed your mind. There is no change of mind in spoofing, and therefore it might be seen as a form of deception. Spoofing works because other trading systems are waiting to see certain market prices before they change their behavior. Spoofing can also slow down the market as it may overload the exchange’s network or matching engine, such a slowdown may put at disadvantage other market participants. The question is: is spoofing ok?

Deception is never far from risk taking in business. Taking risks implies building a mental abstraction that works out the pros and cons of future scenarios, and that concludes that certain actions are more profitable than others. Sharing this understanding with others, might well affect negatively your future outcomes. A first form of deception is to avoid public debate around subjects that would develop information that would be against your interests.

Is business deception morally wrong?

The simple answer is yes, if only because I am unhappy when something I have purchased breaks because of its "fragile design". This emotional state is a sign that a part of me thinks that these companies are behaving badly. Yet the full answer is less obvious.

All companies that take risks are potentially being deceptive. Again the issue is that minimizing risk and maximizing profit leads to minimizing information disclosure. And keeping information “quiet” is a form of deception. The issue is that business is not possible without taking risks or making profits. In fact the whole economy is built on embracing risk and profit, and therefore also built on embracing certain forms of deception.

That our way of life leads to deception and that deception can hurt people, is a reality we would be happier without. And yet as the issue does not go away, people have found ways to deal with it. When confronted with deception, people will either:

• Ignore it, and act as it does not exist. 
• Try to keep it from happening by introducing rules and legislation. 
• Accept it as fact of life. 
• Work around it.
• Find flaws in the method of deception, and take advantage of these flaws. 
• Join in, and deceive others in a likewise manner.

The magic of human beings is that for each “deception situation”, each of us will take one of these approaches, and this choice will be based on personal values. We will either take the escapist approach of ignoring the deception, or take the constraining approach of trying to limit future deception, or take the liberal, somewhat darwinistic, approach of accepting the deception as “a natural part of the system”, or when possible work around or take advantage of the deception, or finally become a deceiver ourselves. Note that the constraining approach has two variants:

• Rules or laws are set up that disallow a form of deception. 
• Rules or laws are set up that limit behavior with the goal to limit the ability of deception. 

Given these remarks, what should be done about spoofing in financial markets? Exchanges prefer either to ignore spoofing or to ban it by disallowing spoofing as a form of deception. I think that is wrong, that spoofing should be allowed, and that throttling mechanism within the market APIs are the best way to deal with exchange overloading causes by trading behavior. Here is my reasoning.

Market prices are the result of equilibrium that happens between buyers and sellers. It is the outcome of competing ideas, competing beliefs, competing interest, competing focuses, competing technologies, etc. Someone or an algorithm may want to buy, others and other systems want to sell, they are all taking risks, often thinking differently, and the outcome is market activity.

Spoofing is an ultra-short-term strategy that provides a healthy counter balance to strategies that are based on speed, and only on speed. The thing is, among all those traders and trading machines, are those that base their decision making more on the current market prices than on real world activity. These market participants are taking short-term risks, risks that often lead to profits because of their extreme speed. These are profits taken from others that are competing on the short term, and also profits taken from those are taking longer-term view. In this jungle of “eat or be eaten”, it is perfectly right that spoofing makes profits by disrupting short-term strategies.

Spoofing that makes a profit by physically overloading the exchange’s matching engines is wrong. It is wrong because it puts the whole exchanges at risk, both in its operational integrity, as in the ability for the exchange to be master of its rules and regulations. It is not right to allow spoofers to purposely affect the processing behavior of other participant’s orders. For example, by purposely causing the exchange to slow down and order queues to build up. Yet here again we are confronted with the difficulty to distinguish between behaviors that are purposely trying to kill exchange performance, and behaviors that are directly business driven, yet end up impacting exchange performance because of the amount of trading activity they generate. Therefore I agree that exchanges should introduce constraints that protect their networks and execution engines, but care must be taken to how this is done. As mentioned above, two approaches are possible: One can simply “outlaw” spoofing, and hope that it is not done, and if done, hope that it will not harm exchange infrastructure; Or, one can change things, to keep spoofing from impacting on the exchange’s execution. Throttling dangerously high trading activity is the natural way to approach this problem.  It is perfectly reasonable that exchanges limit trading activities that disrupts exchange integrity be implementing throttling within the trading APIs. Therefore, it is is right for the exchanges to configure API  throttling limits to limit spoofing activity that would impact too strongly on the exchange's network or matching engine. It is wrong for exchanges to simply ban spoofing, as that would give an unfair advantage to participants that apply trading strategies that are hurt by spoofing. 

Ps: Thanks to Adrian for remarking that deception may lead to deception.

All original content copyright James Litsios, 2015.

Software designs that grow with monads, comonads, and type compatibility

This post is about designing software with internal and external APIs that are robust to future changes. It is therefore about API compatibility, but more importantly it is about the compatibility of a full software design to changes. Not surprisingly, monads and comonads are part of the presented solution, as is a careful approach to use of types.

I had a "aha" moment last year when I watched a video (by Timothy Baldridge)

that showed how an AST for a Clojure compiler was fully based on key-value pairs (nested hash maps), therefore without typed structures nor classes, and was doing very well. The thing is, I have suffered enough times to get the types of the different phases of a compiler to fit together. Therefor the idea of giving up on types and just using key-value pairs, that can easily be added for each compiler phase, seemed really to be an attractive way to write a compiler.

Key-value pairs, duck typing, and many "traditional" functional patterns (think lisp, not Haskell) have all in common their reliance on generic, almost typeless, structures. So while each "atomic element" of these patterns has a type (e.g. int, string, ...), the structures (e.g. struct, list, array, ...) are all generically typed.

Types are what capture the invariants of a design. Giving up on types is in effect giving up on capturing those invariants in a robust manner. Using types normally leads to higher quality software, yet with enough complexity, types no longer capture all the design invariants of a program. Worse, sometime types actually hinder the design by holding it back because they are not powerful enough to capture the desired invariant. This is the case with the difficulty to design an typed AST that "fits" all phases of a compiler. This rigid nature of types is also the hidden problem of API compatibility.

The invariants of APIs are captured with the types of their signatures. When new features or corrections are added to an API, these invariants and associated types evolve. When APIs link different software projects, changing API types is where API compatibility becomes an issue: APIs remain compatible when types change but remain compatible with older types, APIs become incompatible when the types are no longer compatible. An example of type compatibility in OO, is to derive a new class from an existing class, and to add a constructor in this new class from the old class. Unfortunately, that last example is at the source level. At the binary level, compatibility to type change is usually nonexistent, especially when focusing on forward compatibility. Note that API compatibility is not only about types: an API will also become incompatible when the interpretation given to values transferred over the API changes. Therefore to remain compatible, an API can add new value interpretations but must also ensure that past value interpretations never change.

Serializing data for transport and for storage is much about breaking program defined types into language standard basic types, and keeping a strict discipline of interpretation to ensure compatibility. Therefore ensuring both backward and forward compatibility of an API is to maintain a strict growth of value interpretation and to use serializing packages like Protocol Buffers or Thrift. The question is then: how do we ensure the future growth of a complete software design, not just a single API, but a family of APIs? Just like with the single API, the answer also lies in the notion of serialization. The goal is to stitch together the typed lower level design with a higher level untyped design that can change over time.

Data is serialized by walking through it a certain order and breaking it down into its primitive types. Programs can be broken down the same way. Yet to do that, your first need to adopt a functional programming style because it is hard to serialize procedural constructions. In a functional style, only functions need to be "broken down".

In the good old days, scalable software design was about using construction such as data schemas, object patterns, futures, continuation style, etc. Data schemas are still good, but all these other program constructions elements must be revisited with the fact that they can all be implemented with monads and comonads. More importantly, they must be revisited because the bind and cobind operator (and other monadic and comonadic operators) is what serializes functions! Therefore, just like you must serialize your  data schema to ensure future data compatibility, you must "serialize" your functions with monads and comonads to ensure future design compatibility.

Here are few examples of existing designs that do this:

• Injection frameworks are comonadic constructions. 
• Transactional frameworks are monadic constructions.
• Parallel tasks are monadic constructions.
• Embedded queries are monadic constructions.
• Reactive streams are monadic constructions.
• Lenses are comonadic constructions.
• Automatic differentiation (computing analytical derivatives of numerical code) are both monadic (forward differentiation) and comonadic (backward differentiation).

Just like data compatibility is tied to the order in which data is traversed, future design compatibility is tied to the "order" of function serialization. That is to say that each software design is strongly defined by the order in which functions are broken into monadic and comonadic constructions. While monads and comonads have a duality relationship, they fundamentally differ in "character": monads are "trace" centric, while comonads are environment centric. Said differently, conditional expressions can be monadic, while comonadic expressions can abstract away their environment. A design is then a chosen hierarchy of monads and comonads with chosen set of API extension points.

Just like with data compatibility, future design compatibility is tied to the amount in which types can be changed and remain compatible. And again, to differentiate between the need of source compatibility (e.g. for component designs) and binary compatibility (e.g. to design distributed systems). Use strong types to build your design when your programming language offers a type system that ensures that forward design compatibility is supported by forward type compatibility. Limit your reliance on types when these do not provide this forward compatibility. If this limited use of types, or the limits of the type system, do not allow monads and comonads constructions to be expressed with types, then use typeless/generic bind, cobind, and other monadic and comonadic like operators (e.g. duck typing on key-value pairs). 

Finally, use only the language features that allow you to break down your functions into monadic and comonadic constructions. For example, only use destructive assignment if you can model it within your monadic and comonadic design. Also,  do not forget you need also to enforce a "growth" only rule for your interpretation of values.

Now having said all this, you cannot build a design if it costs too much or if you cannot explain it to others. For example,  I wanted once to upgrade an algorithmic trading API around a monadic construction (within a C++ framework). I communicated around prototypes and presentations, (I was CTO), yet failed to get across to my key team members (who where top notch developers), and ended up canning the idea.  And this brings me back to a few important issues:

• Monadic and comonadic constructions are higher order, even if you do not implement them with higher order types, you still need to think of them as higher order invariants hidden in back of your design. This is like a mathematician would "see them", and is not obvious from a classical procedural or OO background.
• The cost to implement typeless/generic monadic and constructions within your language may simply be too high. 

For example, concerning this last point, this week I wanted to implement a duck typing pattern in Java  on top of Protocol Buffers objects (for all the reasons mentioned above), but Protocol Buffers is only efficient if you access fields in a typed manner. So even while I could see how to build a design with a robust algorithmic binary compatibility model, the cost on performance would simply be too high to do it on top of Protocol Buffers.  (Luckily using Protocol Buffers is not a requirement for this project so I can move to another data serialization protocol that supports fast generic field access).

As an end note: I understand that I am not telling you in this post "how" to write monadic and comonad style constructions, especially without a functional language. What I am telling you is that you want to know how to do this if you want your designs to grow well.

All original content copyright James Litsios, 2015.