Sunday, October 25, 2015

Stick man in haskell

Today some feel good code in Haskell:
man :: Diagram B
man =
        circle 1.2 # center
        ((rotateBy (-0.1) (vrule 3)) ||| (vrule 3) ||| (rotateBy 0.1 (vrule 3))) # center
        ((rotateBy (-0.1) (vrule 3)) ||| (rotateBy 0.1 (vrule 3))) # center
  which produces the following svg stick man:

You need to call it like this:
main = mainWith $ man

Sunday, July 05, 2015

The financial market as a magic carpet

(This is a "reaction to Greek no vote" blog.)

A financial market is like a magic carpet.
A financial market is like a magic carpet because it is sustained by the belief of the participants.
When these beliefs disappear, the market fails, and falls just like a magic carpet that has lost its magic.

You would think that most people would be very careful to keep "their" magic carpet flying.
Yet many people do not understand that their pension or salary are provided by the magic carpet.

The Euro is a big magic carpet. Yet some governments have been spending too much. In effect they have been removing some of the magic from the flying carpet. This makes the countries that are careful with their spending unhappy, as they know that the carpet will fall if the magic disappears. So although the Greeks are in a terrible situation (and share the blame with others in getting there), I would argue that part of Europe is seriously tempted to eject them from the Euro, if only to remind others that they better shape up, or else suffer a similar consequence.

Friday, April 24, 2015

Flash crash revisited

If you try to boil water in pyrex glass container, you may well see the water's temperature go past boiling point (100c/212F), because the smoothness of the glass is not providing any starting point for the boiling to start. Then if you take a wooden spoon and drop it in to the water... the boiling starts in an explosif manner, triggered by the spoon.

Would it be right to say that the spoon caused the water to boil? Yes.

Would it be right to say that heating the water caused it to boil? Also yes!

Would it be right to say that choosing a glass container caused the water to boil explosively? Yes.

Would it be right to say that dropping the spoon in caused the water to boil explosively? Also yes!

The analogy here is that the wooden spoon is a single trader. Take lots of spoons, the first one that touches that water will cause it to boil.

Would it be right to say that there was one trader that caused the flash crash? Just as much as that first wooden spoon makes that water boil.

Should this trader be prosecuted?
Well they did burn Joan of Arc...

Thursday, April 23, 2015

Unbearable truths

I found myself explaining to my daughter that people will go a long way to explain why there is value in the products they have purchased, especialy relative to products they have not aquired.

The hard reality is that to invest oneself is to blind oneself.  What happens is that because personal investments set us up to lose under certain scenarios, we will naturally think of how to avoid these scenarios. The thing is, having worked out how to minize loss, we have usually built a whole reasoning on what can happen, and what will not happen. The last thing that we want then is to be proven wrong. It is this avoidance to be proven wrong (among others) that blocks us from revisiting certain scenarios. Indirectly, what happens is that we set up a subconscious barrier as an effort not to be proven wrong.

This is very much black swan territory: events that do not happen often and surprize us in a bad way. But it is actually more than that, it is about being careful not to accept just any solution. Some solutions will kill you! You take them, you explain to yourself why you took them, and by doing so you have locked yourself into a thinking pattern that may harm you.

We cannot reject all because of their futur risks. We must accept new solutions when we can. Still, there are two ways to avoid disasters. First, listen to your gut feeling: if something feels wrong, then it might be wrong, and then better avoid it as a solution. Secondly, listen to your gut feeling: if something feels odd or out of place, then explain it. If you cannot, then one of your solutions may well be wrong and needs revisiting.

One last remark, work hard! It is the hard work that creats those "gut feelings"!

Sunday, March 29, 2015

Understanding certain crazy people

I was saddened by the deaths in Germanwings crash this week.

The natural question to ask oneself is why?  Why kill all these people while committing suicide?

Narcissism is a good place to start with “asocial” behaviors. Yet you must be careful in understanding what narcissism is about. Wikipedia, for example, starts by defining it as: “narcissism is the pursuit of gratification from vanity or egotistic admiration of one's own attributes”. That is only a subset of what narcissism is really about. Therefore, staying within the narcissistic theme, but taking another approach, I would argue that it is best to approach the subject from the “mental bubble boy disease” perspective. (Same for girls too!)

The classical bubble boy disease is about the inability of the immune system to deal with infections. The “mental” analogy is to argue that some people with certain “socialization” issues build a “mental bubble” to protect part of their inner "identity core" from the social world around them. The Wikipedia definition of narcissism (given above) only covers a specific form of “bubble” (the social image thing). Yet it does not capture the scope nor the essence of the problem: bubbles can be of varied content, and once locked-in to a bubble, the person will put as much effort to preserve the bubble, as to preserve themselves. Said differently, the person sees the bubble as part of their self, and will protect it as strongly as they protect themselves.

All of us have one form or another of narcissistic bubbles, they shape our personalities, they give us character. Most are harmless, and even entertaining: we like independence in others that gives character (and is not anti-social). Yet not all bubbles are the same, and some bubbles are  unsustainable, and can be dangerous to others.

Learning, and growing, is a layered process. Therefore, the earlier the setup of the narcissistic bubble, the more a person will build “on top” of this bubble. For example, someone who sets up a bubble before adolescence, may well have introduced a distorted view of relations between the sexes, and may end up understanding the world only through this distorted view. (This is one reason why kids should be protected from sexual implicit content, and why sexual discrimination is so hard to change in a culture).

To be an airline pilot, is to be a skilled pilot and to have built professional and social status that goes with the job. A person that obsesses early in their life about airline pilots, to the point of creating a “bubble” between themselves and the world, centered on the notion of being an airline pilot, is in effect creating a way to “interpret the world” with a “me the airline pilot” as reference. The sad story is that if started young enough, such a person would not have much other understanding of people, things, relations, than an understanding that supports him as an airline pilot. I could try to elaborate here, but the sad truth is that this man did not kill people as you and I feel and live people, he killed people that were not supporting his self-made “me the airline pilot” world. Knowing that he was crazy does not make it better. Still, it is worth understanding, and therefore this blog entry today.

To understand this subject better, I can recommend the book “Identifying and understanding the Narcissistic Personality” by Elsa F. Ronningstam. Add to this a good amount of personal introspection (you cannot understand others if you do not understand yourself) and much people observation.

Tuesday, February 17, 2015

About making bad decisions

I noticed that the link to Sydney Finkelstein video was broken in an old post of mine ("Why Smart Executives Fail"). Therefore, I went looking for a new link on youtube and found something similar, but recorded three years later. So as I have been working all day, all weekend and all evening theses days, I thought I would take some time off to watch the video, in order to know if it would do the job to fix my broken link.

This more recent video is excellent. Here is the link:

It is the same theme, with a few years of extra-wisdom and extra-learning.

Mentioned are four red flags in decision-making:
  • Are your personal experiences misleading you?
  • Is your personal self-interest clouding your thinking?
  • Have you made a dangerous pre-judgment that you are locked into?
  • Are inappropriate attachments pushing you in the wrong direction?
These are very good!

The tricky part is that to really understand them, you need need to have traveled the multiple sides of the human psyche, which many managers have not.

All original content copyright James Litsios, 2015.

Sunday, February 08, 2015

Is order spoofing ok?

Spoofing is the act of generating orders to buy or sell something on an electronic market, but to then immediately cancel these orders in order not to trade. Spoofing is used to generate a burst of activity with the hope that it will cause other algorithmic trading systems to react in a sub-optimal manner, allowing the spoofer to follow through with profit making trading activity. Spoofing happens when you enter orders with no intention of letting those orders trade. Which is different from entering orders in to a market and to then cancel them because you have changed your mind. There is no change of mind in spoofing, and therefore it might be seen as a form of deception. Spoofing works because other trading systems are waiting to see certain market prices before they change their behavior. Spoofing can also slow down the market as it may overload the exchange’s network or matching engine, such a slowdown may put at disadvantage other market participants. The question is: is spoofing ok?

Deception is never far from risk taking in business. Taking risks implies building a mental abstraction that works out the pros and cons of future scenarios, and that concludes that certain actions are more profitable than others. Sharing this understanding with others, might well affect negatively your future outcomes. A first form of deception is to avoid public debate around subjects that would develop information that would be against your interests.

An example of such a business deception is the communication around “phone screens that break”. Looking around, and by my own experience, the number one issue for mobile phone owners is that their phone falls and the screen breaks. I still remember my Note II slipping from my hand and shattering on the pavement, and I sadly remember dropping it again after having fixed the screen. From a phone vendor perspective, broken screens drives profits as it creates a continuous demand for replacement phones. Therefore there is no incentive for the vendors to fix the issue. To the contrary, the bigger the phone and the closer the glass to the edge, the more chances that a phone will break on fall. The consequence is that phone vendors build larger and more elegant phones, which just so happen to break more easily. Making fragile mobile phones is business deception at its best!

Is business deception morally wrong?

The simple answer is yes, if only because I am “pissed off” with mobile phone makers. This emotional state is a sign that a part of me thinks that these companies are behaving badly. Yet the full answer is less obvious.

All companies that take risks are potentially being deceptive. Again the issue is that minimizing risk and maximizing profit leads to minimizing information disclosure. And keeping information “quiet” is a form of deception. The issue is that business is not possible without taking risks or making profits. In fact the whole economy is built on embracing risk and profit, and therefore also built on embracing certain forms of deception.

That our way of life leads to deception and that deception can hurt people, is a reality we would be happier without. And yet as the issue does not go away, people have found ways to deal with it. When confronted with deception, people will either:
  • Ignore it, and act as it does not exist. 
  • Try to keep it from happening by introducing rules and legislation. 
  • Accept it as fact of life. 
  • Work around it.
  • Find flaws in the method of deception, and take advantage of these flaws. 
  • Join in, and deceive others in a likewise manner.
The magic of human beings is that for each “deception situation”, each of us will take one of these approaches, and this choice will be based on personal values. We will either take the escapist approach of ignoring the deception, or take the constraining approach of trying to limit future deception, or take the liberal, somewhat darwinistic, approach of accepting the deception as “a natural part of the system”, or when possible work around or take advantage of the deception, or finally become a deceiver ourselves. Note that the constraining approach has two variants:
  • Rules or laws are set up that disallow a form of deception. 
  • Rules or laws are set up that limit behavior with the goal to limit the ability of deception. 

Given these remarks, what should be done about spoofing in financial markets? Exchanges prefer either to ignore spoofing or to ban it by disallowing spoofing as a form of deception. I think that is wrong, that spoofing should be allowed, and that throttling mechanism within the market APIs are the best way to deal with exchange overloading causes by trading behavior. Here is my reasoning.

Market prices are the result of equilibrium that happens between buyers and sellers. It is the outcome of competing ideas, competing beliefs, competing interest, competing focuses, competing technologies, etc. Someone or an algorithm may want to buy, others and other systems want to sell, they are all taking risks, often thinking differently, and the outcome is market activity.

Spoofing is an ultra-short-term strategy that provides a healthy counter balance to strategies that are based on speed, and only on speed. The thing is, among all those traders and trading machines, are those that base their decision making more on the current market prices than on real world activity. These market participants are taking short-term risks, risks that often lead to profits because of their extreme speed. These are profits taken from others that are competing on the short term, and also profits taken from those are taking longer-term view. In this jungle of “eat or be eaten”, it is perfectly right that spoofing makes profits by disrupting short-term strategies.

Spoofing that makes a profit by physically overloading the exchange’s matching engines is wrong. It is wrong because it puts the whole exchanges at risk, both in its operational integrity, as in the ability for the exchange to be master of its rules and regulations. It is not right to allow spoofers to purposely affect the processing behavior of other participant’s orders. For example, by purposely causing the exchange to slow down and order queues to build up. Yet here again we are confronted with the difficulty to distinguish between behaviors that are purposely trying to kill exchange performance, and behaviors that are directly business driven, yet end up impacting exchange performance because of the amount of trading activity they generate. Therefore I agree that exchanges should introduce constraints that protect their networks and execution engines, but care must be taken to how this is done. As mentioned above, two approaches are possible: One can simply “outlaw” spoofing, and hope that it is not done, and if done, hope that it will not harm exchange infrastructure; Or, one can change things, to keep spoofing from impacting on the exchange’s execution. Throttling dangerously high trading activity is the natural way to approach this problem.  It is perfectly reasonable that exchanges limit trading activities that disrupts exchange integrity be implementing throttling within the trading APIs. Therefore, it is is right for the exchanges to configure API  throttling limits to limit spoofing activity that would impact too strongly on the exchange's network or matching engine. It is wrong for exchanges to simply ban spoofing, as that would give an unfair advantage to participants that apply trading strategies that are hurt by spoofing. 

Ps: Thanks to Adrian for remarking that deception may lead to deception.

All original content copyright James Litsios, 2015.

Sunday, January 11, 2015

Software designs that grow with monads, comonads, and type compatibility

This post is about designing software with internal and external APIs that are robust to future changes. It is therefore about API compatibility, but more importantly it is about the compatibility of a full software design to changes. Not surprisingly, monads and comonads are part of the presented solution, as is a careful approach to use of types.

I had a "aha" moment last year when I watched a video (by Timothy Baldridge)
that showed how an AST for a Clojure compiler was fully based on key-value pairs (nested hash maps), therefore without typed structures nor classes, and was doing very well. The thing is, I have suffered enough times to get the types of the different phases of a compiler to fit together. Therefor the idea of giving up on types and just using key-value pairs, that can easily be added for each compiler phase, seemed really to be an attractive way to write a compiler.

Key-value pairs, duck typing, and many "traditional" functional patterns (think lisp, not Haskell) have all in common their reliance on generic, almost typeless, structures. So while each "atomic element" of these patterns has a type (e.g. int, string, ...), the structures (e.g. struct, list, array, ...) are all generically typed.

Types are what capture the invariants of a design. Giving up on types is in effect giving up on capturing those invariants in a robust manner. Using types normally leads to higher quality software, yet with enough complexity, types no longer capture all the design invariants of a program. Worse, sometime types actually hinder the design by holding it back because they are not powerful enough to capture the desired invariant. This is the case with the difficulty to design an typed AST that "fits" all phases of a compiler. This rigid nature of types is also the hidden problem of API compatibility.

The invariants of APIs are captured with the types of their signatures. When new features or corrections are added to an API, these invariants and associated types evolve. When APIs link different software projects, changing API types is where API compatibility becomes an issue: APIs remain compatible when types change but remain compatible with older types, APIs become incompatible when the types are no longer compatible. An example of type compatibility in OO, is to derive a new class from an existing class, and to add a constructor in this new class from the old class. Unfortunately, that last example is at the source level. At the binary level, compatibility to type change is usually nonexistent, especially when focusing on forward compatibility. Note that API compatibility is not only about types: an API will also become incompatible when the interpretation given to values transferred over the API changes. Therefore to remain compatible, an API can add new value interpretations but must also ensure that past value interpretations never change.

Serializing data for transport and for storage is much about breaking program defined types into language standard basic types, and keeping a strict discipline of interpretation to ensure compatibility. Therefore ensuring both backward and forward compatibility of an API is to maintain a strict growth of value interpretation and to use serializing packages like Protocol Buffers or Thrift. The question is then: how do we ensure the future growth of a complete software design, not just a single API, but a family of APIs? Just like with the single API, the answer also lies in the notion of serialization. The goal is to stitch together the typed lower level design with a higher level untyped design that can change over time.

Data is serialized by walking through it a certain order and breaking it down into its primitive types. Programs can be broken down the same way. Yet to do that, your first need to adopt a functional programming style because it is hard to serialize procedural constructions. In a functional style, only functions need to be "broken down".

In the good old days, scalable software design was about using construction such as data schemas, object patterns, futures, continuation style, etc. Data schemas are still good, but all these other program constructions elements must be revisited with the fact that they can all be implemented with monads and comonads. More importantly, they must be revisited because the bind and cobind operator (and other monadic and comonadic operators) is what serializes functions! Therefore, just like you must serialize your  data schema to ensure future data compatibility, you must "serialize" your functions with monads and comonads to ensure future design compatibility.

Here are few examples of existing designs that do this:
  • Injection frameworks are comonadic constructions. 
  • Transactional frameworks are monadic constructions.
  • Parallel tasks are monadic constructions.
  • Embedded queries are monadic constructions.
  • Reactive streams are monadic constructions.
  • Lenses are comonadic constructions.
  • Automatic differentiation (computing analytical derivatives of numerical code) are both monadic (forward differentiation) and comonadic (backward differentiation).
Just like data compatibility is tied to the order in which data is traversed, future design compatibility is tied to the "order" of function serialization. That is to say that each software design is strongly defined by the order in which functions are broken into monadic and comonadic constructions. While monads and comonads have a duality relationship, they fundamentally differ in "character": monads are "trace" centric, while comonads are environment centric. Said differently, conditional expressions can be monadic, while comonadic expressions can abstract away their environment. A design is then a chosen hierarchy of monads and comonads with chosen set of API extension points.

Just like with data compatibility, future design compatibility is tied to the amount in which types can be changed and remain compatible. And again, to differentiate between the need of source compatibility (e.g. for component designs) and binary compatibility (e.g. to design distributed systems). Use strong types to build your design when your programming language offers a type system that ensures that forward design compatibility is supported by forward type compatibility. Limit your reliance on types when these do not provide this forward compatibility. If this limited use of types, or the limits of the type system, do not allow monads and comonads constructions to be expressed with types, then use typeless/generic bind, cobind, and other monadic and comonadic like operators (e.g. duck typing on key-value pairs). 

Finally, use only the language features that allow you to break down your functions into monadic and comonadic constructions. For example, only use destructive assignment if you can model it within your monadic and comonadic design. Also,  do not forget you need also to enforce a "growth" only rule for your interpretation of values.

Now having said all this, you cannot build a design if it costs too much or if you cannot explain it to others. For example,  I wanted once to upgrade an algorithmic trading API around a monadic construction (within a C++ framework). I communicated around prototypes and presentations, (I was CTO), yet failed to get across to my key team members (who where top notch developers), and ended up canning the idea.  And this brings me back to a few important issues:
  • Monadic and comonadic constructions are higher order, even if you do not implement them with higher order types, you still need to think of them as higher order invariants hidden in back of your design. This is like a mathematician would "see them", and is not obvious from a classical procedural or OO background.
  • The cost to implement typeless/generic monadic and constructions within your language may simply be too high. 
For example, concerning this last point, this week I wanted to implement a duck typing pattern in Java  on top of Protocol Buffers objects (for all the reasons mentioned above), but Protocol Buffers is only efficient if you access fields in a typed manner. So even while I could see how to build a design with a robust algorithmic binary compatibility model, the cost on performance would simply be too high to do it on top of Protocol Buffers.  (Luckily using Protocol Buffers is not a requirement for this project so I can move to another data serialization protocol that supports fast generic field access).

As an end note: I understand that I am not telling you in this post "how" to write monadic and comonad style constructions, especially without a functional language. What I am telling you is that you want to know how to do this if you want your designs to grow well.

All original content copyright James Litsios, 2015.