Beware of the language trap in software development

Consistent design of multiple views of distributed state

The notion of "state" is key to software development. For example, we talk about stateful and stateless designs. Still, there are multiple notions of state. For example, there is the stored state, the communicated state, the received state, the processed state. Therefor when we make a statement like "stateful", we are always just referring to one "notion" of state and not the others. This "abuse of stateful view" was historically not an major issue. Yet with the rise of processing speeds and truly distributed systems, these multiple views of states started more and more to exist "simultaneously" and this caused problems. For example, we might model a trading system as four different states: the state of trading action messages received by the exchange, the current state of the exchange, the state of confirmations received from the exchange, and the current state of our trading system. A question is then: in our concrete code, from which state views should we build our design, and how to do it consistently? We now know this is the wrong question. A trading system is a distributed system, all views of states are real and have business significance. Therefore the question is not which state, but how to design all state views/models as one coded design in a cohesive manner?

We now know we can use higher order types to do this, to capture the multiple views of states "as one model" within a distributed computation. It took me a long time to understand this. One reason being that only C++14 had the explicit template specialization needed to do this purely in C++. And while I had experimented with Prolog to generate complementary C++ code models in the mid-1990s (and was not convinced). And in early 2000, I tried to use generative technics with dynamically typed functional programming (I wrote about a generative programming conference in Dr. Dobb's at the time). It was only when I picked up statically typed FP (initially F# in 2006), that I understood how it could be done (e.g. with monad-comonad adjunctive joins as hinted here pre-Elevence).

Business success

The reason I bring "managing distributed states" in this posting, is that I was retrospecting on the challenge of success in my many software developments. This made me think back to my experience in developing a market making system. (I wrote about Actant in a recent posting).

Early 2000 market and regulatory pressures meant that many new derivative exchanges where being created. This resulted in many "easy" new business opportunities to sell an algo trading and market making system, as one "just" needed to connect a product to these new exchanges. However, what we did not foresee, was that this "lively" exchange business development would also have the exchanges competing "lively" among themselves and be updating their trading and quoting APIs at an unprecedented rate (e.g. every quarter). In parallel, we consistently had mismatches in our different view of distributed state, for the reason mentioned above, but also because we were exposing different state views that were easy on our end-users, but which broke distributed consistency. The result was, we spent most of our development resources maintaining high-performance exchange connectivity and managing a hard to manage "mismatch" of state models, with little resources left over to be strategic.

The language trap

One of my Actant partners once said: C++! It was C++ that hurt us the most. (Again Actant mentioned here). By that he meant "stick to C", but C is a subset of C++, right? So how can C++ be an issue?

Here is a timeline that will help us:

1. Pre-C++98 (templates): we used macros, generative and model driven programming to write consistent models that ran distributed programs as multiple and consistent views of state.
2. Post-C++98: we used templates, type unsafe models, and redundant code to write distributed programs with often inconsistent views of state.
3. Post-C++14 (explicit template specialization): we used templates and explicit template specialization to write consistent models that run distributed programs as multiple and consistent views of state.

Because we chose to adopt the rules of C++, because we did not understand that by doing so we could not code a single consistent model of multiple views of state that we needed for distributed computing, we got "caught" in the 16 year gap of formal inconsistency that C++98 introduced! 

I write "formal inconsistency" because nothing in C++98 says that you couldn't continue to use macros, generative and model driven programming to get around the limitations of templates. The thing is "we do not know what we do not know", so we did not know that it would have been best to ignore the "formal" template solution and stick with our old technics. And that is example of a language trap.

A language trap is when developers choose to adhere to limiting coding language semantics without understanding that they are "shooting themselves in the foot" because they are no longer able to "get things right" when adhering to these limiting language rules. In some sense, a language trap is a technical form of semantic barrier.

Unfortunately, again "we do not know what we do not know". So we may or not be in a language trap, we do not usually know if this is the case. In early 2000, we did not realise how much the choice of purist C++ approach had been a bad choice.

My learning from that period was: never let a language nor language purism restrict you. Because you do not know how that it harming you until it is too late. The safer and future resistant approach is to deconstruct "a single language", and be purist in how you formally compose multiple language constructions.  An advantage of this approach is that it may also be applied consistently across multiple languages.

All original content copyright James Litsios, 2019.

QT Software, the Troika, to insights on co-founding Elevence

In summer 1996, I met Mattias Jansson and Lukas Lüthy (left and Luki middle above, I am at the right), as well as Adrian Lucas (left below) and in what seemed to be less than 15m, they were inviting me to join them set up a software company to develop a non-proprietary derivative market making system. Another key partner in this story is Stig Hubertsson (not shown). Therefore, we found ourselves co-founding QT Software, which later was renamed Actant.

I bring up this past in this blog to cheer excellent individuals and a great team, and give you insights into how QT and Actant led me to co-found Elevence and develop a unique smart contract language (then acquired by Digital Asset).

First the cheering: Still now Actant software is known to be the fastest non-proprietary derivative trading software. And while I don't have the numbers, I could well believe that they compete well with proprietary derivative trading software on classical hardware (meaning CPUs and not FPGAs).
And we might mention golden years of early 2000's when two of the top three European derivative market making firms were relying on Actant's product to do their business.

Mattias, Luki, and I were the core the company's engineering. Like minded, we worked much as one,  relying on voting to resolve the sometimes disagreement, and on the way, shaped a unique company with its specific market niche and customers.

That first picture above was taken in early 2018. Two hours later we were out of the house watching the fireman putting out the fire of the gas barbecue grill. Luckily no one was hurt, the gas tank did not explode, and house suffered only minimal damage. We then cooked our steaks in the oven and enjoyed an evening of great friendship.

The second picture is LIFFE Connect advertisement in the Financial Times!

In 1998, QT software was running on LIFFE Connect and Eurex. For the LIFFE Connect opening, which was first electronic access to the LIFFE exchange, we were asked to help publicise the new exchange offering. Adrian is to the left of that picture, and I on the right. Funny story, we showed up for the photoshoot at the exchange. They offered to walk down to the trading floor, but then proceeding to disallow Adrian's brown shoes, as only black shoes were allowed there!

One insight I gained in market making was how to model distributed systems with "adjoint" relations. The date was October 2008, a difficult period because of the impact of the 2007 crisis on our customers. I was lying in bed at our company's yearly executive get-together at the Palace Hotel Lucerne, and it occurred to me that one could model the market maker as some form of adjoint dual to the market of buyers and sellers. Years later I used that insight to co-found Elevence Digital Finance where we developed a language of rights and obligations sustained by a dual adjunction like relation to a blockchain ledger (see BearingPoint's announcement when Elevence was acquired by Digital Asset).

Anecdote: to whom do you go for a market making product in 2021 for derivative cryptocurrencies? Actant of course, they are leader in non-proprietary cryptocurrency derivative market making systems!

All original content copyright James Litsios, 2019.

Agile process vs Data Management

Modern architecture, for example for large cloud systems, have much in common with agile processes. You may know from past posts that I love agile processes. One reason is that agile processes have mathematical roots. The other is that when you know how to yield them, they deliver. I could talk for two hours on the subject! Here, for your comfort, the focus is just on the basic tie between an agile process and the architecture of modern data management.
Here is the video (9m):

Pictures from the video

Hereunder, the main pictures of the video, in case you want to refer to them in your comments:

1) Concrete vs Abstract (horizontally) and Static vs Dynamic (vertically):

(Note how I say concrete, but write "contract" in the video. That is a mistake, and what happens when you invent smart contracts and focus much on them!).

2) Agile process "functional schema" and data management "functional schema" side by side:

3) Waterfall process (red):

4) Agile iterative process (blue):

5) Traditional database architecture (red):

6) Modern data management architecture (blue):

Yes, in case you are wondering: my left hand is holding my iphone as I record. 

As always, comments are welcome, especially from old friends!

Skunk Works Rules (Kelly Johnson's Rules) Agile Style

#	Skunk Works Rule	Agile Interpretation
1	The Skunk Works manager must be delegated practically complete control of his program in all aspects. He should report to a division president or higher.	Control is delegated to teams. Direction is owned by business/product team leads who report at the business stream level.
2	Strong but small project offices must be provided both by the military and industry.	Leaders are strong and few.
3	The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).	Keep teams small.
4	A very simple drawing and drawing release system with great flexibility for making changes must be provided.	A very simple process to integrate, validate and release the full deliverable must be provided.
5	There must be a minimum number of reports required, but important work must be recorded thoroughly.	Process is minimal and simple, and yet records all important work.
6	There must be a monthly cost review covering not only what has been spent and committed but also projected costs to the conclusion of the program. Don't have the books 90 days late, and don't surprise the customer with sudden overruns.	Process is iterative. Achieved and validated results are systematically compared against projected iteration estimates. Consolidated project estimations are updated with learning from discrepancies observed within estimates of iterations.
7	The contractor must be delegated and must assume more than normal responsibility to get good vendor bids for subcontract on the project. Commercial bid procedures are very often better than military ones.	External dependencies are delegated (not remote controlled) to meet defined requirements that make sense from that external view.
8	The inspection system as currently used by the Skunk Works, which has been approved by both the Air Force and Navy, meets the intent of existing military requirements and should be used on new projects. Push more basic inspection responsibility back to subcontractors and vendors. Don't duplicate so much inspection.	There is just one way to validate that deliverables meet their requirements. This may include multiple categories of criteria (e.g. acceptance criteria, definition of done, test suites, ...). Don't duplicate validation work.
9	The contractor must be delegated the authority to test his final product in flight. He can and must test it in the initial stages. If he doesn't, he rapidly loses his competency to design other vehicles.	Components delegated to external parties can be finally validated within the full (functioning) deliverable.
10	The specifications applying to the hardware must be agreed to well in advance of contracting. The Skunk Works practice of having a specification section stating clearly which important military specification items will not knowingly be complied with and reasons therefore is highly recommended.	Validations process is communicated and accepted by all before work starts.
11	Funding a program must be timely so that the contractor doesn't have to keep running to the bank to support government projects.	Manage budgets "outside" of teams.
12	There must be mutual trust between the military project organization and the contractor with very close cooperation and liaison on a day-to-day basis. This cuts down misunderstanding and correspondence to an absolute minimum.	There must be mutual trust between those that say what the product does (e.g. product) and those that build it (e.g. engineering).
13	Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures.	Communicate your innovation inwards, not outwards.
14	Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay not based on the number of personnel supervised	Reward performance, not the role.

Explanation

The table above is a short analysis of the "Skunk Works Rules". As this is about innovation, I chose to express each rule from an agile perspective.

I love the engineering of the Lockheed Skunk Works. A few years ago, I enjoyed very much listening to Ben Rich's book on the subject (Skunk Works:...), Ben Rich having been the second director of the Skunk Works, (he succeeded its founder Kelly Johnson). In this book, and in many sources, Kelly's management approach and "rules" are mentioned, but not "deconstructed". I made a note to do so, which I do today, and I share that above. Comments are welcome, I took a very direct approach here, and am happy to hear how else to look at this.

Remarks

3 - Size of team

Size of team too big when it can no longer absorb new information from individuals (e.g. learning), or when individuals no longer stay aligned with changes of direction of team.

5 - Reporting work

Too many documents kill productivity. Recording work is ideally done in something as close as possible to what is the work. Yet to be a record of work, there must be more than work, there must be a record of some context of the work. In software development, text that describes work tends to age much too quickly.

7 - Delegating towards specific requirements

How to delegate work and still be agile? How to capture just enough of the future for the teams to be independent within the right amount of dependence? (Think distributed algorithms to answer that!).

14 - About rewarding people

Long term company stability may depend on stability of certain roles and people. Sometimes you need to pay key individuals to ensure that stability.

All original content copyright James Litsios, 2019.

Servant Leadership in seven rules

My favorite leadership approach is servant leadership!

While leadership is needed for different reasons and is expressed in different ways, it is clear that certain styles of leadership are better suited for specific situations, characters of teams, groups and organisations.  Therefore: this blog entry about servant leadership... because it works for me!

Note that this blog entry talks of teams... Teams which could be organizations, be hierarchical, etc... Servant leadership is not about structure but about method.

Let's start by expressing a complementary view: what is not servant leadership. At the extreme we could mention dictatorship, but that would not help us much, and is in fact not complementary. Instead, it is better to bring up followership in which team members follow their leader. Let's now start with three defining aspects of followership:

Leader with followership:

1. Team leader communicates expectations, team members follow with goal to meet expectations
2. Team leader expresses expectations to optimally meet the teams capabilities
3. If the leader receives feedback from team, the leader may review their communication and expectations

The most important part of this very simplified followership leadership is that only the leader adapts! Sadly, many leaders have a hard time with the idea that they must adapt. Therefore, to understand leadership, you need to accept that you too as a leader need to adapt. This is our rule one in servant leadership, a rule that is universal for all leaders:

Servant leaders align communications and expectations to meet their team’s need

Followership leadership is useful for short lived teams, such as one time situations or events, and useful for situations where members don't consider modifying tasks, because the tasks suit everyone or are too hard to change.  It is important  to note that in all situations there will be "moments of followership", moments that are "one off"or for which debate is not wanted, nor learning expected. These moments are very much "leader leads, followers follow" moments. For this reason,  we have rule two of servant leadership:

Sometimes servant leaders lead and their team follows

Much of what you do as a servant leader is to work for others, and yet you are still "the boss"and that means that there are moments where you need to "be the undisputed leader that is followed". This much because it is needed to keep your team healthy and sustainable.

Good leaders know that the greater value is in the team, not in them. That means that we want to boost our team coherence and self drive, in effect to give more leadership ownership to the team. This will be our rule three:

Servant leaders help the team lead itself

 Each team member is a mini-leader! In parallel, each team member is also a follower of all those other mini-leaders. Therefore we take our simple followership process above, and we rewrite it twice, first focusing on mini-leaders, then focusing on team followers. This gives us:

Team members as mini-team-leaders:

1. Team members communicate expectations to the rest of the team
2. Team members express expectations to optimally meet the team's capabilities
3. If each team member receives feedback, they may review their communication and expectations.

Team members as followers of their team:

1. Team members follow the communicated team expectations
2. Team members communicate their capabilities and optimally meet the team's expectations.
3. If team members provide feedback to their teammates, teammates may adapt how they follow goals, and adapt how they communicate their capabilities, and meet expectations.

With these two lists, we have captured something of a "pure unemotional and egoless self-driven team", that is, a hive-like team where everyone is led by everyone else, and is following everyone else. Yet such a team does not exist: teams are groups of individuals, all with their differences, and most importantly with their emotions and ambitions. And a productive team is much fueled by its emotional energy and enthusiasm! Therefore rule four, an obvious one, but as you will see, that has a twist:

Servant leaders encourage emotional energy and enthusiasm

The twist is then the following:  What type of emotions is does the leader encourage? Team emotions or individual emotions? Or both?

Both! Because a team is a group of team members but also of many individuals that are part of group. Here is the thing though,  the team’s emotions and individual's emotions can sometimes be very different. Think extrovert versus introvert here. In fact, they can be so different that they challenge each other, and this often subconsciously! To simplify, what can be happening is that the team feels aggressed when certain individuals do not embrace their emotions, and the individual feels agressed when the team or other individuals act so as to disturb their personal emotions. Therefore, when I say that the leader encourages energy and and enthusiasm, I am also saying that the leader cares about how team and individual emotions interact, and when needed help emotions co-exist together peacefully. We can write this as the following rule:

Servant leaders foster acceptance of team emotions by individuals and of individual emotions by teams 

Helping the productive coexistence of the team and its individual team members is much of what servant leadership is about. In part to help sustain good emotions, but also simply because when a team needs to solve many problems, it helps when the team members are given the independence to come up with solutions. However, just like emotions may hinder and even splinter a team, so can action and ideas. This is in part because giving team members some independence is also allowing them to be mini-self-centered-leaders to some extent. This, added to people's ego, gives you something extra, which can both be valuable, as well as highly lethal to a team. And therefore, an important task of a servant leader is to help channel the effects of giving people some independence. One way to approach this is to equate being independent to following yourself. We already have a followership list above, which we can rewrite as follows, with a self-centered focus:

Team members as followers of themselves:

1. Team members follow their own expectations
2. Team members communicate their capabilities, and optimally meet their own expectations.
3. If team members provide feedback to their teammates, teammates may adapt how they follow their goals, and adapt how they communicate their capabilities, and meet their own expectations.

Note here that I am assuming best intention, and therefore do not include disruptive self-interests above. However, best intention also goes the other way: people tend to bias their communication and expectations to protect their team members independence and ego! And not necessarily on purpose. We can express this as a form of shyness, again rewriting one of the previous followership lists.

Team members as shy mini-leaders:

1. Team members do not communicate expectations that might clash with expectations of others in their team
2. Team members do not expresses expectations to optimally meet the teams capabilities when that optimality might cause issues to others in their team
3. Team member tends not to receives advice when these are too personal towards them, or towards others.

These last two lists give us new tasks for the leader: to coach individuals to help them lead themselves coherently within their team, and to help teams lead themselves without fear of interfering with each other (e.g. social shyness). We might express this as follows:

Leader as personal coach:

• Leader coaches individuals to help them align personal and team expectations

Leaders help deal cross team interference:

1. Leader helps team members communicate expectations that might clash with expectations of others in their team
2. Leader helps team members expresses expectations to optimally meet the teams capabilities when that optimality might cause issues to others in their team
3. Leader provides advice when these are personal towards them, or towards others.

Now as we will see below, we will also want help the team deal with these topics, yet still then, the leader wants to have them in their radar. And therefore we condense all of this into rule six of servant leadership:

Servant leaders coach individuals and teams around expectations

Again, good leaders know that the greater value is in the team, not in them. Therefore, when possible, we want our team to deal with their expectation issues. And therefore, we want to boost our team members and the relations they have within their team.

Now we are almost done. Our last servant leadership rule will be about process, as until now our focus has much been on people, which makes sense as teams are groups of people. However, process is the backbone of the team: if it is weak, the team will be weak, if it is too rigid, the team will be too rigid... and we could go on. Yet the process as a backbone analogy is in fact a great analogy, in part because it is telling us that the team and the process and are symbiotic relation. Processes are not rules to apply, and neither are they routines that we follow, they are productivity boosters! Taking that from an optimisation perspective, processes are mechanised ways to push your team to be better each time. And that is not easy, and very much a job for your servant leader to help you with.

Servant leaders help team grow and pivot their process to be more productive

I'll give you a simple example: Give your team the productivity boost to be on time to your meetings! This may sound like "petty" time dictatorship. But hey, do we need to lose three minutes multiplied by the size of your team with every late meeting start, as always one member comes late? I would argue not.

Here you have it, seven rules for servant leadership:

1. Servant leaders align communications and expectations to meet their team’s need
2. Sometimes servant leaders lead and their team follows
3. Servant leaders help the team lead itself 
4. Servant leaders encourage emotional energy and enthusiasm
5. Servant leaders foster acceptance of team emotions by individuals and of individual emotions by teams 
6. Servant leaders coach individuals and teams around expectations
7. Servant leaders help team grow and pivot their process to be more productive

Note: You do not need to be a purist to apply servant leadershp. Provided here is an "idealized" explanation: in the real world there is friction, things interfere, and things happen differently then planned! Still, what you do have here is a servant leadership tool box that fits all those real world moments.

All original content copyright James Litsios, 2019.