Typed duck typing in Python with Gemini

(picture By Klem - This vector image was created with Inkscape by Klem, and then manually edited by Mnmazur, Public Domain, https://commons.wikimedia.org/w/index.php?curid=3213322 )

Dynamic typing of duck typing

The link to the code produced by Gemini and the topic of this blog.

Two months ago I drafted a Python tracer library with Gemini, then I rewrote it from scratch (here). It is however "clunky". Yet I do not worry, as Python is a computer scientist dream language, and in Python one can quickly implement something meta and probably slow, but then use a cache or two to make it fast enough. That is what I did last week: I sat in front of my screen, stared at it for 30m thinking through my options, and then asked Gemini to draft the concept. And while the result is missing some logic, it is however doing what I want: it dynamically wraps "types" around duck-typed operation. I could try to explain what it does... but I do not need to. Instead, I share what I told Gemini. The key desires are the following:

• let's make a cached wrapper factory, that when given an object, first is looks in the cache, and returns the instantiated wrapper with the object if found, otherwise it uses the above greedy cover, where given a list of mixin classes, it dynamically creates a "best covering wrapper" that is remembered in the cache, and used to wrap the object.
• Ok, that works. Now we want to extend the concept to include some __init__ specify logic. First we need to make sure the __init__ is special and not part of the covering logic. Then we will assume that all mixins will get the target_obj as the first positional argument, and other mixin arguments will come through kwargs arguments.
• Nice. Now a last little bit: A list of additional "context specific" mixins can be given to create_wrapper, these will be will be added as parents in addition to the coverage mixins. Now the caching needs to be extended, as a same target object might be wrapped with different additional context mixins.

          (the full set of prompts is given below)

The idea is the following: Python naturally favors a duck-typing style of programming, yet duck typing is hard to scale without boilerplate code. Therefore one approach is to "find" what types support the duck typing. The types (classes in Python), can then be used to scale the program's design. In effect, we are dynamically typing the duck typing. Caching is then used to ensure performance.

Open vs closed sets in software design

Traditional designs ask the developer to choose a closed set of types, and then enables an open set of data to be created. Trace oriented designs work the other way around: we design a closed set of data traces, from these we derive an open set of types. This is the first level of Yin and Yang of software design. Something you might when you want to optimize high-performance code, such as an array heavy ML system. This technique is explained from a slightly different angle here.

 The Gemini prompt strategy

I did really think about the design for a half an hour before starting on the prompt session below. As mentioned before (the-art-of-prompt and master-class-techniques-for-llm), you really cannot just randomly chat with an LLM to get world class output, you need to think through your strategy, and the key part is that you are really trying to achieve a "one shot" chat with no corrections needed. The reality is that the billions of facts known by the LLM will only collapse into to the desired solutions if you pull it carefully, almost like a glass blower will shape molten class. This comparison is especially true: you only have a limited number prompts before the quality of the LLM fails you! 

The chat start with pedantic info... this is important! You really must ensure that the base concepts are well aligned with end goal of the task!

Note my very chatty style: don't forget: the LLM is trained on human text, you want to resonate with that humanity to pull out quality output.

The prompts (producing this code)

1. What are the rules of double inherentence in python? supposing I have A and B both deriving from X, and now I create Z deriving from A and B, is is a special case for Python, or does it just take it as a normal class hierarchy and simply takes Z of A when there is an ambiguity (supposing that A comes before B in the parent list).
2. Ok, clear. What about the __init__, and multiple inherentence, is it correct that I need to explicity call the parent __init__ , What about X, who calls it?
3. Ok, but then what arugments? Am I pretty much foced to use kvargs so that the parents can pick and choose their arguments?
4. Finally, how do I inspect an object to gets its class hierarchy?
5. Great. What would be the name of programming style where I rely heavily on the MRO? In effect I take all the MRO, and build a dynamic logic that depends on their interdependent structures?
6. so maybe what I want to do is "dynamic mixin" as their will be only dynamic assemblies of interdependent classes.
7. Right, and to be double shure, I can provide any base class list I want to that type constructor, python will work out the MRO that ensures the proper class parent hierarchy order, right?
8. Does itertool have a "generic" tree iterator? I want to get all classes of a module, ideally I want the recursive call to have already been written, and I feed it functions that depend on inspect. Can I do that?
9. And how about: given an object, and a list of classes, use a greedy search to find the subset of classes that "covers" the methods of the given object. The greedy search always tries to first cover with classes with a larger number of methods. Note however that it is not allow to cover using a class with a method that is not present in the given object.
10. Ok, so we follow up on the wrapper: let's make a cached wrapper factory, that when given an object, first is looks in the cache, and returns the instantiated wrapper with the object if found, otherwise it uses the above greedy cover, where given a list of mixin classes, it dynamically creates a "best covering wrapper" that is remembered in the cache, and used to wrap the object.
11. Ok, that works. Now we want to extend the concept to include some __init__ specify logic. First we need to make sure the __init__ is special and not part of the covering logic. Then we will assume that all mixins will get the target_obj as the first positional argument, and other mixin arguments will come through kwargs arguments.
12. Nice. Now a last little bit: A list of additional "context specific" mixins can be given to create_wrapper, these will be will be added as parents in addition to the coverage mixins. Now the caching needs to be extended, as a same target object might be wrapped with different additional context mixins.
13. Right, last steps is to remove the underscore filter, as the mixins are specifically designed for dunder logic.
14. Right, but LengthMixin does not need to test for hasattr __len__ as it will only be brought in dynamically.
15. Et voila. If you remember, we wrote a first wrapper version about two month ago. The issue with that first approach without this dynamic mixin logic is tha the wrapper logic becomes clunky with special cases. This double mixin approach allows the wrapper logic to be added very generically, although at the cost of a more complex and slower execution.
16. Let's add an MIT license with my name James Litsios and year 2025
17. You can add a comment under the license where you share the credit of the work!

What next?

I will rewrite my tracing library for the fourth time using the above!

All original content copyright James Litsios, 2025.

A software mind's eye

I have been writing higher-order functional programming in Python for the last few weekends:

• hofppy (https://github.com/equational/hofppy) will be a Python library. For the moment it is a collection of Jupyter notebooks.
• My initial goal was to have a handy FP toolkit which supports applied math with JAX's JIT. Yet I realise that in fact what I am really doing is reimplementing the generic part of a compiler for a reactive (trading) language I wrote in 2010 in F#, while including a few tricks I picked up since then, the primary one being to think of code as implementing a synchronous execution model.
• There is really very little code like this on the web, therefore why I am doing this open source.

This blog is in part written to mention the above, as already the first JAX JIT supporting monad and comonad models are "nice". Yet this blog is also to bring up the subject of the process of creating new technology.

My recipe to do something new, such as lead a team on a new subject, write software that did not exist before is the following:

1. Create a "mind's eye" imaginary world of "usage" of the end result.
2. Imagine how it should work.
3. Start very quickly to implement features.
4. Review 1, 2, and 3, as you regularly cycle through them, again, again, and again!

I use the word imagine twice above. For a few reasons...

People are constantly asking me "how do you know?", with regards to client's requirements, technology, design, approach, etc. The reality is that I do not completely know: I know much, I have done much, written hundreds and hundreds of thousands of lines of software. Yet when something is new, no one knows fully how things will look in the end. The reality is the newer it is, the less you know. And... the more you think you know, the more you are fooling yourself. However, just like with poker, the optimal approach is to imagine much and often, and to work as hard as possible to build and maintain a mindful view of "new usage" and "new functioning technology".

The more experience, the better! As your imagination is only as good as the real details it contains. I have tons of real-life learning that I use for this higher-order FP in python. I mention here a few of them:

• Lazy software design are easier express in code.
• The most flexible design, is the code that does nothing but accumulates lazily what it could do, until finally because output is expected, it works backwards from output expectations to pull out the desired output.
• Multi-stack semantics is perfectly normal. 
• For example, in the project's above monadic code, the free monads have "their" variables, "normal" Python has its variables, and these are carefully kept separate.
• Multiple flows of causality exist in the real world, stacks of various semantics is the only cheap way to capture these cleanly in software.
• For every Ying, there is a Yang.
• If there is more, there is also less; If there are variables, there are constraints, etc.
• Structure is more important than types.
• I started in functional programming without the types. Then I embraced types. And yet to understand that the sweet spot is somewhere in between. The reality is that it is much easier to use code to constrain software "just a right level" than to use types.
• Math is the most powerful structure.
• Think geometry, topology, and dualities!

 All original content copyright James Litsios, 2024.

Thinking about skills in people and in software

Here is a handy way categorize levels of software sophistication:

1. Software implements features
2. Software  supports software that implements features
3. Software supports software to support software that implements features

Here is handy way to categorize levels of people skills:

1. Person implements features
2. Person supports people that implement features
3. Person support people that support people that implement features

Let us combine some of these people skills and levels of software sophistication:

1. Person implements software that implements features
2. Person implements software that supports software that implements features
3. Person supports people that implement software that implement features
4. Person implements software that support people that implement software that implement features
5. Person support people that support people that implement software that implement features
6. ...

Note how entry 4 is a hybrid of people skills and software expertise!

The above lists actually makes sense, yet they are too simple. We can do better!

We can distinguish between support as:

• Direct support
• Proactive support

For example:

• A unit test is software that directly supports software.
• A library or a framework is software that proactively supports software.

More examples:

• A project manager directly support members of a project with a plan.
• A servant leader proactively support his or her team members.

We can bring this together as:

1. Person implements software that implements features
2. Person implements software that directly supports software that implements features
3. Person implements software that proactively supports software that implements features
4. Person directly supports people that implement software that implement features
5. Person proactively supports people that implement software that implement features
6. Person implements software that directly support people that implement software that implement features
7. Person implements software that proactively support people that implement software that implement features
8. Person directly supports people that directly supports people that implement software that implement features
9. Person directly supports people that proactively supports people that implement software that implement features
10. Person proactively support people that directly supports people that implement software that implement features
11. Person proactively supports people that proactively supports people that implement software that implement features
12. ...

You might be thinking: this stops making sense!

Not so! Each of these categories are different, and imply different levels of skills and experience. Some need more people skills, while some need more software skills. 

Wrapping this up: Yes, it makes little sense to enumerate these categories. However, it makes total sense to understand them, and understand how the micro-structures of software and teamwork combine in different ways and demand different levels of experience and skills. 

Wishing you continued success with your project!

 All original content copyright James Litsios, 2023.

Software with formal properties at a low cost through monads and co-execution

Not so long ago a good friend of mine mentioned his need for 'formal properties' in his code.  

I asked him if he had already made the code purely monadic. He had. "Good", I said, "because the 'cheap route to formal properties' is first to ‘monadize' your code, and then 'reverse it' by bringing in 'co-execution'".  That last part needs some development. 

Formal properties are not only about execution, actually mostly not about execution, but more about formal properties of state. So while you do need to show execution invariants, you often much more need to show that the ‘results’ happen within stateful invariants (e.g. of the ledger). The easiest way to do this is to  deconstruct / co-execute the state model backwards. This 'backwards' logic must 'match-up' with the 'forward execution' (monadic) model. My view has always been comonads are easiest for this, especially if your stateful model (e.g. a blockchain) is already something that is easy to express as a comonad. 

You might note that my last post is all about combining 'forward' and 'backwards' arrows. One way to interpret these arrows is 'right arrow' for monads, and 'left arrows' for comonads. And sure enough, that is an easy way to sketch out higher order systems where adjoint invariant properties (e.g. rights, expectations, authorizations, ...) are assembled 'with good properties' by design. As mentioned in previous slide, monads are easy to express as CPS, and as are comonads with a bit more 'gymnatics' (see previous post).

All original content copyright James Litsios, 2022.

Why Use Arrows and PowerPoint to Express Software Designs?

Dog Drives Car 

Silly me shared the following picture with my sister-in-law yesterday,

as she was stuck in a massive traffic jam with her dog (in photo) in her mini. 
"How did you make the picture" was the question. "PowerPoint" was my reply.

I used PowerPoint for that photo because it took me three minutes, and I had a slide deck open.

Stop, you might say, this has nothing to do with programming, or software designs! It does if we want to design a software where a dog drives. Yet how would we approach that? Here I tell you how I have designed and communicated designs of 'big and bold' softwares quickly and with good properties. And will also sketch out how to approach exotic requirements, such as dogs driving cars.

Arrows Away!

Let me show you how I work. This how I use to draw design 28 years ago (from my PhD thesis):

This is how I would have drawn that design 18 years ago (~2004):

Here I use arrows to represent flows. Arrows are used to capture the Lagrangian interpretation of flows (e.g. as stream oriented design). In this case: for a given set of inputs, there is one or more sets of models, and for each of these sets there is a solver. If this sounds confusing it is, as it was ambiguous. It is ambiguous as it is not clear which flows we are talking about. Are these flows of the actual simulation calculations (e.g. variables of the simulator, this was a simulator), are these flows of structural data dependencies (e.g. how big each array is), are these flows of types (e.g. real vs complex). The solution to that problem in 2004 was to draw multiple flows. For example, here we have the flow of configuration, and the flow of object hierarchy:

Note here I introduced these flows as Lagrangian, meaning we are 'moving along with the flow'. When implementing flows, there are always a 'outside observer' view of the flow, the Eulerian interpretation (e.g. a CPU core's view). These can also be drawn with arrows (e.g. see my 'old' stream processing video). In 2004, I would have used two different arrow types to show the difference of flow representation. For example in this diagram the 'stream processing solver engine' produces 'a stream of variable data': 

Let's mention the strong influence of multi-stage programming on some of these 'notation concepts' (see Walid Taha ~2000 work referenced here) as one way to think of these arrows would be like this:

However, such a notation implies that the 'telescoping flow' has its own semantics, and that is tricky. An easier approach to draw something like this:

The idea being that the solver 'thing' is specialized with a model 'thing' (returning a new flow thing), which we can then specialize again with a variable 'thing', resulting in a 'fully formed flow thing' that includes the solver, model, and variable details. Importantly, we are no longer 'telescoping' things when we do it this way. Therefore one learning is that 'telescoping' is one representation, others are possible.

Holy Trinity Continuum

The notion of telescoping is much associated to monads. While the concept of 'specializing something with something else' is often implemented with with comonads. Yet this is not critical information to know. The reason being that monadic like logic can be expressed with continuation passing style (CPS), and again can also be expressed in defunctionalized forms (see this early ref).  We have something that looks like this:

Even better, we have something that looks like this (correct me if I wrong, I am writing this fast today):

This last diagram is 'my holy trinity' of programming design belief, as I do not need to worry if a design will be implemented with types, closures or functions, as one can always be mapped into another. Instead, the key concept is that a design should express what is invariant no matter  'how it is implemented'. Said differently, designs are best when they express invariants that hold over multiple variants of implementations.

Notes:

• CPS is really functions + a more or less complex scoping logic implemented across the functions.
• Things are a little bit more tricky when we interpret things backwards. Still similar concepts  hold.

Scoping Invariance

A primal invariant to mention is about 'variable scoping'. (The applicability of the concept is broader, and applicable to containers with access by indices versus access by search). Early languages noted two fundamentally different approaches to 'variable scope' semantics, the so called static and dynamic scoping. Dynamic scoping is still much a curiosity to developers in 2022. However, Richard Stallman writes: "Dynamic scope is useful".  So what gives? The trick is that we need to restrict the dynamic scope semantics to make things scale. A good choice to limit dynamic scoping to have only 'one point of variable access'. With that we have the following symmetry:

• Static scoped variables are 'defined at one place' and possibly used many times.

• Dynamic scoped variables are 'defined possibly many times' used used 'at one place'. 

An 'semi-formal' insight is that dynamic scoping is static scoping 'running backwards' (with the restriction above). In a simplified arrow form, this looks like this, indicating an invariance correspondence between static scope going forward and dynamic scope going backwards:

Inspired by this, my notation rule is: arrows that go left-to-right are symmetric to those going right-to-left. Often just the direction is needed. For example, in the following diagram:

If A has a static scope, B has a symmetric dynamic scope, and vice-versa.

Directional Invariants

There are many software invariants. Many of them have their roots in the notion of 'sequence of effects'. For example, if we write something, then to 'read it' is just like 'to write it but backwards'. Here too we can use arrows to capture these relations, for example:

Again, the idea is: if a left-to-right arrow has a design feature, then there is matching 'reverse logic' right-to-left arrow with a 'complementary design feature'.  I have about 10 of these invariant symmetries that I use 'all the time' to design, and a broader list of over 50 symmetries that I like to refer too.

Arrows and Dimensions to Help Design Software

I have been practicing to 'think backwards' for over ten years. (I sound incoherent when I forget to 'switch that mode off'). You may well wonder what 'running backwards' means. Luckily the following 'equivalence heuristics' help:

• While we 'execute forward' we can accumulate a lazy execution of reverse logic which we can run 'at the end' of the forward execution. At which point the reverse execution will be run forward, yet it will still be the reverse logic. (This is how reverse mode auto-diff is done).

• We go 'up' the call stack, after going 'down'.  Similar to the previous bullet, we might say that we go forward as we go down into function calls, and go backwards as we return out of each function. 

• A two phase commit has 'forward computation' in the first phase, and 'reverse computation' in the second phase.

The important insight here is that reverse logic does not need to be run backwards. Yet it does need to be 'arranged in the design' to have a proper 'reversed logic'. Therefore we do not need to draw reverse logic from right-to-left if we have a way to indicate that it is 'forward execution that manages backward logic'. I use the U-turn arrow for that. The previous example can be drawn as:

Nothing imposes directions, therefore the following is a perfectly valid design sketch:

By which I am saying: the design will verify things, accumulating a reversed execution logic which will be interpreted. 
You may note that I am being 'very one dimensional', limiting myself to a left and right direction. We can in fact introduce as many dimensions as we want. Two dimensions (horizontal and vertical) often suffice to describe large architectures. (Sometime more dimensions are brought in with 45" or 30" and 60" degree arrow directions). For example, we may indicate our I/O logic 'vertically':

When we combine the two, we might indicate the following two designs:

Which one to choose?
Both!
One way to read these designs are as follows:

• 1 top: while we write we accumulate to confirm the overall 'write commit' only when 'all writes' are valid from a 'reverse perspective'.

• 1 bottom: While we read we interpret.

• 2 top: To verify we read 'the past':

• 2 bottom: we can 're-interpret' the past by writing.

A careful reader might note that while we may want 1 and 2, we cannot just have them! The issue is that when combined, we are mixing 'left-to-right' and 'right-to-left' arrows 'of the same thing'. For example, 1 has Verify as a U-turn and 2 as Verify as a left-to-right arrow. That will not work, as one is the 'reverse direction' of the other. 

Further Directional Invariants

There are many important software design concept. Here are three more, expressed with complementary arrows:

In simple terms: A coherent design does not have complementary design concepts 'at the same level'.  What the above is saying is:

• Both a local and global design must be expressed, and both are complementary. 

• Providing authentication allows us to 'look back' with trust.

• If new 'things' are 'freshly' created, then at some point they become 'stale'. And two complimentary design logics must be designed to deal with this. 

With that in mind, the previous two designs can be brought together as as single coherent design:

To note that this is how I was designing software early-2015 (see my pre-Elevence post).
(I leave the reader the reader the exercise to bring in authentication, trust, freshness and staleness).

Semantics, Tools, and Teamwork

Given a team of developers, I might suggest that design 1+2 be implemented with local monads and a global comonad (e.g. as a blockchain). Yet once I do that I lose flexibility. I also lose my audience. Instead I draw arrows, and explain what I mean with these arrows. I say what properties we 'the development team' are looking for, and sure enough we end up with a decent design that is robust and strong.  Drawing design 1+2 takes just a few minutes in PowerPoint, and maybe would have needed a few tens of minutes to sketch out before (e.g. on a sheet of paper). Typically, I duplicate my 'master design slide' many times, to then add annotations to show how we might 'slip in' extra design requirements. For example, the notion of trust and authentication, and here, the logic to deal with 'staleness'. 
I started this post with a picture of a dog driving a car. The important property of a software design is to be 'consistent by design'. Were I to design a 'dog driving car' software design. I would start by sketching out the complementary properties that I would need to bring in to the design. These might me:

Then I would draw these in PowerPoint and iterate and grow to create a good design.

All original content copyright James Litsios, 2022.

Healthy, noisy, productive Python

Asked about my feelings about Python, “speckle noise” came to mind.  The bigger subject here is Python's limited types and inherent noisiness in capturing deeper invariant properties (and therefore my speckle remark). 

Speckle noise, is the noise that arises due to the effect of environmental conditions on the imaging sensor during image acquisition. The analogy is that there is inherent noise in how one captures ideas in Python, and that this noise has "good Pythonic character". By noise, I mean something that cannot be precisely mastered.  Note that "real" speckle noise is not "good" or "bad", it just is.

All programming languages are "noisy". Yet to the developer, the way that noise affect you varies greatly. "Messiness" of computer languages may hurt you, as it may also help you (e.g. by improving your team's productivity). Said differently, sometime "cleanliness" is unproductive.  The main idea is the following: 

People naturally care about uncertainty.  Therefore, sometimes, we naturally focus on things that are not certain. As a bonus, we are naturally social around uncertain topics (think of the weather!), in part because we are happy to share when no one has an absolute truth, but also because sharing helps us deal with these uncertainties. Finally, there are many situation where an "external" nudge is needed to move out of a local minima. (I mention here the suggestion that financial markets need a bit uncertainty to be healthy, and here how I was once stuck in a bad local C++ design).

People naturally build on certainties. And when we do so, we in part lock ourselves in, because it would cost us to change what was certain and rebuild things.

This game of "certainty", "uncertainty", "building and locking ourselves in", "not finding a solid base to build", is what happens when we program.  Our choice of software language strongly affects how this happens. I have programmed and managed teams using Python, Haskell, F#, C++, Pascal, C, and Fortran (plus exotic languages like OPS5). Each of these languages is "robust" at a different level, some with more impact than others. 

Python, for example, is a language where expressions and functions come first, and types (e.g. list, object, classes, ...) are much a thin way to group functions and data together.  To contrast with Haskell,  where types are more important than expressions. The result is that new concepts are quickly captured in Python, and are considerably harder to capture in Haskell. However, it is quite difficult to capture deeper invariant properties of new concepts in Python, something that is easy to do in Haskell, with its strong types.

We might summarize by stating that Python has noisy types. At least that is often the way I feel when "dragging" concepts from one expression to another using "glue" list, dictionaries, objects or tuplets structures, just to make it work. Also to mention Python's limited dispatch logic, forcing yet more ad hoc constructions into your expressions Yet the magic of the real world, is that such noise creating properties is not necessarily bad!

I few years ago, I hired Haskell developers to build a system with "crystalline design properties". This had been one of my goals since being responsible for a "pretty messy design" in the late 90's. Therefore I co-founded a company with in part the goal of building a "single coherent distributed system". It is not easy to create a system where every complementary concerns fit precisely together, and all exists within coherent contexts. In fact, it only makes sense if you need it, for example to ensure trust and security. Now imagine the developers in the team working on such a "single coherent design". In such a development, no engineer can take independent decisions. In such a design no code can be written that does not fit exactly with rest of the code. How then to create common goals that map into personal team member tasks so as to avoid a design deadlock?  The simple answer might be make sure you have failed before in many ways, so as to avoid repeating those failures. Yet still, that does not avoid design deadlock. The hint of the approach is that for every dimension of freedom that you need to remove to guarantee the strength of your design, make sure to add an additional free non-critical dimension to help individuals still have a form of personal independence. In addition, I will add that it took me a lot of micro-management of vision and team dynamics to make that development a great success.

With “speckle noise”, especially at the type level, no such problems! There is no single crystalline unified software design. There is no coherency that is assured across your system. Python naturally accumulates imperfection which are just too expensive to keep precisely tamed with each addition of new code. This means that developers can agree on similar and yet different Python designs, In some sense one agrees to compare naturally fuzzy design views. And by doing so, one naturally protects one’s ego, as there is always a bit of room to express individual choices.

This may sound like Python bashing. It is not. This expensive “to design right” property is common to most programming languages.  This post is in fact a “praise Python” post.  If Python only had “a certain fuzziness”, it would be not much better than Visual Basic (to be slightly nasty).  Python is not “just another language”, it is a language where design logic is cheap to change because of the messy types are in fact naturally "spread apart". That is, the "noisy" Python type property results in "not too dense type relations", allowing changes to be made in one (implied) type without effecting the core of the other (implied) types. 

ps: I mentioned Fortran above really only because I like numpy's stride_tricks.as_strided and it reminds me of my large equivalence array structures which I used in Fortran when I was a teenager.

All original content copyright James Litsios, 2020.

25'000 views; Some topics I am passionate about

 This week a took a day off to walk in the alps with my son. And while doing so noticed that my blog view count was 24999. So I quickly asked my son to be the next view! 

My blog has been around for more than ten years, and a tenth of a second is all a site like Google would need to achieve the same hit count. Still, I write this infrequent blog because it helps me find what is relevant to me and to others, and helps me communicate better. And encourages me to do my best, because I do care about my audience.

I write infrequently mostly because I do not have the time. Also, my topics tend to be complex, and sometimes even sensitive, and therefor take time. Still, I often jot down a title, sometimes with a page or two of content. These then become public if I happen to have free evening or weekend. For example, these are my unpublished blog titles going back to early 2019:

• Balancing productivity, tempo, and success
• Program = complementary
• Some retrospectives in business of innovation
• Type centric vs ...
• The instantaneous view of the real world has no math
• Control and flow in innovation
• Lessoned learned: writing "coherent software"
• Careful Software Architecture (in Python)
• Absent minded at work
• Choose your inner truth: run your organization like a hedge fund
• Dealing with time
• Computer languages as data for machine learning
• My reality? Or your reality?
• The Peter principle in software development
• Innovate, but don't waste you time
• ...

Out of these, you might notice topics that I am passionate about:

• Innovation
• Productivity
• Formal software and architecture
• Teamwork and organization processes
• Modelling the real (and unreal!) world

The thing is: You cannot innovate if you are not productive. You cannot be productive if your teams and organizations do not work well together. You cannot work well together if you do not understand how to express the real world in your work, and to be precise and correct when needed. These are topics that I care about, and what this blog has mostly been about.

Cheers to all, I am looking forward to sharing some more writing!

Thinking versus brute force "doing" in software design in 2020

About thinking versus doing

(I put a subset of this on YouTube as: https://www.youtube.com/watch?v=_kXpyctbmrQ)

Is it ok to program by "brute force" and put aside your brain?

A few years ago, I wrote on the pros and cons of programming with explicit and implicit type declarations. In 2020, I feel that this question is almost irrelevant. Therefore to ask the question:

Should we think about software design when we program? 

There is a trend in software development which states "developing software products is not about thinking but about doing".  I know many venture people who believe very much in this statement. Yet the reality is that software is a process of thinking (in part at least). Still, from a pure seed-venture-money game, the rule of "it is not about thinking" is true"! Because the number one rule is "be fast" and thinking too much will very much slow you down.

Is it then both true and untrue, that we should both use and not use our brains when we write software? Yes, and yes! And it is exactly this subtle challenge that makes software management in innovation a tricky game.

Thinking can be done different ways, different approaches can be taken. Some are efficient, some are a waste of time. Some will help your software grow, others will block you. Some will help you meet your deadline, others will invariably lead to failure. The game is then only to think in the ways that help you!

Some of my colleagues have heard me condense these concepts into one statement:

It is about thinking backwards!

There are many ways to do this. I will cover just a few here, there are in fact as many more.

Play to win above all else

In recent blitz chess games, FM Lefong Hua repeats "it is not about calculating the end's win game", (because that is not a guarantee win), instead is about about "flagging your opponent". By which he means, winning is not about thinking within the classical rules of chess, it is about meeting the "win conditions", which in blitz chess games is much about not running out of time (being flagged).

Transposed into a software development, that means it is not about meeting the classical rules of software design, it is about thinking through how to always meet your goals and deliver results. Or to be more specific, it is about thinking backward, to find a path back from your "runnable software" goals, to your current status that guarantees your development success.

The heart of your software comes first

I have mentioned that "One of the little understood properties of programming is that at the lowest level things tend to be single dimensional".  A good software design builds a "break" or "separation" in that unique dimension "just about where your main product abstraction is", to achieve a sandwich like design where new product abstractions are added "by reassembly" of the two separated parts.

There are few ways to do this, domain specific languages (DSL) and functional programming (FP) being my favoured approaches.  While all my major software developments had DSLs, it was only much later in my career that I understood that what was working for me was not the DSLs but the effort to maintain separation between interpreter and interpreted abstractions. This separation is naturally achieved with a DSL, yet can also be achieved with adjunctive higher order FP. (The "ultimate tool" is then to combine the two concepts as adjunctive FP based DSLs, which was the base of Elevence's contract language).

Be functional and agile

Agile development is a lot about prioritizing your requirements and implementing them in work iterations. That means that the first level of your design, what might be seen as the first draft of primary keys of your data base, are directly tied to the most important business concept (taken from OO vs FP vs agile). The trick is then not to see this as a database of data, where keys are data, but a database of code, where keys are functional properties such as APIs with specific invariants (e.g. somewhat like class types in Haskell). The extended trick is then also to make this a normalized design (just like a normalized classical DB schema, but in a the "functional space"), for example by using linear types.

Brute force yes, but with the right pieces

Let us put this all together, our first conclusion is when building new software:

Use brute force and minimal brains to assemble data and call external APIs 

Importantly, keep your program structure as flat as possible when you do this. Which pretty much means to shun OO patterns, and to use as many array and container like constructions as you want.

Then, and this is the tricky part:

Incrementally invest effort to separate interpreter and interpreted abstractions in your code.

Always make sure that these abstractions fit within the plan to your deliverables. That means you are able to think backwards from your deliverable goal, and work out the path to your current development state, the path which you will follow to build your code. An agile process and a good team will make this an easier job. Also, all those arrays and containers will need to fit into a single unified concept (e.g. such as indexed tensors).

It's complicated

Sometimes things are just too complicated. Many developers do not know how to "separate interpreter and interpreted abstractions".  Here I will be honest, it is not easy. And just to make this clear, the last DSL I wrote took me multiple tries until "I got it right".  Also, to mention that embedding product abstraction in higher order FP is even harder than writing DSLs.

Worse, the process is "hack brute force" to assemble data and use external APIs, and only then think about how you can slice your interpretive abstractions. These means that initially, software development is about doing, not thinking. Tricky...

Welcome to 2020

I have been writing this blog for almost 15 years. Interestingly, a major change over these years is that there is so much well integrated open source software (OSS) out there that it is cheaper to "just try" than to "think" (when using OSS and third party APIs). And in part it was this reality that led me to write this blog today. Enjoy, and stay safe!

All original content copyright James Litsios, 2020.

Lean Startup is about market making your product

I am a big fan of lean startup, and will therefore tell you why. Still, the method is not universal, and has its limits, I will tell you about that too.  If there is one thing you should remember is that lean startup gives you a simple recipe that can shape a new product both to meet the needs of its customers and to be profitable. To do that, the method needs customer interaction, and without enough of it, the method is likely to fail.

Lean startup is an iterative process. Yet unlike other development processes that only focus on the "how", lean startup gives you some very precise rules to follow on the "what".  Once you understand how the method works, these rules make sense. Here are the important "whats":

• Value is defined relative to a reverse income statement (defined as the "engine of growth"). In effect you are expected to model both your product and your customers. You can use a simple model of fix price values per events (such as client clicks), or you can be more creative and use some real financial engineering tricks to build your notion of value.
• Risk is to be minimized, and as risk is hard to measure, it is up to you to judge your uncertainties, and then to diminish your largest know risk/uncertainty at each development iteration.
• Customers and products are understood through the data that can be collected out of them (this is the notion of "measure"). All value assumptions must be validated through the collection of data. Risk assumptions are not validated.
• Continuous deployment means that software changes go directly into production (with some smart test automation in between). 
• Software is "staged" into production. So new changes go into production, but normally only to be initially used by few. As data comes in that validates the code change and expected value improvements, the change will be made available to more users. Staged deployment makes it possible to simultaneously collect data on different versions of your product. This is the notion of split testing. If you have done enough math you will know that split testing, well implemented,  gives you the most reliable measure of how the customers are effected by your code changes.  Other ways to collect data will include much more noise..
•  The careful observer will have noted that split testing within a "learn and change" loop is in effect a quasi-Newtonian optimization process. That is, we a looking for the best way to improve our product (equivalent to the steepest gradient). As we do not know that best way, and only know "a way", when the split testing confirms that the new version of the product is better than its previous version, we have a "quasi" method. As Newtonian methods are susceptible to getting stuck in local minimas, we need something more, and that is inspired by linear programming with the the notion "pivot": To pivot is to swivel variables into constraints and constraints into variable. Said differently, to pivot in business terms, is to change assumptions, to change your beliefs.
• Data and data collection are core values in Lean Startup, and therefore the notion of pivot. In lean startup you must note all your assumptions down, and any change to these assumption (any "pivot")  must be noted down too. This type of detail may sound pedantic but that is how smart learning happens. And more importantly that is how good team learning happens, by making the effort to express and note down your assumptions, to validate them, and make the effort to update your notes (data) when you change these assumptions.

Expressed in this way, lean startup is one big algorithmic machine. People still have a role to play as they decide the changes they might want to bring to their income model and to their product. The lean startup algorithm makes sure that changes really do have impact on revenue and on product perception. The originality of the method is to be expressed within one recipe.

I do not know what made the creator of lean startup put his method together, but if you have worked in finance, and especially market making and algorithmic trading, then you have already been using  very similar methods. The big differences is that the method is meant for material products, and not immaterial financial products, and that these material products normally only have a single sided market, unlike financial markets where you can buy and sell. These two aspects do make the method a bit tricky when the income model is not obvious. For example, when income comes from advertisement, you can equate each advertisement screen refresh with income, therefore it is pretty simple to map customer events to income values. But suppose you want to move to a subscription model, now your customer is paying a monthly fee, how to map your products features to this income? If you have done enough math and worked in finance, you can think up a few ideas on how to do this, but nothing is obvious when you do not have this math experience. And that can make the method tricky to apply.

I like the method's smart tie into  financial theory. For example, simple portfolio theory say that it equivalent to maximize return under fixed variance, as to minimize variance under fixed return. So lean startup chooses the second option: minimize risk (variance) while having asked you to express return (inverse income statement, growth engine). To then make sure that you validate that return model with your collected data. As a full notion of variance would be impossibly complicated, lean startup says: focus on the largest risk. That is a simplest approach, but also the smartest approach.

Another tie to financial markets is that you need to make a market in your product for lean startup to exist. Making a market means that you attract customers to use your product. And because lean startup relies on a "market fitting" approach, that is, you fit your product and income model to your market of customer, the type and amount of customers you have will make all the difference in how your product evolves. This is a real issue because the good clients are not the ones that want to spend their time playing with small applications that only have a subset of functions. Therefore, a critical difficulty with lean startup is in how to bring valuable customers to a product that is not yet really valuable enough to them. This bootstrap phase can be most difficult and that explains why you need to cheat at bit and sometimes offer what you do not have to get the necessary feedback to validate your product's evolution. Unfortunately, customers will hesitate to come back to your product if the first time they came they felt deceived. So offering what you do not have, in order to get a feed back on the demand, will also tarnish you ability to grow later. This is especially true with small niche markets.

Market dynamics are much the higher order properties of a product. Many just ignore them and hope for the best when launching their new product. Lean startup makes the effort to try to learn about these market forces while you develop the product. It is not easy, it is not obvious, but it is still the right thing to do.

OO design patterns from a functional programming perspective

Here my attempt to match up design patterns with functional programming constructions:
Chain of responsibility
Fold request across the chain of handlers, at each step returning either (using Either type) request to continue fold (possibly with transformed request), or returning handled request indicating that fold is finished.

Command
Request is held in data (e.g. as discriminant union)

Interpreter
Language is available as typed data structure (e.g. discriminant union) and is interpretable with functions calls.
Alternative:Language is defined in monadic form (e.g. as function calls)  and is interpreted as part of the monad's interpretation.

Iterator
Exposed traversal functions hide data structure of containers. Functions typically work on zipper focus adapted to the containers structure.
 
Mediator
Exposed functions stay at top/root of exposed data structures.

Memento
Use immutability and exposure of intermediate states to allow recovery, restart or alternate computation from previous intermediate states.

Observer
Associate container of data with container of queries (e.g filter plus callbacks). Changes to data container (insert, update, removal) triggers matching query callbacks, change to queries may trigger callbacks. Callbacks are either in a monadic structure or pass around a state context.

State
Data context position is indexed (e.g. by one or more keys), associated state is stored in corresponding state monad where individual states are found with index. 

Strategy
Single function signature implemented in multiple ways.

Template Method
Use type classes to refine implementation by type variant.

Visitor
Break down algorithm into sub-functions; Provide these sub-functions in argument to algo; Use them within algo; Provide different ones if you want to change behavior of algo.

Adapter
Adapter functions that takes one or more functions with given signatures and returns one or more functions with adapted functions.

Bridge
Clients accesses a bridge data type that holds functions.

Composite
Use recursive data types.

Decorator
Data has a dynamic map of functional operators.

Facade
API presents no internal types and is as "flat" as possible. This may mean that certain functional constructions have been "collapsed" into a purely data centric view in order to limit the "depth" of the API.

Flyweight
Flyweight data structure are often provided as "first" arguments of functions. These functions can then curry their first argument so that the flyweight shared data is provided but does not need to be "passed around" everywhere.

Proxy
Data type that "hides" another type. Possibly relies on using an index/key reference and a map like container to store original data.

Abstract Factory
Factory hides the finer type of created data and functions. Losing the inner type can be tricky because the  exposed signatures must still need to be powerful enough to do everything you want with the exposed API (as we assume that no dynamic type casting is allowed). This may mean that you need powerful type features to make the abstract factory concept work (e.g. GADT).

Builder
Define functions to construct data. This is in fact the "normal" way to write functional style.

Factory Method
Use polymorphism, types are determined by inner types of arguments.

Prototype
(Immutable data does not need to be copied).

Singleton
Encapsulate the access to the singleton as a mutable variable within a function; Use this function to access singleton. In immutable setting, store singleton within a  state monad.

(I'll revisit this one if I have the time to try to put side by side an OO definition of the pattern with a functional definition of design pattern. Also it is a bit minimalistic).

All original content copyright James Litsios, 2013.

Eventual consistency: dual software models, software with gaps

Maintaining consistency within parallel, distributed, high-performance, and stream oriented programming is all about accepting one thing: when it comes to consistency, that is dealing with failures, don't promise too much too soon! And that is the whole notion of eventual consistency: you sometimes need to wait for things to get resolved. "How much you wait" depends typically on how much money you have invested in infrastructure, yet it also depends on how fine your system captures failures.

Let's start with the eventual consistency.  In traditional transactional models, consistency, the knowledge that your system is "doing well", is assured by approaching each operation "transactionaly": if an operation fails, make sure nothing is messed up and that consistency is maintained.  Yet in the "new" world of massive parallelism, need for high-performance, and the like of streaming models, it is often simply impossible to maintain consistency. The simple explanation is that as operations are split along multiple cores and computer, when something goes wrong it takes "more" time to clean things up. The key theory is that the "cleanup time" after failed operations is "not plannable" (lookup "CAP theorem"). Said differently, failed operation can get "eventually" cleaned up, if your systems puts in the effort!

Eventual consistency is often brought up in the context of distributed databases. Yet in my world of real time streaming systems, system designs is even more limited than with DBs. The simple statement is that streaming system are one dimensional. And that often mean that you only have one way to look at things. Or again, you can view things as different models, but often not bring these view back together at a reasonable cost. 

One pattern to manage failure is simply to kill the process in which the failure was detected. The idea is then to let that process or another recover the operation. I remember once arguing with another developer that "life" is not always that simple. It is ok to use this pattern if the scope of "operations" is small enough. In fact the pattern is also economically good because it simplifies the whole automated QA approach. Yet, if the operations that you are working with have too long a lifespan, the cost to kill processes and restart the failed operations may be simply too high. In these cases you can add lots of redundancy, but that too will cost you, and your solution might simply be non-competitive with a solution that uses a smaller footprint. The last resort is to refine the scope of what you kill: only kill a process if it is corrupted, kill only the broken part of operations, keep as much of the operation that is "still good", incrementally recover the rest.

Some of you may know that duality is still one of my favorite subjects. In the context of linear programming, a dual model is one where variables are used to "measure" the "slack" associated to each constraint. In a software system, the constraints can be seen as the transactions, or the transactional streams, that are being processed. The dual model of a software system is then a model in which you track "how wrong" transactional operation are. That is what I call the "gap model". In a stream oriented model, the gap model is also a stream oriented model. The finer the gap model, the more failures can be described precisely, and the more you can limit the corrective measures needed to fix things. Yet like with all good things, if your model is too "fine", it simply becomes too expensive to work with. So compromised are needed.

(PS: my blog work will be considerably lower this year as I decided that I was more fun to write books (and big theories) than blogs. I will still try to keep you entertained from time to time. And as ever, requests are taken seriously! So do write comments.)

Introduction to state monads

I went to my first Zurich FSharp meeting the other day and made a presentation on how to program with state monads in F#.

The presentation is designed to pull you off the ground, to a higher order programming style, using the state and maybe monad as stepping stones. It finishes with stateful lifting.

Here is a link to a slightly improved version: youtube .

Link to sources are below but I would recommend that if you really do not know this type of programming and want to learn, then you should type it back in and allow yourself some creative liberty. This is usually a much better way to learn, than to just copy code!

You can watch it here if you want:

ModularStateMonadsInFSharp.pdf
XState1.fs
XState2.fs
Script.fsx

Where is my state?

People often associated functional programming with statelessness. The reality is that there is always a notion of state, functional programming is about stateful programming. The ambiguity is probably the result of state appearing in different ways than in a procedural style.

In a procedural style, state is associated to memory and you change it by using the writable property of memory.

In a functional style, state is still in memory  but the “classical” approach is to disallow yourself from changing it by writing again over memory. This is the immutable programming style. It goes with functional programming because it is very difficult to build on immutability without some functional programming concepts (I have tried!).  So if you have state and you cannot write over it, the only thing you can do is to read it, and if you want to change the state, you need to write a new memory location with your new state value (a non-functional  near immutable approach is to use arrays and grow them). The problem for beginners is “where should that new state be”?  It needs  to be accessable, and while you maybe knew how to access the old state, where should you put the new state so as to be able to work with it, instead of the old state?
Recursiveness is then the first  way  to manage state; The idea is that if you have a function F that works with state, then if you create a new version of the state you can simply call F recursively  with the new state, and again and again with each new version of the state.

Recursiveness is always somewhere nearby if you are using immutability; Yet you do not need to expose it to manage state. Again, suppose a function F uses state, have it return its state after use, if the state has not changed, then  return it unchanged, if the state changed, then return its new version. The trick is then to have a loop that calls F with the state, then takes the returned value to call F again, and so on. As this is immutable code, the loop is a small tail recursive function.

State can be big and heavy, and not for the wrong reasons, your application may be a big thing. The last thing you want to do is to copy that whole structure just to make a small change. The simple approach is to reuse parts of your old state to make a new state, changing only what needs to be changed. Yet  as the state is usually a tree like structure, if what is changing is deep in the original state, then it will cost you to create a new state because you will need to rebuild much of the hierarchy of the state. One way around this is to take the old state, encapsulate it in a “change of state” structure and use this as the new state  This approach is what I call a differential approach: the state change is the difference. By stacking differences you build a complete state. Usually the differential approach is combined with the “normal” partial copy approach. The reason is that if pursued forever, the “stack” of state changes become too big and starts both to consume memory and take time to access.

In the end your new state will either leave your function “onwards” into recursion or backwards by return, and often the same algorithm may use both methods.
When beginning with the functional style, state seems to be forever a pain to drag around. But with time patterns appear. First you may notice that not all states have the same lifetime, and they do not all have the same read write properties. It is a good idea to try to separate states that have different access and update properties.  You may then note that the hierarchy of functions aligns itself to with the access patterns of your state. Yet if you tie state to function in an layered like fashion that is classical to procedural programming you realize that what you have is not so much state as parameters. That is, while you are working in an inner function you have no ability to modify the outer “state”, so it is more a parameter like than state like. Nevertheless, with situation where states have a strict hierarchy, this approach is ok.

At some point, carrying around state becomes such a standardized routine that  it makes sense to look for help to go to the next level. And the next level is… monads. Here is the deal, I said before that states will either exit your function through a further recursion or by return.  That means that there is a set of compatible patterns that can be used to assemble little bits of code that work on state, so that they nicely fit and the state zips around as an argument or as a return value. This pattern appears naturally when your work with states: it is the idea that the state is passed as last argument and that the stateful functions always return a state. As you want the function to do something useful, you systematically return a tuplet with a state and a function specific return value. Before understanding monads,  I wrote whole applications like this. What monads allow you to do is to standardize so much this method of call pattern that it becomes possible to strip the exposed code of these “peripheral” state going in and state going out. The result is code that looks stateless but that isn’t.  The value of monadic state management is that you make much less mistakes and you can specialize your monad further to give it special properties. Still, from a stateful perspective the main gain of monadic states is productivity: hiding the state mean you can focus on more important things.