Thinking about parallel and distributed computation

A classical way to look at parallel / distributed systems is in terms of:

1. Work-Efficiency
2. Parallelism
3. Locality 

This is one of those inconsistent triads (or trilemma), where only two among three properties can be chosen. For example, a system can have both work-efficiency and parallelism, or parallelism and locality but cannot be fully work-efficient, parallel and local.

This is not the only inconsistent triads to think about when designing scalable systems.

Here are more:

1. Velocity
2. Resources
3. Work

1. Momentum
2. Quality
3. Traffic

And of course the CAP theorem's:

1. Consistency
2. Availability
3. Partition tolerance

And the important:

1. Fast
2. Good
3. Cheap

All of these are ultimately important, yet some of these are more practically useful. Especially when you realise that you can weaken the inconsistent of these triads by introducing additional complementary properties. 

To illustrate this with examples:

• Two people must work together vs an engineer and a sales person must work together
• Two time constraints must be considered vs a measurement must last one microsecond and be done in the next two weeks.

The idea here is that by clarifying dependencies and scales we are 'loosening' our constraints. The second example above is about distinguishing what is short versus long (in duration), and also that a measurement is typically done before things are said to done. Therefore if we partition what is of 'short duration' vs what is of 'long duration', or what is 'done before' and what is 'done after', we soften the inconsistent of the triads above. For example, we could have:

• Work-Efficiency of all durations
• Parallelism of long durations
• Locality of short durations

As a result, we 'never have all three' for short durations, or for long durations.

In their most primitive form, these are binary properties. For example, things can be:

1. Small
2. Large

1. Slow
2. Fast

1. Short
2. Long

1. Light
2. Heavy

1. Precise
2. Approximate

1. Predictable
2. Unpredictable

1. Efficient
2. Inefficient

1. Up-stream
2. Down-stream

1. Present
2. Future

And so on...

These properties make sense when they are properties that build the inconsistent triad. Using arbitrary properties would not work. For example, tall vs short does not help us. And to note the triads above are different. For example, efficiency is not part of the CAP theorem, but does effect the other triads (and this depends also on how efficiency is measured).

Finally, all of these inconsistent triads are interdependent. You want to understand how. 

For example, my previous post was in fact based on two tightly depend triads, which I illustrate as follows with their 'limits' within a distributed system view:

The cost of a cloud architecture is strongly tied how these complementary inconsistent triads are approached.

All original content copyright James Litsios, 2022.

Six ways to improve your organisational agility

Failing smart 

 My agile mantra is much:

"Failures must bring you closer to your vision"

It is a mantra I have often used. And very much how I approach agile innovation (e.g. see Search and Vision for Systematic Innovation)

However...

It is only partially true, as failure may happen simply because you are disorganised, with no relations to your vision. Fixing organisation issues helps avoid drifting further away from your vision, it does not bring you closer to your vision! Still, unmanaged organisational issues will eventually consume you 'from within', therefore they too must be addressed.

Can we focus on fixing organisation issues? Can we identify a subset of failures as 'organisational failures', and others as 'non-organisational failures on the way to our vision'? The simple answer is yes! Yet we most know what we are looking for.

Six types of failures

When I act as agile manager I try to distinguish between six types of failures. These are:

1. Fail Goal: Fail to achieve previously promised goals
2. Waiting: Fail deadline because of waiting on others
3. Underutilisation: Fail to use all your team members
4. Exhaustion: Fail to stay productive because of exhaustion
5. Inconsistency: Fail because of misalignments
6. Queuing: Fail because of past work no longer relevant, or past work never finished.

Failure 2 to 6 are all organisation failures!

Failure 1, not achieving goals, might be an organisational issue, yet may also be tied to the difficulty of the the task at hand.

Six ways to improve your organisation

We can express specific actions for each type of failure observed: 

#	Failure type	Observation	Corrective action
1	Fail Goal	Fail to achieve previously promised goals	Review / pivot how resources approach goals
2	Waiting	Fail deadline because of waiting on others	Review how resources approach work
3	Under-utilisation	Fail to use all your team members	Review how goals are broken down into work
4	Exhaustion	Fail to stay productive because of exhaustion	Review how goals are picked up by resources
5	Inconsistency	Fail because of misalignments	Review how work is shared across resources
6	Queuing	Fail because of past work no longer relevant, or past work never finished.	Review how common work impacts different goals

You may note the common patterns both in the observations and in the corrective measures above. This is because they all refer to the same system! This is important, and maybe the most important learning from this post. When managing fast greenfield innovation projects:

Comparing failure types is as important as to address specific failures!

This is because when a failure type happen more often than others, we can take organisational actions even before we understand the specific details of each failure! 

A final note: the approach is scalable.

All original content copyright James Litsios, 2022.

Reversible contextual Python with strong properties

I previously mentioned my effort to learn how best to do semi-formal development in Python. Here I report my progress, done during my end of year 2021 holiday break. 

The effort is to find a 'best design' to achieve certain properties, and I am doing this by rewriting code again and again to progress a 'design' towards a 'better design'. At rewrite 21 we have:

• A systematic way to write reversible, context aware code.
• Context specific algorithms can be defined over containers as well as over higher order functional applications.
• Logical and declarative 'free-variable' oriented style is possible.
• Function applications with context can be 'saved' to state and to be continued later or elsewhere.
• Quoted expressions can help define equalities and inclusion properties.

Some insights:

• This could be Java or C++! Python is handy with a focus on 'abstract' arguments, but traits would carry us through an OO implementations.
• What about types? The question is better expressed as: what about types within a formally strong stack? Types are good, but they too must be expressed 'with strength', otherwise they hinder you.
• How much value is there in the strong typing of Haskell and Scala? Much if you use it for formal properties. However if most of your types are 'structural boilerplate', then you are not actually gaining much. Formal properties with types means 'type level' programming, and that is not cheap, nor easy to scale. 
• This code does 'nothing real'!? It is not meant to! Its purpose is to show how design cost effective code with formal properties in 2022!

Here is a quick run through my efforts. To note each of these efforts are a 'throw' away effort with the goal of learning how to write 'stronger' more 'formal' code. Twenty-one rewrites later (about sixty hours of work), the effort was worth it:

1. This effort starts with a hard coded functions to swap a positional argument to keyword arguments and vice versa (the other way around).  Swapping is reversable, and our goal here is to write reversible code. 
• I built this 'keyword argument' constructor/destructor initially to bridge between the 'product' type nature of keyword arguments and the 'sum' property of positional arguments. Later, I realised that keyword are named, positional arguments are not (conceptually!), and that this first focus allows me to move freely between a nameless point-free style of programming with positional args, to a more classical 'named' style. This explains why I am still point-free when I use identifiers, because these can be removed with well chosen operators.
2. A generic keyword argument' constructor/destructor function can be implemented by simply 'accessing kwargs'. However, Python has no efficient ways to remove keys from a dict, and that forces us to use 'ugly' copy constructs to 'pass on' a kwargs, minus the chosen keyword. Therefore  version two uses eval and lambda to generate a an effcient generic keyword argument' constructor/destructor. 
• Retrospectively, I would have avoided the eval, and stuck with kwargs and copying the dict to remove an arg.
3. Lambdas are a pain to debug and to type annotate,  version three uses callable objects instead of lambdas.
4. None of the above is composable nor 'stateful' (from a 'traditional' perspective), as all happens within function arguments. Version 4 adds function composition and introduce a AKW class, to 'statefully' hold function arguments. It is 'reversible', in that we can 'call' (cocall) a function with the object's data as arguments. 
• AKW stands for Args KWargs, as it holds both the args tuple and the kwargs dictionary.
5. This is all dynamic typing, and initially at least, with no type conditional expression. However, to 'feel the ground', version 5 does the following: if two complementary functions/lambdas are expressed as callable object, then we can give their type classes a class attribute which programmatically provides this complementary tie. For example, given d_k1() an operator from keyword arg k1 to posional arg, one can find its complementary operator c_k1(), from a positional arg to a keyword arg k1.
• This works, we do not need it (yet).  
6. Focusing on function arguments is somewhat 'stream oriented'. The AKW class introduced in 4, is more of a classical 'stateful' approach. Version 6 shows how function composition can be constructed on this stateful view.
7. Functions and callable objects are nice. Yet by default they force us to create 'return value states'. These return states 'semantics' are different and do not naturally fit within our simple 'call' semantics. Another aspect is they are 'invisible' to the notion of reversible code.  Therefore version seven takes a first go at using tail contination to avoid 'return states' of functions. 
8. Continuation passing can express a 'deeper' execution context with the help of a 'stack' of tail continuations. Version 8 introduces a continuation 'list' for this purpose.
• I believe Gerald Jay Sussman's keynote and discussion - reClojure 2021 mentions using a stack of tail continuations in his recent work. I don't have much scheme/lisp practical experience, and therefor do not know how much stacking tail continuations 'is common'. 
9. Version 9 extends operators to have something equivalent to the notion 'quoted' expressions in Scheme/Lisp.
• While not futher developed here, 'quoting' is an 'extra dimension' and extremly valuable to define notions of equalities, and of inclusion (e.g. conbined with matching).
10. Version 10 introduces a basic map function. 
• Implemented for tuple, list, sets and dicts, the mapping is not always reversible. Which highlights that certain operations need more support to guarantee properties of reversibility. For example, mapping the set 1,2,3 to 1,1,1 is not reversible.
11. The classical tail continuation is only telling us where the output of a function should go. With a 'stack' / 'list' of continuations, we can carry more information, for example the 'index' or 'key' of the element that is currently being mapped. Version 11 shows how this is done, and shows how we can make a map able to recurse over hierchies of containers. Also this rewrite introduces a first 'partition' 'unpartion' operator.
12. Here we partition and unpartition classes of dataclass fields. 
• Object orientation is stronger when 'done backwards'. That is, composing general objects quickly breaks. However, given a large object that expresses a real world scenario, we can sometimes partition it 'with good formal results'.
13. Logical programming allows us to perform 'or' and 'and' like operations, and often have something similare to a 'not'. Version 13, shows how logical programming is 'easy' with tail continuation.
• Five class/functions are defined Success, Fail, First, All, and Fails. 
• Sucess always succeeds.
• Fail always fails.
• First is succeeds when the first of its arguments succeeds.
• All succeeds when all of its arguments succeed.
• Fails succeeds when its single argument fails, and fails when it succeeds.
• Here __call__ is success while 'fail' is called on failure. Later __call__ with the named keyword 'fail'=True indicates failure.
14. Adding a 'match' operator (with free-variable) to logical programming gives us a simple way to write declarative programs (here in Python).
•  The 'larger' Prolog program I wrote was a prototype compiler to assign 'topologically' senstive types (e.g. eges vs face vs element, etc.) and perform other transformations to a domain specific modeling language for semicondutor device simulations (applied math mid-90s).
• My current view about declarative programming (logical programming is a form for declarative programming): They only scales when embedded in a more general semantic stack (as done here for example).
15. Logical programming is more powerful with lambda expressions (or callable objects). Version 15 defines the operator SelectApplyN1, which apply multiple operators to a single argument, and keeps those that succeed. The goal is to validate that we can mix declarative and lambda like function application.
• Example: 
• Given: SelectApplyN1((Above(3), Above(5), Above(7), Above(8)))(6, continuations=...
• We get: (((7, 6), (8, 6)),) having defined an appropriate 'Above' operator.
16. In the previous code rewrites, spcial keyword arguments are defined (e.g. for continuation passing, for the free-variable list) and are explicitely defined in all the function signatures. Version 16, pushes this approach in a slightly non-sensical way by adding the keyword argument 'state' to the reference call signature, and explicitely adding this 'state' argument throughout the code. This results in the question: 'why be explicit'?
17. Version 17, only declares keyword arguments explicitly when used locally. However, the tail continuation list is an explicit keyword argument, and expressed everywhere. The 'standard' call signature looks like this:
• def __call__(self, *args, continuations: Optional['Continuations'],  **kwargs):
18. At which point I ask myself: "why not go all the way"? Therefore version 18 moves the continuation stack to a positional argument, as follows:
• def __call__(self, continuations: Optional['Continuations'],  *args, **kwargs):
• To note that we are back to a more 'classical' point-free formulation, as 'continuations' is positional and not keyword based.
19. Context annotation is revisited in version 19, and expressed more generically. To be specific, the continuation stack gets dynamically decorated with contextual information that can be queried 'within context'. These decorators are referred to by 'type'.
• The example code tags the logical operataors First and All (see 13), so that within the arguments of these functions can query 'which argument index are we'? To the example presented in 15 above, we get (((7, 6, 2), (8, 6, 3)),), because the successful operators were the third and fourth (starting from 0).
20. Continuing on context annotation, the recursive map of version 11 is adapted in version 20 and a 'depth' context specific function is introduced. The goal is to show that while "positional" context annotations is a handy nice, even more valuable is the ability to algorithmically extend your contexts.
21. Finally, we conclude this series with version 21 and a careful change ensures that the complete context stack is captured when capturing the state of compuation (e.g. with AKW as introduce in version 4).  This context becomes active again when we 'continue' compuations from each saved state (with cocall).

All original content copyright James Litsios, 2022.