Thinking about parallel and distributed computation

A classical way to look at parallel / distributed systems is in terms of:

1. Work-Efficiency
2. Parallelism
3. Locality 

This is one of those inconsistent triads (or trilemma), where only two among three properties can be chosen. For example, a system can have both work-efficiency and parallelism, or parallelism and locality but cannot be fully work-efficient, parallel and local.

This is not the only inconsistent triads to think about when designing scalable systems.

Here are more:

1. Velocity
2. Resources
3. Work

1. Momentum
2. Quality
3. Traffic

And of course the CAP theorem's:

1. Consistency
2. Availability
3. Partition tolerance

And the important:

1. Fast
2. Good
3. Cheap

All of these are ultimately important, yet some of these are more practically useful. Especially when you realise that you can weaken the inconsistent of these triads by introducing additional complementary properties. 

To illustrate this with examples:

• Two people must work together vs an engineer and a sales person must work together
• Two time constraints must be considered vs a measurement must last one microsecond and be done in the next two weeks.

The idea here is that by clarifying dependencies and scales we are 'loosening' our constraints. The second example above is about distinguishing what is short versus long (in duration), and also that a measurement is typically done before things are said to done. Therefore if we partition what is of 'short duration' vs what is of 'long duration', or what is 'done before' and what is 'done after', we soften the inconsistent of the triads above. For example, we could have:

• Work-Efficiency of all durations
• Parallelism of long durations
• Locality of short durations

As a result, we 'never have all three' for short durations, or for long durations.

In their most primitive form, these are binary properties. For example, things can be:

1. Small
2. Large

1. Slow
2. Fast

1. Short
2. Long

1. Light
2. Heavy

1. Precise
2. Approximate

1. Predictable
2. Unpredictable

1. Efficient
2. Inefficient

1. Up-stream
2. Down-stream

1. Present
2. Future

And so on...

These properties make sense when they are properties that build the inconsistent triad. Using arbitrary properties would not work. For example, tall vs short does not help us. And to note the triads above are different. For example, efficiency is not part of the CAP theorem, but does effect the other triads (and this depends also on how efficiency is measured).

Finally, all of these inconsistent triads are interdependent. You want to understand how. 

For example, my previous post was in fact based on two tightly depend triads, which I illustrate as follows with their 'limits' within a distributed system view:

The cost of a cloud architecture is strongly tied how these complementary inconsistent triads are approached.

All original content copyright James Litsios, 2022.