Discover Natural Language Processing with the help of ChatGPT

The latest generation of machine learning is of magical education value:

• They can "show you that something is possible".
• They are quick: New concepts can be explored in seconds.
• They are very powerful.

Best to show by example. This post-Xmas 2022 week I am refreshening my natural language processing skills. Therefore I start my learning sessions with the following request:

Can you tell me the little red riding hood story while including the phrase structure name and the most important phrase structure properties of each phrase structure of the story.

https://chat.openai.com/chat responds with: 

Sure! Here is the story of Little Red Riding Hood, with the phrase structure name and the most important phrase structure properties included for each phrase structure:
Once upon a time, there was a little girl (noun phrase) who lived in a village (noun phrase) near the forest (noun phrase). She had a red hooded cloak (noun phrase), which she always wore (verb phrase, finite, past tense) on her daily walks (noun phrase) to visit her grandmother (noun phrase) who lived on the other side of the forest (prepositional phrase, place).
One day, her mother (noun phrase) said (verb phrase, finite, past tense) to her (verb phrase, finite, past tense, indirect object), "Take (verb phrase, imperative mood) this basket (noun phrase) of goodies (noun phrase) to your grandmother (verb phrase, finite, present tense, indirect object). She is not feeling well (adjective phrase) today (adverb phrase, time)."
The little girl (noun phrase) set off (verb phrase, finite, past tense) through the forest (prepositional phrase, place), and on the way (adverb phrase, time), she encountered (verb phrase, finite, past tense) a wolf (noun phrase). "Where (adverb phrase) are you going (verb phrase, present tense), little girl (verb phrase, present tense, direct object)?" the wolf (noun phrase) asked (verb phrase, finite, past tense)...

I cannot remember what are the typical phrase properties. So I ask, and get the following phrase structure properties:

• Head: the central word or group of words in a phrase that gives the phrase its main meaning
• Modifier: a word or group of words that provides additional information about the head of a phrase
• Function: the grammatical role that a phrase plays in a sentence
• Complementation: the addition of a complement (a word or group of words that is necessary to complete the meaning of a verb) to a verb phrase
• Dependency: the relationship between one phrase and another, in which one phrase relies on or depends on the other for its meaning
• Voice: the form of a verb that indicates whether the subject of the verb is performing the action or is the recipient of the action
• Tense: the form of a verb that indicates the time when the action described by the verb takes place
• Mood: the form of a verb that indicates the attitude or mode of the speaker or writer toward the action described by the verb
• Finiteness: the property of a verb phrase that indicates whether it is complete or incomplete
• Agreement: the correspondence in number, person, or gender between the subject and verb of a sentence
• Person: the form of a verb that indicates the perspective of the speaker or writer (first person, second person, or third person)
• Number: the form of a verb or pronoun that indicates whether it refers to one or more than one person or thing
• Gender: the form of a pronoun or noun that indicates whether it refers to a male or female
• Case: the form of a noun or pronoun that indicates its grammatical function in a sentence (subject, object, possessive, etc.)
• Aspect: the form of a verb that indicates the extent or completeness of the action described by the verb
• Formality: the level of politeness or formality in the language used
• Emphasis: the focus or emphasis given to a particular word or phrase in a sentence
• Syntactic role: the function of a phrase in the sentence
• Negation: the presence or absence of a word or phrase that indicates the opposite or negation of the action of the verb
• Emotion: the expression of feelings or emotions in the language used
• Quantity: the amount or quantity of something mentioned in the phrase
• Quality: the characteristic or quality of something mentioned in the phrase
• Condition: the state or condition of something mentioned in the phrase
• Reason: the cause or reason for something mentioned in the phrase
• Manner: the way in which something is done or happens mentioned in the phrase
• Place: the location or place where something is or happens mentioned in the phrase
• Time: the point in time when something is or happens mentioned in the phrase
• Comparison: the comparison of one thing to another mentioned in the phrase
• Possession: the ownership or possession of something mentioned in the phrase
• Purpose: the intended or desired result or purpose of something mentioned in the phrase

I then do the same for the phrase structures. Here is a descriptions and their major properties of the first major fifteen:

1. Noun phrase: a phrase that functions as a noun in a sentence and consists of a head noun and any modifiers. Major properties: head, modifier, function, agreement (person, number, gender, case).
2. Adjective phrase: a phrase that functions as an adjective in a sentence and consists of an adjective as the head and any modifiers. Major properties: head, modifier, function
3. Adverb phrase: a phrase that functions as an adverb in a sentence and consists of an adverb as the head and any modifiers. Major properties: head, modifier, function.
4. Verb phrase: a phrase that consists of a verb and any modifiers or complements and functions as the predicate of a sentence. Major properties: head, modifier, complementation, function, tense, mood, finiteness, agreement (person, number).
5. Prepositional phrase: a phrase that consists of a preposition, its object (a noun or pronoun), and any modifiers of the object, and functions as an adjective or adverb in a sentence. Major properties: head, object, modifier, function.
6. Infinitive phrase: a phrase that consists of an infinitive (the base form of a verb preceded by "to") and any modifiers or complements and functions as a noun, adjective, or adverb in a sentence. Major properties: head, modifier, complementation, function.
7. Gerund phrase: a phrase that consists of a gerund (a verb form ending in "-ing" that functions as a noun) and any modifiers or complements and functions as a subject, direct object, subject complement, or object of a preposition in a sentence. Major properties: head, modifier, complementation, function.
8. Participle phrase: a phrase that consists of a participle (a verb form ending in "-ing," "-ed," or "-en" that functions as an adjective) and any modifiers and functions as an adjective in a sentence. Major properties: head, modifier, function.
9. Appositive phrase: a noun or noun phrase that renames or explains another noun or noun phrase, and is set off by commas or parentheses. Major properties: head, modifier, function.
10. Absolute phrase: a noun phrase, participle phrase, or adjective phrase that modifies the whole sentence, and is not directly connected to the rest of the sentence. Major properties: head, modifier, function.
11. Participial phrase: a phrase that consists of a participle and any modifiers and functions as an adjective in a sentence. Major properties: head, modifier, function.
12. Prepositional object phrase: a phrase that consists of a noun or pronoun and any modifiers and functions as the object of a preposition. Major properties: head, modifier, function.
13. Subject complement phrase: a phrase that follows a linking verb and complements the subject of the sentence by renaming or describing it. Major properties: head, modifier, function.
14. Object complement phrase: a phrase that follows a direct object and complements it by renaming or describing it. Major properties: head, modifier, function.
15. Adverbial phrase: a phrase that functions as an adverb in a sentence and consists of an adverb as the head and any modifiers. Major properties: head, modifier, function.

For inspiration, I ask for an ascii text representation. (I bet that using ASCII is not very modern!). Which produces:

Once upon a time | noun phrase | there was | verb phrase / \ finite past / \ a little girl / | \ | \ a an the little | | noun noun | | who who / | \ | \ lived in a village | | | noun noun noun | | | near the near | | noun noun | forest | noun | she | noun | had | verb | a | noun | red | adjective | hooded | adjective | cloak | noun

                                             ... 

Then I ask for a Prolog representation:

% Characters
character(little_red_riding_hood).
character(wolf).
character(grandmother).

% Relationships
relationship(little_red_riding_hood, grandmother, granddaughter).

% Locations
location(forest).
location(path).
location(grandmother's_house).

% Actions
action(little_red_riding_hood, start_journey, forest).
action(little_red_riding_hood, go_through, path).
action(little_red_riding_hood, arrive_at, grandmother's_house).
action(wolf, start_journey, forest).
action(wolf, go_through, path).
action(wolf, arrive_at, grandmother's_house).
action(wolf, disguise_as, grandmother).
action(wolf, eat, grandmother).

% Items
item(basket).
item(cake).
item(wine).

% Facts
has(little_red_riding_hood, basket).
contains(basket, cake).
contains(basket, wine).

A which point I noted that this is the 'fabric' of the little red riding hood story, but much is missing. So I ask the machine learning to produce an alternative Prolog version that captures an object centric view and the emotional tension of the story. Which it produce promptly!

% Objects
object(little_red_riding_hood).
object(wolf).
object(grandmother).
object(basket).
object(cake).
object(wine).
object(forest).
object(path).
object(grandmother's_house).

% Object properties
property(little_red_riding_hood, human).
property(wolf, animal).
property(grandmother, human).
property(basket, container).
property(cake, food).
property(wine, drink).
property(forest, nature).
property(path, nature).
property(grandmother's_house, building).

% Object relationships
relationship(little_red_riding_hood, has, basket).
contains(basket, cake).
contains(basket, wine).

% Emotional tension
emotion(little_red_riding_hood, excited, start_journey).
emotion(little_red_riding_hood, worried, arrive_at_grandmother's_house).
emotion(wolf, hungry, start_journey).
emotion(wolf, satisfied, eat_grandmother).

Hmmm... that is pretty good. The last thing we need is a bit more temporal and spatial details. Which ChatGPT produces as:

% Temporal dependencies
before(start_journey, go_through).
before(go_through, arrive_at).

% Spatial dependencies
adjacent(forest, path).
adjacent(path, grandmother's_house).

Finally, we can ask for a few query examples, which ChatGPT produces as:

Who is Little Red Riding Hood's grandmother?
?- relationship(little_red_riding_hood, Grandmother, granddaughter).

What is in Little Red Riding Hood's basket?
?- has(little_red_riding_hood, Basket), contains(Basket, Item).

What is the emotional state of Little Red Riding Hood when she arrives at the Grandmother's house?
?- emotion(little_red_riding_hood, Emotion, arrive_at_grandmother's_house).

What actions must Little Red Riding Hood take before arriving at the Grandmother's house?
?- before(Action, arrive_at).

What locations are adjacent to the forest?
?- adjacent(forest, Location).

PS: Prolog because it is compact and easy to express in production with a monadic FP style.

PS: In the first version of this post, ChatGPT produced incompatible snippets, but as I tried to clean up the post's style, the new snippets all fit together, and the post was updated.

James Litsios, 2022.

Managing agile from technology to business

 This how I tend to look at management ownership in agile development (see previous blog entry for more on the alternate views of agile):

Where:

• The CTO owns technology and most often engineering teams.
• The Engineering Manager may be an inwards looking CTO and also own technology and engineering teams, but is often less owning technology and more owning agile processes and engineering teams.
• The Product Owner owns the team's product vision and possibly the customer strategy. She/he is a member of the development team or has a representative that is part of the development team.
• The Technical Product Owner owns the customer view of technology, may be a full fledged product owner, or may be a support member of a larger product owner team.

Individuals become more and more key enablers as your product and technology base become more advanced. A Servant Leader is then a manager (e.g. engineering manager) that helps individuals contribute in independence yet within orchestrated flows. This figure illustrates where the servant leader provides critical guidance (the red corners!):

Here, from the central diagonal outwards:

• The team iterates a technology and product development and review cycle, again, and again.
• In parallel, the team learns and explores productivity and innovation. Product does the same with their customers and market segment.
• Productivity issues are monitored. In a physical model of agile processes 'high momentum' is high productivity. Servant Leaders help monitor for issues such as unsustainability, diverging focus, low or over capacity, and help teams and product avoid or resolve these issues.

 My experience is:

• Product Owner and CTO have too much "skin in the game" to be effective servant leaders. Therefore engineering managers and technical product owners (when they have only partial ownership) have a critical servant leader role to play to help all deliver better.

All original content copyright James Litsios, 2022.

Agile from business to technology

This is how I scale agile towards business and towards technology.:

Where:

• In the center we have an agile process that blends team process and product process.
• To the right we have a business process that ties into the product process.
• To the left we have a technology process (expertise process to be general) that ties into the team process.

To be honest, this is my minimal theoretical business to technology agile process. In practice, a specific agile process is chosen, with process steps and process artefacts. People like to say that agile is about people, not process. Yet people need to communicate, and the common vocabulary for that is given by a process.

Deepening in to the product business view (right side above):

Where:

• The core agile process (blue) delivers features and deliverables that are part of a product and business goals.
• These deliverables are picked-up by customers
• The features and deliverable sustain (or not) the customers competitivity within its market segment
• The product owner (in scrum terminology) is the expert that guides the product with regards to the customer needs.

We can do that also for technology/engineering (left above):

Where:

• The core agile process (blue) includes experts within the team.
• The experts enable/produce the technical/engineering needs of the features and deliverables.
• The expert individuals may be experts in different domains.
• Coordinating technology/engineering frameworks allow experts to work both as individuals and in coordination (within their team).

Many domains of technical expertise are possible. Above are the ones closer to my experience.

Note:

• The complete process above needs to coordinate at least SIX streams.
  A minimal agile process only presents three streams (team vs product vs customer). 
• Each of these (six) streams will naturally be driven at their own different rate!
• Your job, as CTO / engineering lead is to work with individuals, teams and product to pace work within these different streams to achieve an optimal throughput!
•  Think laminar flow! Productive is as fast as you can go before things become turbulent!

I wrote this up as a (much longer) slide deck, and am always happy to share my experiences!

All original content copyright James Litsios, 2022. 

The future of software development (end 2022)

 

My end of 2022 predictions for software development:

1. Structure of code becomes more important than code
2. Partial code becomes more valuable than finalized code
3. Proprietary code gains in value
4. Contextual programming gains in traction
5. IDEs become ever more ML driven

And therefore:

1. Dynamic typing gains over static typing
2. Duck-typing grows in value
3. IDE security and spyware become an ever more important topic
4. Functional programming is still the more powerful programming paradigm
5. Proprietary IDE with in-house ML engines will grow in importance

All original content copyright James Litsios, 2022. 

Wild in the forest: the tricky limits of ML

As a proof of concept, I had our favorite ML based chat write the following song for me:

Once upon a time in a forest so green
There was a girl with hair so red, you know what I mean
She was looking for some fun, and she found it with the woodmen
They were drunk and wild, but she wasn't afraid
They ran and wrestled, and had a blast 

(the song goes on...)

(and song ends with)

Red riding hood, she never looked so good

Hanging with the woodmen, in the forest she stood

They were rough and rowdy, but she loved them just the same

Red riding hood, she was never one to play it safe. 

Note I did not suggest "fun", "wild", "rough and rowdy", "ran and wrestled", "had a blast", "play safe", ...
I just asked to replace the wolf/wolves with 'drunk woodmen'.

Asked to summarize, the chat is happy to tell me:

This song tells the story of Little Red Riding Hood, but with a twist. Instead of encountering a wolf in the forest, the young woman meets a group of drunken woodmen and has a wild and exciting time with them. The villagers are shocked by her behavior, but she doesn't care and continues to have fun with the woodmen. The song celebrates her freedom and adventurous spirit, and portrays her as a rebellious and independent young woman who is not afraid to take risks and live life on her own terms.

Asked of inappropriate elements for children in the song, the chat correctly states:

The song contains references to alcohol and drunkenness, which may not be suitable for children. In addition, the woodmen are described as rough and rowdy, which could be seen as potentially disturbing for younger audiences. Overall, the song has a rebellious and slightly dark tone that may not be suitable for younger children.

Yet the ML misses the obvious. And of course it does: there is no adult behavior within the scope of little red riding hood, and therefore the ML has no local framework to constrain the rephrasing. And the global framework is not able to capture the issues within the strong relations of the story elements. 

A fascinating challenge to enhance ML to be able to capture subtle innuendos and cultural contexts. 

Note: The chat suggested the title: "Wild in the forest" to "capture the sense of adventure and freedom that the young woman experiences in the forest with the woodmen".

All original content copyright James Litsios, 2022.

Everybody can be a science fiction writer with ChatGPT

Let's write some science fiction with ChatGPT (https://chat.openai.com/chat).

I gave it three requests:

1. Can you help me write a short science fiction story? Use two stories as guidelines. The reference story is  the little red riding hood with the wolf as an a scary but nice alien race. The second reference story is Cinderella with the step mother as bad but beautiful and charming member  of the aliens. Also, these aliens use flying skateboards to travel.
2. Can you rewrite with Bela wearing a mini-skirt and the prince having muscles?
3. Can you make it longer and have Bella and the prince romance and kiss a bit?

FYI:

1. I recently spent way too many science fiction hours listening to something like sixteen S. H. Jucha audio books in the Silver Ships series (to give up on a very poor choice of narrator).
2.  The hero in S. H. Jucha is the 'strongest man around' and his girlfriend is 'incredible beauty in thin textile'
3. Romance and kissing is ok in science fiction. Still important is not to have too much of it.

The results:

Once upon a time in a faraway galaxy, there lived a young girl named Bella. She lived with her stepmother, who was a beautiful but ruthless member of an alien race known as the Wolfen. The Wolfen were feared throughout the galaxy for their sharp teeth and fearsome howls, but Bella's stepmother was particularly cunning and charming.

One day, Bella's stepmother tasked her with delivering a basket of food to her sick grandmother on the other side of the planet. Bella put on her red mini-skirt and set off on her flying skateboard. As she flew through the forest, she encountered a Wolfen wearing a hooded cloak.

The Wolfen smiled warmly at Bella and offered to lead her to her grandmother's house. Bella, feeling safe with the Wolfen, agreed and followed him through the forest.

But as they approached her grandmother's house, the Wolfen revealed his true form and attacked Bella. Bella managed to fight back and escape, but not before the Wolfen took a bite out of her red mini-skirt.

Bella reached her grandmother's house and told her of the encounter. Her grandmother, who had been expecting the Wolfen, revealed that he was actually a prince from a rival alien race who had been cursed to take on the form of a Wolfen.

Bella and her grandmother banded together to break the curse and free the prince. In the end, the curse was lifted and the prince was restored to his true form, revealing his muscular physique. Bella and the prince fell in love and began to spend their days together, traveling on their flying skateboards and exploring the galaxy.

As their romance blossomed, Bella and the prince shared many tender moments and even shared their first kiss. But their happiness was short-lived, as Bella's stepmother continued to plot against them.

In a desperate attempt to get rid of Bella and marry the prince herself, the stepmother hired a group of bounty hunters to capture Bella. But Bella and the prince were able to outsmart the bounty hunters and escape.

In the end, justice was served and Bella and the prince ruled over the kingdom together, using their flying skateboards to travel across the galaxy and spread peace and prosperity. And they lived happily ever after, their love stronger than ever.

All original content copyright James Litsios, 2022. (plus a bit of ChatGPT content this time!)

How to Get the Best Out of an ML Chat System

Some recommendations to improve your interactions with an ML based system, such as a ML chat like ChatGPT https://chat.openai.com/chat:

1. Structure your requests
• Structure your requests in a way that is explicit and large in structure. This will help the ML algorithm "unroll" your request and provide a more structured and in-depth response. Avoid using too much implicit structure, as this may confuse the ML. 
2. Rephrase your request with previous reply content
•  Rephrase your request using words and themes from the previous replies, as the ML may provide additional content to explain its inability to respond. By using these words and themes, you can bring your request into a form that the ML can provide response content for.
3. Change the content order within your request
• Try changing the order of the structure of your requests, as the ML's approach to the response may be overly constrained by the input order. 
4. Use analogies to structure your request
• Use analogies to structure your request can also help strengthen the structure of the response. You can do this by asking the ML to find an analogy between the subject of your request and a reference that has a lot of structure already assimilated by the ML. The request might be phrased as: 
• How is unstructured-request related to structured-knowledge?
5. Anchor the request structure in known complementary relations
• A powerful way to structure a request is to provide complementary input. The idea is that you provide a well know implied relation, or you implicitly provide a well know relations such as that A is different to B, and you guide the ML's response by telling it to use the provided relation as structural base for the response. Additional complementary attributes can be added to shape the response.  The request might be phrased as: 
• How is structured-input related to unstructured-request?
• How is complementary-input-1 different from  complementary-input-2 in the context of unstructured-request?
• If structured-input is true, how is unstructured-request false?
• How does structured-input  imply unstructured-request?
• Debate unstructured-request as a dialog between reference-profession and complementary-profession?
6. Generalize, refine, strengthen, weaken your request
• Add extra words and context to generalize, refine, strengthen, or weaken your request to shape the response. This adds implied structure and helps guide the ML in its response. 
7. Use negations
• Use negations in your request to help broaden the scope of the ML's replies.
8. Change the point of reference of the request
• Change the point of reference of the request to change the basis on which the response is built. For example, request 'how does A become B' might be rewritten as 'how does B derive from A'. 
9. Limit implied structure within requests
• Limit implied structure within your requests, as the ML may struggle to infer this structure on its own and may make mistakes.
10. Reset the chat thread
• In a chat mode, previous requests may also over-constrain the ML's replies, so it may be best to reset the chat session before continuing. 
11. Request alternate reply forms
• Request the answer in 'reverse order', 'from other point of view' (see change of point of reference above), 'as a semantic graph', 'as code' (e.g. Python). Request reply as a 'rap song',  a 'fairy tale', a 'debate/dialog between ...'.
12. Request alternate views without key elements
• Rephrase the request it include a removal of a key element of the 'normal' request response. This will allow the system to present how that key element relates to the rest of the response. The request might be phrased as:
• What is input without key-element-from-previous-reply-to-input?
13. Limit lower bounds of request details
• Request 'over X reply-category'. For example, a semantic graph with over ten nodes and one hundred edges.
14. Use your imagination!
• Finally, remember to use your imagination and think creatively about how to qualify your requests to shape the response of the ML system. There are many ways to do this, and using a variety of these techniques can help improve your interactions with the ML system.

All original content copyright James Litsios, 2022.

Nouriel Roubini's 7 Cs of crypto. Some FTX thoughts.

Pondering if I should write something about FTX, I fell on this recently posted Nouriel Roubini's 7 Cs of crypto, which he states as:

1. Concealed
2. Corrupt
3. Crooks
4. Criminals
5. Con man
6. Carnival barkers
7. C... with response here

Should we add:

8. Credulous
9. Careless
10. Childlike
11. ...

Yet all of these have positive roots:

1. Privacy
2. Liberty
3. Freedom
4. Independence
5. Society
6. Economy
7. Entrepreneurs
8. Optimism
9. Adventurous
10. Youth
11. ...

We might call that first extended Roubini list the crypto-challenge list, and that second, the crypto-potential list.

A challenge is that most of us learn to appreciate the crypto-potential list before we learn how to avoid the crypto-challenge list. Ranting about the crypto-challenge list does not help. And in many cases, much experience is needed to develop strategies to address the crypto-challenge list.

A bigger challenge is that risk management is done by projecting past negative events into future possibilities. Part of the game of fraud is to recast the past into a different present, and to avoid risk management by doing so, as no past negative event exists of the presented product. Caveat emptor, buyer beware, as they say. 

Twin tower memories

 My summer 2000 lunches were taken sitting under the twin towers in New York. I was there for the recently opened Actant's NY office, and would either have lunch at Battery Park or looking up at the towers.

I have sometimes described my work as "if I were an architect, I would design skyscrapers", to then add "it is a pride thing, but also life is short, so let's push things forward". To eat my lunch below the towers, was in part to bask in the marvel of engineering tall buildings, but also to feel the ego that drives us to surpass ourself, and often others too. 

These are some pictures of the towers I took:

When the towers went down, I truly could not believe it. My old black and white TV had been switched on in the office (in Zug, Switzerland), my colleagues were huddled in front of it, I walked over, to hear someone say "one of the towers is down", to which I answered: "that is not possible, it must be hidden by the other". It was a sad day. 

Recently, I was asked "how does it look now?". And to note, I had chosen not to remember, even though I've been there a few times since then. I can visualise the inner mall, elevators, and escalators, yet my core memories have them standing tall, and I enjoying a sandwich below.

All original content copyright James Litsios, 2022.

Learning as a personal experience: James Taylor vs George Polya

How best to learn? How best to build deep personal growth? 

I noted this 1995 advice from James Taylor (in this YouTube), on how to become a 'musician like him':

Be as self-contained as possible, to keep the overheads to a minimum... spend as much time free and lonely as you can. ... I think lonely and free go together... somehow. And then you can’t help but evolve, if you keep your mind open, and you are not denying your experience, not shutting yourself down, not sort of clamping yourself down, or letting other wastes of time come in and claim all of your time. If you keep yourself open and free, then things will happen to you, and you will be educated by them and you will evolve, you will grow.

Is this good advice? Does deep personal growth only happen alone?

To contrast this with George Polya who states in 1966:

Teaching is giving opportunities to students to discover things by themselves.

We might rephrase this as:

To learn, students must be given the opportunity to discover things by themselves.

So in fact Polya is also saying that learning happens by actions of the students, not anyone else.

There are a few things to mention here:

1. We may learn by discovery, we may learn by imitation. Both James Taylor and George Polya are referring to learning by discovery. Learning by imitation, by definition, cannot be done alone.
2. Polya is not excluding that students 'discover together'. What is excluded is learning where each student does not personally experience the discovery from 'not learned' to 'learned'.
3. For Polya, the teacher is responsible that each student experiences their learning.
4. For James Taylor, the student is responsible to 'shut-away' others to experience alone their learning.
5. Polya is referring to learning math, James Taylor to learning to be a singer song writer.
6. Learning math can be organized with others, learning the humanity and introspection that goes into song writing is much about "maintaining one's aloneness", which some might call narcissism, remoteness, egoism, detachment, a religious experience, or even being a hermit.

Both advices are similar: learning is best experienced personally, and sometimes experiences learned alone change what is learned. And one cannot help to note that James Taylor naturally expresses the individual nature of learning in an asocial 'push others away' manner, while George Polya stays in a social and organized setting of teachers and students.

All original content copyright James Litsios, 2022.

Software with formal properties at a low cost through monads and co-execution

Not so long ago a good friend of mine mentioned his need for 'formal properties' in his code.  

I asked him if he had already made the code purely monadic. He had. "Good", I said, "because the 'cheap route to formal properties' is first to ‘monadize' your code, and then 'reverse it' by bringing in 'co-execution'".  That last part needs some development. 

Formal properties are not only about execution, actually mostly not about execution, but more about formal properties of state. So while you do need to show execution invariants, you often much more need to show that the ‘results’ happen within stateful invariants (e.g. of the ledger). The easiest way to do this is to  deconstruct / co-execute the state model backwards. This 'backwards' logic must 'match-up' with the 'forward execution' (monadic) model. My view has always been comonads are easiest for this, especially if your stateful model (e.g. a blockchain) is already something that is easy to express as a comonad. 

You might note that my last post is all about combining 'forward' and 'backwards' arrows. One way to interpret these arrows is 'right arrow' for monads, and 'left arrows' for comonads. And sure enough, that is an easy way to sketch out higher order systems where adjoint invariant properties (e.g. rights, expectations, authorizations, ...) are assembled 'with good properties' by design. As mentioned in previous slide, monads are easy to express as CPS, and as are comonads with a bit more 'gymnatics' (see previous post).

All original content copyright James Litsios, 2022.

Why Use Arrows and PowerPoint to Express Software Designs?

Dog Drives Car 

Silly me shared the following picture with my sister-in-law yesterday,

as she was stuck in a massive traffic jam with her dog (in photo) in her mini. 
"How did you make the picture" was the question. "PowerPoint" was my reply.

I used PowerPoint for that photo because it took me three minutes, and I had a slide deck open.

Stop, you might say, this has nothing to do with programming, or software designs! It does if we want to design a software where a dog drives. Yet how would we approach that? Here I tell you how I have designed and communicated designs of 'big and bold' softwares quickly and with good properties. And will also sketch out how to approach exotic requirements, such as dogs driving cars.

Arrows Away!

Let me show you how I work. This how I use to draw design 28 years ago (from my PhD thesis):

This is how I would have drawn that design 18 years ago (~2004):

Here I use arrows to represent flows. Arrows are used to capture the Lagrangian interpretation of flows (e.g. as stream oriented design). In this case: for a given set of inputs, there is one or more sets of models, and for each of these sets there is a solver. If this sounds confusing it is, as it was ambiguous. It is ambiguous as it is not clear which flows we are talking about. Are these flows of the actual simulation calculations (e.g. variables of the simulator, this was a simulator), are these flows of structural data dependencies (e.g. how big each array is), are these flows of types (e.g. real vs complex). The solution to that problem in 2004 was to draw multiple flows. For example, here we have the flow of configuration, and the flow of object hierarchy:

Note here I introduced these flows as Lagrangian, meaning we are 'moving along with the flow'. When implementing flows, there are always a 'outside observer' view of the flow, the Eulerian interpretation (e.g. a CPU core's view). These can also be drawn with arrows (e.g. see my 'old' stream processing video). In 2004, I would have used two different arrow types to show the difference of flow representation. For example in this diagram the 'stream processing solver engine' produces 'a stream of variable data': 

Let's mention the strong influence of multi-stage programming on some of these 'notation concepts' (see Walid Taha ~2000 work referenced here) as one way to think of these arrows would be like this:

However, such a notation implies that the 'telescoping flow' has its own semantics, and that is tricky. An easier approach to draw something like this:

The idea being that the solver 'thing' is specialized with a model 'thing' (returning a new flow thing), which we can then specialize again with a variable 'thing', resulting in a 'fully formed flow thing' that includes the solver, model, and variable details. Importantly, we are no longer 'telescoping' things when we do it this way. Therefore one learning is that 'telescoping' is one representation, others are possible.

Holy Trinity Continuum

The notion of telescoping is much associated to monads. While the concept of 'specializing something with something else' is often implemented with with comonads. Yet this is not critical information to know. The reason being that monadic like logic can be expressed with continuation passing style (CPS), and again can also be expressed in defunctionalized forms (see this early ref).  We have something that looks like this:

Even better, we have something that looks like this (correct me if I wrong, I am writing this fast today):

This last diagram is 'my holy trinity' of programming design belief, as I do not need to worry if a design will be implemented with types, closures or functions, as one can always be mapped into another. Instead, the key concept is that a design should express what is invariant no matter  'how it is implemented'. Said differently, designs are best when they express invariants that hold over multiple variants of implementations.

Notes:

• CPS is really functions + a more or less complex scoping logic implemented across the functions.
• Things are a little bit more tricky when we interpret things backwards. Still similar concepts  hold.

Scoping Invariance

A primal invariant to mention is about 'variable scoping'. (The applicability of the concept is broader, and applicable to containers with access by indices versus access by search). Early languages noted two fundamentally different approaches to 'variable scope' semantics, the so called static and dynamic scoping. Dynamic scoping is still much a curiosity to developers in 2022. However, Richard Stallman writes: "Dynamic scope is useful".  So what gives? The trick is that we need to restrict the dynamic scope semantics to make things scale. A good choice to limit dynamic scoping to have only 'one point of variable access'. With that we have the following symmetry:

• Static scoped variables are 'defined at one place' and possibly used many times.

• Dynamic scoped variables are 'defined possibly many times' used used 'at one place'. 

An 'semi-formal' insight is that dynamic scoping is static scoping 'running backwards' (with the restriction above). In a simplified arrow form, this looks like this, indicating an invariance correspondence between static scope going forward and dynamic scope going backwards:

Inspired by this, my notation rule is: arrows that go left-to-right are symmetric to those going right-to-left. Often just the direction is needed. For example, in the following diagram:

If A has a static scope, B has a symmetric dynamic scope, and vice-versa.

Directional Invariants

There are many software invariants. Many of them have their roots in the notion of 'sequence of effects'. For example, if we write something, then to 'read it' is just like 'to write it but backwards'. Here too we can use arrows to capture these relations, for example:

Again, the idea is: if a left-to-right arrow has a design feature, then there is matching 'reverse logic' right-to-left arrow with a 'complementary design feature'.  I have about 10 of these invariant symmetries that I use 'all the time' to design, and a broader list of over 50 symmetries that I like to refer too.

Arrows and Dimensions to Help Design Software

I have been practicing to 'think backwards' for over ten years. (I sound incoherent when I forget to 'switch that mode off'). You may well wonder what 'running backwards' means. Luckily the following 'equivalence heuristics' help:

• While we 'execute forward' we can accumulate a lazy execution of reverse logic which we can run 'at the end' of the forward execution. At which point the reverse execution will be run forward, yet it will still be the reverse logic. (This is how reverse mode auto-diff is done).

• We go 'up' the call stack, after going 'down'.  Similar to the previous bullet, we might say that we go forward as we go down into function calls, and go backwards as we return out of each function. 

• A two phase commit has 'forward computation' in the first phase, and 'reverse computation' in the second phase.

The important insight here is that reverse logic does not need to be run backwards. Yet it does need to be 'arranged in the design' to have a proper 'reversed logic'. Therefore we do not need to draw reverse logic from right-to-left if we have a way to indicate that it is 'forward execution that manages backward logic'. I use the U-turn arrow for that. The previous example can be drawn as:

Nothing imposes directions, therefore the following is a perfectly valid design sketch:

By which I am saying: the design will verify things, accumulating a reversed execution logic which will be interpreted. 
You may note that I am being 'very one dimensional', limiting myself to a left and right direction. We can in fact introduce as many dimensions as we want. Two dimensions (horizontal and vertical) often suffice to describe large architectures. (Sometime more dimensions are brought in with 45" or 30" and 60" degree arrow directions). For example, we may indicate our I/O logic 'vertically':

When we combine the two, we might indicate the following two designs:

Which one to choose?
Both!
One way to read these designs are as follows:

• 1 top: while we write we accumulate to confirm the overall 'write commit' only when 'all writes' are valid from a 'reverse perspective'.

• 1 bottom: While we read we interpret.

• 2 top: To verify we read 'the past':

• 2 bottom: we can 're-interpret' the past by writing.

A careful reader might note that while we may want 1 and 2, we cannot just have them! The issue is that when combined, we are mixing 'left-to-right' and 'right-to-left' arrows 'of the same thing'. For example, 1 has Verify as a U-turn and 2 as Verify as a left-to-right arrow. That will not work, as one is the 'reverse direction' of the other. 

Further Directional Invariants

There are many important software design concept. Here are three more, expressed with complementary arrows:

In simple terms: A coherent design does not have complementary design concepts 'at the same level'.  What the above is saying is:

• Both a local and global design must be expressed, and both are complementary. 

• Providing authentication allows us to 'look back' with trust.

• If new 'things' are 'freshly' created, then at some point they become 'stale'. And two complimentary design logics must be designed to deal with this. 

With that in mind, the previous two designs can be brought together as as single coherent design:

To note that this is how I was designing software early-2015 (see my pre-Elevence post).
(I leave the reader the reader the exercise to bring in authentication, trust, freshness and staleness).

Semantics, Tools, and Teamwork

Given a team of developers, I might suggest that design 1+2 be implemented with local monads and a global comonad (e.g. as a blockchain). Yet once I do that I lose flexibility. I also lose my audience. Instead I draw arrows, and explain what I mean with these arrows. I say what properties we 'the development team' are looking for, and sure enough we end up with a decent design that is robust and strong.  Drawing design 1+2 takes just a few minutes in PowerPoint, and maybe would have needed a few tens of minutes to sketch out before (e.g. on a sheet of paper). Typically, I duplicate my 'master design slide' many times, to then add annotations to show how we might 'slip in' extra design requirements. For example, the notion of trust and authentication, and here, the logic to deal with 'staleness'. 
I started this post with a picture of a dog driving a car. The important property of a software design is to be 'consistent by design'. Were I to design a 'dog driving car' software design. I would start by sketching out the complementary properties that I would need to bring in to the design. These might me:

Then I would draw these in PowerPoint and iterate and grow to create a good design.

All original content copyright James Litsios, 2022.

What to study? Machine learning vs Computer Science vs Cyber Security vs Financial Engineering

Today I helped a student review his study options (Bachelor level).

We discussed and looked at the courses the university offered in four areas: machine learning, computer science, cyber security, financial engineering. Debating the 'deeper analytical learnings', I mentioned the following:

• Machine Learning
• ML is much about models of models in a very mathematical sense.
• No one knows how to teach it that way (yet).
• One tends to learn more about ML tools and technics than theory.
• Computer Science
• Few model in computer science because expensive, very hard to manage and hard 'get right'. 
• However, applied math and crypto/blockchain domains are easier to approach in a 'modelled CS approach'. Functional programming is a highly effective way to be successful in doing that. 
• The CSL language from Elevence, a functional language, was a rare 'purely modelled CS effort' (inspired by complementary higher-order type properties, now named DAML). It was also a rare effort in covering all of the four domains of this post!
• Cyber Security
• In cyber security, we model the present and the past. We adjust our current state when we detect a security breach in the past.
• Financial Engineering
• In financial engineering, we model the present and future. We adjust our current state to be optimal in the 'modelled' future. 

Talking about job opportunities, I drew the following:

The figure above is meant to convey the following:

• Machine Learning
• Much demand as had a broad application and quick return on investment.
• Computer Science
• Good demand but by fewer companies as there is a certain truth that writing software takes time and is expensive. 
• Cyber Security
• Much demand as has a 'quick' return on investment, but only applicable on IT solutions.
• Financial Engineering
•  Lesser demand. Applicability is more limited. When 'finance' is accounting you do not need 'engineering' or 'complex math'. 

All original content copyright James Litsios, 2022.

Three step research process

Use your brains when developing abstract innovation. It is a game of 'balance': you think-up what is needed, you set goals that take advantage of your thinking, then you work towards your goals. If on the way something does not fit, review your goals or thinking. It is an iterative effort, you try, you repeat, until it works.

Here are the three iterative steps in more details:

1. State what you believe is true
2. Express implementation goals that will leverage your beliefs
• Check consistency of goals with respect to your environment and to underlying beliefs
• Correct your beliefs if inconsistencies are found between goals and environment
3. Develop towards your implementation goals
• Check consistency of development with respect to environment and underlying goals and beliefs
• Correct goals or beliefs if inconsistencies found between development and environment

At first, one tends often to 'juggle' and develop just one idea at a time, when developing this way. Yet with experience it becomes clear that it is the iterations that take much time, and therefore it is often worthwhile to think-up more than one idea, and bring them forward in parallel until the environment gives feedback, and one idea turns out to be better than another. Lean is therefore not always agile when working with abstract development.

All original content copyright James Litsios, 2022.

A laptop, desk, beamer and flip chart is all you need!

July 1 2015,  I moved into a single office room at the Zurich Technopark as the sole founding employee of Elevence Digital Finance (11ce). I assembled my sit-stand-desk. I used my personal laptop to write the specs for an 'authenticated contract logic database'. Also, I setup my beamer and flip chart stand! I mention this because I found the following note today dated July 1, 2015, where I 'sold' those objects to 11ce:

We used this beamer internally, yet more importantly, we used it to pitches the contract modelling language we developed that year. 11ce was crazy successful and acquired by Digital Asset in 2016.

All original content copyright James Litsios, 2022.

Some favorite early programs I wrote

These are all pre-2000, and in chronological order:

1. A 'Hunt the Wumpus' like maze adventure program on my Commodore P100 programable calculator.
• Given the max of 72 program steps and 10 memory cells of the P100, this was probably the first time I used a rudimentary hash noise function (in this case to generate the maze 'on the fly', and not need to store it).
2. A primitive LISP like eval loop on my HP-41C, with symbolic derivatives and simplification.
• The integer part of the memory cells were the 'car' part, and the fractional part being the 'cdr' part (with an appropriate scaling).
3. A Fortran 2d finite element solver, 'mesher' and plotter on my schools 'new' VAX/VMS.
• The only early program that I still have, as it is printed as an appendix to my 'hand typed' BS EE thesis.
• A bridge calculation:
  
  
  
  
4. A two player 3d wireframe helicopter against tank simulator/shooter game (C and assembly, on my 4MHz Intel 8088 Cannon AS-100
• The screen and keyboard were split in two,. The game animated a bit more than 100 segments for both the helicopter and the tank. "Space dots" were projected in front of each 'camera' to improve the feeling of motion. Frame rate was about 4 frames per second.
• Fun fact: the 3d rotations matrix was 'compiled' in real-time to shift and add/sub instructions, as this was faster than using the CPU's multiply instructions.
5. A Warnock algorithm to render only spheres on my Canon AS-100. Example, 3d triangles were many small spheres, cubes were a recursion of well placed spheres of various sizes.
• The resulting rendering where very 'organic' because of all those 'spheres' meant that everything looked 'soft',
• The 'radius testing' code was somewhat similar to the (later) doom distance calculation code, as it was one table lookup of the scaled argument, plus one fix-point iteration to refine the result.
6. A scan line based design rule checker and layout 'compactor' (at the CSEM, for the analog CAD system we built)
• Initially Manhattan (orthogonal) geometries, then 45'' angles too, then any 2d geometry.
• It implemented 'boolean' and 'sizing' operators on 2d polygon geometries which my colleague Thomas Schwarz used to implement a window manager 'for fun' on the serial Tektronix color terminal.  (Slow but ultra-cool effect!)
• Mentor Graphics integrated this CSEM development into their ECAD suite: 
• 
7. A symbolic determinants simplifier using minimum weighted matching, used to symbolically extract dominant poles and zeros out of analog circuit sizings.
• The better algorithm came out the same year and worked on weighted matroids. 
• It is this type of experience in applied math that gives you a 'bottom-up' understanding of  Laplacian matrices. 
8. A multi-model hybrid-grid-graph simulator (as part of my PhD). 
• The generic sparse matrix assembly based on priority queues meant that all sparse matrix formats and parallelization schemes were supported efficiently for very large systems (think supercomputers). As a by-product, compact models could efficiently be extracted from large grid based simulations, which ensured the simulator's and ISE AG's commercial success.
• Grid properties could be computed using Jean-Bernard Lasserre's recursive simplex volume algorithm.  This generic n-dimensional algorithm was a bit slow but quite useful as many meshing systems provided incorrect volume/area/length values to the simulator for difficult mesh geometries.
• Some early ISE AG goodies (ISE was later acquired by Synopsys): 
  
  
  
9. A distributed object protocol with futures in C++ (early Actant).
• Prior to Eurex only VMS and AIX based APIs existed to quote options on Deutsche Börse. I had a good experience in serializing C++ objects to files (for my PhD). I adapted this code to make a remote object socket protocol to allow our Windows based option quoting system talk to VMS and we used this until Eurex became live (1998 I believe).
• Remote calls could be chained as a 'future protocol'.  While 'cool', I later disabled the 'futures' functionality having learned that too much generality is sometimes too expensive to maintain. 
• Some Actant AG goodies: 
  
  
  

All original content copyright James Litsios, 2022.

Thinking about parallel and distributed computation

A classical way to look at parallel / distributed systems is in terms of:

1. Work-Efficiency
2. Parallelism
3. Locality 

This is one of those inconsistent triads (or trilemma), where only two among three properties can be chosen. For example, a system can have both work-efficiency and parallelism, or parallelism and locality but cannot be fully work-efficient, parallel and local.

This is not the only inconsistent triads to think about when designing scalable systems.

Here are more:

1. Velocity
2. Resources
3. Work

1. Momentum
2. Quality
3. Traffic

And of course the CAP theorem's:

1. Consistency
2. Availability
3. Partition tolerance

And the important:

1. Fast
2. Good
3. Cheap

All of these are ultimately important, yet some of these are more practically useful. Especially when you realise that you can weaken the inconsistent of these triads by introducing additional complementary properties. 

To illustrate this with examples:

• Two people must work together vs an engineer and a sales person must work together
• Two time constraints must be considered vs a measurement must last one microsecond and be done in the next two weeks.

The idea here is that by clarifying dependencies and scales we are 'loosening' our constraints. The second example above is about distinguishing what is short versus long (in duration), and also that a measurement is typically done before things are said to done. Therefore if we partition what is of 'short duration' vs what is of 'long duration', or what is 'done before' and what is 'done after', we soften the inconsistent of the triads above. For example, we could have:

• Work-Efficiency of all durations
• Parallelism of long durations
• Locality of short durations

As a result, we 'never have all three' for short durations, or for long durations.

In their most primitive form, these are binary properties. For example, things can be:

1. Small
2. Large

1. Slow
2. Fast

1. Short
2. Long

1. Light
2. Heavy

1. Precise
2. Approximate

1. Predictable
2. Unpredictable

1. Efficient
2. Inefficient

1. Up-stream
2. Down-stream

1. Present
2. Future

And so on...

These properties make sense when they are properties that build the inconsistent triad. Using arbitrary properties would not work. For example, tall vs short does not help us. And to note the triads above are different. For example, efficiency is not part of the CAP theorem, but does effect the other triads (and this depends also on how efficiency is measured).

Finally, all of these inconsistent triads are interdependent. You want to understand how. 

For example, my previous post was in fact based on two tightly depend triads, which I illustrate as follows with their 'limits' within a distributed system view:

The cost of a cloud architecture is strongly tied how these complementary inconsistent triads are approached.

All original content copyright James Litsios, 2022.