Posts

Run Your Engineering Team Like a Hedge Fund

Embracing Asymmetric Risk What is the common economic theme between: Lockheed Skunk Works (1943) ( see my analysis here ) Netflix's Migration to AWS and AWS S3 Storage (later 2000s onward) High-Performance Software Engineering  (my previous post) Embracing engineering as  high-risk, high-return investment where risk is asymmetric, bad things can happen, and no less than exponential growth is expected! The key primary word is RISK ! The key secondary word is EMBRACE ! And therefore why I wrote that  High-Performance Software Engineering is like  "running your development like a hedge fund " Some history, three dimensions Let's start with the following claims:  Google invented the foundational math and architectural blueprints (GFS, MapReduce, Borg) that made massive distributed computing possible. AWS commoditized that architecture, transforming physical hardware into infinitely scalable, programmable infrastructure . Netflix pioneered the operational c...

High-Performance Software Engineering Explained

Is failure ok? I was recently asked: " What companies have a culture where failing is ok "? I cheekily answered " None "!  The art of high-performance engineering This is the recipe I have been applying for the last fifteen years: Never f ail to achieve previously promised goals Never f ail deadline because of waiting on others Never f ail to use all your team members Never f ail to stay productive because of exhaustion Never f ail because of misalignments Never f ail because of past work no longer relevant, or past work never finished. To which mostly people say: are you kidding? There is a twist! These are extreme technology projects. Agile deconstructed An agile process typically maintains delivery drive and failure avoidance with process cycling over: Features (what we discuss with our stakeholders) Stories (what we work with) Code (what we deliver) These are primary driving efforts.  I skip the 10+ supporting efforts, such as validating, analyzing, etc. I skip ...

Meeting the many different communication structures in and across organizations

Sixt y years ago, Melvin E. Conway wrote: Organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. — Melvin E. Conway, How Do Committees Invent? (quoting from Wikipedia) How does Conway's finding affect software language design? How does one specify a language that meets the many different communication structures of one or more organizations and yet still preserve formal properties such as security and provability? How do I make a language that does not end up communicated as list of hundreds of recommended design patterns? How does one zigzag between OO, FP, and other programming approaches? How is this related to machine learning and LLMs? Recently I was asked to help on a project. My immediate response was: Tricky, knowing what I know now, I would write a DSL to produce the necessary content for the project, but we cannot afford that for this project, so no DSL. ...

Semantics of semantics in Python with tracing code

Writing software that actually behaves like pure math is harder than it looks. Usually, state changes, loops, and memory quirks get in the way, turning elegant logic into a tangled mess of side effects. A modern tracing library fixes this by acting as a silent observer. It watches your program execute, sweeps away all the temporary memory junk the computer needed to do the work, and leaves behind a perfect, permanent map of how your data actually flows. By completely separating how the code runs from what the math is, you get a flawless, auditable blueprint of your system. It gives you the exact precision needed to stop fighting your language's quirks and start engineering truly smooth, reliable software. And so last year I wrote a first draft of a Python tracing library (see original https://github.com/equational/hltpy ). This code works, but fails to be elegantly re-entrent: it is tricky to trace interpretations of traces. So today, pressured by a needy use case, I am rewriting t...