Equational

I went to my first Zurich FSharp meeting the other day and made a presentation on how to program with state monads in F#. The presentation is designed to pull you off the ground, to a higher order programming style, using the state and maybe monad as stepping stones. It finishes with stateful lifting. Here is a link to a slightly improved version: youtube  . Link to sources are below but I would recommend that if you really do not know this type of programming and want to learn, then you should type it back in and allow yourself some creative liberty. This is usually a much better way to learn, than to just copy code! You can watch it here if you want: ModularStateMonadsInFSharp.pdf XState1.fs XState2.fs Script.fsx

A recent economist reviews a book on the "The Great Chinese Famine 1958-1962" and writes the following: "The Great Leap Forward was the high point of ignorant Maoist folly. Chairman Mao said in 1957 that China could well overtake the industrial output of Britain within 15 years. People left the fields to build backyard furnaces in which pots and pans were melted down to produce steel. The end product was unusable. As farmers abandoned the land, their commune leaders reported hugely exaggerated grain output to show their ideological fervour. The state took its share on the basis of these inflated figures and villagers were left with little or nothing to eat. When they complained, they were labelled counter-revolutionary and punished severely. As the cadres feasted, the people starved. Mr Yang calculates that about 36m died as a result." When I read this, I could not help thinking that too many software development teams are starved from doing the right thing bec...

I decided my life needed a little bit more math, and so to decided to add a few differential equations in some F# code. Math in code means applied math and lots of arrays, but arrays are a pain functional style. So heres my appoach. I have written many numerial applications, mostly in C++, yet also some in Fortran. I mention Fortran because it has the notion of equivalence, where you can map arrays on to each other.  Equivalence is like C unions for array and there to  reminds us is that arrays do not need to be managed as dynamic memory. A functional style of programming is a lot about preserving your invariants with immutability, yet immutable vectors are not a solution (yet). Still, numerical code has its own invariants, and therefore some immutability is possible, and should be exploited to write robust code.  In my case, I have vectors and matrices; Their sizes are invariant, and so is the structure of t...

 There is nothing special to greenfield projects yet teams that can tackle greenfield project have something special! It is not the projects that matter as much as the approach to resolve unknowns when they appear in the projects that matters.   My goal here is to give you a few insights on how do this. Greenfield projects, or greenfield areas of non-greenfield project, is really about hitting the limits of what you know, it is about meeting unknowns. When that happens, and it does happen decently often, you should not apply a different process, but continue with the one you have, one that already has those “off road tires” that allow you to move in greenfield pastures.   This concept of “one process” fits them all is really the hardest thing to get across to teams that have not really managed to deepen their agility. Yet processes like Scrum works equally well on greenfield projects than on non-greenfield projects.   Best explain thing in act...

A tricky problem is to deal with software legacy. I think I have never been anywhere where I did not need to deal with legacy some way or another. In fact my first task on my first job was to improve the performance of "old" design rule checker (DRC) by upgrading its geometric store with a KD-tree. That was during a summer job, and later as employee, I was asked to write a new DRC to replace the old one. (A DRC is used to check geometric rules in layouts of integrated circuits). The new DRC was better than the old one, and part of an automated layout tool for analog circuit, which... ultimatly failed. Commercially the project did break even, but other systems worked better. Looking back, with the hindsight of knowing which system succeeded, this CAD sytem failed to approach the problem the right way, which was incrementaly. For example, the requirements for the DRC were: do what the old software does but better and faster, yet they really should have been "make it incre...

People often associated functional programming with statelessness. The reality is that there is always a notion of state, functional programming is about stateful programming. The ambiguity is probably the result of state appearing in different ways than in a procedural style. In a procedural style, state is associated to memory and you change it by using the writable property of memory. In a functional style, state is still in memory   but the “classical” approach is to disallow yourself from changing it by writing again over memory. This is the immutable programming style. It goes with functional programming because it is very difficult to build on immutability without some functional programming concepts (I have tried!).   So if you have state and you cannot write over it, the only thing you can do is to read it, and if you want to change the state, you need to write a new memory location with your new state value (a non-functional   near immutable approach is to ...

I had a magical insight recently about something that was bugging me almost two years. The question was: why the hell was I so unproductive creating type classes! So here is the deal: when you develop, you combine invariants and transformation properties of your  different pieces to make something new. If you are someone like me, you live your code, and that means that the relations between the different concepts throughout the code drive me "emotionaly". Yesterday, it occured to me that what is special with type classes is that they are "open".  They are open because their properties are not always obvious from the definitions. Worse, new type classes often have unambiguous properties. I say open because by contrast if I avoid type classes by explictly defining type specific functions or if I create encapsulating types to replace the functional polymorphism by data variants, then I am in effect "closing" the desi...

A market making and brokerage firm, Knight Capital Group,  lost all their money (~400M$)  when their automated trading system went crazy and it seems repeatedly bought high and sold low, as fast as it could, across many instruments. My guess is that they turned on a test app on the live market. This would explain why they were not feeding the trades into their portfolios. An algo to buy the spread makes sense if it is used to act as a counterparty to test other algos such as a market making application. Then again, a test app normally has hard coded detections of the live market that turn it off. So who knows?! This event is interesting in a few different ways: What fail-safe code do you want to include in your algo system? What fail-safe development management process do you implement to try to guarantee that those failsafe coding standards are met? Trading is really about keeping your profits, not about making them! Organic systems are assymetric...

This blog entry is more than of a little essay than a blog posting, the question I am trying to answer is what are my key learnings in managing software development. Software development management is about achieving business success, as well as team and individual success, by leveraging skills and process while focusing on productivity and learning. I know this because I have experienced it and applied it as developer, architect, product owner, project management, scrum master, development manager and CTO. Software development is a self-sustained process: much of what needs to be done depends on what has been done, much of future productivity depends on existing team culture and past investments. It is when starting up greenfield software projects that this “chicken and egg” nature of development is especially noticeable, developers starting in completely new domains are often acutely aware of what they do not know. They may not know, for example, who are their users, what are their...

Helping agile team with "out of team" work is always a challenge, they naturally tend to rely only on their own resources.  It is really a tricky management problem, and I have had my fair share of failures around teams that do not know that they would be better off with help. Yet my current thinking is that it is possible to set up parallel intra-team processes, but for these to work they need to include a large amount of inquiry activity from the teams; By inquiry I mean that people have questions for which they want answers.  So when team members have questions, and these do not find answers within the team, that is when supporting them with intra-team processes works. In "out of the book" scrum, the product owner and the scrum of scrums are meant to be the "only" intra-team process. But the bigger the domain, the more both the product owner and the scrum of scrum become bottlenecks. The insight is then that you can add more interfa...

Have you ever noticed that some development efforts seem to go nowhere? Maybe because they are just too complicated? Well what I have noticed is that code that I can program while watching TV tend to age much better than code that is complex enough that it needs my full attention!

I like this   paper in CIDR 2011 which shows you how to use the convex properties of functions to implement distributed algorithm with provable properties. The idea is that if functions are logicaly monotonic then they have nice properties. This paper fills in the details of what we already knew but with the exact details of what these good properties are. FYI: logically monotonic implies that inputs only add to outputs. Said differently incremental implementations of functions based on selections, joins and projections all have the logical monotonicity property. And this brings us to convexity because that is exactly why these properties exist: selections, joins and projection is all about preserving the convexity property of the function. And finally this is why I like monadic and stream oriented programming: its about implementing incremental algorithms and with enough care you can preserve nice provable properties! On a side note: type infe...

I recently read twitter's scala style recommendations and could not help being somewhat unhappy about their recommendation to use futures. They basically say: "Use Futures to manage concurrency. Fifteen plus years ago I wrote a futures library which I used in a derivative trading system for a long long time. All the basic functionalities were there (trigger on future, future timeouts,  merge futures) and some more advanced like boolean trigger conditions (e.g. trigger if futureA or futureB), as well as futures across processes and networks.  It was a nice library! Yet ten years later, we removed the use of futures! Here is the reasoning... When a future is used, its state can be seen as part of a higher order concurrent logic. The comfort of using futures, is that we do not need to model nor design this higher logic, it is implicitely defined and managed for us by the future library. There are situation where this lack of "bigger...

Double-checked locks are normally seen as a form of optimization, they reduce the overhead of their acquisition by first testing their locking criterion. I recently used double-checked locking to support non locking collaboration   between real-time processes. Here is how it works:   all processes agree upon a future collaboration point, each then sets up a lock to protect themselves from going past that point without the right information, they then proceed to exchange enough information in a finite number of transactions, this completes enough their knowledge that when they meet the collaboration point the lock is not activated. Well-tuned and repeated, the processes exchange information in a semi-synchronous manner without ever waiting on the locks!

A little bit more math to help us into the new year. This article is really very nice: What Shape is your Conjugate? A Survey of Computational Convex Analysis and its Applications  (by Yves Lucet) This article is beautiful because it brings together many, and I mean really many, algorithms and unifies them all through applied convex analysis. For example, there is something in common among: Computing flows through networks Properties of alloys Image processing for  IRM, ultrasound, ... Dynamic programming The magic in back of all of these concepts is convexity and how certain  mappings preserve certain properties. The article is probably a bit fast for many readers, but truly insightful because of the scope of its coverage. It really priovides a common way to approach convexity in its many different forms.