KOERS

KOERS stands for Kadaster Objecten En Rechtenregistratie Systeem (Cadastral Objects and Rights Registration System) and is the system for the maintenance of the Basic Registration Cadastre, the BRK. This is the story of the development and highlights of the key success factors of the eponymous project in which this system was built. The goal was to replace and retire the 40-year-old mainframe system …

How it began — 1832

The Kadaster (Dutch Land Registry) traces its origins to the French era: in 1811 Napoleon decreed that the Netherlands should have a land registry and public registers, so that it would be clear which plot of land belonged to which owner. The Kadaster has officially existed since 1832 and grew into the organisation responsible for registering and publishing information on real estate and the rights attached to it. The Kadaster is therefore a public body with a core responsibility for legal certainty. That responsibility was later enshrined in law through the Cadastral Act, which identifies legal certainty as its primary duty. This makes modernising the registration immediately a societal and legal operation, not merely a technical one.

The Basic Registration Cadastre, or BRK, contains information on parcels, ownership, mortgages, limited rights such as leasehold, building rights and usufruct, and utility networks. The cadastral map is also part of it, including plot boundaries, plot numbers, surface areas and the boundaries of the state, provinces and municipalities. The BRK is therefore the authoritative source for this domain and connects with other basic registrations and data exchange with notaries and government bodies. The BRK is not just a database, but the operational heart of the legal and geographical reality of land and rights in the Netherlands.

The situation — 1962-2014

The Kadaster was often at the forefront of technological developments in the Netherlands. Early on, mechanical aids such as the mechanical calculator and later automation were deployed to make cadastral registration more accurate and efficient. As early as the 1960s, the Kadaster took its first steps towards automation. Around 1970, the first location introduced the automated register, and approximately ten years later the last branch had also been converted. With the completion of AKR, Automatisering van de Kadastrale Registratie (Automation of the Cadastral Registration), in 1990, the paper-based land accounting system came to a definitive end.

After the introduction of AKR, renewal at the Kadaster continued. The organisation received its statutory footing in the Cadastral Act in 1992, became an independent administrative body in 1994, and during the nineties took further steps towards digitising maps, registrations and data exchange. In the period up to 2014, various studies and ‘disentanglements’ were carried out to reduce and ultimately replace AKR with newer technologies that could keep pace with the demands of a changing world. Yet all initiatives stalled at the heart of the BRK: the mainframe system AKR. It was too deeply embedded in too many connections, with too much domain knowledge baked in.

That was the situation until something changed in 2014 …

The new course — 2014

After the necessary studies and trials, a vision and strategy were chosen to tackle the stubborn knot of AKR head-on. A deliberate choice was made to keep ownership and expertise in-house as the foundation for the many people — including external parties — who would become involved in this renewal. The strategy was:

Building fixed teams working with a continuous flow to construct a new, flexible and future-proof system, in which business and IT together strive for a gradual transition without synchronisation

Fixed teams refers to the agile approach to the entire project. The aim was to steer work towards teams, in place of organising people around the work. These were multidisciplinary teams in which business expertise and software development expertise had to be present in combination. The emphasis on business and IT together reflects this.

A continuous flow and a flexible and future-proof system point to DevOps and Continuous Delivery as methodologies for delivering the new system. In 2014, this was by no means the norm, and to a large extent the Kadaster had to figure it out for themselves. Even so, the decision was made at the time to start with a Continuous Delivery pipeline to a Platform-as-a-Service (PaaS). Only after that was set up technically did development of the first functionality begin.

Gradual transition and ‘without synchronisation’ refer to the introduction of the new system alongside the old one — parallel running, including data exchange between the two systems. Trials had been conducted earlier, but the AKR system was so much a single unit and so central to the system landscape that this was considered infeasible. The strategy was therefore explicitly: no synchronisation.

Project phase — 2014-2018

Agile is often experienced as chaotic… and the project was indeed ’not boring’! As is natural in any project, there were different phases and many changes. By setting up the project organisation to be flexible and agile, it was possible to adjust course at every turn. We started with one development team in which the key business analysts were directly involved. In the first phase, much was tried, searched for, changed, adapted and established. At a certain point it made sense to scale up to multiple teams, organised around components with specific functionalities: a frontend team, a registration team, an information provision team and an integration team. The business was needed everywhere and became the work preparation team. Later, as the components became sufficiently robust and stable, we could focus more on functionalities across the whole. The teams became more fluid and followed business functionalities rather than system structures. Agile. Flexible and responsive. Brilliant!

Decoupling, separation of responsibilities and concerns are architecture questions that also affect team scalability, among other things. CQRS — the separation of the command side and the query side — was not applied from the very start, and that created many dependencies. After the first months of research and experimentation, we did start applying this separation clearly, supplemented with Event Sourcing. Here, changes to the system are recorded as events on the command side and then (only) processed on the query side. This provided a very robust separation in system functionality and also in team focus. The registration team handled the complex business functionalities in processing new instructions — the command side. The information provision team handled the translation and processing of those events into the (many!) different interfaces the system had to supply with data flows.

Moreover, events turned out to fit extremely well with the business way of processing! The Cadastral Registration is the processing of notarial deeds. These contain legal acts, also called legal facts. The registration is then based on an information model in which various entities are modified by these legal facts. And the business rules had already been designed as reusable units, with multiple legal facts triggering the same or similar processing on those entities. Concretely: a deed of transfer — a ’normal transfer of ownership’ — and a ‘gift without consideration’ essentially do the same thing under the same conditions (business constraints), with a few small differences. And this turned out to be the case for a whole range of legal facts which, legally speaking, have genuinely different backgrounds and effects, while at the same time having more or less the same effect on the Cadastral Registration. Events expressed in business terms describe the effects of legal facts on business entities very precisely and effectively. Magnificent!

Through the use of Continuous Delivery — a continuous flow of changes from the development teams to the production environment — it was possible to roll out new functionalities quickly and continuously (what’s in a name) to the test teams. Improvements to existing functionality could also be brought swiftly ‘into the hands of users’. Early in the project, employees from the deed processing teams were asked to process the same deed in KOERS alongside their normal processing in AKR. Outcomes and results were then compared. This was called shadow running.

A precondition for shadow running is that the starting positions in both systems — AKR and KOERS — must be equal. Synchronisation had been explicitly ruled out for this purpose. We chose to produce a fortnightly export from AKR — naturally at the weekend — of the data. We then developed migration software to distil events from this export (dump), which formed the starting point of the new system. This created a fortnightly rhythm in which data was fully aligned on the first Monday and gradually became more out of date thereafter. Yet this was sufficiently usable to allow more and more AKR users to trial run — shadow run — with KOERS. This was not only a test of the new system, but also a training mechanism for our production staff!

What was particularly special is that we had built up so much experience and mutual trust that we dared to take more risks. For four years we had gained experience with Continuous Delivery. Deploying to the production environment was genuinely no longer exciting — it was more of a given. We had also built experience and confidence in our analysis, design, development and testing expertise. We knew we could take a new legal fact and handle it thoroughly and concretely at a certain pace, including deployment and actual processing in the production environment. Furthermore, we had developed and made available recovery capabilities — and trained people in them — so that even without new development work, we could handle a great deal of variety. This gave us the background and context to go live without having developed 100% of the legal fact types in the system! 😳 An earlier analysis had been done of the variety of legal facts, and some legal facts from the past would never occur again in the future, some might possibly still occur but were unlikely, and a fair number occur only once every 10 years. Well… when you build a new system, you need to support everything, right? Wrong!! 😎

Live — Oct 2018

Then the moment drew near that we were complete enough in functionality, the migration had been sufficiently tested and failures had been resolved. In October 2018 the big weekend arrived. On Friday, processing of notarial deeds was halted early, naturally within the legally available provisions. Over the weekend, the normal procedure was followed — the same one we had been working with every two weeks for over two years — to carry out the migration from AKR to KOERS. That went as expected and planned… with the usual small details and checks. After the export from AKR we were able — as usual — to load the new system with the new events, which this time would truly form the foundation of the new era. Exciting! And yet… the most exciting thing was that we had to get used to stopping the fortnightly pattern we had followed for over two years. And then to stop. That was the strangest feeling of all. 😄

All the connections to the surrounding systems had been tested and trialled multiple times in various stages. But now, of course, it was truly the moment when everything had to go right and keep going right. That was very nerve-wracking, and it didn’t go smoothly on its own. By Sunday afternoon we were past the point of no return; we could only go forward. Unfortunately, Monday was a day full of stress in which the outside world — in particular the notarial profession — also experienced disruption from our go-live. But still, by the end of Monday we had caught up on the delays accumulated during the day. In the rest of the week we still had to put out many small fires, but ultimately — for the outside world — it went by fairly quietly. One article in Computable, and otherwise: silence. That’s how it should be. And yet we had performed an ‘open-heart transplant’ in the application landscape of the Kadaster, in which an important and old, trusted core — AKR — was replaced with a ‘big bang’. There we go! What an achievement!!

Project completion

Wow, what a journey. Four years of building. Four years of discovering how we could collaborate, how we could adjust course, how we could get a grip on the totality of the work… But we pulled it off. And how! What a wonderful and intense trajectory we covered. With many people involved, both internally at the Kadaster and externally through hiring, as well as audits, advice, checks and additional expertise brought in…

And what an achievement! An open-heart transplant with a big-bang introduction in one weekend — OK, plus one Monday, but even so! Replacing a mainframe system surrounded by around 100 interfaces… and one that had been the beating heart of legal certainty around property in the Netherlands for years. To be able to continue that stably and with quality, there had to come a point where AKR had a solid successor. And that’s what happened: KOERS!

Key success factors

This project would not have been possible without an agile approach. That may be a somewhat general term, and there are 100 ways of ‘being agile’. But the primary focus on people — agile manifesto rule #1 — and on collaboration, on understanding the different needs and how to support them as well as possible, truly delivered maximum results for us. 💪

We also made the other rules of the agile manifesto tangible, among other things through Continuous Delivery. We started on day 1 and never stopped. At a certain point it took 30 minutes for an accepted feature to run through the entire pipeline of tests and steps and for the functionality to be usable in production. And that eventually became a bottleneck! The various teams, and later feature teams, were delivering so many small changes that a queue built up for changes going to production. We came up with solutions there too, to get things to production faster, more smoothly and more easily. That worked brilliantly!! 😎

The internal architecture is primarily based on CQRS and Event Sourcing and provided a wealth of possibilities:

• Business and IT (developers) developed the same language. Business analysts even started helping to write functional tests, because there was mutual understanding of the functionality, the naming of things, the commands, events and business constraints. This is truly business and IT working together! When developers start poring over articles of legislation and asking questions that analysts cannot immediately answer — and, vice versa, business analysts start writing functional tests… 💯
• The decoupling of the command side and the query side, with events in the middle as the core API (interface), was powerful! This created separations in the technical components that were designed along business lines. That produced business-aligned separations between teams, collaborations and interfaces. This delivered enormous robustness in traceability, repeatability, testability, integration and evaluation. Love it! 🚀
• By recording events as the primary result of processing changes, it is possible to add multiple, different, and even retrospective projections while preserving history. The initial focus in the project was naturally on processing legal facts for updating the Cadastral Registration. Later in the project, business requirements also emerged — for example, invoicing. On the processing side — the command side — NOTHING needed to change! We could ‘simply’ add a new projection that, based on existing events, arrived at the set of data needed for invoicing!! BAM! 🤘

Lessons learned

Did everything go well? No, of course not. Without airing all the dirty laundry, there are a few lessons worth noting:

• Agile is difficult. Agile is… also still unfamiliar. Really?!? Yes, unfortunately. Despite the fact that the IT industry has been working with agile for many years, the audit world, for example, is still completely unaccustomed to it. This was a large project, so there was critical oversight and there were audits. But how do you answer ‘waterfall questions’ when you’re running an agile project? 💥
• CQRS and Event Sourcing are still largely unknown. Really?!? Yes, unfortunately. Despite the fact that they are actually quite old patterns — comparable patterns were already described in the 1970s and 80s — asynchronicity, the separation of ‘mutation side’ and ‘information side’, and especially synchronisation by means of events, are still rarely part of many people’s experience. Not architects. Not designers. Not developers. 🥲
• Automation, Continuous Delivery, Platform-as-a-Service are all good things of our time. In 2014 they were not yet at the level they are now. We made mistakes there too and had to figure out a lot of things ourselves. But… these days, that’s no longer necessary! The tools and infrastructure have improved enormously. Strangely enough, the methodologies are still insufficiently known outside the developers who set these things up and maintain them. Strange… 🤷‍♂️

What’s next?

Well, in October 2018 KOERS went live… and has not gone offline since. After all these years we still do Continuous Delivery and develop the system a little further every day. Not as intensively as during the project. In terms of teams and people we have also scaled down. And we have found and addressed the necessary issues and made technological improvements. At the same time, that’s business as usual — every challenge, question and requirement is designed, built and deployed to production. The underlying platform has changed. The delivery tooling has changed. The mechanism has not. Nor has event sourcing. Upgrades, yes, of course. But above all: steady business 😊

My role in the project was that of solution architect for the internal software architecture of KOERS. The introduction of CQRS and Event Sourcing originated with me. The foundation of Continuous Delivery and the first pipelines were also my work. I had significant influence on the design of the components, the interfaces between them, and the organisation of the teams around them.

Based on my experiences in the project and with the KOERS system, I have developed ideas about what the future might look like. I have worked these out — together with colleagues — in, for example, protocol thinking. I am also documenting my experience in the “Guidance for Reliable Registers” being delivered in the project From a reliable source.