Start a new project with an Event Sourcing Architecture

Getting started with an Event Sourcing project in PHP, using an example event sourcing project. This is a step by step guide to get you up and running with Event Sourcing.

By Keith Mifsud · 16 Jul 2019

I firmly believe that most, if not all, real-life process-driven applications can greatly benefit from Event Sourcing. If a system needs to know what happened in the past, then Event Sourcing is a good architecture fit.

I wish it was that simple🧞!

I find that a lot of Engineers, Product owners and developers, believe, or assume, that implementing Event Sourcing as an architectural pattern for their system is a costly decision. It can be, yet it may not.

Building a new system with an event-sourced domain is not too costly, in my opinion, it is not a lot of work. Less work than, for example, changing the persisted state directly from some GrapghQl API. Yes, you do have to write some more classes to represent Events and Commands and what not. All of which can be easily generated using some CLI code file generation tool. Even if you don't use generators, given the consequences of not having a model, representing the business domain, the problem you are solving or the business opportunity you are providing, is simply careless and unprofessional.

I am not saying that you should implement an Event Sourcing architecture in all your projects, or, that you are not professional if you don't. I am, however, saying that you should always have a separate Domain Model. DBAL, Eloquent or whatever you use to abstract your persistence mechanisms should never be your source of truth. There are a lot of reasons for why this statement stands true, too many to list them here. However, I am saying that Event Sourcing should be a candidate for discussion when planning the software architecture of your new project. Since you are a professional and you understand that most process-driven systems require a Business Domain, then, your choice is whether your Domain should follow a basic DDD (Domain Driven Design) architecture or take it further by separating the Command and Query responsibilities using a CQRS architectural pattern. You can also choose to combine all of these reliable architectures and retain as much information as possible for the present, and more importantly, for future consumption through Event Sourcing.

Data gathering and retention benefits should factor in your new project's cost/benefit analysis.

If you, the Engineer, building a new product, have decided to implement the Event Sourcing pattern as the primary architecture of your system's Domain, then, this article will help you get started. Even more so, this article is the first in a series I am writing, to show you, how to successfully and cost-effectively start and finish an event-sourced application.

When I am consulting to software development agencies and software engineering laboratories that are building an Event Sourcing system for the first or the second time, I get asked a particular set of questions, such as, which PHP Event Sourcing framework they should use or if they can still use the Symfony or the Laravel application framework. These questions are, in a way, redundant as the Domain should not be dependent on the accessing application. However, this is not so when gazing through other perspectives. As I will also explain in this article, planning your external dependencies is very important, not planning for implementations and integrations can be very costly.

Event Sourcing example project

The best way I can help you, the Engineer, the Product Owner or as I will be throughout this series of articles, both, is with an example event sourcing project. There are several components required in building an event sourcing project and the best way for you to be able to plan and estimate your project is with understanding the amount of work required to build and adapt these components.

To illustrate what it takes to plan and build an event-sourced application, I chose to use a simple To-Do application... I'm kidding 😂, To-Do applications are all over the web and they are too basic to justify this pattern. Instead, I, together with you, will be building a Classified Listing web-based application. A scaled-down version of GumTree.

With this example project, I will focus on the main business problem, the Listing Domain. Throughout a series of articles, I will show you the process I will follow to plan, design and develop an event-sourced Listing application. I intentionally chose this domain (or problem) because I know it as much as you do, unless, of course, you created GumTree yourself 😂.

In this series of articles, I will explain everything needed to have a Listing Web API, providing Members with the ability, of drafting and publishing a Listing, to advertise an item they wish to sell or purchase. Therefore, I will write about planning, how to solve the problems (or provide the opportunities) for this business, setting up a development environment, developing a Common/Shared Model, developing the first Aggregate, applying and dispatching events using a CQRS approach implementing concrete implementations for the system's requirements, testing, cross-domain consequences, read models and the access through the application.

I should get on with it, right? But where to start? From a business and service offering perspective, I would start with Event Storming. However, implementing the infrastructure and Integrations with external services should also be planned to ensure the feasibility of the project.

We are in luck as our Domain and system do not make use of external services. However, I want to highlight that this is a rare scenario! Depending on the product you are building, you may experience integration issues. Just recently, I was working on a Smart Condominium system whose internal hardware provider did not have a public API. The point is that, yes, most common and general service providers have an API but in the real world, a lot of vendors protect their IP (Intellectual property) by not offering an accessible API.

Whether you're developing an Event Sourcing Domain or an application using any pattern, it pays to determine the integration possibilities upfront.

This example Event Sourcing project does not have any external services' requirements and the Infrastructure requirements will be easy to implement because they are quite common.

Event storming

Same as in DDD (Domain Driven Design), brainstorming an event sourced-domain should also start with the domain. The riskiest part of developing a new enterprise system is not understanding the business domain and thus throwing away days, weeks or even months of work. It makes sense to start designing the system from the Domain upwards. Starting your design from the Domain also forces the project's stakeholders to start talking a ubiquitous language, drawing lines for terms within a Bounded Context and ensuring the feasibility of the system.

Event Storming our Listing domain is going to be trivial because we understand the Domain and because I'm limiting a lot of features and conditions 😜

Events are the first-class citizens of an event-sourced domain and in an Event Storming exercise, I start with following the process an Actor will go through to achieve the desired function. I will not cover authentication and membership in this series of articles, however, I will still include a basic Member domain in the code because the OwnerIdentifier in the Listing domain will essentially be a MemberIdentifier. Also, because I will illustrate how a Domain can listen to Events from another, and, thus I need more than one Domain.

Process a Member follows to publish a listing

Presumably, a registered Member will click on a Create Listing button and navigate to a screen containing a form for describing the listing. A Listing will have some required details and some optional ones.
Then, the Member can also add some images so that the viewer of the listing can understand the listed item better.
The Member can then publish the Listing for other Members to see.
The Member can cancel the listing at any time. This is normally done when the listed item is sold or obtained or if the Member decided to not sell or require it.
Listings have a life-time of fourteen days. However, a Member can pause the Listing and then resume it as long as the total number of days the Listing was published, does not exceed fourteen.
A Listing becomes automatically expired after fourteen days. Expired Listings are not listed on the public website.
The Listing Owner can change the Listing's title.
The Listing Owner can place the Listing in a different Category.
The Listing Owner can detach an image from the Listing description.
The Listing Owner can reprice the Listed Item.

So, the process is simple enough. I left a lot of possible features. Obviously, in real life, this type of application will allow for comments, messaging and other ways for an interested Member to converse with the Listing Owner.

Extracting events

With the process in mind (or written on post-it notes stuck to a board), I then start extracting the valuable events from the process. I came up with the following events:


ListingDescribed(...)
ListingImageAttached(...)
ListingPublished(...)
ListingPaused(...)
ListingResumed(...)
ListingCancelled(...)
ListingExpired(...)
ListingRenamed(...)
ListingReCategorised(...)
ListingRePriced(...)
ListingImageDetached(...)

The process itself tells us a lot more than just which events the Member will perform on a Listing. Although I will not be doing this now, the process as I have detailed, also tells us what Read Models we need and some of the business rules that need to be met before Events can occur.

We will come back to this process description very often throughout the project's development.

Infrastructure plan

As I've already explained and although planning the application and infrastructure implementations is not related to the business domain and in fact, the domain is segregated from this layer, if you (including teams) are working with Event Sourcing for the first time, or if your system requires some rare implementations, it is very wise to plan for these scenarios as early as possible.

Our example project does not make use of any quirky third party API or any rare database engines. Thus, I don't need any detailed plans for the system's infrastructure. In the next article of this series, we will be setting up our development environment which will be similar to the production one if we were going to deploy the system to a cloud or remote deployment. So, subscribe and I will email you when I publish it.

When you are planning how your Event Sourcing project should be distributed, you will consider a lot of different distribution strategies. I suggest to you, as I do every time I am leading a team, to not just choose a strategy because of its popularity but focus on the value to the product owner and the user using the product. For example, even if I did release this example project to the public, I will not opt for a Microservices architecture. This type of system may become busy and thus it may need to scale vertically by adding more resources, but it serves a specific domain and will not require a major growth in features.

This type of vertical growth is easy to scale with modern technology. Our project will use containers for individual infrastructure requirements. Therefore, we can easily replicate and increase each instance if we use a container orchestrator such as Kubernetes.

On the other hand, say the product owner decides to start offering a new feature using an external service. Such as a review system for corporate members integrated with TrustPilot (FYI, I don't like this service, it is easily spoofed). The Engineer can build and run the integration in a separate microservice accessible by our system through a REST API or a Pub/Sub remote messaging API.

Which Event Sourcing framework should I use in PHP?

There are several renowned and some less popular frameworks providing helpers for various elements of an Event Sourced application, including testing helpers. However, I found that I prefer to not use any off-the-shelf framework because of several reasons. Some of these reasons are specific to a particular framework, however, overall, I found that using a framework, I end up extending vendor classes inside my domain source code. I don't want any dependencies, other than the runtime programming language in my domain source code. The internals of a framework also makes it very hard, if not impossible to use the Adaptor Pattern for adapting the framework within the domain.

Broadway

In 2014, Qanditate open-sourced Broadway, a package providing several Infrastructure and Testing helpers for CQRS and Event Sourced applications. When I was faced with designing & developing the first enterprise-scale application written in PHP, I found this package very useful. At the time, I had already worked with Event Sourcing in production, but not using PHP. As you probably already know, I like PHP and nowadays I write more code in PHP than I do in any other programming language. However, back in the days, one would get laughed at for simply mentioning these three letters (P.H.P.) in a Sprint planning meeting. Nowadays, I have developed half a dozen event-sourced systems using Broadway.

The main reason I like Broadway is that the way this framework wants you to structure your code is very similar to how Event Sourcing as a Pattern of Enterprise Application Architecture is documented by Martin Fowler. So, it is very easy to pick up and for me to feel confident about the system I'm building.

What I don't like about Broadway is very much the same to what I don't like about any other Event Sourcing framework or helper library. Specifically, I don't like the amount of wiring up I have to do when not using the Symfony framework. The Scenario testing helpers are great until they aren't. These helpers cannot be extended without having to either rewrite the concrete classes for say InMemoryEventStore because it is declared final or adding even more external dependencies to your domain source code.

Just to elaborate this fact with an example, if you wanted to write an integration test between subdomains. I.e. one domain listens to another's Event. You cannot. You can, however, write your own Scenario test case as long as you are ready to, the very least, write your own cache-based event store. This anti-pattern keeps adding up and finally you end up with a copy of Broadway under your domain's namespace.

Prooph

Prooph is a set of tightly lose CQRS and Event Sourcing components. They are very well written, however poorly documented and in my personal opinion, not intuitive. I have tried implementing Prooph four times on clients projects and a few more times on personal or sandbox projects. Unfortunately, this was never successful. There are some things that I like about Prooph, but overall, I find it to be hard and cannot justify the extra work.

I find that Prooph is very concrete and focused on the message buses. I don't want my domain to be Infrastructure dependant. I understand that they are gives and takes in every situation. But I believe there's more gives than takes when using Prooph in my domain.

Hopefully, it is needless to say, my opinions are simply that, opinions. I believe that everyone who contributes to Open Source is a legend 🏆 To both Broadway and Prooph, thank you, and please do keep sharing your work with all of us.

I believe that most Engineers and developers will agree with me when you hack Event Sourcing in a language, it is not that hard to develop the Infrastructure and testing abstractions. It is then up to us to design how we will implement these abstractions. Whether using libraries such as Prooph and Broadway or building our own. I find it easier to build my own, but, this is mainly because I don't want external dependencies in my domain source code, not because any of these libraries are specifically bad.

I am sure that I would not be such a successful Engineer in building complex business systems in PHP without the help of Broadway. Unfortunately, Prooph arrived a lot later and I was already accustomed to the way Broadway wants to me to structure the code, from back in the days of working with .Net languages.

Throughout this series of articles, I will be using custom Event Sourcing infrastructure abstractions and who knows, maybe I'll also implement a few third party libraries for implementing these abstractions.

Can I still use Symfony or Laravel?

Certainly, you can use either and more or less all other application frameworks, micro framework, custom HTTP library, you name it.

The question of whether a team can use this or that makes me cringe. This is because the domain source code should not have concrete dependencies. Believe you me these type of questions are very common.

I use Laravel, a lot, it helps me by providing a lot of helpers and a structure for my application. Yes, application and maybe some Infrastructure configuration but nothing else. My domains don't depend on Laravel, or Symfony or Zend Framework or even DBAL or some HTTP router. My application does, rightly so, it is the sole purpose of the application, isn't it?

Application frameworks help me with providing a gateway to my software. Maybe even with Dependency Injection, binding interfaces to concrete classes and general bootstrapping. Laravel also helps me with structuring my HTTP layer with Controllers and request handlers, application-level validation, authentication and other ACL requirements. This is all I use Laravel and Symfony for. I don't want to develop a Router and Sessions mechanism from scratch since Taylor Otwell and Fabien Potencier have already mastered these hardly ever-changing aspects of my systems.

There is no way I'm sacrificing months, even years of work in establishing a source code that speaks the business domain. The problem it solves and the blood and sweat of all the project's stakeholders just because I don't want my team to add anything to a framework. It only cost you a project to come up with a system that works for you and only one major upgrade of your framework to cost you all your projects and businesses. Use your framework for what it is intended for.

In this series of articles, you will see how I use Laravel, how much it is an afterthought and how frameworks are disposable.

Development workflow for Event Sourcing in PHP

The process we will take to build this example Event Sourcing project is probably not ideal for you and your team. I will be building the Models first and then proceed to the Infrastructure implementation with the Application glue.

However, depending on the size of your team and the separation of stack skills, you would most probably have one or more Engineers building the Models and other developers implementing the abstraction to the Infrastructure layer and binding things together in the Application layer. There's no one size fits all manner of how you structure your workflow. This topic is team and project dependant. If you, your team or your company need guidance for managing an Event Sourcing project across a team, please get in touch, I can help you.

Have a good one

In this article, the first in a series of Event Sourcing articles, I've explained the initial steps you should take when implementing an event-sourced domain in your system. To recap, the domain planning starts with Event Storming but you or other team members should always forecast integrations and implementations needs. This planing normally comes before the deciding on which architectural pattern to use in your enterprise application and should serve as a feasibility study. The study ensures that the product owner is getting the best possible value, at present and for the foreseeable future. Taking into account, both vertical ad horizontal growth, cost and value.

This article and this series are aimed for new Event Sourcing projects. Modernising Legacy Systems has other variables which are not covered here. I can help you with modernizing your Legacy System.

In the next article, we will plan the infrastructure and implementations requirements. Followed by another article in which I will develop the Common (or shared) Model. I will try my best to publish the next article very soon, you can subscribe if you want to get an email from me when I finish it.