Architecture · The Empower Stack

Overview

A standard architecture using the Empower Stack includes the following services:

The Golang services and their databases, implementing the models, responding to CRUD operations, and in general implementing the logic of your application.
The GraphQL gateway, which aggregates the API of your services into a publicly accessible GraphQL API.
The Admin GUI, which consume the GraphQL API to perform CRUD operations easily.
The other services needed in the architecture, NATS, Jaeger, Hydra, the Login service.
The tools for implementing your DevOps process and managing the deployment of your services, like Gitlab and Kubernetes.

The Golang services

The Golang services are the heart of your architecture, defining your models, managing your databases, and exposing CRUD operations through a GRPC API. The Empower Stack manage your Golang services by providing a set of libraries, in https://gitlab.com/empowerlab/stack/tree/master/lib-go.

To use them, you need to describe your models as a Definitions struct in the ./orm/definitions.go file. A generator included in the libraries can then use these definitions to generate .go files with the Protobuf of your models and the ORM functions to perform CRUD operations. Theses Protobufs structs are then used to send GRPC messages and to serialize data.

The lib-go repository contains the following packages:

The libdata package includes the basic structure of your data, and the drivers to interact with data sources like PostgresQL/CoackroachDB databases, local memory / Redis caches, GraphQL/REST/GRPC API, NATS events etc...
The liborm package, built on top of libdata, contains the CRUD functions to interact with your data and also the ORM code generator.
The libgrpc package, built on top of liborm, expose your ORM CRUD functions as GRPC functions and also contains the GRPC code generator
The libgraphql package, built on top of libgrpc, contains the libraries which will generate the code used by the GraphQL gateway to redirect the GraphQL requests to your GRPC APIs

We use code generation to provide development productivity similar to a framework, while still using only libraries so you keep a full understanding of the code you are writing. You retain complete control on the main.go files of your services, and can decide to directly use the librairies without using the code generation.

The GraphQL gateway

In your gateway service, you specify the GRPC services which need to be aggregated, and the gateway will import the generated files containing the GraphQL resolvers and generate the GraphQL schema. To name the CRUD operations, it uses OpenCRUD, a standard proposed by Graphql.cool and Prisma.

The GraphQL endpoint will be exposed to the Internet, and will be used by client web/mobile/IoT applications (GRPC is designed for internal communications, not public use). By using precise verbs, REST APIs often force client applications to do multiple queries for a single action. We prefer GraphQL over REST APIs because using a GraphQL API is like using a query language, clients request what they need so the complexity of building data is managed by the backoffices and not the client applications which will use our API.

The GraphQL gateway also performs security operations, checking the provided token against the Hydra authentification service, and attaching access groups to the request so the services can decide if the requests are allowed.

The Admin GUI

Much inspired by the open-source ERP Odoo, the Golang libraries already provide everything you need to build the ERP of your organization, with the microservices patterns.

But to fill the requirement of a true ERP, we also need to have the administrative GUI an ERP usually provide, allowing your organization's users to easily access the data and perform CRUD operations through a clear and normalized interface.

We use another open-source project for this, React Admin, a great project which provides an admin GUI to any API. This is what's very interesting with this project, it's not tied to any specific backend and a driver exists to use it with a GraphQL endpoint using the OpenCRUD schema. Hopefully with time, React-Admin will evolve to become as awesome as the Odoo GUI, allowing the stack to become a full-fledge ERP.

The other services

NATS is an asynchronous messaging system, we use it to allow your services to send events that are then redistributed to the other services. By default, any CRUD operation will submit an event. All services also keep an event table in their database, and can keep complete track of the life of each record. This allows us to implement the event sourcing pattern in the stack. Thus a consumer service does not have to work together with the source service to get the data, it just has to subscribe to the corresponding event in NATS, enabling loose coupling between your services.

Jaeger is used for distributed tracing. When a new request hits your gateway, a new trace starts and can be followed during all its life through each of your services. This is extremely important to monitor your infrastructure, know when there is a sudden rise of errors (for example after a deployment or when servers go down), and for debugging. Debugging is especially tricky in a microservices architecture because you'd need to inspect the log of each service, which could be close to impossible. Distributed tracing is the answer to this problem, and is a critical component of your architecture.

Hydra is the service managing your OAuth2 authentification. When attempting to login, your GUI will redirect the user to Hydra, which will itself redirect him to a consent web application. We also call this consent application the Login service, and is basically another Golang service, built with our libraries, where the user will authenticate itself. After the consent service gives his assent, Hydra will provide a new token and redirect the user to the Admin GUI. As you can see, Hydra is the central component managing authentication and access rights in the stack, implementing the OAuth2 standard we definitely don't want to reimplement ourselves. And it does so beautifully, by redirecting the authentification to a consent app we need to develop, keeping full control on the authentication user experience.

DevOps

Finally, we also provide a DevOps example process to manage the development cycle of your services.

Most of this process is managed by Gitlab. We made an active choice toward Gitlab with the Empower Stack because, despite having an open core business model they have a real dedication to open-source, usually open-sourcing features if their community is asking. Also, as a company, they have an amazing story being almost entirely remote, and even their HR resource is entirely published online. On the other end GitHub, used by nearly all the open-source community, is and always were a closed service. Gitlab provide free services even for private repositories, and has features far more advanced than GitHub, there is no reasons to stay on GitHub except for the network effect. That's why the development of the Empower Stack happen on Gitlab.

Of course that's also because Gitlab is itself a part of the stack. Gitlab will provide a friendly interface to your developers, where they can inspect code, issues, and merge requests. It also implements a full-fledge CI/CD features, described in the .gitlab-ci.yml file, which will build your service images, store them in the Gitlab registry, and finally test and deploy them in your staging and production environment.

At deployment, your images will be sent on the Kubernetes cluster you have configured on your Gitlab repository. Kubernetes is now the de facto standard for containers orchestration, getting traction everywhere you can now easily find a Kubernetes service in almost any cloud provider. Kubernetes will manage most of the essential things of your architecture, like scaling your applications into multiple nodes, the storage of your data into volumes, the network communication between your services and their securitization. Kubernetes is what making microservices a reality, and must be included in any serious stack implementing this pattern.

Finally, we try to implement the monorepo pattern. Not to be confused with a monolith, this means that the code of your services is centralized in a single Gitlab repository, to avoid having too many of them and stay able to see all modifications at one glance. The monorepo pattern brings some challenges, for example it's great to test and deploy only services that got modified in the latest commit. We are also not enforcing the monorepo pattern, you can move your service into another repository if you want, but our example will be on a monorepo to ensure we are able to manage this pattern and also ease our own development process. We don't believe in a monorepo containing the source code of the whole organization, but having one monorepo per autonomous team feels great to us.