Preamble

Some of you may know that I wrote a book – Cloud Native Patterns: Designing Change-tolerant software. The book was published in 2019 – yup 5 years ago – yet I suggest the content is as relevant today as when the first books rolled off the presses. That’s because it is a book about software architecture, not a specific technology. Sure, I’ve made it pragmatic by implementing the patterns using specific technologies like Java, Spring Boot and Kubernetes, but the focus is on the patterns, not the tech.

But tech evolves (thank goodness for the sanity of this change junky). I’ve recently started revisiting the book with an eye toward updating the implementations to newer tech. This includes updating to the latest version of Java (yes, Java is still SUPER relevant so I’m sticking with that) and Spring Boot (same), but I want to go beyond that.

For example, I’ve become super interested in Temporal as of late. This is crazy powerful technology that my gut tells me could change the way that we write software in a fairly substantial way (others, like Paul Nordstrom thinks so too). True, for some people, this is not new tech – it’s been around in some form for nearly a decade – but the penetration into the developer population is still minimal. For all intents and purposes, it’s new tech.

What you see in this post is my initial application of Temporal to the example that runs through the book. I learned quite a bit in the process. I hope that it can be helpful to some of you as well.

Introduction

The goal of the first code-centric chapter of the book is to make clear that there are two primary approaches for designing the distributed systems that are cloud native applications. The style that most folks likely default to is request/response, where a service implementation makes calls to downstream services, waits for and then processes those responses, generating its own response. Alternatively, an application can deliver exactly the same capabilities when implemented in an event driven style where a service implementation generates its response by processing data that has been pushed to it preemptively, as a result of some events having occurred.

I won’t contrast these approaches in detail here – I invite you to read chapter 4 for that – rather, what I will do in this first post is leverage Temporal to implement the former – the request/response style. (While predicting a future blog post is a bit like saying “the kicker has never missed one from this distance” – almost assures a miss 😜 – I’m going to promise that “Applying Temporal to Event-driven Microservices” post in the near future. I should also say that I am a big fan of the latter – I’ve actually asserted in the past that event-driven is the “right” approach for most microservices – but we still need to understand how to do request/response better.)

The Post Aggregator

The application that we will be looking at is quite simple – it allows a user to follow any number of bloggers they are interested in, and a post aggregation service aggregates the posts from all of those authors. The following figure depicts that simple application topology.

Taking a Temporal 101 view of this example, the starting point is pretty obvious. The Connections’ Posts service implements business logic that obtains a list of user IDs that a given individual follows, and then fetches blog posts from each of those users, producing an aggregated response. That sure sounds like a workflow, and, given the calls to the Connections and Posts services happen over the network, implementing activities to assist with each of these downstream service invocations also seems quite obvious.

And the tl;dr; is, it was pretty straight forward. Let’s have a look at the code:

You can find the final implementation of this example here, and you can have a look at the commit history to see the set of incremental steps I took. In brief:

• I made a copy of the chapter 5 code which implements separate microservices for each of the above depicted back end services.
• I took code that had been in the Connections’ Posts controller and moved some of it out into a workflow. Initially, I didn’t implement any activities, instead making HTTP requests directly from within the workflow implementation.
• Then I created activities to add some resilience to those network calls.

Let me cover the second and third of these in a bit more detail, and share a bit of what I learned in the process.

Creating the Workflow

The original implementation of the Connections’ Posts service had the logic that retrieved connections and then posts, all within the controller. And it didn’t have a lot of (any) resilience baked in. My aim was to explore what type of resilience Temporal would bring.

Since seeking resilience was precisely the goal, I started out looking at the places in the code where things could go wrong. Of course, calls to any downstream services could fail, so I started there. I left the HTTP request parsing, and HTTP response construction in the controller and moved the bulk of the business logic into the workflow.

And I’ll confess, that in doing so it felt like the implementation, or perhaps better said, the structure of my application became quite a bit more complex. Before adding the workflow I had a very typical Spring Boot microservice implementation with a couple of controllers that served HTTP endpoints and a java class with a main method that launched the Spring Boot app.

Sure, bringing workflow into the implementation only adds java interface and implementation classes for the workflow and the activities, something that ultimately isn’t terribly complicated, but I did start to get a bit wrapped around the axel because the workflow is its own application. That’s right, the workflow, or perhaps more accurately stated, the worker that will execute the workflow isn’t a part of my Spring Boot microservice. Or is it?

The answer, I believe, is that the workflow absolutely IS a part of my Connections’ Posts microservice, even while it is NOT a part of the Spring Boot controller. True, there are two different Java apps that form my microservice implementation, and true, these two apps communicate via the Temporal server (see the following picture), but together they absolutely do provide the implementation of my microservice.

Okay then, one microservice, two java apps. How do I structure my project then?

I am not at all sure that I did this the “right” way; I do not have enough experience with Temporal yet to know what the best practices are, but let me tell you what I did.

I created three separate modules within my microservice repository folder:

• One for the controller – this is a spring boot application (a servlet) that serves HTTP endpoints.
• One for the workflow that includes the workflow and activities classes as well as the java app for the worker. I’m leveraging spring boot here too because it is just so good at packaging, dependency management, and property injection.
• And a module that houses the resources that are shared across these two implementations.

I create two fat jars (with mvn clean install) that I then run using java -jar connectionposts-controller-2.0-SNAPSHOT.jar and java -jar connectionposts-workflow-2.0-SNAPSHOT.jar, respectively.

Sidebar: If you are leveraging the resources from Temporal to learn, you will notice that generally they start apps – workflow launchers and workers – on the command line with something like a mvn exec:java ... (I am doing all my work in java). This is great for getting started, but real-world deployments will almost always need to think about packaging. That is effectively what I’ve been doing here – packaging jars that can then be deployed in a number of ways. There is a lot to be said about this – ==stay tuned==.

Adding the Retry Pattern Through Activities

In my first cut at migrating the biz logic from controller into the workflow, I just moved all of the downstream HTTP requests directly into the workflow. Of course, making a call over the network introduces a point for potential failure – one that activities are exactly suited to address.

The good news is that introducing activities into my implementation was a non-event. I had to create the Activity interface and implementation, and register the activity with the worker (um, yeah, I won’t admit to how much time I burned because I forgot this 😵‍💫 – total noob move that I won’t be repeating). Of course, I had to configure the retry and timeout options – all very straight-forward (and the temporal docs are rife with examples). And then it just worked.

My application is now deployed through the launch of four java apps:

• The Connections microservice
• The Posts microservice
• The Connections’ Posts controller
• The Connections’ Posts workflow

Once all of those service are running, the Connections’ Posts service can be invoked with the following commands:

curl -X POST -i -c cookie localhost:8080/login?username=cdavisafc
curl -b cookie localhost:8080/connectionsposts | jq

On the command line you’ll see results coming from some sample data, and of course, you can see the workflow execution in the Temporal console.

I mentioned earlier that the book implementation of chapter 5 wasn’t particularly resilient – this was intentional, as later chapters introduce patterns incrementally. Retry is one of those patterns and is added in chapter 9 through the use of Spring Retry. What we’ve done here with Temporal serves the same purpose. You’ll notice that the config looks pretty similar between the two approaches (Spring Retry, Temporal) (i.e. setting max attempts and backoff values), and at a high level, the outcomes are the same.

The implementations are different, however; Spring is a framework, Temporal is a platform. While the developer isn’t responsible for the implementation of retries using either approach, they need only configure, there are some subtle differences in the characteristics of the solution. Yet another topic for a future post.

You can find the final solution here.

Summary

My goal with this exercise was twofold. First, I wanted to keep learning Temporal, and applying it to my own problem rather than just following a tutorial absolutely deepens my understanding. Second, I am very keen to update the tech for the code samples in my book, and I want to find key technologies that today’s developers need to be aware of.

On the former, here are some of the key things I learned (and I hope this post is helping you also understand):

• Workflows and Activities BOTH need to be registered with your worker. (I know this is obvious, but stating the obvious can be helpful 🙂)
• Microservice implementations can themselves be distributed systems. In the example here the Connections’ Posts service is implemented via two java apps – the microservice API/controller and the workflow that implements the business logic.
• We can package a workflow into a form factor that allows for deployment into environments like Kubernetes. I found Spring Boot to be well suited for this and I packaged it as a jar. (I haven’t done the Kubernetes deployment yet but will very soon.)

On the latter, I have to admit that I am not yet convinced that using Temporal for web services such as these is the right approach. Sure, adding retries through configuration adds a level of resilience, but there are other, lighter-weight means of doing this, like Spring Retry. I’m not convinced that the logic of the posts aggregator needs the type of durability that Temporal is specifically targeted at. Does the added complexity of a Temporal server (even if you let Temporal.io manage it for you by using Temporal Cloud) bring enough benefit? If you have some thoughts, please do either comment here or send me a DM. In any case, I’m going to keep exploring – keep an eye on my new git repo to track my progress.

I hope that my musings here are of value to others. I love feedback.

(Bluesky, LinkedIn)

This has been cross-posted to my medium hosted blog as well.