Writing an Interval-based Worker Pool using Generics in Go

Photo by Yk K on Unsplash

I recently needed the ability to pull data from a database at a specific interval and process that data. Not only that, but I also wanted to spread the processing to different goroutines (workers) in case the other goroutines were busy, much like a worker pool. This article will describe the code I wrote to achieve this task.

The Basics

First, let’s cover the basics by defining what we are building, then talking about the most basic data structures we need.

To start, I knew that I wanted to process work from a database table at an interval. To do this, I knew I’d probably use Go’s time.Ticker struct. For every N interval, the ticker will write to a channel that we can consume and do what we want.

Now I knew that I needed to define a basic unit of work. At a minimum, we’ll need to know what data we are working with and a function to handle that data for processing. I decided to use generics to make this component as general-purpose as possible. Here’s what that looks like.

This snippet defines a new struct type named WorkItem. Notice that it uses generics and accepts a type of any. The idea here is that we can work with any type of data we want.

The next thing I’d need is a way to describe how to retrieve and process work. I decided to create a struct that would be used to configure this “work ticker”.

As you can see, this struct has two fields for assigning functions to retrieve and handle work. But what do those look like?