Writing an Interval-based Worker Pool using Generics in Go
I recently needed the ability to pull data from a database at a specific interval and process that data. Not only that, but I also wanted to spread the processing to different goroutines (workers) in case the other goroutines were busy, much like a worker pool. This article will describe the code I wrote to achieve this task.
The Basics
First, let’s cover the basics by defining what we are building, then talking about the most basic data structures we need.
To start, I knew that I wanted to process work from a database table at an interval. To do this, I knew I’d probably use Go’s time.Ticker struct. For every N interval, the ticker will write to a channel that we can consume and do what we want.
Now I knew that I needed to define a basic unit of work. At a minimum, we’ll need to know what data we are working with and a function to handle that data for processing. I decided to use generics to make this component as general-purpose as possible. Here’s what that looks like.
