cb codes

A Tutorial To Stuart Sierra's Component

A lot of people have already heard that they should use Stuart Sierra's component library to manage state in their Clojure applications. But most don't know why and how you would incorporate it into your software.

This how to guide will give you all the tools you need to start managing state in your Clojure software.

What is component?

Component is a Clojure library that can be thought of as a micro dependency injection framework. I use it in all my web projects to manage runtime state.

What are alternative(s) to component?

Component Strengths

Component Weaknesses

An Example Of Component

Let's say you're planning on building a new service for managing emails in your application. The work involved is taking messages off of a queue, doing some work, writing data to a database, and then putting things onto other queues. Everything will be asynchronous.

Imagine we had a queue that had requests for messages that needed to go out. An outgoing queue. You have other services that put things onto this queue when they need to send an email. You might have your website's user authentication functionality queue up emails to go out when a user signs up or forgets their password. You might also have a background job that queues up product recommendations to go out to users every week.

The job of this service is to take those messages, send off a request to an email service provider, log the messages and their statuses in a database, and queue up messages for other services to consume.

How would you break this down into components?

These are the components I see us needing:

The database and queue should be super obvious candidates for componetizing. The worker process is basically a handler. The core async channels and email service are probably not so obvious components.

Let's sketch out some code. I'll explain as we go.

Note: This will just be snippets of code that I haven't fully tested as I don't plan on writing an entire microservice. If you find major errors, go ahead and send me an email.

;; Note again, I didn't actually run this code so there ~may~ WILL be
;; errors and mistakes. The point is to demonstrate the power of
;; component in an example!
(require '[clojure.core.async :as async]
         '[com.stuartsierra.component :as component])

;; Here's the system we'll implement (defn ->system [config] (component/system-map :input-chan (async/chan 1024) :result-chan (async/chan 1024) ;; This is the queue of emails that need to go out, which is the ;; INPUT to our service. We're never going to queue anything here ;; so we give it a simple chan to satisfy dependencies :outgoing-emails (component/using (map->Queue (assoc (:outgoing-emails config) :outgoing-messages-chan (async/chan))) {:incoming-messages-chan :input-chan}) :db (map->Database (:database config)) :mailgun (map->EmailService (:mailgun config)) :worker (component/using (map->Worker {:work-fn log-and-send-emails}) {:input-chan :input-chan :db :db :email-service :mailgun :result-chan :result-chan}) :sent-emails (component/using (map->Queue (assoc (:sent-emails config) :incoming-messages-chan (async/chan)) {:outgoing-messages-chan :result-chan}))))

;; We're going to pretend we have some library for some queue that ;; allows us to connect, subscribe, and publish messages (defrecord Queue [config conn incoming-messages-chan outgoing-messages-chan] component/Lifecycle (start [this] ;;Put whatever messages that come in onto our incoming channel (subscribe-to-queue config (fn message-handler [msg] (async/put! incoming-messages-chan msg))) (let [conn (connect-to-queue config) stop-chan (async/chan 1)] ;; Start a loop that goes on until we put something onto the ;; stop channel (async/go-loop [] (async/alt! outgoing-messages-chan ([msg] (publish-to-queue config msg) (recur)) stop-chan ([_] :no-op))) ;; Assoc the stop-chan onto the record so we have access to it ;; in stop function (assoc this :stop-chan stop-chan :conn conn))) (stop [this] ;; Stop the go-loop (async/put! (:stop-chan this) :stop) ;; Close the connection (close-connection (:conn this)) (assoc this :config nil :incoming-messages-chan nil :outgoing-messages-chan nil :stop-chan nil)))

;; The database example is a common one so I'll be really brief. In ;; fact, Stuart Sierra has one in the readme of component (defrecord Database [config conn] component/Lifecycle (start [this] (let [conn (connect-to-db config)] (assoc this :conn conn))) (stop [this] (close-db-conn (:conn this)) (assoc this :conn nil)))

;; Here's an interesting example and use case. You can implement more ;; protocols on records than just the Lifecycle one, something I don't ;; see very many people doing.

;; For instance, we're going to have a generic MessageService protocol ;; that our EmailService can implement. (defprotocol MessageService (send [this msg opts]))

;; Our EmailService can implement MessageService (defrecord EmailService [config] ;; I explain at the bottom that all java objects implement ;; Lifecycle. Since this is going to be stateless here, this is ;; OPTIONAL component/Lifecycle (start [this] this) (stop [this] this)

MessageService (send [this msg opts] (smtp-send! config msg opts)))

;; The good thing about implementing a message protocol for our ;; EmailService is that in testing we can mock it out by making a ;; fixture that implements it. This fixture just adds the emails to an ;; atom we can inspect later to see that our message has been "sent" ;; in testing. Also, with the component model, you can just take your ;; application's system map and replace your EmailService with the ;; MockEmailService and it'll just work. (defrecord MockEmailService [sent-emails-atom] MessageService (send [this msg opts] (swap! sent-emails-atom conj msg)))

;; Next up is the Worker component

;; I want to note important something here. In the system below, we ;; give the Worker a db and email service on top of the input/result ;; chans and work-fn. Components are just records. You don't have to ;; explicity say in your defrecord that your record needs a db and an ;; email service. As long as you specify that in your system map, ;; component will assoc them onto the record for you. What does this ;; mean in practice? You can have a generic Worker component that ;; takes things off of a channel and puts the result onto another ;; channel. I provide the Worker component's work-fn with the worker ;; itself (work-fn this result). This way, the work-fn has access to ;; the worker's dependencies if it needed to, and our record ;; definition is general. (defrecord Worker [input-chan result-chan work-fn] component/Lifecycle (start [this] (let [stop-chan (async/chan 1)] (async/go-loop [] (async/alt! input-chan ([result] (async/put! result-chan (work-fn this result)) (recur)) stop-chan ([_] :no-op))) this)) (stop [this] (a/put! stop-chan :stop) (assoc this :input-chan nil :stop-chan nil :result-chan nil :work-fn nil)))

;; Some work function we'll provide to our worker (defn log-and-send-emails [worker email-msg] (smtp-send! (:email-service worker) email-msg {}) (write-status-to-database (:db worker)) ;; Return some message to be queued to sent-emails {:status :sent :email-msg email-msg})

;; I'm going to omit the sent-emails queue as it'll be the same as the ;; other queue (but with different config).

;; Here's our system again for reference (defn ->system [config] (component/system-map :input-chan (async/chan 1024) :result-chan (async/chan 1024) ;; This is the queue of emails that need to go out, which is the ;; INPUT to our service. We're never going to queue anything here ;; so we give it a simple chan to satisfy dependencies :outgoing-emails (component/using (map->Queue (assoc (:outgoing-emails config) :outgoing-messages-chan (async/chan))) {:incoming-messages-chan :input-chan}) :db (map->Database (:database config)) :mailgun (map->EmailService (:mailgun config)) :worker (component/using (map->Worker {:work-fn log-and-send-emails}) {:input-chan :input-chan :db :db :email-service :mailgun :result-chan :result-chan}) :sent-emails (component/using (map->Queue (assoc (:sent-emails config) :incoming-messages-chan (async/chan)) {:outgoing-messages-chan :result-chan}))))

Tips and Tricks

(defrecord SomeRecord

component/Lifecycle (start [this] this) (stop [this] this))

Component provides a no-op implementation of the Lifecycle protocol on all java objects. What that means is, if you need to provide an atom, map, or something else as a dependency, there's no need to make a record for it.

Just put it in your system map as-is.

   ;; Position Tracking
   :position (atom 0)
   :position-tracker (component/using (map->Worker {})
                                      {:position :position}))

(start [this]
    (assoc this
           :config nil
           :incoming-messages-chan nil
           :outgoing-messages-chan nil))