clj.orcery

Language, Expression and Design

Saturday

14

June 2014

The abstract container pattern

by Chris Zheng,

Clojure has very powerful tooling for working with native java objects and classes. It has numerous ways to take advantage of the java inheritence model. However, one feature that is missing from the language is the ability to define abstract classes. Therefore there is no built-in construct for building multi-tiered class abstraction hierachies in clojure as it is done in java. In my own transititon from thinking in terms of objects to thinking in terms of dataflow, I have not really missed this particular feature that much. However now that I have more experience building bigger and more complex programs using clojure, I have come back to revisit the importance of having such a layer in the design of complex systems.

The diagram below shows how functionality is extended in Java. Objects inherit functionality in a hierarchical manner. There has been a rather strong backlash against Object Orientated design methology in recent years (myself included), but there are a number of well designed java libraries that I greatly admire. Those that I'm been studying more recently are jscience, asm and javaassist.

Java Inheritence

Clojure is dynamically typed, with emphasis on representing data as maps, vectors, sets and lists. However, for a more type-safe approach to programming, the language advocates the use of protocols. Protocols allow flexiblity in the extension of functionality to existing classes and custom clojure defined types. However, it is a flat, two-tiered inheritence model that does not allow building up of object hierarchies. A diagram of such a pattern is show below:

Clojure Inheritence

Code Contracts

One way to look at design using abstract classes (or any language feature for that matter) is that it is a programming contract that is strictly enforced by the language itself. When we inherit functionality from an abstract class, we make a binding contract with the compiler. The contract stipulates that we have to implement a specific set of functions and we have to do it in a certain way. The compiler is meticulous in what it accepts as valid implementations of this contract. However, if we are able to satisfy its constraints, we can leverage all the functionality of a library built around the abstract implementation. The apache pool api for instance is a great example of how abstract classes are used to great effect. The string buffer example shows just how trivial it is to add pooling functionality for improved optimisation.

Although clojure generally does not restrict the programmer to form these contracts, we are still free to use this contract within our own code base. Although we don't have a compiler to check that our contract is conformant (though it could still be done through macros), we just have to be disciplined with our code and tests so that our code achieves the same effect of code reuse, flexibility that java forces us to use.

The Abstract Container

We in the clojure world are quite spoilt with the tools dealing with what is commonly know as the expression problem - multimethods and the afore mentioned protocols:

  • protocols are much like java interfaces, taking advantage of Java's class hierarchy to allow extension of operations. Protocols also have type safety which is very useful.
  • multimethods allows polymorphism on the return value of a dispatch function. This allows it more flexibility and users can implement a custom hierarchy for polymorphic dispatch.

There two paradigms are dual-wield through the abstract container pattern. The pattern uses an abstract layer to carry out domain specific functionality between an interface and a set of concrete implementation. Essentially, it mimicks the structure of how abstract classes are defined and used in object orientated design through simple use of protocols and multimethods. The pattern can be outlined as follows:

  • A single deftype acts as the abstract container, extending one or more protocols
  • A set of methods defined through defmulti that is used within the deftype form act as abstract methods
  • The abstract methods all dispatch on map keys (usually :type).
  • defmethod implementation of abstract methods act as concrete classes. We will refer to them as adaptors
  • An concrete implementation of functionality is just a map that is passed in to a generic constructor.

Having used the pattern now for the past couple of projects as well as having recieved valuable feedback from friends and colleges, I thought that it would be good to share this technique amongst the wider clojure community. This particular pattern has made my systems much easier to reason about, extremely flexibile, type safe (at least partialy), and also reduced the total lines of code.

A diagram of how a typical codebase should be arranged can be seen in the diagram below:

Abstract Container

Design of Server Framework

Lets look at an actual implementation of the abstract container pattern. This is shown below for a small and extensible http-server helper framework. The code and working example can be found on github.

The design guidelines of the helper framework are listed as follows:

  • Different types of servers (jetty, http-kit, tomcat, undertow) could be running at the same time (for comparison purposes)

  • The type of server can be set through an external config that is read in.

  • Servers that are running on a particular port can be tracked and stopped. I have to say, this was the feature that I wanted the most, which motivated the framework's design. The annoying thing about development in emacs is that I have to be careful of not losing the reference to the server. Since there was no way of stopping it unless the repl is restarted. I wanted to implement a registery for references to running servers to be saved.

  • A unified way of stopping, starting and querying servers. Although most clojure servers follow the ring spec, they have different ways of starting and stopping.

    • jetty calls run-jetty to start, returning a jetty instance. Stopping the server is done by calling .stop
    • lamina and httpkit use run-server, returning a function. Stopping the server is done by invoking the returned function with no arguments.

Actual Usage

We want to write the following code to define, start and stop a server:

(def serv (server {:type :http-kit
                   :port 8082
                   :handler (fn [req] {:status 200
                                       :body "Hello http-kit!"})})

(start! serv)  ;=> starts the server

(all-running)  ;=> shows an instance of the server on port 8082  
(stop!  serv)  ;=> stops the server

Implementation

Lets look at the actual implementation of the framework (under 100 loc). The overall design can be seen below:

Abstract Container Example

Interface

The interface is a very simple one, consisting of methods to start and stop a server as well as to query whether a server is stopped or started. Because we are using only one deftype, we can create type-safe methods by implementing them through protocols (where we get an exception if the method is called on a type that does not implement the protocol).

(ns server.common)

(defprotocol IRunnable
  (start! [system])
  (stop!  [system])
  (restart! [system])
  (started? [system])
  (stopped? [system]))

The server framework

The server framework can be split into a couple of parts:

  • abstact methods which are just defmulti forms.
  • deftype form implementing IRunnable providing the abstract container
  • other supporting framework methods.

For our server framework, we have two defmulti abstract methods - server-start and server-stop. Both multimethods dispatch on the value of :type of the map, and both implement a :default behaviour that just throws an exception if the method has not been implemented.

(defmulti  server-start :type)
(defmethod server-start :default
  [server]
  (throw (Exception. (format "server-start not implemented for {:type %s}" (:type server)))))

(defmulti server-stop :type)
(defmethod server-stop :default
  [server]
  (throw (Exception. (format "server-stop not implemented for {:type %s}" (:type server)))))

We define an atom as our register for all the running server instances.

(defonce ^:dynamic *running* (atom {}))

A server type is defined, implementing the IRunnable protocol along with methods for printing (toString) and lookup access (valAt). Comments are added within the code.

(deftype Server [state]
  Object
  (toString [server]               ;; This is to be able to pretty print the contents of a Server container 
    (format "SERVER::<%s>"
            (-> state
                (dissoc :instance :handler)
                (assoc  :started  (started? server))
                (->>
                 (map (fn [[k v]]
                        (str (name k) "=" v)))
                 (clojure.string/join ", ")))))

  IRunnable
  (start! [server]                 ;; First checks if the server is running, then updates the atom attached 
    (cond (or (started? server)    ;; to the :instance key of the state map with the  
              (get @*running* (:port state)))
          false

          :else
          (let [instance (server-start server)]   ;; Call to start `abstract method`
            (reset! (:instance state) instance)
            (swap! *running* assoc (:port state) server)
            true)))

  (stop! [server]
    (if (stopped? server) false
        (do (server-stop server)                  ;; Call to start `abstract method`
            (reset! (:instance state) nil)
          (swap! *running* dissoc (:port state))
          true)))

  (restart! [server]
    (stop! server)
    (start! server))

  (started? [server]                             
    (not (stopped? server)))

  (stopped? [server]                              ;; If there is nothing in the :instance atom, then the server 
    (nil? @(:instance state)))                    ;; is not running (indirect but effective)

  clojure.lang.ILookup
  (valAt [ele k]
    (if (or (nil? k)
            (= k :all))
      state
      (get state k))))

(defmethod print-method Server
  [v w]
  (.write w (str v)))

The actual call to construct a server instance is just a function. Note that settings should just be a map containing a key called :type whose value will be dispatched through the multimethod. Other helper methods like all-running and stop-all-running become very easy to define with the abstractions that have been put into place:

(defn server [settings]
  (Server. (assoc settings :instance (atom nil))))

(defn all-running []
  @*running*)

(defn stop-all-running []
  (do (reduce-kv
       (fn [i k v] (stop! v))
       nil @*running*)))

So that's it. Most of the logic is in the deftype itself. There are two places where the abstract methods are called (stop! and start!)

Concrete Implementations

Lets look at the code to extend the framework to use the http-kit server. As can be seen, not much code is needed:

(ns server.adapter.http-kit
  (:require [org.httpkit.server :as httpkit]
            [server.common :refer
             [server-stop server-start]]))

(defmethod server-start :http-kit
  [server]
  (httpkit/run-server (:handler server) (:all server)))

(defmethod server-stop :http-kit
  [server]
  ((-> server :instance deref)))

An equally short amount of code extends the jetty server:

(ns server.adapter.jetty
  (:require [ring.adapter.jetty :as jetty]
            [server.common :refer
             [server-stop server-start]]))

(defmethod server-start :jetty
  [server]
  (jetty/run-jetty (:handler server) (assoc (:all server) :join? false)))

(defmethod server-stop :jetty
  [server]
  (-> server :instance deref (.stop)))

Usage

Now that implementations for http-kit and jetty are defined, we can have a play around:

(ns server.example.http-kit
  (:require [server.common :refer :all]
            [server.adapter.http-kit]))

(def serv-http-kit
  (server {:type :http-kit
           :port 8082
           :handler (fn [req] {:status 200
                              :body "Hello http-kit!"})}))


(println serv-http-kit) ;;=> SERVER::<started=false, type=:http-kit, port=8082>

(start! serv-http-kit) ;;=> true

(all-running)          ;;=> {8082 SERVER::<started=true, type=:http-kit, port=8082>}

(stop! serv-http-kit)  ;;=> true

(all-running)          ;;=> {}

As can be seen, other clojure servers (such as undertow, lamina and tomcat) can be extended in the same manner using very little code and used within this smallish framework.

More Examples

The abstract container pattern has been used on some widely differing applications such as:

Thanks

A warm shoutout to Tushar, Lyndon, Dean, Alan, Hank, Derek, and all the guys at clj-melb that gave feedback and helped flesh out this rehash of OO design.

comments powered by Disqus