Caveman2 - Lightweight web application framework

Usage

(defparameter *web* (make-instance '<app>))

@route GET "/"
(defun index ()
  (render #P"index.tmpl"))

@route GET "/hello"
(defun say-hello (&key (|name| "Guest"))
  (format nil "Hello, ~A" |name|))

About Caveman2

What's different from Caveman "1"?

Everything. Caveman2 was written from scratch.

These are noteworthy points.

Is based on ningle
Has database integration
Uses new, separate configuration system (Envy)
Has new routing macro

The reason I wrote it from scratch:

One of the most frequently asked questions was "Which should I use: ningle or Caveman? What are the differences?" I think these were asked so frequently because Caveman and ningle were too similar. Both of them are called "micro", and had no database support.

With Caveman2, Caveman is no longer a "micro" web application framework. It supports CL-DBI, and has database connection management by default. Caveman has started growing up.

Design Goal

Caveman is intended to be a collection of common parts of web applications. With Caveman2, I use three rules to make decisions:

Be extensible.
Be practical.
Don't force anything.

Quickstart

You came here because you're interested in living like a caveman, right? This isn't Disneyland, but we can start here. Let's get into a cave!

Installation

Caveman2 is now available on Quicklisp.

(ql:quickload :caveman2)

Generating a project skeleton

(caveman2:make-project #P"/path/to/myapp/"
                       :author "<Your full name>")
;-> writing /path/to/myapp/.gitignore
;   writing /path/to/myapp/README.markdown
;   writing /path/to/myapp/app.lisp
;   writing /path/to/myapp/db/schema.sql
;   writing /path/to/myapp/shlyfile.lisp
;   writing /path/to/myapp/myapp-test.asd
;   writing /path/to/myapp/myapp.asd
;   writing /path/to/myapp/src/config.lisp
;   writing /path/to/myapp/src/db.lisp
;   writing /path/to/myapp/src/main.lisp
;   writing /path/to/myapp/src/view.lisp
;   writing /path/to/myapp/src/web.lisp
;   writing /path/to/myapp/static/css/main.css
;   writing /path/to/myapp/t/myapp.lisp
;   writing /path/to/myapp/templates/_errors/404.html
;   writing /path/to/myapp/templates/index.tmpl
;   writing /path/to/myapp/templates/layout/default.tmpl

Start a server

This is an example that assumes that the name of your application is "myapp". Before starting the server, you must first load your app.

(ql:quickload :myapp)

Your application has functions named start and stop to start/stop your web application.

(myapp:start :port 8080)

As Caveman is based on Clack/Lack, you can choose which server to run on -- Hunchentoot, Woo or Wookie, etc.

(myapp:start :server :hunchentoot :port 8080)
(myapp:start :server :fcgi :port 8080)

I recommend you use Hunchentoot on a local machine, and use Woo in a production environment.

You can also start your application by using clackup command.

$ ros install clack
$ which clackup
/Users/nitro_idiot/.roswell/bin/clackup

$ APP_ENV=development clackup --server :fcgi --port 8080 app.lisp

Routing

Caveman2 provides 2 ways to define a route -- @route and defroute. You can use either.

@route is an annotation macro, defined by using cl-annot. It takes a method, a URL-string, and a function.

@route GET "/"
(defun index ()
  ...)

;; A route with no name.
@route GET "/welcome"
(lambda (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

This is similar to Caveman1's @url except for its argument list. You don't have to specify an argument when it is not required.

defroute is just a macro. It provides the same functionality as @route.

(defroute index "/" ()
  ...)

;; A route with no name.
(defroute "/welcome" (&key (|name| "Guest"))
  (format nil "Welcome, ~A" |name|))

Since Caveman bases on ningle, Caveman also has the Sinatra-like routing system.

;; GET request (default)
@route GET "/" (lambda () ...)
(defroute ("/" :method :GET) () ...)

;; POST request
@route POST "/" (lambda () ...)
(defroute ("/" :method :POST) () ...)

;; PUT request
@route PUT "/" (lambda () ...)
(defroute ("/" :method :PUT) () ...)

;; DELETE request
@route DELETE "/" (lambda () ...)
(defroute ("/" :method :DELETE) () ...)

;; OPTIONS request
@route OPTIONS "/" (lambda () ...)
(defroute ("/" :method :OPTIONS) () ...)

;; For all methods
@route ANY "/" (lambda () ...)
(defroute ("/" :method :ANY) () ...)

Route patterns may contain "keywords" to put the value into the argument.

(defroute "/hello/:name" (&key name)
  (format nil "Hello, ~A" name))

The above controller will be invoked when you access "/hello/Eitaro" or "/hello/Tomohiro", and name will be "Eitaro" or "Tomohiro", as appropriate.

(&key name) is almost same as a lambda list of Common Lisp, except it always allows other keys.

(defroute "/hello/:name" (&rest params &key name)
  ;; ...
  )

Route patterns may also contain "wildcard" parameters. They are accessible by using splat.

(defroute "/say/*/to/*" (&key splat)
  ; matches /say/hello/to/world
  (format nil "~A" splat))
;=> (hello world)

(defroute "/download/*.*" (&key splat)
  ; matches /download/path/to/file.xml
  (format nil "~A" splat)) 
;=> (path/to/file xml)

If you'd like to write use a regular expression in a URL rule, :regexp t should work.

(defroute ("/hello/([\\w]+)" :regexp t) (&key captures)
  (format nil "Hello, ~A!" (first captures)))

Normally, routes are tested for a match in the order they are defined, and only the first route matched is invoked, with the following routes being ignored. However, a route can continue testing for matches in the list, by including next-route.

(defroute "/guess/:who" (&key who)
  (if (string= who "Eitaro")
      "You got me!"
      (next-route)))

(defroute "/guess/*" ()
  "You missed!")

You can return following formats as the result of defroute.

String
Pathname
Clack's response list (containing Status, Headers and Body)

Redirection

Redirect to another route with(redirect "url"). A second optional argument is the status code, 302 by default.

Reverse URLs

When you defined routes with names, you can find the URL from a name with (url-for route-name &rest params).

The function will throw an error if no route is found.

More helper functions

Structured query/post parameters

Parameter keys containing square brackets ("[" & "]") will be parsed as structured parameters. You can access the parsed parameters as _parsed in routers.

<form action="/edit">
  <input type="name" name="person[name]" />
  <input type="name" name="person[email]" />
  <input type="name" name="person[birth][year]" />
  <input type="name" name="person[birth][month]" />
  <input type="name" name="person[birth][day]" />
</form>

(defroute "/edit" (&key _parsed)
  (format nil "~S" (cdr (assoc "person" _parsed :test #'string=))))
;=> "((\"name\" . \"Eitaro\") (\"email\" . \"e.arrows@gmail.com\") (\"birth\" . ((\"year\" . 2000) (\"month\" . 1) (\"day\" . 1))))"

;; With assoc-utils
(ql:quickload :assoc-utils)
(import 'assoc-utils:aget)
(defroute "/edit" (&key _parsed)
  (format nil "~S" (aget _parsed "person")))

Blank keys mean they have multiple values.

<form action="/add">
  <input type="text" name="items[][name]" />
  <input type="text" name="items[][price]" />

  <input type="text" name="items[][name]" />
  <input type="text" name="items[][price]" />

  <input type="submit" value="Add" />
</form>

(defroute "/add" (&key _parsed)
  (format nil "~S" (assoc "items" _parsed :test #'string=)))
;=> "(((\"name\" . \"WiiU\") (\"price\" . \"30000\")) ((\"name\" . \"PS4\") (\"price\" . \"69000\")))"

Templates

Caveman uses Djula as its default templating engine.

{% extends "layouts/default.html" %}
{% block title %}Users | MyApp{% endblock %}
{% block content %}
<div id="main">
  <ul>
  {% for user in users %}
    <li><a href="{{ user.url }}">{{ user.name }}</a></li>
  {% endfor %}
  </ul>
</div>
{% endblock %}

(import 'myapp.view:render)

(render #P"users.html"
        '(:users ((:url "/id/1"
                   :name "nitro_idiot")
                  (:url "/id/2"
                   :name "meymao"))
          :has-next-page T))

If you want to get something from a database or execute a function using Djula you must explicity call list when passing the arguments to render so that the code executes.

(import 'myapp.view:render)

(render #P"users.html"
        (list :users (get-users-from-db)))

JSON API

This is an example of a JSON API.

(defroute "/user.json" (&key |id|)
  (let ((person (find-person-from-db |id|)))
    ;; person => (:|name| "Eitaro Fukamachi" :|email| "e.arrows@gmail.com")
    (render-json person)))

;=> {"name":"Eitaro Fukamachi","email":"e.arrows@gmail.com"}

render-json is a part of a skeleton project. You can find its code in "src/view.lisp".

Static file

Images, CSS, JS, favicon.ico and robot.txt in "static/" directory will be served by default.

/images/logo.png => {PROJECT_ROOT}/static/images/logo.png
/css/main.css    => {PROJECT_ROOT}/static/css/main.css
/js/app/index.js => {PROJECT_ROOT}/static/js/app/index.js
/robot.txt       => {PROJECT_ROOT}/static/robot.txt
/favicon.ico     => {PROJECT_ROOT}/static/favicon.ico

You can change these rules by rewriting "PROJECT_ROOT/app.lisp". See Clack.Middleware.Static for detail.

Configuration

Caveman adopts Envy as a configuration switcher. This allows definition of multiple configurations and switching between them according to an environment variable.

This is a typical example:

(defpackage :myapp.config
  (:use :cl
        :envy))
(in-package :myapp.config)

(setf (config-env-var) "APP_ENV")

(defconfig :common
  `(:application-root ,(asdf:component-pathname (asdf:find-system :myapp))))

(defconfig |development|
  `(:debug T
    :databases
    ((:maindb :sqlite3 :database-name ,(merge-pathnames #P"test.db"
                                                        *application-root*)))))

(defconfig |production|
  '(:databases
    ((:maindb :mysql :database-name "myapp" :username "whoami" :password "1234")
     (:workerdb :mysql :database-name "jobs" :username "whoami" :password "1234"))))

(defconfig |staging|
  `(:debug T
    ,@|production|))

Every configuration is a property list. You can choose the configuration which to use by setting APP_ENV.

To get a value from the current configuration, call myapp.config:config with the key you want.

(import 'myapp.config:config)

(setf (osicat:environment-variable "APP_ENV") "development")
(config :debug)
;=> T

Database

When you add :databases to the configuration, Caveman enables database support. :databases is an association list of database settings.

(defconfig |production|
  '(:databases
    ((:maindb :mysql :database-name "myapp" :username "whoami" :password "1234")
     (:workerdb :mysql :database-name "jobs" :username "whoami" :password "1234"))))

db in a package myapp.db is a function for connecting to each databases configured the above. Here is an example.

(use-package '(:myapp.db :sxql :datafly))

(defun search-adults ()
  (with-connection (db)
    (retrieve-all
      (select :*
        (from :person)
        (where (:>= :age 20))))))

The connection is alive during the Lisp session, and will be reused in every HTTP request.

retrieve-all and the query language came from datafly and SxQL. See those sets of documentation for more information.

Set HTTP headers or HTTP status

There are several special variables available during a HTTP request. *request* and *response* represent a request and a response. If you are familiar with Clack, these are instances of subclasses of Clack.Request and Clack.Response.

(use-package :caveman2)

;; Get a value of Referer header.
(http-referer *request*)

;; Set Content-Type header.
(setf (getf (response-headers *response*) :content-type) "application/json")

;; Set HTTP status.
(setf (status *response*) 304)

If you would like to set Content-Type "application/json" for all "*.json" requests, next-route can be used.

(defroute "/*.json" ()
  (setf (getf (response-headers *response*) :content-type) "application/json")
  (next-route))

(defroute "/user.json" () ...)
(defroute "/search.json" () ...)
(defroute ("/new.json" :method :POST) () ...)

Using session

Session data is for memorizing user-specific data. *session* is a hash table that stores session data.

This example increments :counter in the session, and displays it for each visitor.

(defroute "/counter" ()
  (format nil "You came here ~A times."
          (incf (gethash :counter *session* 0))))

Caveman2 stores session data in-memory by default. To change this, specify :store to :session in "PROJECT_ROOT/app.lisp".

This example uses RDBMS to store session data.

      '(:backtrace
        :output (getf (config) :error-log))
      nil)
- :session
+ (:session
+  :store (make-dbi-store :connector (lambda ()
+                                      (apply #'dbi:connect
+                                             (myapp.db:connection-settings)))))
  (if (productionp)
      nil
      (lambda (app)