First, I should state that we have had very few instances of spam on our site. Any user accounts must be verified by email and require the user to input a captcha from the reCaptcha project. Once this is done, the user is free to make edits to the API documentation and to post new topics and replies on the discussion forums. However, the process of posting spam on the site is very difficult to automate due to the honeypots, field hashing, and post tokens that a default Sputnik installation employs. That’s not to say that the software is bullet-proof, but in practice it seems to work for my needs on a real site.

The first few accounts, I simply removed from my authentication system via a simple mysql query (although removing it from a default installation would be a easier, a simple edit of the sputnik/passwords node.) But then I realised I was giving up information about the spammers, including the email addresses they were using to register, something I could share with some of the spam registration databases that exist. It was clear that I needed an easier way to stop those user accounts from posting on the site without simply nuking their user accounts from the system. Really what I wanted was a way to ‘ban’ a user account and prevent them from making edits on the site.

Permissions in Sputnik

Each node in a Sputnik document store is a Lua script that is compiled and assembled into a Lua object at runtime. Consider the following simple node:

    title = "Some Page"
    content = [[This is a simple test node for a Sputnik installation.
 
    Hello World!]]

Sputnik nodes can be more than just a symbolic representation of text data. For example, you can restrict the permissions of a node by adding a permissions field. Consider the following definition:

    permissions    = [[
      deny(all_users, all_actions)
      allow(all_users, show)  -- show, show_content, cancel
      allow(Authenticated, edit_and_save) -- edit, save, preview
      allow(all_users, "post")  -- needed for login
      allow(all_users, "rss")
      allow(Admin, all_actions)
    ]]

When this field is loaded, it’s obviously a string that we can then further process. Specifically, we load it in an environment where a few helpers functions are defined. The all_users function returns true for all users, all_actions does the same for all action types. Authenticated is a special function that returns true if the current user is authenticated, while Anonymous can be used when the user is not logged in. There’s also a special function called Admin that queries the authentication metadata for the current user and checks if the is_admin flag is set.

When I started looking for a way to ban users, I started looking in this section of the code and found the following gem that Yuri sneaked in after the definition for the Admin group:

      local groups = self.saci.permission_groups -- just a local alias
      -- Define group "Admin" as any user that has is_admin set to true
      groups.Admin = function(user)
         return user and self.auth:get_metadata(user, "is_admin") == "true"
      end
 
      -- Define any group that starts with "is.<group>" as including all users
      -- for whom "is_<group>" is set to true. For example, if is_clown is set
      -- to true than user is a member of group "is.clown" and we can set:
      --
      --     allow(is.clown, "show")
      --
      local groups_mt = {
         __index = function(table, key)
            return function (user)
                      return user and self.auth:get_metadata(user,
    "is_"..key) == "true"
                   end
         end
      }
      groups.is = setmetatable({}, groups_mt)

Bingo! With this bit of code, without making any changes to the Sputnik core, I had a way to ban users in my system, by setting the is_banned flag in their authentication metadata. For a simple authentication system (the default) this is just a matter of setting the given flag in the user’s authentication entry, whereas in a mysql authentication system I just add a row to the database. This was only part of the puzzle, since Sputnik doesn’t know anything about the new user group. I had to make a few edits to the @Root prototype to add the following at the end:

    deny(is.banned, edit_and_save)

Since earlier in the permissions script the edit_and_save permissions are granted to all authenticated users, I needed to make sure the denial runs after that point, since the final ‘permission’ is what really matters. Since I have custom forums in my system, I needed to make the same one-line change in a few other places.

It took a total of about 6 minutes to familiarize myself with the code and make the necessary changes, and now I have an easy way to ban a user from being able to post anywhere on my website.

Down with the spammers!

Making a website just for yourself is one matter; doing it for others brings into play a range of quite separate considerations, that affect even the technical matters of design and implementation. You are the pig in the middle, the bacon in the sandwich, sitting between the site on one side and its commissioners on the other. I am a slow learner, but after a dozen years of experience of different commissions I have come to some definite general conclusions.

First, even for modest websites, plenty of time is needed for preparation, before you even start thinking about design or implementation; time for establishing a proper protocol between yourself and those who have commissioned you. Ensure that all communications between yourself and them pass through a single intermediary, a spokesperson who can shield you from the disagreements and internal rivalries into which it may be fatal to be drawn. Your commissioners may well have a committee charged with responsibility for the proposed website, and it will probably delegate its duties to a number of people: Mr X the timetable, Miss Y the graphics, Dr Z the document archive, … . Fine. Nice to know you, Mr X, Miss Y, Dr Z, … Have nothing to do with these arrangements. Insist that they all communicate only through the spokesperson. The spokesperson will have to channel information in both directions. That way, with any luck, you will not have to explain more than a dozen times why web pages look different in different browsers, and why enormous TIFF files intended for high quality printing are not appropriate for graphics on web pages, because they will take rather a long time to download.

To be realistic, it is too much to expect the more important members of the website committee to stick to this regime. In any case, out of friendliness they may want to express themselves personally. It is for that reason that I recommend that during the development phase of the website each page should contain within its header the tags

<meta  http-equiv="pragma" content="no-cache" />
<meta  http-equiv="cache-control" content="no-cache" />

because otherwise you will have the committee chairman sending you messages accusing you of not getting on with the job: “Hi Gavin, Mr X says he sent you that stuff about emergency car-parking three weeks ago and it still is not up on the website.”; and then you will have to explain to a deeply suspicious technophobe all about how browsers cache webpages and that he is still looking at the pages that his browser cached three weeks ago, and no, you cannot tell him how to clear the cache for himself because you do not know what operating system or browser he is using (“operating system? what is that, old boy?”).

I struck an extraordinary glitch when I was making the website for the twenty-first International Congress of Byzantine Studies, which took place in London from 21 to 26 August 2006. It was a prestigious affair, with the Prince of Wales as patron and a long list of eminent sponsoring organizations. Clarence House had been persuaded to send me a facsimile of a signed letter of welcome from HRH, in receipt of which I had to sign an undertaking that I would not permit any improper use of it. I put a link to it from the sponsors’ page.

The organizing committee of the conference had obtained a lovely jpeg from the British Museum, of a Byzantine wooden carving of an angel. They used this to make a poster. I used a jpeg of that to make the entry page of the site. Clicking on the angel was to bring one to a page listing the sponsors, including HRH, and inviting the viewer to choose either the English or the French version of the website. In my innocence, I called the page “sponsor.html”. When it was up for review by the organizing committee I began to get puzzled queries from them. “I clicked on the angel, and nothing happened”, about half of them reported. I had no problem, nor did the other half. It was very mysterious. Eventually research showed the culprit to be an arrogant piece of Norton AntiVirus software running on the computers of some committee members. It had decided that “sponsor” was obscene or dangerous, apparently. Thus was HRH’s letter censored by Norton. In the end I changed the file’s name to “spons.html” and the problem went away. The moral, I suppose, is that you have to fathom some pretty sick minds if your webpages are to be viewable by the public.

It is time to turn from the human to the coding side of things. I have always believed that the best principle is to keep all as simple as possible. As none of my sites have been designed for huge traffic I have always shunned client-side scripting. It is usually unnecessary. One should also keep in mind that many users may not have the latest browser. In any case, adoption of WAI (the Web Accessibility Initiative ) is mandatory for .gov.uk websites, and it makes sense for any site that is to be used by the public, especially an academic public that is not likely to have the latest hardware.

I always create my websites on my own machine. I generally divide the directory for website-creation into two subdirectories: “site”, to contain all the stuff to be uploaded to the server, and “data” to hold the gubbins from which the website is to be created, usually as a “make” project. The contents of the webpages I classify into three groups:

1. furniture – the graphics and css stuff that determine a common theme;
2. permanent content – usually text;
3. formatted mutable data – usually held as a textual database, as lists of labelled records, for ease of updating with a text editor.

I use Lua format for the databases and Lua scripts to input from the data-directory and output the finished pages to the site-directory. To begin with, for the Art Historians’ Millennium bash, I used a Lua program, called Weave, which I had developed for creating web pages. Over time I improved it by generalizing to the production of any sort of marked-up text. Eventually it simply got replaced by a Lua library. I must make another blog about this, because Lua’s way with strings was a key factor in the design of this software. I also produced a Lua program, Infuse, to perform the abstract task of pouring content into template files – a sort of mail-merge. It was the actual experience of coping with the commissioned websites that drove me to keep reinventing. The frustrating thing was that I dared not change over to a newer regime until the project was completed. By starting sufficiently early I could create my templates for the approval of the commissioners, before the flood of content hit me. It was important to have a design sufficiently flexible to cope with sudden changes of tack. The Lua format for labelled records was good in this respect, because extra fields could always be added. By contrast, CSV format would have been hopeless.

Lua’s extreme portability was handy. My own machine ran neither Windows, Linux nor MacOS. Yet when I did a website for an artist, whose agent needed to take over the running of it, even though the agent used Windows, there was no problem getting everything working on his machine.

For the Byzantine Congress I was told that I had been appointed ἐπὶ τοῠ κανικλείου , guardian of the imperial inkstand, so named because it was in the shape of a little dog (canicula). To find out more about this prestigious office, use Google.

Gavin Wraith

Ropes are better than Strings: A Technical Appendix

In Lua strings are immutable values. This means that you should avoid building up strings out of small pieces, because although they will eventually be garbage-collected, all the intermediate stages of your construction will still be there taking up storage. Building up a string of n characters by consecutive concatenations will use a number of bytes varying quadratically with n. So concatenations are something to be avoided. What about the construction of web pages, where markup and text must be intertwined? Surely that will involve lots of concatenations?

Actually, no – it need not. Whether we think of an html document as text with markup inserted, or whether we think of it as a structure of nested tag-pairs with text inserted, either way we are led to the concept of strings-with-holes. To be more formal we define the datatype of “ropes” recursively by:

rope = string | list of rope

In Lua syntax, a rope either looks like

[[ ..... ]]

or like

{ ........ }

where the contents of the braces are a comma separated list of ropes.

Put another way, a rope is a tree whose leaves are strings. We can think of it as a string that has been broken apart leaving gaps. If we walk over a rope, depth first, printing out the leaves to a file, then we can construct our marked-up text without doing any string-concatenation at all.

So this code

<a href="http://luanova.org/">LuaNova</a>

can be obtained by printing out the rope

     {
     {
      [[<a ]],
      {
       [[href="]],
       [[http://luanova.org/]],
       [["]],
      },
      [[>]],
     },
     [[LuaNova]],
     [[</a>]],
     }

If we define

     link = function (url)
            return function (label)
                    return {
     {
      [[<a ]],
      {
       [[href="]],
       url,
       [["]],
      },
      [[>]],
     },
     label,
     [[</a>]],
     } end end

then our rope is just the value of

    link "http://luanova.org/" "LuaNova"

There are two sorts of markup: “monotags” and “tags”.

By a monotag I mean something like

<tag_name attr />

and we can implement this as a function from ropes to ropes:

    function(attr) return {"<tag_name ",attr," />"} end

By a tag I mean something like

<tag_id attr>stuff</tag_id>

which we can implement as a function from ropes to functions from ropes to ropes:

     function(attr)
      return function(stuff) return
             {"<tag_id ",attr,">",stuff,"</tag_id>"}
             end
     end

To simplify construction of webpages I invented a little language, which I called Weave, as a fragment of Lua using strings and ropes and named functions for the tags and monotags of HTML (see Lua technical note 30 .) After a while I broadened Weave’s abilities so that it could cope with any sort of markup and no longer had HTML’s tags and monotags built in. I introduced empty tables called “TAG” and “MONOTAG” with metatables, so that using the __index key I could define tag and monotag functions with notations like “TAG.a” and “MONOTAG.img” and so on. This was more flexible, but it did mean that the user had to define her own tags before building a webpage. That did not matter, because by then CSS had appeared and most webpages needed only a handful of tags and monotags. Eventually I did away with Weave as a separate entity and simply used the Lua module “xhtml” instead.

To create a very trivial webpage “foo.html” in the current directory, run a Lua script as follows:

     #! lua
     require "xhtml"
     do
     local mydoc = BEGIN "foo.html"
     local hstyle = [[style="text-align: centre;"]]
     local myheader, link = xhtml.TAG.h1, xhtml.link
     mydoc.BODY = {
                   myheader (hstyle) "Try this";
                   link "http://luanova.org/" "Lua Nova";
                  }
     END(mydoc) -- write the rope out to file
     end -- do

OK, this takes almost as long as creating the webpage by hand, for this trivial example. But it should not take too much imagination to see the possibilities. Practically every webpage I have created for the last ten years has used this method.

Here are the relevant modules:

weave.lua

     --[[
         The rope datatype is defined recursively:
 
         data rope = string | list of rope.
 
       Use:  status,err = rope.walk(rope,function)
             print(table.concat(err,"."))
       Error message prints out index at each level.
     ]]
     local pcall = pcall
     local type = type
     local ipairs = ipairs
     local tostring = tostring
     local assert = assert
     local setmetatable = setmetatable
     local open = io.open
 
     module "weave"
 
     -- ropes -------
 
     local t_err = "<bad type: %s>"
     local a_err = "<bad arg: %s>"
 
     walk = function(x,f)  -- apply f to leaves of tree x, depth first
          local g
          g = function(x,e) -- e = error stack
             local x_type, status = type(x)
             if x_type == "string" then
              status = pcall(f,x)
              if not status then e[1+#e] = a_err:format(x) end -- if
                  return status,e
              elseif x_type == "table" then
                  e[1+#e] = ""
                  for i,y in ipairs(x) do
                    e[#e] = tostring(i)
                    status = g(y,e)
                    if not status then return nil,e end -- if
                  end -- for
                  e[#e] = nil
                  return true
                else
                  e[1+#e] = t_err:format(x_type)
                  return nil,e
                end -- if
              end -- function
              return g(x,{})
        end
 
       local file_err = "cannot open %s"
       out = function(x,file)
             local o = assert(open(file,"w"),file_err:format(file))
             local f = function(s)
                       o:write(s or "")
                       return true
                       end -- function
             local status,err = walk(x,f)
             o:close()
             return status,err
             end -- function
 
       ------ markup stuff --------------
       --[[
       Two types of tag:
 
         monotag    <foo attributestring />
         tag    <foo attributestring> rope </foo>
 
       where attributestring could be empty.
         ]]
 
       local monotagger = {
       __index = function (_,tag)
                 assert(type(tag) == "string")
                 return function (attr)
                        local o = {"<",tag}
                        if attr and (#attr > 0) then
                          o[1+#o] = " "
                          o[1+#o] = attr
                        end -- if
                        o[1+#o] = " />"
                        return o
                        end -- function
                  end -- function
                 }
 
       local bitagger = {
       __index = function (_,tag)
                 assert(type(tag) == "string")
                 return function (attr)
                        return function (rope)
                         local o = {"<",tag}
                         if attr and (#attr > 0) then
                          o[1+#o] = " "
                          o[1+#o] = attr
                         end -- if
                         o[1+#o] = ">"
                         o[1+#o] = rope
                         o[1+#o] = "</"
                         o[1+#o] = tag
                         o[1+#o] = ">"
                         return o
                  end end end
                  }
       MONOTAG, TAG = {},{}
       setmetatable(MONOTAG,monotagger)
       setmetatable(TAG,bitagger)

xhtml.lua

       --[[ The xhtml library exports
 
         TAG, MONOTAG, BEGIN, END, TEXT, SPACE, REM and link
 
       ]]
 
 
       require "weave"
 
       local assert = assert
       local type = type
       local pairs = pairs
       local print = print
       local this = arg[0]
       local leaf = this:match("\\([^\\]*)$")
       local weave = weave
       local walk = weave.walk
       local out = weave.out
 
       local concat = table.concat
 
       module "xhtml"
 
       MONOTAG = weave.MONOTAG
       TAG = weave.TAG
 
       local doctype = [[
       <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">]]
 
       local HTML = TAG.html [[
       xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"]]
 
       local nl = "\n"
 
       BEGIN = function(filename)
               if filename then
                 assert(type(filename) == "string","BEGIN takes a string or nil")
               end -- if
               return { FILE = filename or this..".html"; }
               end
 
       local META,LINK = MONOTAG.meta,MONOTAG.link
       local HEAD,BODY,TITLE = TAG.head "",TAG.body,TAG.title ""
       local css = [[rel="stylesheet" type="text/css" media="screen" href=""]]
 
       END = function(rope)
             local file = rope.FILE
             local style = rope.STYLE
             local doc = {
              rope.DOCTYPE or doctype;
              nl;
              HTML {
               nl;
               HEAD {
                nl;
                TITLE (rope.TITLE or leaf);
                nl;
                META [[name="Generator" content="Lua"]];
                nl;
                META [[name="MSSmartTagsPreventParsing" content="TRUE"]];
                nl;
                rope.HEAD or "";
                style and LINK {css;style;'"'} or "";
                    };
                nl;
                BODY (rope.ATTR or "") (rope.BODY or "");
                nl;
                  };
                          }
              status, err = out(doc,file)
              if not status then print(concat(err,'.')) end -- if
              end --function
 
       local entity = {
         ["<"] = "&lt;",
         [">"] = "&gt;",
         ["&&"] = "&amp;",
                      }
       local catchall = "(.)"
       local entify = function (c)
                      local n,fmts = c:byte(),"&#%s;"
                      return (n > 127) and fmts:format(n) or c
                      end -- function
 
       TEXT = function(rope)
              local o = {}
              local f = function(s)
                  local status = type(s) == "string"
                  if status then
                    for symb,ent in pairs(entity) do
                      s = s:gsub(symb,ent)
                    end -- for
                    s = s:gsub(catchall,entify)
                    o[1+#o] = s
                  end -- if
                   return status
                 end -- function
              walk(rope,f)
              return o
              end -- function
 
       SPACE = function(n)
               assert(type(n) == "number","SPACE takes a number")
               local s = "&nbsp;"
               return s:rep(n)
               end -- function
 
       REM = function(s) return { "\n<!-- ";s;" -->\n"; } end -- function
       link = function(url) return TAG.a { 'href="'; url; '"'; } end -- function

Orbit is a lightweight framework for Web applications written in Lua. Unlike Sputnik, Orbit gives you the basic tools for constructing applications built on WSAPI. If you want authentication, standard look-and-feel, wiki-like functionality, you are better off starting with Sputnik; but Orbit is a satisfying way to build web applications from scratch.

The easiest way to install Orbit on a Unix machine is using LuaRocks, but here for a change I will assume that you already have Lua for Windows which already has most of the dependencies for Orbit. Then download orbit-2.0.2-all.zip, unzip it somewhere preserving folder names, and run lua install-orbit.lua from this directory in a command prompt.

The examples in this tutorial are then in the orbit-2.0.2\samples\basic folder.

The Basics

The simplest Orbit application is one that serves up a single HTML page:

    -- basic1.lua
    require"orbit"
 
    -- declaration
    module("basic1", package.seeall, orbit.new)
 
    -- handler
    function index(web)
      return render_index()
    end
 
    -- dispatch
    basic1:dispatch_get(index, "/", "/index")
 
    -- render
    function render_index()
       return [[
        <head></head>
        <html>
        <h2>Pretty Easy!</h2>
        </html>
        ]]
    end

An Orbit application is a Lua module, declared in the usual way except for the extra ‘orbit.new’. Lua’s module function can take a number of extra arguments, which are all functions which modify the module table in some way: package.seeall makes the global environment accessible to the module, and orbit.new adds some extra methods to the new module (you may think of this as ‘inheritance’.)

Any HTML request is mapped onto handler functions – in this case, the result of doing a GET request on ‘/’ or ‘/index’. index just asks render_index to actually generate the HTML for this particular page. You can then run this application using the default Xavante server:

c:\basic> orbit basic1.lua
Starting Orbit server at port 8080

And you can view the result by going to http://localhost:8080 in your favourite browser.

Not perhaps very impressive, seeing as the result could be achieved by a simple static page, but the render function can return any dynamically created string. And this example will work with Xavante, FastCGI or CGI, because Orbit is a WSAPI application.

All the usual web request variables are parsed and available to you. The web variable returned from the handler contains a table GET which has all the parameters passed to the server from a GET request. Call this script with http://localhost:8080/?cat=felix&dog=fido and you will see that cat and dog are fields of web.GET, appropriately translated from URL encoding.

    -- basic3.lua
    require"orbit"
 
    module("basic3", package.seeall, orbit.new)
 
    basic3:dispatch_get(function(web)
        local ls = {}
        for k,v in pairs(web.GET) do
            table.insert(ls,('<li>%s = %s</li>'):format(k,v))
        end
        ls = '<ul>'..table.concat(ls,'\n')..'</ul>'
        return ([[
        <html><head></head><body>
        Web Variables <br/>
        %s
        </body></html>
        ]]):format(ls)
    end, "/", "/index")

This is not a pretty script; it’s only virtue is that it is short. It is ugly for two basic reasons; first, generating HTML with regular string operations is awkward, and second, it does not separate the logical parts out neatly. The HTML problem will be discussed in the next section, but the next example separates the various operations into their own sections.

This also shows how Orbit can serve up static content as well, in this case anything inside the /images directory:

    -- basic2.lua
    require"orbit"
 
    module("basic2", package.seeall, orbit.new)
 
    function index(web)
      return render_index(collectgarbage("count"))
    end
 
    basic2:dispatch_get(index, "/", "/index")
    -- any file in this directory will be served statically
    basic2:dispatch_static ("/images/.+")
 
    local template = [[
    <head></head>
    <html>
    %s
    </html>
    ]]
 
    function render_page(contents)
        return template:format(contents)
    end
 
    function render_index(mem)
       return ([[
        <img src="/images/lua.png"/>
        <h2>Memory used by Lua is %6.0f kB<h2>
       ]]):format(mem)
    end

Note that dispatch_static is passed a Lua string pattern, not a file mask; in this case, anything in the directory.

We are now presenting true dynamic content (the amount of memory managed by Lua) and made a first start to prettifying the result. It is convenient to only generate the ‘inner’ HTML and use a template to generate the full page, so render_page has been factored out.

Orbit follows the Model-View-Controller (MVC) pattern that separates an application into three distinct parts. The Controller acts as the executive, the Model manages the data, and the View presents the data to the user; here the dispatch rules calls index which uses the renderer render_index to present the HTML form of the data. This pattern comes from a basic principle, Separation of Concerns. You do not want to mix database access, application logic and visual presentation together because the result is not clear and may become unmaintainable quickly.

HTML the Orbit Way

The HTML problem is usually solved using templates, such as Cosmo as discussed in the last article on Sputnik. Orbit provides another option, which is that Lua code can generate HTML:

    -- html1.lua
    require"orbit"
 
    function generate()
        return html {
            head{title "HTML Example"},
            body{
                h2{"Here we go again!"}
            }
        }
    end
 
    orbit.htmlify(generate)
 
    print(generate())
 
    ==>
    C:\basic>lua html1.lua
    <html ><head ><title >HTML Example</title>></head><body ><h2 >
    Here we go again!</h2></body></html>

This certainly reads better if you are used to Lua, but the magic involved needs some explanation. You may imagine that Orbit has introduced a whole bunch of little functions into the module, but this is not how it is done. Instead, orbit.htmlify modifies the environment of the function generate so that any undefined variable is assumed to be a HTML tag and becomes a suitable function. So you can generate anything, even with unknown tags. So if generate was:

    function generate()
        return sensors {
            sensor {name="one"},
            sensor {name="two"},
        }
    end

Then the result would be this syntactically valid XML:

<sensors ><sensor name="one"></sensor>
<sensor name="two"></sensor></sensors>

In presenting the resulting markup I’ve inserted a few line breaks for making reading a bit easier. Naturally you might like to beautify the output for debugging purposes; web browsers do not care. HTML Tidy is another option. which will also give you a host of other diagnostic checks.

One problem to watch out for is that a mistyped tag name will give you odd HTML, not a Lua error. And there are some HTML tags which are valid Lua globals, such as table – the solution is to say H(‘table’), etc. But HTML can now be generated without making your eyes bleed, as with the ugly web variables script. This little snippet uses the fact that tag functions take table arguments:

    function generate (web)
        local list = {}
        for name,value in pairs(web.GET) do
            table.insert(list,li(name.." = "..value))
        end
        return ul(list)
    end

Any technique can produce over-elaborate code, as this program which generates an HTML form formatted with a table:

    require"orbit"
 
    function wrap (inner)
        return html{ head(), body(inner) }
    end
 
    function test ()
        return wrap(form (H'table' {
            tr{td"First name",td( input{type='text', name='first'})},
            tr{td"Second name",td(input{type='text', name='second'})},
            tr{ td(input{type='submit', value='Submit!'}),
                td(input{type='submit',value='Cancel'})
            },
        }))
    end
 
 
    orbit.htmlify(wrap,test)
 
    print(test())

Not quite eye-bleeding, but getting there! The solution is to capture common patterns in a library. The util module in the basic folder makes for much more readable code:

    function show_form (web)
        return wrap(form {
            htable {
                {text("First name",'first')},
                {text("Second name",'second')},
                {submit 'Submit', submit 'Cancel'},
            }
        })
    end

Databases the Orbit Way

Orbit can create an Object-Relational Mapping (ORM) between the tables of a relational database and Lua tables. The restriction is that the SQL table is required to have a numeric key called id. In the following discussion, we must switch between two very different meanings of the word table, so I’ll use ‘array’ to mean ‘an array-like Lua table’ and ‘map’ to mean ‘a hash-like Lua table’.

Consider the following SQL table definition found in the blog example database:

CREATE TABLE blog_post
       ("id" INTEGER PRIMARY KEY NOT NULL,
       "title" VARCHAR(255) DEFAULT NULL,
       "body" TEXT DEFAULT NULL,
       "n_comments" INTEGER DEFAULT NULL,
       "published_at" DATETIME DEFAULT NULL);

This code uses LuaSQL’s sqlite3 driver to dump out a particular row of this table by defining a mapper:

    -- ORM1.lua
    require "orbit"
    require "luasql.sqlite3"
    local env = luasql.sqlite3()
 
    function dump (t)
        if #t > 0 then
            for _,row in ipairs(t) do
                dump(row)
            end
        else
            for field,value in pairs(t) do
                if type(value) == 'string' then
                    value = value:sub(1,60)
                end
                print(field,type(value),value)
            end
            print '-----'
        end
    end
 
    mapper = orbit.model.new()
    mapper.conn = env:connect("../blog/blog.db")
    mapper.driver = "sqlite3"
    mapper.table_prefix = 'blog_'
 
    posts = mapper:new 'post'  -- maps to blog_post table
 
    -- print out the second post
    second = posts:find(2)
 
    dump(second)
 
    ==>
    published_at	number	1096923480
    title	string	The Care And Feeding Of Your Sleeping Bicycle
    id	number	2
    n_comments	number	0
    body	string
    Thunder, thunder, thundercats, Ho! Thundercats are on the m
    -----

posts:find(2) gets the database row with id equal to 2 and returns a map with corresponding fields and values.

SQL is a more strongly-typed language than Lua, so that both published_at and n_comments are mapped to the number type. However, the mapper does preserve this information in its field meta; for instance, mapper.meta.published_at.type is ‘datetime’.

Orbit’s ORM mappers simplify database access by bridging the gap between the underlying database and the language; the most involved code here is just to dump out Lua maps and arrays.

As always, the interactive prompt is the best way to explore the available mapper methods:

C:\basic>lua -lmapper
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> posts = mapper:new 'post'
> dump(posts:find_first('n_comments > 0'))
published_at    number  1097769720
title   string  'Star Wars' As Written By A Princess
id      number  4
n_comments      number  1
body    string
Ten years ago a crack commando unit was sent to prison by a
-----

find_first also returns a row, but by an explicit condition; find_all is similar, but returns an array of all matching rows. Note that the condition is a SQL expression, not Lua; this is effectively a more friendly way to create an SQL query:

    dump(posts:find_all("id = ? or id = ?",
        {2,4,order = "published_at asc"}))

ORM mapping works particularly nicely with dynamic languages, but there are some downsides. consider the case that the table blog_post had many thousands of rows; the default mapper will grab all the columns, including the potentially rather large body. So opportunities for optimizing queries are lost, if we were only interested in searching the title for a match. It’s possible to reorganize things so that blog_post only contained ‘meta’ data about each post (plus perhaps a short summary) and a new table blog_text would contain the actual text of each post, indexable using the same id.

Putting it Together

The next example will bring together these three concerns: request pattern matching (Controller), ORM mapping to a database (Model) and Orbit htmlification (View). It uses the database from the blog example but presents it in summary form. For each row, it puts out a nicely formatted summary. The actual rows are selected by a user set of keywords.

All our HTML-generating functions begin with ‘render_’. orbit.htmlify can be called in another way, where you give the module first, and then specify a string pattern to match the functions in the module that you want to htmlify.

	orbit.htmlify(blog1,'render_.+')

Initially, the actual model-controller part of the application is simply this:

    function index(web)
      return render_index(posts:find_all())
    end
 
    blog1:dispatch_get(index, "/", "/index")

And the view part:

    local function limit (s,maxlen)
        if #s > maxlen then
            return s:sub(1,maxlen)..'...'
        else
            return s
        end
    end
 
    function render_page(contents)
        return html {
            head { title 'Blog Example' },
            body (contents)
        }
    end
 
    function render_post(post)
        return div {
            h3 (post.id .. ' ' .. post.title),
            p (limit(post.body,128)),
            em(os.date('%a %b %d %H:%M %Y',post.published_at)),
            ' ',post.n_comments,' comments'
        }
    end
 
    function render_index(posts)
        local res = {}
        for i,post in ipairs(posts) do
            res[i] = render_post(post)
        end
        return render_page(res)
    end

The heart of the view is render_post, which formats each row from the model. You will only achieve date formatting happiness once you have learned the formating characters for os.date; these are the same as those used in the C strftime function.

All in all, not bad for a 55-line program. The next step is to add simple keyword searching, which takes an extra line:

    function index(web)
      local keyword = web.GET.keyword or ''
      if keyword == '' then return {}
      else
        return render_index(posts:find_all('title like ?',{'%'..keyword..'%'}))
      end
    end

The SQL like operator works with patterns, so that finding a title containing the word ‘Chicken’ would be “title like ‘%Chicken%’”, where ‘%’ means ‘any sequence of characters’. The keyword is passed as a GET request variable: enter the following URL to see the results of the search: http://localhost:8080/?keyword=Chicken.

It would be useful to have some user interface, especially when we start passing multiple keywords. render_index becomes:

    function render_index(posts)
        local res = {}
        append(res,p (form {
            input {type='text',name='keyword'},
            input {type='submit',value='Submit'},
        }))
        append(res,hr())
        append(res,h2(('Found %d posts'):format(#posts)))
        for _,post in ipairs(posts) do
            append(res,render_post(post))
        end
        return render_page(res)
    end

By default, HTML forms submit their values to the same URL as the page, so this works fine.

This introduction was intended to help you begin reading the Orbit samples, documentation and tutorials. In the next in this series, we will discuss building more complicated applications, caching generated pages, and Orbit Pages, which is a powerful Lua-powered template engine for generating HTML.

We haven’t heard a lot about Lua on the web, even though there are a number of developers working with Lua on the server-side. Lets take a brief look.

The Kepler Project

Kepler is a “web development platform” named after astronomer Johannes Kepler. It looks to be the most ambitious project of the bunch. Kepler is written as a number of modules that fit together, and includes its’ own web server named Xavante.

There is also an endeavor to build WSAPI, in similar fashion to Python’s WSGI interface, that support pretty much any web server and configuration (CGI, FastCGI, SCGI, etc, etc.)

Nanoki

When I joined to Kepler mailing list, “Petite Abeille” was the first person to respond. He has been developing a Wiki called Nanoki.

However, it is not built on top of Kepler. Instead there is a small built-in web server or will run atop the Unix tcp command. All the source code is released under a MIT license, and is instructive in building a light, focussed web application vs. the broad scope of Kepler.

mod_wombat

Brian McCallister spear-headed a project to embed Lua in an Apache HTTP Server module. Going through Brian’s slides is quite inspiring as to just how good a fit this is. Apache does all the heavy-lifting, such as providing a fully multi-threaded “worker MPM” (multi-processing module) and a portable runtime (APR) environment. Lua is a thread-safe language, with “share-nothing” Lua states, so it fits right in and is super-efficient.

Skimming through The Apache Modules Book is even more inspiring, as you can see the potential of mod_wombat if it took advantage of Apache’s database connection pooling (DBD) and the various other facilities of Apache HTTP Server.

Wombat is still a little rough around the edges, but you can find the Apache-licensed source code at: http://svn.apache.org/repos/asf/httpd/mod_wombat/trunk.

Building

First, you need to have Apple’s Developer Tools installed. They should be your Mac OS X disc, under Optional Installs -> Xcode Tools. Run the XcodeTools package to install. You can also download Xcode if you sign up for a free ADC account (but note: they are large!).

Next, download Lua source code and unpack the archive in Finder. Or you can use Terminal to download the current release (5.1.2) to your Downloads folder (or a suitable location of your choice):

cd Downloads curl -O http://www.lua.org/ftp/lua-5.1.2.tar.gz tar xzvf lua-5.1.2.tar.gz cd lua-5.1.2

Either way, you need Terminal (from Application/Utilities) to do the rest. Make sure you are in the lua-5.1.2/ folder, and type: make macosx

Several lines should scroll by. On Leopard you will see some deprecated warnings in loadlib.c. Don’t worry about it. To make sure everything is okay, run: make test You should see “Hello world, from Lua 5.1!”

Installing

It is possible to run the Lua interpreter from right here, but let’s install it the rest of the way. For this we will need to run “make install”. By default Lua installs to /usr. This will work, but it’s recommended to install under /usr/local.

So from the lua-5.1.2/ folder, run: sudo make install INSTALL_TOP=/usr/local and provide your password.

It appears that Leopard ships with /usr/local/bin in your path. You can check by running: (make sure PATH is uppercase) env | grep PATH

If you don’t see it there, you need to modify your .profile file. pico ~/.profile

and include:

export PATH="/usr/local/bin:/usr/local/sbin:$PATH" export MANPATH="/usr/local/man:$MANPATH"

If you are using TextMate, and will be installing the Lua bundle later, you may also want to include:

export SVN_EDITOR="mate -w" export LC_CTYPE=en_US.UTF-8

With pico, press Ctrl-X followed by Y to exit and save.

The changes will take affect when you open a new Terminal window.

Running

With Lua installed in your path, you can run it in Terminal from any folder. Just type: lua

This brings up the interactive interpreter. You can type in Lua code: print "Hi" = 2 + 3

Press Ctrl-C to exit when you’re done.

Now that Lua is installed, you could remove the Downloads/lua-5.1.2 folder. But you may want to check out the test/ folder for some example code. You can run these examples from within the lua-5.1.2/test/ folder like this:

lua factorial.lua

A local copy of the Reference Manual can be found under doc/manual.html.

TextMate

If you are using TextMate, there is a Lua bundle to give it color syntax highlighting and a few snippets. Unfortunately the Lua bundle isn’t included by default. You can either use GetBundle, or you can grab it with the command line as follows.

The bundles are stored in a Subversion repository, which requires Subversion (svn) on your computer. Leopard comes with Subversion, but for Tiger or prior you need to install it. I’ve had good success with Martin Ott’s Subversion package.

As per the TextMate manual:

mkdir -p /Library/Application\ Support/TextMate/Bundles cd /Library/Application\ Support/TextMate/Bundles svn co http://svn.textmate.org/trunk/Bundles/Lua.tmbundle

You need to restart TextMate or navigate to to Bundles -> Bundle Editor -> Reload Bundles in the menu.

Files with a .lua extension will be associated with the Lua language bundle. One nice feature is that you can press Command-R to run them.

So that’s about it for getting setup. Now we just need to write some code!

Lua was started in 1993, near the same time as Matz began working on Ruby. It was designed to be a small and efficient scripting language that could easily be embedded in C/C++ on practically any platform.

Aiming for simplicity, Lua implements a single collection type called “tables”. PHP also takes this approach, with Arrays that can act as hashes (associative arrays) and standard arrays. Python uses its dictionaries (hashes) for namespaces and passing named parameters to functions. Lua does this and more, making extensive use of tables.

One of Lua’s precepts is to “provide mechanisms instead of policies.” You won’t find Ruby’s emphasis on object-oriented programming in Lua, but it still is easy enough to OOP. In fact, Lua works somewhat like the prototype system in today’s JavaScript.

First-class functions allow functions to be passed as arguments, stored in tables, and basically treated like any other value. Proper tail calls allow for recursion without stack overflows. Closures are a bit hard to explain without an example. You are free to make use of all these mechanisms, which originated in functional programming, a distinct discipline from the typical C-style of programming.

Lua isn’t alone in mixing programming paradigms, but its overall simplicity as a language makes it a good place to begin computer programming.

In 1996, Lua found its way into Dr. Dobb’s Journal and from there into adventure game Grim Fandango and many other games since the 1999 Game Developers’ Conference, including the ever-popular World of Warcraft.

Lua has grown in response to the demands of game programmers. It now posesses an incremental garbage collector to avoid long pauses. Coroutines provide programmers with co-operative multitasking. Lua’s virtual machine is register-based, making it one of the fastest interpreted languages available.

Due to a rather strict adherence to ANSI C, Lua doesn’t run multi-core out of the box. But Lua can take advantage of your OS-dependent threading code, having a fully reentrant API. Lua states isolate each thread, so scripters don’t need to concern themselves with the typical locking complexities of concurrency. Perhaps a future version of Lua 5.2 could see an optional library with OS-specific threading support.

If you want to dig deep into Lua’s history, take a look at The Evolution of Lua paper (26 pages), as was presented at the 2007 History of Programming Languages (HOPL) Conference. I will conclude with a quote from the paper:

“It is much easier to add features later than to remove them. This development process has been essential to keep the language simple, and simplicity is our most important asset. Most other qualities of Lua — speed, small size, portability — derive from its simplicity.”

And contrast it with a quote from Yukihiro “Matz” Matsumoto from Beautiful Code:

“When simpler tools are used to solve a complex problem, complexity is merely shifted to the programmer, which is really putting the cart before the horse.”

While there is validity to his statement, the beauty of Lua is how it manages to be a quite powerful language, despite its simplicity.

Adobe Lightroom won out for me. It provided enough of the Photoshop functionality (curves, etc.) that I often wouldn’t need to launch Elements. Just as important, the reviews proclaimed Lightroom to be more responsive and less of a resource hog. So what does all this have to do with Lua?

Adobe Lightroom is programmed with C/C++/Objective-C and… 40% Lua. An interpreted scripting language making up 40% of the code, and it feels more responsive and uses less resources?! That deserves a second look…

Adobe Lightroom puts Lua’s coroutines to good use, keeping the user interface responsive while images visually rotate or exports process. Coroutines are a form of “cooperative multitasking,” where the programmer controls task-switching rather then being preempted as with “real” threads.

I’ve taken a great interest in the Lua language. The more I learn, the more intrigued I am. This blog exists in order to share these findings with you. Lua is Portuguese for moon, and “lua nova” means new moon.

Welcome to Lua, welcome to the moon…