Nim Meta Programming

published: [nandalism home] (dark light)

Nim - Meta Programming

The Nim programming language is a statically typed, compiled language with python-like syntax and C++-like templates, function and operator overloading.

It also has one more very interesting feature, meta-programming, which is common in Lisp but not in compiled languages.

What is Meta Programming?

Imagine a world where you could add missing features to your language, by writing compiler plugins. If you want some language feature which is not present then you write a compiler plugin to convert that new syntax into code in the current language, which implements the new feature.

Sounds difficult? Well, now imagine it were really easy. That is Lisp meta-programming. Nim meta-programming is not quite as easy but still great.

Many current language improvements are actually just simple syntactic sugar which you could have implemented yourself had you had access to meta-programming in that language. Instead you must wait for the next language version.

"But wait!", I hear you cry, "doesn't that mean the language is different for everyone and things get really confusing?". Well, it doesn't work out that way in practice.

Lisp programmers will rarely write a program without writing at least one "compiler plugin". It's just so useful, easy and commonplace.

Meta-programming produced code is natively compiled just as any other Nim code is. The meta-programming transform phase happens before the normal compile phase and the compiler just sees the Nim code produced. The later compiler stage is unaware of whether the code it sees was manually written or meta-programming produced code.

Meta-programming capable languages must have the feature that they can run their own language at compile time. This is quite a big feature to ask for, and most compilers don't have it. However, Nim uses a VM at compile time to run Nim code, thus making meta-programming possible.

Now let's look at a practical example of meta-programming.

Web Server Templating

Lots of web frameworks provide a templating language, allowing you to mix html and code of some sort to generate a page dynamically at server-render-time. Invariably, these templates are stored in separate files and use a special purpose language designed for html templating.

They are also, and this is important, effectively interpreted, string-based, programming languages, which are interpreted and run each time that templated page is rendered.

Meta programming allows us to write code which is re-written at compile time into other code. If we could express our templates using meta-programming, we would have a compiled version of our template instead of a slow, run-time interpreter.

Here is an example of how this might look:

ht.divv({style:"display:flex;flex-direction:row;justify-content:space-between;margin:0;padding:0;"}):
  ht.ul({id:"terminal"})
  ht.divv():
    ht.ul({class:"userlist", id:"active_users"})
    ht.ul({class:"userlist", id:"passive_users"}):
      for u in members:
        ht.li({style:"display:flex;flex-direction:row;adjust-items:center;"}):
          ht.img({src:fmt"/avatar?i={u.avatar_id}"})
          ht.span(txt(u.name))

Note the for loop and the interpolation of values (e.g. fmt"..." and otherwise) are just normal Nim code constructs. We don't have to re-invent these basic forms, whereas a normal template language does, in its special, interpreted, programming language.

There is one downside to pre-compiling the template and that is that we cannot make changes at server runtime. This is a small price to pay for not having to repeat the interpretation of the template language on each render. (Also, Lisp hot-code-loading experts will be laughing at this restriction, which they don't have).

The Nim Macro

Finally, we will explore the Nim macro which implements the template language, above. How does this magic work?

Nim has a standard html templater (also macro based). It generates the entire template, in memory, as a string. I think a more efficient way to do things is to write to a stream. We can then stream the template over the web connection without holding the entire thing in memory. We also have the option, using a StringStream, to render the template as a string if we wish. The best of both worlds. So, my templater writes to a stream, called 'w', which must exist in the scope of the template. (I will discuss the trade offs later).

I have designed the system so that all html tags take a "map" (Nim TableConstructor) of attribute:value as the first (optional) argument, followed by the child nodes. Nim has a nice syntax feature where sub-blocks (with colon) are passed as a final argument to a function/form/macro. This means I can have simple e.g. <h1> elements with just text or more complex ones with sub elements, using the same macro definition.

Since the format of all nodes is the same (except html/txt, the base text renderers) I can use a single function generictag to implement all the macros. This makes is trivial to add new element types. The parameterization so far, relates to cosmetic newline output and whether shortcut close forms are allowed for a particular html element.

macro txt*(content: string) = quote do:
  w.write_escaped_html(`content`)

macro html*(content: string) = quote do:
  w.write_safe_html(`content`)

macro divv*(e: varargs[untyped]) =
  generictag(e, "div", newlined=2, separate_close=true)

macro pre*(e: varargs[untyped]) =
  generictag(e, "pre", newlined=2, separate_close=true)

macro p*(e: varargs[untyped]) =
  generictag(e, "p", newlined=2, separate_close=true)

macro h4*(e: varargs[untyped]) =
  generictag(e, "h3", newlined=1)

macro hr*(e: varargs[untyped]) =
  generictag(e, "hr", newlined=1)

macro li*(e: varargs[untyped]) =
  generictag(e, "li", newlined=1, separate_close=true)

macro ul*(e: varargs[untyped]) =
  generictag(e, "ul", newlined=2, separate_close=true)

macro a*(e: varargs[untyped]) =
  generictag(e, "a")

... and many more

The '*' after the macro names is just Nim's way of marking a module function/macro exported for use outside the module.

The Code Produced

Ignoring generictag for the moment (where all the magic happens), let's look at what happens when we use this templater.

This is what the template example above would produce during compilation. Note that the developer never sees this code, I have captured it using the [expandMacros] debug tool in Nim's [import std/macros].

The template is expanded to a sequence of write() calls, which are compiled as usual Nim code into the function which produces the template output when called with the relevant data parameters (lists of users etc.).

write(w, "<div")
write_escaped_attr(w, "style", "display:flex;flex-direction:row;justify-content:space-between;margin:0;padding:0;")
write(w, ">\n")
write(w, "<ul")
write_escaped_attr(w, "id", "terminal")
write(w, ">\n")
write(w, "</ul>\n")
write(w, "<div>\n")
write(w, "<ul")
write_escaped_attr(w, "class", "userlist")
write_escaped_attr(w, "id", "active_users")
write(w, ">\n")
write(w, "</ul>\n")
write(w, "<ul")
write_escaped_attr(w, "class", "userlist")
write_escaped_attr(w, "id", "passive_users")
write(w, ">\n")
for u in items(members):
  write(w, "<li")
  write_escaped_attr(w, "style", "display:flex;flex-direction:row;adjust-items:center;")
  write(w, ">")
  write(w, "<img")
  write_escaped_attr(w, "src"):
    var fmtRes_436207675 = newStringOfCap(33)
    add(fmtRes_436207675, "/avatar?i=")
    formatValue(fmtRes_436207675, u.avatar_id, "")
    fmtRes_436207675
  write(w, "/>")
  write(w, "<span>")
  write_escaped_html(w, u.name)
  write(w, "</span>")
  write(w, "</li>\n")
write(w, "</ul>\n")
write(w, "</div>\n")
write(w, "</div>\n")

Here we also get to see how [fmt] strings are expanded (fmt is also implemented as a standard macro), and how for loops are expanded, the items() call is added.

(Yes, this would only be efficient if the write stream were buffered, and we assume it is here.)

There is also a open question as to whether a write sequence/streaming is better than producing the output in memory and then dumping it to the stream. I may write a follow up post comparing the two styles. Building a string also requires calling a sequence of append functions at its lowest level, and that is equivalent to a sequence of write calls. So the write() style is unlikely to be slower and has the advantage of avoiding the creation of a large in-memory string.

The Magical generictag function

And yes, generictag is a function, not a macro. This can be confusing at first. Remember that a macro is just a function you get to inject into the compiler, which the compiler will run for you while parsing the source code (as soon as it sees your macro being used in the code).

This function is required to take code (in pre-parsed abstract syntax tree form, not as a string) and produce more code (again in AST form).

The macros I have defined above are letting the compiler know to inject them, what they do inside is normal Nim code (running in a VM at compile time). So I can call helper functions like generictag. As long as they are taking AST (the NimNode type) and producing output AST, they are doing the work of the macro.

proc generictag(args: NimNode, tag: string, newlined=0, separate_close=false): NimNode =
  proc common_write(code: NimNode, tag: string, attrs: NimNode, newlined: int, tag_close: string) =
    if isNil(attrs):
      let element = newStrLitNode(fmt"""<{tag}{tag_close}{(if 2==newlined: "\n" else: "")}""")
      code.add quote do:
        write(w, `element`)
    else:
      let pre = newStrLitNode("<" & tag)
      let suff = newStrLitNode(tag_close & (if 2==newlined: "\n" else: ""))
      code.add quote do:
        write(w, `pre`)
      code.add attrs_to_writes(attrs)
      code.add quote do:
        write(w, `suff`)

  result = newStmtList()
  var attrs: NimNode
  if args.len >= 1:
    let first=args[0]
    if nnkTableConstr == first.kind():
      attrs = first
      args.del()
  let has_children = args.len > 0
  if has_children or separate_close:
    common_write(result, tag, attrs, newlined, tag_close=">")
    for arg in args:
      result.add arg
    let element_close = newStrLitNode(fmt"""</{tag}>{(if newlined>0: "\n" else: "")}""")
    result.add quote do:
      write(w, `element_close`)
  else:
    common_write(result, tag, attrs, newlined, tag_close="/>")

The helper function which writes html attribute lists.

proc attrs_to_writes(attrs: NimNode): NimNode =
  result = newStmtList()
  for pair in attrs:
    assert nnkExprColonExpr == pair.kind, fmt"expected key:value for attr, found {$pair.kind}"
    let k = pair[0]
    let v = pair[1]
    assert k.kind in {nnkStrLit, nnkIdent}, fmt"expected ident key for attr, found {$k.kind}"
    let sk = newStrLitNode(k.strVal)
    result.add quote do:
      write_escaped_attr(w, `sk`, `v`)

Briefly, the generictag function is organized as follows:

Notes

Some may say that macros capturing a write stream with a fixed name 'w' is bad practice. However, it means we don't need an overarching macro to create the stream. Since these templates will be used in a function, we can pass 'w: Stream' as a parameter and the leakage of 'w' is very localized.

There is a low-level NimNode api in [std/macros] and it can be used to generate all the code. Indeed, the initial version of generictag did just that. There was no used of [quote do]. I later re-wrote using [quote do] since I think it makes the macro code more readable. However, [quote do] is not necessary, and not magic, we are simply creating a NimNode tree and returning it. Knowing the low-level NimNode api is enough to do everything you want; which is why I learned that first.

The macro is named 'divv' not 'div' since 'div' is a keyword in Nim language. Nim allows keywords to be used, quoted like `div`, but I preferred 'divv'.

site built using mf technology