Lydia

Background

There are so many static website builders. Each with catchy names like Fimmy, Dogg or Splendish (I’m making these up). Here are some of them listed on colorful web cards.

The concept of a static website builder is promising. You keep working on your writing or programming projects in the usual way, in some folder hierarchy on your computer. Don’t worry about HTML or anything, just use markdown text files for example. And then the site builder transforms whatever is in your project folders into a web of HTML pages that you can publish on a webserver, asset files and graphics included.

Most of these tools let you chose and contribute nicely designed CSS themes for your website. They also provide a way for you to create page templates, into which your writing gets inserted, so you don’t have to type that much and so that your website has a uniform, polished look.

I despondedly icked out of my “try-it-out” installation of Hugo, to name a particularly popular site building tool. I know from looking into it in some detail, that Hugo is a robust and very useful tool – it just isn’t for me in my current, simplistic mindset.

Three things happened: First, I got the impression that in chosing a theme, I would have to provide all the content that the theme assumed I had.

Then I realized that installing a very basic theme (so I could build my own from it) had littered my site structure with Lorem placeholder content, with no obvious way to know where it was all stored in order to remove it.

And thirdly, I opened one of the template files, only to find a wall of impenetrable line noise that resisted my eyes parsing or even scanning it.

So since Hugo was out, and that is the name of my grandpa, I sat there in my Wezterm in the dumps for a while thinking about what to do. I decided to write my own tiny, hypocritally hubristic builder named after my grandma, Lydia.

What I learned

This might be the wrong order of things, but let me start-out by writing about my personal findings.

For one thing, I found a sweet spot for dealing with HTML by taking the role of a “director” and my shell to compose it for me. It may be a round-about way, but in doing so, I don’t have to constantly switch gears in my mind, since I am only ever doing one thing: composing text, either literally, or by obtaining it from a script.

The way I do this in Lydia is by running a top-level build-script that execs (thus sharing environment variables) all kinds of helper scripts using the source command. I was surprised how tersely you can put things together. Using only here-documents and command substitution allows you to work with a library of snippets and “plugins” that you can curate while you go along.

# FILE build-script.sh

export PROJECT_NAME="sloppy"
lib/sh/date-header.sh > website/index.html

# FILE date-header.sh
    
# Use here-documents to inline literal HTML
# with command substitution to capture the output
# of any shell command
cat << EOF
/* My great $(echo "$PROJECT_NAME")HTML page */
/* published on $(date) */
EOF

Running build-script.sh then creates a stub for your index.html:

/* My great HTML page */
/* published on Wed Jul 29 08:56:57 CEST 2026 */

Using this approach, the distance towards converting your markdown project file to an HTML fragment using Pandoc, say, and injecting it into your page skeleton with sed -i shrinks to about two lines of code.

What I find appealing about working in the shell, is the fluidity of putting things together.

Even with Perl, which was specifically designed for this, you do have a small “setup fee” when working with command line arguments, environment vars and file descriptors. It’s minimal, but as with any other scripting language, you quickly get sucked into just using it instead of keeping it flat. Eclectic tool-chain vs. swiss-army knife.

Swiss-army knives and their confusing Tower of Babel. One-trick tools don’t compete or vie for attention, by definition they are good at doing their job, nothing more, so you can accumulate any number of them and be good. Having two or more swiss-army knives is like having two or more tooth brushes.

Awk for querying text files

I you are reading this and haven’t heard of or never used Awk, go and experiment some with it before reading on, honestly. Awk is a small language that’s been with Unix since almost the beginning. It is a Posix standard tool, so it should already be available on your system.

In Lydia, I am maintaining what is essentially a csv file, but with a pipe symbol as the field separator. This file (pages.db) serves as my website page database, with each line of the file representing one of the pages.

Using this “database”, I can generate page lists – for example by category – using Awk. I tell it to match all records/lines where the CATEGORY field matches the category I am interested in, and build up an an HTML list structure with links to those pages, using the other fields of each matching record.

The database text file is generated during the build process. Each project folder has a file called “metadata” with project-specific information that gets included into the pages.db database.

A method to this madness - JAM

An acronym I’ve come across in the sphere of static site builders is JAM, which stands for Javascript, API and Markup. The idea is that instead of a backend like Express, your static pages can get by just fine by injecting text converted from markdown files, and adding some Javascript that communicates with a remote API.

And Lydia does use two API endpoints from my Express website: /permalink and /tags, although it does not use JavaScript for this. Since both Lydia and Express are on the same machine, Express can access the database text file I mentioned. Each row in this file, as I said, contains data fields for a particular page.

One of these fields is a unique string, a hex number called PERMAKEY. Although in theory I could move around my project folders in the Lydia hierarchy or change their names, the PERMAKEY will stay the same, since it’s encoded in the metadata file inside the project folder.

Now, when I hit my Express server endpoint /permalink/<PERMAKEY>, the backend redirects the client browser to the appropriate page of the Lydia static website. It implements this by reading the pages.db text file and building a hash, mapping permalinks to URLs.

Similarly, the /tags endpoint uses a hash map to map tag strings to URLs, allowing me to render an HTML page with a list of Lydia pages that mention this tag in their metadata file. Both Lydia and Express are behind the same NGINX reverse proxy server, and so these API calls are not obvious to the user.

Pandoc

I am using the pandoc command line tool to convert the markdown files in my project folders to HTML. Pandoc is mature, easy to use and rich in helpful features. You can write Lua filters, and I wrote one that automatically rewrites my markdown image references to responsive srcset picture elements when exporting HTML.

I hadn’t looked at Pandoc for a decade or so, and the markdown extensions it provides, for example regarding syntax highlighting, tables, Latex etc are compelling.

Status

For the moment, I’m still testing out if Lydia is a viable tool. Most importantly for me, it needs to be scalable and get out of the way. I’ll write some more about it in a year or so.

Currently, it has the following features.

• Keep your landing pages like index, about, etc. in folders:

    landings/
    └── <pagename>/
        └── <subpage>/
            └── PROJECT

  Each landing page can own subpages. That means that although the subpages don’t appear in the pages.db database file, they are generated as physical HTML pages in the website which the landing page can reference. This helps organising, since I can slice and dice my landing pages the way I like and keep them together in the same parent folder.

• File recurring projects into:

    series/
    └── <name>/
        └── <part>/
            └── <sortcode>_<projectname>/
                └── PROJECT

  Now, part will most often be the running year (YYYY), but could be any ascending series. For each PROJECT folder you add, an additional landing page is automatically created which you then need to “link in” manually into your home page by placing a link. Lydia can’t really know where you would want the link on your page.

  Then on the landing page for the series, you will want to place a list of all the posts in your series by capturing the HTML output from an Awk script, provided in the lib/sh/... component library I mentioned in the opening paragraph.

  The script iterates over all the post records in ´pages.db´ that match your series name and generates an unordered list of links using the remaining fields in each matching record. The script also handles outputting the list in chunks per year of publication with appropriate title elements. Awk is fine for this as it does have hash maps, but the nice thing is that it acts as a tool, not an army knife, so you could replace the script with Python with zero fuss.

• Put projects that belong to the same “section” into folders:

    section/
    └── <name>/
        └── <sortcode>_<projectname>/
            └── PROJECT

• CSS “bundling” on steroids weed

  There is a folder called automerge/css. Put your CSS files into that folder, prefixed by <number>_. The build tool will concatenate these in numeric order to obtain the full CSS file referenced by the HTML templates. I’ve added this for purely mnemonic reasons, since I can now have a text file for each HTML component or feature that I am using, allowing me to find things more quickly.

• Templating based on symlinks

  The PROJECT folders mentioned above have an intro.sh file, a page.md markdown file, and an outro.sh file. Since I am using Pandoc to generate the HTML output from the markdown file, I have good control over the HTML I get from the markdown. It’s relatively easy to customize the output, even from within the markdown file itself.

  Now, intro.sh and outro.sh are there to generate project-specific HTML content using the lib/sh components I mentioned. You may not even need this, so by default, both files are just symbolic links to default scripts that are located in a parent folder. Then if you do need to customise one of those, just replace its symlink by a copy of the default file and modify it.

Testing

I’ve been using Lydia to generate a static website for my personal website (Swirly Tidings). I serve these pages using a location block in my NGINX server.