Building a custom SQLite search engine for my Hugo site

This website is managed and published by Hugo, which is an outstanding static site generator with incredible speed and flexibility. And, since it generates static websites, having a search tool can be something of a challenge.

I had been using a javascript solution derived from the Book theme. On building the site, it generated an index of al content and saved it as a .json file. The Search page then had some javascript that ran in your browser to download the .json index and perform local queries on it. It worked great, and was super fast. But as the .json index file continued to grow, it became unwieldy. The search script had to download the entire .json index before being able to search, which meant pausing while a 6-7-8MB file was downloaded. And, since the index was rebuilt when the site was published again (to include new content and any modifications), caching wouldn’t help.

I needed to look for another solution, likely server-side. This would break the whole “Hugo generates static sites that can run on a potato” ethos, but it also holds up to the “and the website first and foremost has to be useful” one.

I started looking at various search engine options - LunrJS, Pagefind, Fuse.js and others. But they all felt so overly engineered and big for my tastes. I just wanted a simple php file that could run anywhere, running on my own server.

Since my current typeset theme was built using Claude Code, I figured I’d see what it could do if I asked it to build a custom lightweight search engine.

I want to build a search tool for a Hugo website. It needs to index all content, and run on my FullHost shared webserver. I currently use a solution that creates a single .json index of all content, but that’s grown to be a 7 MB file that gets downloaded in the browser. Not ideal. How can I build a scalable search tool that can be either self-hosted on my FullHost account, or run in the browser?

Claude chewed for a bit and came up with some options. I said:

I like the For Best Performance option. What would that look like? How could I build the index when I build the site? How would I integrate the search interface in the /search page of my site? (note that this project folder doesn’t include my actual Hugo website - that has something like 9,000 pieces of content etc so I wanted to start this exploration with a fresh Hugo site)

It started building a PHP/SQLite solution. Hugo can’t directly generate SQLite .db files, so it had to figure out how to convert the .js search index that was generated from my previous browser-based solution into the .db format that could be used with SQLite on the server. There was some back-and-forth to get around build errors and whatnot, and then it just worked. A fast, effective, customizable search engine with a good browser UI, with searches managed by a single PHP script on the server.

I tested it a bit and had some ideas for improvement - like adding full-text and fuzzy searches. No problem!

🤖 CLAUDE: You’re very welcome! This was a fun project. You now have a seriously powerful search system:

What we built:

• 🚀 FTS5 full-text search with BM25 ranking
• 🔍 Advanced query syntax (OR, phrases, field searches, date filters)
• ✨ Content previews with smart highlighting
• 📚 Built-in help documenta…

…tion? Claude kind of trailed off there. It said “you’re very welcome” because I said thanks in the previous prompt. Not because I think Claude is sentient, but because if the You Are What You Repeatedly Do mantra holds true, not being grateful in conversation with someone (even a simulated someone) who is performing an act of service would be something that becomes ingrained, and I don’t want that.

But it was still having the Hugo theme trigger generation of a .json index that was then converted in a post-processing step into the SQLite .db file. When I removed the now-unused code to load the search interface javascript from my theme, Hugo stopped generating the .json index. So I asked Claude to just build the .json index as part of the build process (as it should have been anyway, but the original Book theme did it as a trigger so that’s how it got inherited through all of the transformation I’d made over the years).

Prompt

we built a custom search engine in my Hugo website - it used to build a search index triggered in baseof.html but I’ve removed that from the html template because it was redundant with our new search engine. Is there another way to build the search index that used to be created by this code in baseof.html ?

<!-- SEARCH FUNCTIONALITY -->
   {{- if eq .Layout "search" -}}
   {{- if default true .Site.Params.BookSearch -}}
     {{- $searchJSFile := printf "%s.search.js" .Language.Lang -}}
     {{- $searchJS := resources.Get "search.js" | resources.ExecuteAsTemplate $searchJSFile . | resources.Minify | resources.Fingerprint -}}
      <script defer src="{{- "flexsearch.min.js" | relURL -}}"></script>
      <script defer src="{{- $searchJS.RelPermalink -}}"></script>
   {{- end -}}
   {{- end -}}

Response

The search index for Hugo is typically generated via a **custom output format** rather than being triggered in a template. You can configure Hugo to build a JSON search index automatically.

Add this to your config.toml (or config.yaml):

yaml
outputs:
  home:
    - HTML
    - RSS
    - JSON

outputFormats:
  JSON:
    baseName: index
    isPlainText: true
    mediaType: application/json
    notAlternative: true

Then create layouts/_default/index.json to define the search index structure:
...

I then asked it to go ahead and make those changes to my theme and config, and hey presto it worked great.

Anyway. I just wanted to document this process. I’m not sure how portable this is, but I might play around with making a version of it that could be kinda dropped-into an existing Hugo site. I definitely don’t have the bandwidth to manage an open source project at the moment though.