OfflineShelf: Choosing Which Wikipedia Goes Offline

There are schools with a room full of computers and no reliable internet. There are clinics where the connection comes and goes with the weather, and libraries where it costs more per month than the books. For places like these, a free app called Kiwix can carry Wikipedia offline: you copy one file onto a laptop, a phone or a small server, and the articles open with no connection at all.

Copying is the easy part. The whole of English Wikipedia with pictures is a download of roughly a hundred gigabytes. That won't fit on an old phone, and most of it won't matter to a nursing course or a secondary-school science class. Someone has to choose. And once they've chosen, the next questions arrive: can we make the same list again next term? What changed since last time? Can we add the biology articles but leave out the biographies?

OfflineShelf is my answer to that choosing step. You describe a list, it builds it from Wikipedia, keeps every version, rebuilds it every month if you ask, and exports it in a form you can carry. I built it at the end of October 2026, across 26 pull requests, and every feature works end to end. It isn't public yet. The launch at offlineshelf.ahmershah.dev is planned for February 2027, after the testing I describe at the end.

This is how it fits together, what fought back, and the two rules I wrote down before anything else.

---

#Two rules before any code

Store the choice, not the content. OfflineShelf keeps article titles and their ID numbers, never the text. Wikipedia already keeps the text, updates it every minute and records who wrote what. If I copied it into my own database, I'd be running a stale mirror with none of that history. The text is only fetched at the very end, when someone asks for an export, and it goes straight into the file.

A failed build never replaces the last good list. Every time a list is built, the result is saved as a new version next to the old ones. If the build fails halfway, because Wikipedia was slow or a category was renamed, the previous version is still there, untouched and still downloadable. A teacher who prepared a USB stick last month should never find that this month's list is half empty because a server hiccupped.

Both rules made the code longer. Both rules are why I'd trust it with somebody else's classroom.

#Four ways to ask Wikipedia for a list

People already describe what they want in different ways, so there are four builders.

A category. Wikipedia already sorts its articles into categories, like "Vaccines" or "Rivers of Pakistan". Categories contain smaller categories, which contain smaller ones, and some of them loop back on themselves. The builder walks down to a depth you choose, from 0 to 5, remembers where it has been so it never goes round in circles, and stops once the list is big enough.

A Wikidata query. Wikidata is Wikipedia's sister project: a database of facts, like "this is a university" and "it's in Pakistan". You write a query in its language, SPARQL, and the builder turns each result into the matching article on your chosen Wikipedia. When a result has no article in that language, the report says so instead of quietly leaving it out.

A pasted list. Sometimes you already have the list: a syllabus, a teacher's reading list. You paste the titles, and the builder cleans them up the way Wikipedia does, follows redirects (so an old or alternative title lands on the article it points to), removes duplicates and reports anything that doesn't exist.

The most-read articles. The top articles by pageviews for a given month, straight from Wikimedia's statistics, with the main page and the special pages filtered out.

The first time I ran all four against the live English Wikipedia, the reports were the most useful part. Three pasted titles came back with two redirects followed, one missing page and one that wasn't an article at all. A Wikidata query for universities in Pakistan returned eight results, of which seven had English articles. The pageviews list for August 2026 had already dropped the main page and the special pages that always crowd the top. None of that was hidden from the person asking.

#Being a good guest on Wikimedia's servers

Wikipedia is run by a non-profit and its servers are paid for by donations. Its API has rules for tools like mine, and I decided early to follow every one of them, because a tool that gets blocked stops working for everybody using it.

Every request goes through one small client. It introduces itself with a name and a contact address, so Wikimedia can reach me if something goes wrong. It sends a setting called maxlag, which means "if your database replicas are falling behind, tell me to wait". When the servers say wait, it waits, and tries again later.

I saw this happen for real. While I was running the example lists in Docker, Wikidata was lagging. The client backed off, the job was put back in the queue, and a few minutes later the list built normally. Nobody had to do anything. That's the boring outcome I was hoping for.

There's also a rule for my own tests: none of them may call the live APIs. They replay real responses I recorded once, and a guard fails any test that tries to reach Wikimedia. CI runs about two hundred backend tests on every change, and none of them puts any load on a volunteer-funded server.

#Lists that combine

Real collections are rarely one simple query. A health worker might want everything about vaccines, minus the pharmaceutical companies. A science teacher might want physics and chemistry together. So lists can be combined: union (in any of the lists), intersection (in all of them) and difference (in the first, in none of the others), from two up to ten lists.

A combination is a list like any other. It has versions, a history and exports. And when one of its source lists gets a new version, the combination rebuilds itself.

Under the hood, every source list is read already sorted by page ID, a little at a time, and merged in a single pass. That sounds like a detail, but it means combining two lists of 100,000 articles uses about as much memory as combining two lists of ten.

#What changed since last month?

Wikipedia changes every day. Articles get merged, renamed, deleted and created. A list that was right in March can be quietly different by June, and the person who carries it into a classroom deserves to know how.

So whenever a new version is ready, OfflineShelf compares it with the last good one and stores what was added and what was removed. In the app, the build history shows those two numbers next to each build, and you can open either list and read the titles. The comparison runs inside the database, so even big lists are never loaded into memory to work it out.

Monthly rebuilds work the same way. A small scheduler wakes up every hour and asks: is there a monthly list that hasn't been built since the 1st? If the server was switched off on the 1st, the next hourly check catches up instead of waiting a whole month.

Also readAIT: Building a University Admissions Portal From Scratch
I Invented 14 People to Break My University Website

#Three ways to carry it

A list is only useful once it leaves the browser, and different places need different things.

• A TSV file: one row per article with its title, ID and revision, which opens in any spreadsheet and feeds other tools, including Kiwix's own. The first lines carry the licence and where the list came from.
• An HTML bundle: a zip with one web page per article, an index and an attribution page. Unzip it onto a USB stick, double-click the index, and it works with no server and no internet. Links between articles on the list open the local copy; links to anything else point back to Wikipedia.
• A ZIM file: the same pages in the single searchable file that the Kiwix apps read on phones, computers and Kiwix hotspots.

Two honest limits. Bundles and ZIM files are capped at 1,000 articles and expire after seven days, to keep the server's disk small; you can make a fresh one from the same version and get the same articles. And images are linked rather than copied for now, so offline pages show text without pictures.

The files inside a bundle are named by page ID, not by title. My first instinct was titles, until I remembered that a title like "Batman: The Animated Series" contains a colon, which Windows refuses in a file name, and that macOS treats "Red Dwarf" (the TV series) and "Red dwarf" (the star) as the same file. Numbers never have either problem.

Every page also ends with where it came from: the article, the exact revision, the licence and a link to its history. Wikipedia's text is shared under a licence called CC BY-SA, which lets anyone reuse it as long as they credit the authors and share it the same way. An offline copy that drops the credit breaks the deal, so every export carries it.

#Big lists, small memory

Most lists will be a few hundred articles, but I wanted to be sure a very large one wouldn't take the server down. So I wrote a test that pushes a list of 100,000 articles through the real build job into the real database, then streams the whole export back, and measures memory the whole way.

The build peaked at about 1.6 MB and the export at about 3.3 MB. Holding all 100,000 articles in memory at once would take about 19 MB, so the test fails if either step ever goes above 8 MB. My first limit was 20 MB, and then I measured the baseline and realised that limit would have let the exact mistake I was guarding against slip through. I tightened it.

#What I got wrong

A restarted worker left lists stuck forever. While running the example lists, I restarted the background worker in the middle of a build. The build never finished, never failed, and sat in "building" forever, which blocked every new build of that list. The same thing would happen after a deploy or if the server ran out of memory. Now a recovery job runs every hour and marks builds whose worker has died as failed, with a readable reason, so the list can be built again. It's careful to leave alone the jobs that are only waiting for a retry.

Combinations failed while their sources were still building. The example set has two lists and their union. When the first source finished, the combination was queued straight away, found the second source still empty, failed, and would have emailed its owner about a problem that was going to fix itself a minute later. You can see that failure in the build history below. Combinations now wait until every source has a good version.

Titles are messier than they look. Wikipedia capitalises the first letter of a title, except on Wiktionary, and except for letters like the German "ß", which turns into two letters when capitalised. Special pages have no page ID, which crashed my first lookup code, and the most-read list is full of them. And on the German Wikipedia, subcategories come back as "Kategorie:", which my code would have turned into "Category:Kategorie:Something". Each of these is now a test.

I had an option to send email without encryption. OfflineShelf emails the owner when a build fails. I'd left a setting that let a self-hoster turn encryption off for their mail server. A code-quality scanner flagged it, and it was right: there's no good reason to keep that door open. Mail now always goes out encrypted.

#By the numbers

• 4 ways to build a list, plus combinations of up to 10 lists
• 3 export formats, every one with attribution
• 8 Wikimedia projects in any language: Wikipedia, Wiktionary, Wikibooks, Wikinews, Wikiquote, Wikisource, Wikiversity and Wikivoyage
• 235 automated tests: 197 on the backend, 38 on the frontend
• 100% branch coverage on the list logic, enforced by CI
• 100,000 articles built and exported in a test, with memory staying under 4 MB
• 26 pull requests, each checked by CI and CodeQL before merging

#What's next

The features are done; what's left is proving them under pressure. Before launch I want end-to-end browser tests of every path, from signing up to downloading a ZIM file, smoke tests on every release, and a load test with 15 to 20 simulated users building, combining and exporting at the same time while I watch the queue, memory and how hard we're leaning on Wikimedia. Then it goes live in February 2027.

After that, the work moves outside the code: the first school or library that actually uses it, and the things they tell me I didn't think of. Pictures in exports are near the top of that list already.

If I started again, I'd write the same two rules on day one. And I'd restart the worker in the middle of a build in the first week, on purpose, instead of finding out by accident.

The source is on GitHub, and I wrote about the idea itself, for people who have never thought about offline Wikipedia, in A library that fits on a USB stick.