7 tools, the same 4,044 audiobooks, every result checked by hand
Each tool was given the same real audiobooks with their original, never-edited tags and asked to find
each book's listing and write its details. Then every result was checked: was it the right book, and did the book end
up with the right title, author, narrator and series? The checking was done with Claude, an AI model, under human
supervision (how).
2026-11-08
4,044 books, 3,622 scored
7 tools
LibraForge build 527247d
At a glance
What each tool left on the 3,622 books with a confirmed listing, best first. Writes is how much of
them it wrote; fully correct and right book are shares of what it wrote.
Fully correctRight book, a field offWrong bookHeld for reviewNothing written
LibraForgepublic catalog + Goodreads
Writes
99%
Fully correct
98.1%
Right book
100.0%
AudioSilo
Writes
97%
Fully correct
82.5%
Right book
100.0%
Audiobookshelf Quick Match
Writes
99%
Fully correct
72.6%
Right book
75.3%
ReadMeABook
Writes
76%
Fully correct
81.4%
Right book
83.0%
beetsaudible plugin
Writes
46%
Fully correct
92.6%
Right book
95.1%
Grimmory
Writes
93%
Fully correct
44.5%
Right book
77.0%
BookOrbit
Writes
92%
Fully correct
36.9%
Right book
85.0%
Fully correctThe right book, and every field it ends up with is right.
Right book, a field offThe right book, but the title, an author, the narrator or the series is wrong or missing. Another recording of the book counts here too.
Wrong bookAnother book's details were written.
Held for reviewThe tool found a candidate but waited for you to confirm it.
All the numbers
Tool
Books
Writes
Fully correct
Other recording
A field off
Wrong book
Held, right
Held, wrong
Nothing written
Run time
Searches at once
LibraForge
3,622
3,57998.8%
3,51096.9%
130.4%
561.5%
00.0%
220.6%
00.0%
210.6%
11 min
5
AudioSilo
3,622
3,52697.3%
2,91080.3%
30.1%
61216.9%
10.0%
661.8%
90.2%
210.6%
30 s local copy2 min 15 s online
6
Audiobookshelf Quick Match
3,622
3,59699.3%
2,61072.1%
270.7%
691.9%
89024.6%
00.0%
00.0%
260.7%
4 h 44 min
1
Grimmory
3,622
3,36492.9%
1,49741.3%
551.5%
1,03928.7%
77321.3%
00.0%
00.0%
2587.1%
4 h 23 min
1
ReadMeABook
3,622
2,74275.7%
2,23161.6%
441.2%
00.0%
46712.9%
00.0%
00.0%
88024.3%
1 h 41 min
1
BookOrbit
3,622
3,34492.3%
1,23534.1%
170.5%
1,59143.9%
50113.8%
00.0%
00.0%
2787.7%
15 h 53 min
1
beets
3,622
1,66546.0%
1,54142.5%
280.8%
140.4%
822.3%
00.0%
00.0%
1,95754.0%
50 min
1 per import (4 run)
Run time is the whole run, start to finish. Searches at once is how many books the tool works on in parallel.
The tools
How each one matches and writes, which also decides what "correct" means for it, with our notes from running it.
LibraForge
public catalog + Goodreads
Matches against
Audible public catalog (no login), with Goodreads when Audible has no good listing
How
Reads the tags, the folder path and the runtime together, runs several searches (the file's ASIN, title and author, keywords) and scores every candidate on title, author, narrator, series, book number and runtime.
Review
Applies confident matches and holds the rest for review, with the reason
Writes
Writes the listing's fields to the files. When the runtime or the narrator says the file is another recording, keeps the file's narrator, ASIN, year and ISBN. A series-only match writes the series alone.
Speed
11 min for 4,044 books, 5 at once (yes, Search workers 1 to 30)
Works best with the Goodreads fallback on, which finds books Audible lists badly or not at all. When the runtime or narrator says a file is another recording of the book, it keeps the file's own narrator instead of writing the listing's. 0 of the 22 books it held for review were the wrong pick.
AudioSilo
Matches against
Its own community database (audiosilo-meta, CC0), not Audible
How
Sends the path, the tags and the runtime together; a file's ASIN or ISBN is a hint it checks, not a match by itself. Applies a match at score 90 or more, clearly ahead of the next candidate, with the runtime close enough; since 2.6 also without an ASIN or ISBN when the author, series position and runtime agree.
Review
Holds everything else for review
Writes
A confident match attaches the ASIN and fills only blank fields, so the file's title, narrator and series stay. Everything else waits in review.
Speed
30 s for 4,044 books with its local copy of the database, 2 min 15 s against its online service; 6 at once (no, a match run checks 6 books at once)
Part of its own metadata project. The whole database can be downloaded locally (mirror mode, a download of about 500 MB, refreshed daily), so matching is fast and never polls their service. Coverage is only what the community has added, and it fills only blank fields, so a file's wrong title or series stays. 9 of the 75 books it held for review were the wrong pick.
Audiobookshelf Quick Match
Matches against
Audible (Audiobookshelf's provider, US store)
How
Quick Match per book: the file's ASIN when it has one, else one search and its first result. No runtime check.
Review
None, applies the first result
Writes
Replaces the book's details with the listing's, in Audiobookshelf's database.
Speed
4 h 44 min for 4,044 books, 1 at once (no, Match All goes one book at a time)
A match is only as good as Audible's first result, so later books of a series often get another book's listing.
Grimmory
Matches against
Audible (switched on and put first), then Goodreads, then Google
How
Per provider, one keyword search and its first result; the file's ASIN when its tags carry one (it reads the iTunes copies of the tags, which carry none here).
Review
None
Writes
Fills or replaces each field from the provider's top result, field by field.
Speed
4 h 23 min for 4,044 books, 1 at once (no, one book at a time, providers in turn)
Audible is off by default. No runtime check, so later books in a series often get Book 1.
ReadMeABook
Matches against
Audible
How
Bulk Import scan: one search on the album, album artist and composer tags, first result.
Review
None
Writes
Picks a listing per book and writes nothing, to files or a library. Judged only on whether the pick is the book.
Speed
1 h 41 min for 4,044 books, 1 at once (no, 1.5 s pause between searches)
Groups folders by their tags, so a series sharing one album tag is folded into a single book. Reads no series.
BookOrbit
Matches against
Its default providers
How
Library bulk metadata fetch.
Review
None
Writes
Fetches the listing into its own records (title, authors, narrators, series), not into the files.
Speed
15 h 53 min for 4,044 books, 1 at once (no, 2 to 5 s pause between books)
Slow by design: 2 to 5 seconds between books. Titles often differ from the book's own.
beets
audible plugin
Matches against
Audible (beets-audible plugin)
How
beet import -q with the plugin's recommended settings, scoring candidates on track length.
Review
Quiet mode skips anything that isn't a strong match
Writes
Writes the listing's fields to the files when it accepts the match; skips the rest.
Speed
50 min for 4,044 books, 1 per import (4 run) at once (only by running several imports)
Cautious: most of what it writes is the right book, but it writes under half the library. It can't read a file's ASIN from the iTunes tags.
Why AudioSilo matched every book in 30 s
The other tools ask a web service for every book: each search is a round trip to Audible or Goodreads, often
several per book, and most pause between books so they aren't blocked. AudioSilo's mirror mode downloads its whole
community database once (about 320,000 recordings in one file), so matching is a query on the same disk, with no
network and nothing to wait for. The trade-off is coverage: it can only find what the community has added.
Run against its online service instead, the same 4,044 books took 2 min 15 s: every book's lookup then makes the round trip to the service.
Where LibraForge went wrong
Of the 3,579 books LibraForge wrote, 3,510 came out fully correct and 0 were the wrong book. The rest were the right book with something off:
19The wrong book number
16A title that isn't the listing's
13Another recording's details written
11No narrator
6A narrator the listing doesn't credit
3An author the listing doesn't credit
2No book number
1A series made from the title
1A series the listing doesn't name
1An author missing
1No series
How it was tested
4,044 audiobooks in their original state, before any matching: the tags LibraForge saved before it first
wrote anything to a library's books, or another library's files as they were. Each book was matched again from those
original tags, its folder path and its runtime, with no login, against the live catalog.
Every tool got the same books, as silent files carrying the original tags at the books' real folder paths (and,
for tools that check runtime, as long as the real book), and each was run the way its own documentation suggests.
The right answers
A book's right answer is the listing confirmed as that book, as Audible's catalog itself records it, read by its
ASIN: its title, every author, every narrator, every series it is listed in with its number there, and its runtime.
A listing counts as confirmed when it was applied to the book in a library and reviewed there, or, for
675 books from another library, confirmed by hand from the listings the tools picked, against the file name,
runtime and narrator. A book listed in several series is answered by any of them. 422 books have no
confirmed listing (not on Audible, or no listing of the file's own recording): every tool's result is shown for them,
but they are not scored.
How results were judged
Checked with Claude, under human supervision. The book-by-book review was done with
Claude, an AI model, following one written set of rules for every tool and every book. A person supervised the
work and checked random books by hand along the way, but did not personally review every one of the
4,044 books. The second audit below measures how often the review got a verdict wrong.
A tool that chose the confirmed listing's ASIN chose the right book. Any other choice was judged by hand, once per
book and listing: another listing of the same recording (the right book), another recording, or the wrong book. What
the tool then left on the book was rebuilt from its own way of writing and checked field by field against the
listing's record, by one script for every tool: a field is off when the book ends up with something the listing
doesn't say (written, or kept from the file) or without something it does. Two tools that leave a book the same way
always get the same verdict.
To check the audit, 100 random books were audited a second time:
20 of 700 verdicts (2.9%)
were wrong, and every kind of error found was corrected for all books. A fresh 100 books audited
after that had 6 wrong (0.9%), also corrected.
Books Audible's search couldn't show
Audible's search only returns a book where Audible sells it, while looking a book up by its ASIN works everywhere.
Some listings are sold in only some of Audible's markets (only the UK and Commonwealth stores, only Australia and New
Zealand, only the US and Canada), so a search made outside those markets doesn't return them. Checked against Audible's
own sales-rights data, 122 of the 3,622 books with a confirmed listing were hidden from
search where these runs were made (3.0% of all books). 80 of those
files carry their own ASIN, which a tool can look up directly.
That is the margin of error this adds: run where those books are sold, a tool that searches Audible could do
better on at most these 122 books, 3.0% of the library. On them,
LibraForge got 119 right, AudioSilo 114 (it reads its own database, so
search didn't hide anything from it), Audiobookshelf 95, BookOrbit 108,
Grimmory 48, ReadMeABook 43 and beets 18.
Setup per tool
LibraForge (build 527247d, run 2026-11-04): Metadata Forge's matcher over the files, Audible public catalog with the Goodreads
fallback on, 5 search workers.
AudioSilo (2.6.0, run 2026-11-08): "Match automatically" over the library, in mirror mode (a local copy of the community
database), and timed once more against its online service. Confident matches count as written, "to review" as held.
Audiobookshelf Quick Match (2.35.1, run 2026-10-29): Quick Match per book with the Audible US provider, which takes the
first Audible result, or the file's ASIN when it has one. One book at a time, as Match All does.
BookOrbit (image of 2026-10-20, run 2026-11-02): a library bulk metadata fetch with its default providers.
beets (2.14, beets-audible, run 2026-10-28): beet import -q with the plugin's recommended settings, which applies strong
matches and skips the rest. It was given each book's real runtime, since it scores on track length.
Grimmory (3.5.0, run 2026-10-29): a library metadata refresh with Audible enabled and first for every field
(Goodreads, then Google behind it), applied without review.
ReadMeABook (1.2.12, run 2026-10-29): the Bulk Import scan, which picks an Audible match per book. It groups folders by
their tags, so books it merged into another folder count as nothing written. Its scan returns no series.
Every book
Each book's original tags, its right answer and what every tool left on it, field by field, with
each tool's verdict. Filter by any tool's verdict, or combine two: LibraForge "Wrong book" with another
tool "Fully correct" shows where that tool did better.
Download every book as JSON (the tags, the right answer and every
tool's verdict, pick and what it left on the book) · Download the mock files (each book as one second of silence carrying its original tags, at its folder path).
and
Loading…
Help improve this comparison
Every tool is scored the same way, on the same books.