Building Shepherd - Preview the new rec system + ads
Building Shepherd - Preview the new rec system + ads
Discovering a new book should be a magical experience where the search is part of the fun. That is what we are creating. We give readers fun ways to find amazing books.
Summary: July 25th to August 5th 2022
The last two weeks were rewarding!
We finished July with 128,000 visitors. That is a 24% increase over June. It is satisfying to see the work I’ve been doing over the last four months start to accelerate growth.
We also had an excellent mention in Jane Friedman’s Hot Sheet newsletter (which I highly recommend). I’ve been a subscriber since the start of Shepherd as I try to learn more about the publishing industry. I’ve included that below! 🤗
The grunt work that powers topics...
I went through 1,800 Wikipedia topics to decide if they belong on Shepherd. Our machine learning system assigns topics to books, but I ultimately have to determine whether the topic is interesting or useful. I’ve cleared every topic that is assigned five books or more. I review this weekly to keep it updated. I also banned thousands of topics that don’t belong in the system. There are a lot of false flags around author names, book names, songs, bands, tv shows, and other weird stuff.
I went through 1,400 Library of Congress topics to map those to our Wikipedia topics. That was time-consuming, as sometimes you get a fragment like “Presidency of,” and you have to decide based on research where it should be assigned. Or you get a vague concept like “loss,” and you have to determine where the falls. I’ve cleared every topic that is assigned five books or more. I review this every week to keep it clean. I am working to clear everything assigned four books or more (450 LOC topics). There are 12,000 assigned at least one book, but I don’t think I’ll be able to do that. I’ll probably stop at three books.
The most time-consuming work is reviewing possible bookshelves and publishing those. I went through 800 pending bookshelves, published 452, deleted some, and have ~250 waiting for a feature or further review.
For example, I had a pending bookshelf around “chemistry,” but the problem is it was picking up the word chemistry around love/relationships. So that will have to wait until we have genre data to help fill it out.
What is the result of this work?
The recommendation system quality improves.
The quality of the bookshelves has improved.
The search feature improves.
I create more ways for people to find books they are interested in.
Preview the new recommendation system!!!
Just add “?new-recommendations” to the end of any book list to see the new recommendation system we are implementing. The developer is about 80% done, and we will roll this out in a few weeks once we finish some more UX tweaks.
Here is an example with Ian Dunbar's list on the best books for teaching dog tricks (scroll to the bottom of the page).
The new system is based on our topic engine and much more accurate. It doesn’t consider genre or age focus, but we will add that toward Winter.
We suggest five related book lists, three related bookshelves, and three books the reader might want to check out if they are still searching after the recommendations Ian made.
Adding ads to Shepherd.com
I decided it was time to add ads to Shepherd.com.
I dislike ads, and I would like to keep the site ad-free. But, to cover our costs, we need them. Over time I hope to make the ads on the website purely book-focused. You will start to see ads this weekend.
If you are wondering about our business model, I have a lengthy write-up here. I aim to be profitable by the end of 2023 and this is a big part of what will get us there.
What is going on outside of Shepherd?
It is nice to be home in Portugal. I have been doing a lot of bike rides as I prepare for a 1,000km bike tour later this year. Beyond that, just a lot of good time with my wife and son.
What have I been reading?
The Buried: An Archaeology of the Egyptian Revolution (by Peter Hessler) - I spent a year living in Zamalek Cairo, and reading Peter Hessler's account is a pleasure. I've read every single one of his books.
