Add SQLite library with API, library desktop apps and MobyGames scraper

README.md

@@ -23,10 +23,11 @@ The site recreates my [blomarchy](https://git.sirblob.co/SirBlob/blomarchy) setu
 - **btop** - a system monitor window with my real GitHub commit graph, repos as processes and some very scientific CPU and caffeine readings.
 - **Image viewer** - click any image to open it in an imv window with zoom, pan, rotate, pinch and swipe.
 - **Kitty terminals** - a real command line with history and tab completion. Try `fastfetch`, `whoami` or `cowsay`.
-- **Desktop icons** - grouped into pages, terminal scripts and socials (GitHub, LinkedIn, Devpost, Discord).
+- **Desktop icons** - grouped into pages, terminal scripts, apps (tcg, movies, shows, books), games and socials (GitHub, LinkedIn, Devpost, Discord).
 - **13 themes** - Cerberus, Flats, Tokyo Night, Night Cat, Catppuccin, Arch Linux, Gruvbox, Nord, Kanagawa, Wintry, Rose, Catppuccin Latte and Rose Pine Dawn, each with its own wallpaper.
 - **Portfolio** - about, portfolio, projects (open source, packages, hackathons, research, organizations), blog with RSS and links with QR codes.
-- **Live data** - GitHub stars and forks, npm downloads, AniList anime stats and OMDb posters for shows and movies.
+- **Library apps** - movies, shows and books open as their own windows with status filters, search, sorting, details and a monthly watch and read history. Entries in the same series or collection are grouped together.
+- **Live data** - GitHub stars and forks, npm downloads, AniList anime stats and Last.fm listening stats collected into SQLite.
 - **Mobile friendly** - on small screens windows go fullscreen with tabs to switch between them.
 
 ## Keybinds
@@ -69,6 +70,7 @@ Alt stands in for Super, since browsers can't use it.
 | `theme [name]` | list or switch themes |
 | `cowsay <text>` | moo |
 | `btop` | system monitor with my GitHub stats |
+| `movies` / `shows` / `books` | open the library apps |
 | `pacman -S <pkg>` | install things (not really) |
 
 ## Running
@@ -87,7 +89,7 @@ The music player needs `ffmpeg` installed on the server.
 docker compose up -d --build
 ```
 
-The image builds the site with Bun and includes `ffmpeg`. It reads `.env` and serves on port 3000; set `HOST_PORT` to publish it on a different port.
+The image builds the site with Bun and includes `ffmpeg`. It reads `.env` and serves on port 5000; set `HOST_PORT` to publish it on a different port. Shows, movies and books live in the `library-data` volume and Last.fm data in the `lastfm-data` volume.
 
 ## Configuration
 
@@ -96,11 +98,29 @@ Copy `.env.example` to `.env` and fill in what you need. Every value is optional
 | Variable | Used for |
 | --- | --- |
 | `DISCORD_TOKEN`, `DISCORD_USER_ID`, `DISCORD_GUILD_ID` | Live Discord status in the bar |
-| `OMDB_API_KEY` | Posters and details for shows and movies |
+| `OMDB_API_KEY` | Looking up shows and movies in the content editor |
+| `LASTFM_API_KEY` | Collecting Last.fm scrobbles for the listening stats |
+| `LIBRARY_API_KEY` | Secret the content editor sends to change shows, movies and books. Set the same value on the server and locally |
+| `LIBRARY_API_URL` | Site the content editor talks to, `http://localhost:5000` by default |
+| `LIBRARY_DATA_DIR` | Folder for `library.db`, `data/library` by default |
+| `LASTFM_DATA_DIR` | Folder for `lastfm.db`, `data/lastfm` by default |
 
 ## Content
 
-Personal content and config live in `content/`. Data the tools fetch or generate from it (scraped book details and OMDb details) lives in `data/`, and downloaded images go in `static/`. Both are committed so the site never has to call Goodreads or OMDb for saved entries. `data/lastfm` is the exception: the server fills it at runtime and it stays out of git.
+Personal content and config live in `content/`. Shows, movies and books are the exception: they live in a SQLite database at `data/library/library.db` (set `LIBRARY_DATA_DIR` to move it) that the server owns, so adding one never needs a commit. Last.fm scrobbles and album covers are collected at runtime into `data/lastfm/lastfm.db` (set `LASTFM_DATA_DIR` to move it). On first start an older `scrobbles.jsonl`, `covers.json`, `track-covers.json` and `images/` in that folder are moved into the database, and those files can be deleted afterwards. Both databases stay out of git.
+
+### Library API
+
+| Request | Does |
+| --- | --- |
+| `GET /api/library` | Every show, movie and book, public |
+| `PUT /api/library/media/<imdbId>` | Add or change a show or movie |
+| `DELETE /api/library/media/<imdbId>` | Remove a show or movie |
+| `PUT /api/library/books/<goodreadsId>` | Add or change a book |
+| `DELETE /api/library/books/<goodreadsId>` | Remove a book |
+| `GET /api/library/images/<id>` | A saved poster or cover |
+
+Changes need `Authorization: Bearer <LIBRARY_API_KEY>`. A `PUT` only changes the fields it sends, and `null` clears one. When a status changes to watching or reading the start date is set, and to watched or read the finish date, unless the request sends its own. Posters and covers are downloaded by the server and kept in the database.
 
 ### Content editor
 
@@ -117,7 +137,10 @@ bun run content
 | --- | --- |
 | `1` to `9` | Switch between Music, Shows, Movies, Books, Profile, Experience, Leadership, Projects and Themes |
 | `a` | Add (YouTube link, IMDb link or Goodreads link) |
+| `S` | Add books from a Goodreads series |
 | `e` / `Enter` | Edit the selected entry (or change the rating for shows, movies and books) |
+| `g` | Put a show or movie in a group, which the library apps and about page show together |
+| `h` | Set the started and finished dates for a show, movie or book |
 | `f` | Toggle featured on a project |
 | `p` | Preview a theme in the editor |
 | `t` | Switch the editor to the next theme |
@@ -125,29 +148,31 @@ bun run content
 | `d` | Delete |
 | `K` / `J` | Move a song or entry up or down |
 | `u` / `U` | Refresh the selected book, or every book, from Goodreads; `u` also refreshes a show or movie from OMDb |
-| `C` | Download every missing book cover, or every missing show and movie poster and details |
-| `r` | Reload from disk |
+| `C` | Save every missing book cover, or every missing show and movie poster and details, into the library |
+| `r` | Reload from disk or the library |
 | `q` | Quit |
 
-Show and movie titles are looked up on OMDb with `OMDB_API_KEY` from `.env` once when added. The details go into `data/mediaDetails.json` and the poster into `static/covers/media`, so the site only asks OMDb about titles that have nothing saved; fill in the name field to skip the lookup. Book covers are saved into `static/covers/books`. Books are scraped from Goodreads once when added: your shelf, rating and series go into `content/books.json`, and the scraped details into `data/bookDetails.json`. To check a Goodreads link on its own, run `python3 tools/content/goodreads.py <url>`.
+The Shows, Movies and Books tabs read and write the library through the API at `LIBRARY_API_URL` using `LIBRARY_API_KEY`, so point it at the live site to update it from anywhere. Show and movie titles are looked up on OMDb with `OMDB_API_KEY` from `.env` when added; fill in the name field to skip the lookup. Books are scraped from Goodreads when added. To check a Goodreads link on its own, run `python3 tools/content/goodreads.py <url>`.
+
+A game in `about.json` can have a `mobygames` link. Run `python3 tools/content/mobygames.py <url> --save` to scrape its release date, platforms, developers, genres, scores, description and images into `data/gameDetails.json`, which the game card shows. Leave out `--save` to only print the result, or use `--debug` if MobyGames blocks the request.
 
 | File | Holds |
 | --- | --- |
 | `profile.json` | Name, bio, socials, skills and photos |
-| `about.json` | Hobbies, games, shows, movies and the AniList username |
+| `about.json` | Hobbies, games and the AniList username |
 | `experience.json` / `leadership.json` | Work experience and leadership roles |
 | `projects/*.json` | Open source, packages, hackathons, research groups and organizations |
 | `music.json` | YouTube links for the music player |
-| `books.json` | Your books: Goodreads link, shelf, rating and series |
 | `wallpapers.json` | Wallpapers for the picker, written by `bun run wallpapers` |
 | `desktop.json` | Workspaces, desktop icons, the startup command and the default theme |
 | `themes.json` | Color themes for the site and the content editor |
 | `blog/*.md` | Blog posts |
 
-| Generated file in `data/` | Holds |
+| File in `data/` | Holds |
 | --- | --- |
-| `bookDetails.json` | Title, authors, cover, description, pages and genres scraped from Goodreads |
-| `mediaDetails.json` | Year, genre, runtime, plot and poster path from OMDb |
+| `library/library.db` | Shows, movies, books, their history dates and saved images |
+| `lastfm/lastfm.db` | Last.fm scrobbles, album and track cover links and saved cover images |
+| `gameDetails.json` | Video game details scraped from MobyGames |
 
 ### Wallpapers