feat: add emote import from legacy source with dry-run mode
Add POST /manage/import (auth-protected) that fetches emotes from a legacy JSON endpoint, downloads each image, uploads it to S3, and inserts it into the DB with the original timestamps preserved. - Skip emotes whose name already exists (best-effort duplicate detection across SQLite and PostgreSQL via error code + message fallback) - Validate source_url against a configurable host allowlist ([import] allowed_hosts in config, default ["smutba.se"]) - dry_run: true previews the import without writing to S3 or DB; result statuses are "would_import" / "would_skip" instead of "imported" / "skipped" - Add db.name_exists() for efficient per-name existence checks used by dry-run - Add reqwest (rustls-tls + json) and url dependencies - Integration tests: auth guard, allowlist rejection, mirror + skip-duplicates, dry-run no-persist
This commit is contained in:
@@ -1,2 +1,57 @@
|
||||
# mikebase
|
||||
A Rust-based emote database and API.
|
||||
|
||||
## Importing legacy emotes
|
||||
|
||||
Use the protected management endpoint to mirror emotes from a legacy JSON feed.
|
||||
|
||||
### Endpoint
|
||||
|
||||
POST /manage/import
|
||||
|
||||
- Auth: HTTP Basic (same credentials used for other protected routes)
|
||||
- Content-Type: application/json
|
||||
- Body:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_url": "https://smutba.se/emoji/json/"
|
||||
}
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
- Fetches source JSON in the format `{"emotes": [{name, url, created, modified}, ...]}`
|
||||
- Downloads each image URL and uploads bytes to this app's configured S3 bucket
|
||||
- Inserts emote rows preserving source `created` and `modified` timestamps
|
||||
- Skips entries where `name` already exists locally
|
||||
- Continues processing after per-item failures and returns a batch summary
|
||||
|
||||
### Example response
|
||||
|
||||
```json
|
||||
{
|
||||
"source_url": "https://smutba.se/emoji/json/",
|
||||
"total": 2,
|
||||
"imported": 1,
|
||||
"skipped": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{"name": "legacy_new", "status": "imported", "reason": null},
|
||||
{"name": "legacy_duplicate", "status": "skipped", "reason": "Name already exists"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Allowlisted hosts
|
||||
|
||||
Import is restricted to hosts in configuration:
|
||||
|
||||
```toml
|
||||
[import]
|
||||
allowed_hosts = ["smutba.se"]
|
||||
```
|
||||
|
||||
Environment override example:
|
||||
|
||||
- `APP__IMPORT__ALLOWED_HOSTS=["smutba.se","legacy.example.org"]`
|
||||
|
||||
Reference in New Issue
Block a user