the backend for ktistec algorithmic feeds just landed in commit 627c9292.

it's a proof-of-concept implementation of hashtag-mention-keyword feeds popular in other servers with any/all/none clauses—not very algorithmic but it gets the plumbing right and tested. it builds on top of the materialized views support i implemented a few releases ago.

#ktistec #fediverse

does anyone have any experience with any of the existing fediverse hashtag relays?

there are a lot of open tabs in my life...
  20+ in my browser
  15+ in my terminal
  40+ in my editor

i'm sure inline json-ld contexts with server extensions seem like the wrong thing, but they're preferable to identical copies of the same context hosted externally and served by a dozen or more instances of a server or family of servers.

i could be persuaded to change my mind if a server allowed meaningful customization. but if a server is publishing a canonical vocabulary of extensions for that server, it should be hosted in a single, well-know location.

#fediverse #activitypub

I started to work on algorithmic feeds but was side-tracked by interoperability work. No complaints. It turned out to be a productive detour.

Here's the full changelog for release v3.7.0 of Ktistec:

Added

• Support FEP-2c59: Discovery of a Webfinger address from an ActivityPub actor.
• Support ActivityPub Update activities for actor profile changes.

Fixed

• Disambiguate reblog IDs from status IDs. (fixes #151)
• Correct the quote_policy mapping to public/nobody values.
• Ignore malformed pagination parameters instead of raising.
• Treat "cannot be reconnected" errors as connection failures.
• Infer a media attachment's type when mediaType is missing.
• Faster, case-insensitive, actor username lookups.
• Faster statuses_count using an approximate count.

Changed

• Resolve JSON-LD contexts by matching their digest against a bundled copy.

The first version of algorithmic feeds won't be very algorithmic—it will let you create a feed that filters by keywords, hashtags, and mentions. That covers a lot of ground for me personally, and lays the groundwork for future enhancements.

#ktistec #crystallang #activitypub #fediverse

I dislike the U-Haul rental experience—I feel like every time I rent I run into problems. This time it was sending me 20 miles into the country to pick up a truck in an area with cellular service so weak I couldn't upload my driver's license.

That said, the human I got on the phone worked with me for a hour to troubleshoot and then to move my reservation. What a champ! As long as they use humans in support, I will continue to rent from them.

It is said that there are only two hard things in computer science: cache invalidation and naming things. The story goes: you have something that is expensive to compute, so you compute it once and then you cache it and use the cached value in the future. But the inputs to that computation change, and so the cached value grows stale. You have to decide when and how to recompute that value.

In Ktistec, presenting accurate tag counts is expensive because not every tagged post counts. Posts are deleted, actors are blocked. My own drafts don't count, but when they're published they do. A post tagged with the same hashtag more than once, must count as one. And tag cardinality is not uniform: #3dprinting has hundreds of thousands of posts, others have one or two. Even with indexes, there is no single query that counts all cases in an acceptable amount of time.

So I reached for a cache, counted once and then cached the count. Because I didn't want to maintain adjustments from every place in the code that changed something that touched the count, I settled for eventual consistency and recomputed counts after every server restart.

As it turns out, that's not good enough. On a server with reasonable traffic, an event that affects some tag's count happens every few hours. Days or weeks later there is significant drift. Worse, the implementation didn't recompute on first read, it recomputed on first write (a new tagged object arrives).

This release fixes all that. Counts are still eventually consistent, but all counts are recomputed in a regular background task, so they really are eventually consistent, and care was taken in constructing the query to minimize database (read) locking to ~100-200msec.

Is it better? Yes! Is it perfect? Probably not. Cache invalidation is hard.

Here's the full changelog for this release:

Added

• Background task to reconcile tag statistics.

Fixed

• Prevent model hook callbacks from interleaving.
• Add spacing between content and the sticky footer.

Changed

• Replace Semantic UI with Fomantic UI.
• Cache the PURL and GoToSocial JSON-LD contexts.
• Reduce database lock time when reconciling tags.
• Block npm dependency install scripts.

Removed

• The unused idx_relationships_type database index.

In the next release, I'm going to fix a few bugs in the Mastodon-compatible API. These require an internal redesign, so I've held off until a few other things were out of the way. And I'm turning my attention to reading and better tools for surfacing and finding interesting content.

#ktistec #crystallang #activitypub #fediverse

I need to use a prefix to namespace status IDs vs. boost IDs in the #ktistec Mastodon-compatible API. In Mastodon, a boost is just a status and they share the same ID namespace. Ktistec predates its Mastodon-compatible API, so statuses and boosts are maintained in different tables. I wanted to use an emoji (✍️ vs. 📣) to distinguish them, but that breaks too many clients.

Shame...

hung up three sets of curtains… hit studs on every hole. it’s surprising what makes me happy.