There are two significant new additions in release v3.8.0 of ktistec.

First, the actor cards on the followed/following pages show relevant actor status. On the followed page, it tells you how long it has been since that actor published an activity (create, announce, like, etc.) that was sent to your server. It's a proxy for how active they are (or are not). On the following page, it tells you how long it has been since you have been able to send to that server. It's a proxy for whether that server is reachable or that actor is still alive.

Second, the backend for user-defined algorithmic feeds is in place, along with a keyword/hashtag/mention feeds implementation. You can't set up a feed via the user-interface, but the feeds work if you set one up directly in the database—which is how I've been previewing them. I plan to release the frontend next week.

Here's the full changelog:

Added

• Display activity status on actor cards.
• Back-end support for user-defined algorithmic feeds.
• Apply community-relayed moderator deletes received as a Group's wrapped Announce.
• Follow a web page's rel="alternate" link when searching.

Fixed

• Avoid loading entire has_many collections when constructing child records.
• Evaluate the same-origin fetch gate against an embedded node's own identifier.
• Accept a delete of an uncached object or actor without verification.
• Catch MIME::Multipart::Error in local file-upload handling.
• Map malformed request-body parse failures to Bad Request.

#ktistec #crystallang #activitypub #fediverse

the backend for ktistec algorithmic feeds just landed in commit 627c9292.

it's a proof-of-concept implementation of hashtag-mention-keyword feeds popular in other servers with any/all/none clauses—not very algorithmic but it gets the plumbing right and tested. it builds on top of the materialized views support i implemented a few releases ago.

#ktistec #fediverse

I started to work on algorithmic feeds but was side-tracked by interoperability work. No complaints. It turned out to be a productive detour.

Here's the full changelog for release v3.7.0 of Ktistec:

Added

• Support FEP-2c59: Discovery of a Webfinger address from an ActivityPub actor.
• Support ActivityPub Update activities for actor profile changes.

Fixed

• Disambiguate reblog IDs from status IDs. (fixes #151)
• Correct the quote_policy mapping to public/nobody values.
• Ignore malformed pagination parameters instead of raising.
• Treat "cannot be reconnected" errors as connection failures.
• Infer a media attachment's type when mediaType is missing.
• Faster, case-insensitive, actor username lookups.
• Faster statuses_count using an approximate count.

Changed

• Resolve JSON-LD contexts by matching their digest against a bundled copy.

The first version of algorithmic feeds won't be very algorithmic—it will let you create a feed that filters by keywords, hashtags, and mentions. That covers a lot of ground for me personally, and lays the groundwork for future enhancements.

#ktistec #crystallang #activitypub #fediverse

It is said that there are only two hard things in computer science: cache invalidation and naming things. The story goes: you have something that is expensive to compute, so you compute it once and then you cache it and use the cached value in the future. But the inputs to that computation change, and so the cached value grows stale. You have to decide when and how to recompute that value.

In Ktistec, presenting accurate tag counts is expensive because not every tagged post counts. Posts are deleted, actors are blocked. My own drafts don't count, but when they're published they do. A post tagged with the same hashtag more than once, must count as one. And tag cardinality is not uniform: #3dprinting has hundreds of thousands of posts, others have one or two. Even with indexes, there is no single query that counts all cases in an acceptable amount of time.

So I reached for a cache, counted once and then cached the count. Because I didn't want to maintain adjustments from every place in the code that changed something that touched the count, I settled for eventual consistency and recomputed counts after every server restart.

As it turns out, that's not good enough. On a server with reasonable traffic, an event that affects some tag's count happens every few hours. Days or weeks later there is significant drift. Worse, the implementation didn't recompute on first read, it recomputed on first write (a new tagged object arrives).

This release fixes all that. Counts are still eventually consistent, but all counts are recomputed in a regular background task, so they really are eventually consistent, and care was taken in constructing the query to minimize database (read) locking to ~100-200msec.

Is it better? Yes! Is it perfect? Probably not. Cache invalidation is hard.

Here's the full changelog for this release:

Added

• Background task to reconcile tag statistics.

Fixed

• Prevent model hook callbacks from interleaving.
• Add spacing between content and the sticky footer.

Changed

• Replace Semantic UI with Fomantic UI.
• Cache the PURL and GoToSocial JSON-LD contexts.
• Reduce database lock time when reconciling tags.
• Block npm dependency install scripts.

Removed

• The unused idx_relationships_type database index.

In the next release, I'm going to fix a few bugs in the Mastodon-compatible API. These require an internal redesign, so I've held off until a few other things were out of the way. And I'm turning my attention to reading and better tools for surfacing and finding interesting content.

#ktistec #crystallang #activitypub #fediverse

I need to use a prefix to namespace status IDs vs. boost IDs in the #ktistec Mastodon-compatible API. In Mastodon, a boost is just a status and they share the same ID namespace. Ktistec predates its Mastodon-compatible API, so statuses and boosts are maintained in different tables. I wanted to use an emoji (✍️ vs. 📣) to distinguish them, but that breaks too many clients.

Shame...

I really enjoy optimization. Release v3.5.0 of Ktistec doesn't drop significant new features, but it does deliver a ~15% smaller executable and significantly faster queries on anonymous endpoints. The two are intertwined.

The size reduction comes from replacing a poorly designed, custom rules engine with a materialized view layer that uses SQL to define membership in a collection. The rules engine worked well enough but required a lot of supporting code to present rules as a DSL (Domain Specific Language) over the domain objects in ktistec. The driving realization was that SQL is a DSL and membership in a collection is just a query and domain objects are just rows. Voilà!

Query performance improvements came from using this new view layer to materialize two very popular but expensive-to-query views: the instance's public timeline and public hashtag pages. Because both are public pages they receive more traffic than internal pages.

The problem with the original queries was that performance was not uniform. Querying for posts with popular tags was okay. Querying for posts with sparse tags was very slow. I could have added more indexes, but that's its own cost. After the change, endpoints all respond in a consistent ~10msec timeframe and the CPU barely registers when a crawler hits. (I don't want to make things easier for bots, but I don't want to pay a tax for their activity either—ask me about my new nginx configuration.)

Here is the full changelog:

Added

• Lightweight probe endpoint for authenticated sessions.
• max-id and min-id pagination links on web pages.

Fixed

• Correct the notifications collection's JSON representation.
• Accept both single-value and array forms of JSON-LD properties.
• Handle variation in schema.org property mapping.

Changed

• Faster timeline, public, hashtag, and notification collections.
• Adjust the layout of actor profile properties.

Removed

• The school dependency; replaced by activity processors and materialized views.
• The openssl_ext dependency; vendored in.

There are still a few slow queries. In the next release I'm going to see if I can get everything under 10msec, and maybe release a new feature, too. 🚀

#ktistec #crystallang #activitypub #fediverse

I just finished working on improvements to #ktistec that cut about 15% off the built executable size, and speed up some of the more common public queries by 2x to 5x (they were already fast, so this is headroom).

It does this by replacing a poorly designed, feature poor, custom rules engine with a materialized view layer that uses SQL as its DSL (domain specific language).

I am about to smoke test it on my own site. If it’s not available—well, you know why! 😀

bitten

https://github.com/mastodon/mastodon/issues/18361

#ktistec #jsonld

This release fixes a small number of bugs found in recent releases.

The full changelog:

Fixed

• Prevent runaway recursion when handling filtered posts.
• Ensure profile header and header_static images are always present.
• Render the inline replies collection for local objects.
• Exclude blocked actors from object statistics and notifications.

Changed

• Return 410 Gone instead of 404 Not Found for missing actors.

Removed

• Tag counts on public pages.

This release fixes a hard-to-exploit but potentially server-crashing bug. If you're running v3.3.9 or v3.4.0, you should upgrade.

#ktistec #crystallang #activitypub #fediverse

The biggest change in release v3.4.0 of Ktistec is cursor-based pagination for all web-navigable collections (timeline, notifications, etc.). Offset-based pagination will be removed completely in the next release.

Offset-based (e.g. page/size) pagination works well on collections that don't change. But, what does "the second page" contain in a dynamic timeline? Support for cursor-based pagination is required by the Mastodon-compatible API, but has been a desirable feature for quite a while.

While updating queries to paginate by cursor, I also made performance improvements to the queries themselves, as mentioned elsewhere. Scrapers and bots have already adapted—sort of. I now see odd hybrid requests in the log like /tags/xyz?page=7&min_id=123. Overall CPU usage under normal load is now sitting at 0-1%.

Here is the full changelog for the release:

Added

• Cursor-based pagination for web-navigable collections. (fixes #122)
• Mastodon-compatible API: /api/v1/timelines/tag/:hashtag endpoint.

Fixed

• Negative replies count when viewing a post that is also a reply.
• Order cached actors' posts by published rather than id.

Changed

• Report 401 and 403 as distinct errors in Ktistec::Network.get.

Removed

• Unused paginated query methods.

Enjoy!

#ktistec #crystallang #activitypub #fediverse