I really enjoy optimization. Release v3.5.0 of Ktistec doesn't drop significant new features, but it does deliver a ~15% smaller executable and significantly faster queries on anonymous endpoints. The two are intertwined.

The size reduction comes from replacing a poorly designed, custom rules engine with a materialized view layer that uses SQL to define membership in a collection. The rules engine worked well enough but required a lot of supporting code to present rules as a DSL (Domain Specific Language) over the domain objects in ktistec. The driving realization was that SQL is a DSL and membership in a collection is just a query and domain objects are just rows. Voilà!

Query performance improvements came from using this new view layer to materialize two very popular but expensive-to-query views: the instance's public timeline and public hashtag pages. Because both are public pages they receive more traffic than internal pages.

The problem with the original queries was that performance was not uniform. Querying for posts with popular tags was okay. Querying for posts with sparse tags was very slow. I could have added more indexes, but that's its own cost. After the change, endpoints all respond in a consistent ~10msec timeframe and the CPU barely registers when a crawler hits. (I don't want to make things easier for bots, but I don't want to pay a tax for their activity either—ask me about my new nginx configuration.)

Here is the full changelog:

Added

• Lightweight probe endpoint for authenticated sessions.
• max-id and min-id pagination links on web pages.

Fixed

• Correct the notifications collection's JSON representation.
• Accept both single-value and array forms of JSON-LD properties.
• Handle variation in schema.org property mapping.

Changed

• Faster timeline, public, hashtag, and notification collections.
• Adjust the layout of actor profile properties.

Removed

• The school dependency; replaced by activity processors and materialized views.
• The openssl_ext dependency; vendored in.

There are still a few slow queries. In the next release I'm going to see if I can get everything under 10msec, and maybe release a new feature, too. 🚀

#ktistec #crystallang #activitypub #fediverse

This release fixes a small number of bugs found in recent releases.

The full changelog:

Fixed

• Prevent runaway recursion when handling filtered posts.
• Ensure profile header and header_static images are always present.
• Render the inline replies collection for local objects.
• Exclude blocked actors from object statistics and notifications.

Changed

• Return 410 Gone instead of 404 Not Found for missing actors.

Removed

• Tag counts on public pages.

This release fixes a hard-to-exploit but potentially server-crashing bug. If you're running v3.3.9 or v3.4.0, you should upgrade.

#ktistec #crystallang #activitypub #fediverse

The biggest change in release v3.4.0 of Ktistec is cursor-based pagination for all web-navigable collections (timeline, notifications, etc.). Offset-based pagination will be removed completely in the next release.

Offset-based (e.g. page/size) pagination works well on collections that don't change. But, what does "the second page" contain in a dynamic timeline? Support for cursor-based pagination is required by the Mastodon-compatible API, but has been a desirable feature for quite a while.

While updating queries to paginate by cursor, I also made performance improvements to the queries themselves, as mentioned elsewhere. Scrapers and bots have already adapted—sort of. I now see odd hybrid requests in the log like /tags/xyz?page=7&min_id=123. Overall CPU usage under normal load is now sitting at 0-1%.

Here is the full changelog for the release:

Added

• Cursor-based pagination for web-navigable collections. (fixes #122)
• Mastodon-compatible API: /api/v1/timelines/tag/:hashtag endpoint.

Fixed

• Negative replies count when viewing a post that is also a reply.
• Order cached actors' posts by published rather than id.

Changed

• Report 401 and 403 as distinct errors in Ktistec::Network.get.

Removed

• Unused paginated query methods.

Enjoy!

#ktistec #crystallang #activitypub #fediverse

Release v3.3.9 of Ktistec continues the security hardening work from recent releases, with further progress on the Mastodon-compatible API.

Of note: all network connections now go through a new Ktistec::Network module. This allows Ktistec to limit the size of HTTP bodies it reads, on both inbound and outbound requests, and ensures it only opens connections to valid remote IP addresses.

Here's the full changelog:

Added

• New Mastodon-compatible APIs.

Fixed

• Close DNS rebinding window for outbound HTTP requests.
• Limit the size of HTTP bodies the server reads.
• Sanitize RSS feed output to prevent CDATA breakout.
• Destroy all sessions and access tokens on account termination.

Changed

• Ensure all GET and POST requests utilize Ktistec::Network.
• Process local recipients in-process in inbox/outbox activity processors.

As always, it's worth upgrading for the security fixes!

#ktistec #crystallang #activitypub #fediverse

This release continues my focus on security instead of new features. As I wrote earlier this week, I rebuilt the template framework Ktistec uses with type safety as a central principle. What does that mean?

Imagine that you have an instance of a String that holds federated data. Where can you safely render that in a browser, and what operations (sanitization, escaping, etc.) do you need to do first?

The only way to answer that is to look carefully at the lineage of that data: where it came from, how it was stored, how it was transformed, and where it's rendered. A name holds text; an href or src attribute holds a URL. If you want to render a name inside an HTML element you should HTML escape it. You should escape href and src, too, but the escaping rules for URLs are slightly different from the HTML rules. It's easy to make mistakes.

Ktistec uses four "safe" types to express the contracts:

SafeHTML: A String wrapper marking HTML markup safe to emit raw into HTML data slots (text content, between tags).

SafeAttrValue: A String wrapper marking a value safe to emit raw inside a double-quoted HTML attribute (attr="..."), other than URL or event-handler slots.

SafeURI: A String wrapper marking a URL safe to emit raw into a URL attribute slot (href, src, action, etc.).

SafeJSON: A String wrapper marking JSON output safe to emit raw into the body of a <script type="application/json"> block.

Using the wrong type at a call site is either a compile-time error, or it triggers automatic sanitization of the underlying string value.

Here's the full changelog:

Added

• String safety framework with typed "safe" strings.
• New Slang template engine with compile-time safety checks.
• Vendored WebFinger and HostMeta client shards.

Fixed

• Prevent delivery to unknown IRIs.
• Narrow Like/Dislike addressing to the liked object's author.

I have at least one more cleanup pass to do, and then I'll turn my attention back to the Mastodon-compatible API and a few features I've been looking forward to—like scheduled posts.

#ktistec #crystallang #activitypub #fediverse

Release v3.3.7 of Ktistec fixes several bugs and introduces two enhancements.

Security is a focus in this release. Every gap in input sanitization or escaping is a potential vulnerability, and I've been systematically closing them. I am also carefully, and maybe conservatively, restricting things like supported URL schemes and uploaded file types.

The two enhancements improve compatibility with Mastodon-compatible clients. Mastodon's OAuth tokens don't expire, and Mastodon clients don't know how to handle tokens that do. Sliding expiration ensures that tokens in active use stay alive, while unused tokens eventually expire.

Here's the full changelog:

Added

• Sliding token expiration for OAuth2 access tokens.
• Mastodon-compatible API: /api/v1/accounts/update_credentials endpoint.

Fixed

• Prevent pinning of (and auto-unpin) private objects.
• Don't save a quote if the quoted actor cannot be dereferenced.
• Fix rendering of federated actor profile attachment values.
• Remove href attributes with unsafe schemes from sanitized HTML.
• Escape interpolated values in view helpers and the actor icon streaming refresh.
• Restrict upload extensions and serve uploads with X-Content-Type-Options: nosniff.
• Escape publicKey and scrub Tag.href.
• Sanitizer no longer permits single-quote attribute injection.
• Ensure bearer-token sessions cannot reach the web UI.
• Require client authentication on the OAuth token endpoint.

I'm working on performance improvements for the next release. A rewrite of the Slang template library looks like it will cut both build time and executable size by around 10%!

📡 Stay tuned!

#ktistec #crystallang #activitypub #fediverse

This release is a maintenance update: a few bug fixes, a security mitigation worth paying attention to, and some performance improvements for users on slow connections.

It's worth updating to pick up the SSRF (Server-Side Request Forgery) mitigation.

Fixed

• Prevent SSRF when dereferencing externally supplied IRIs.
• Timeline entry no longer becomes stale when an announce is undone.
• Correctly represent boosted posts on the home timeline in API clients.

Changed

• Defer downloading of script files.
• Defer loading of image, video, and audio attachments.
• Move poll vote notification for voters into the outbox processor.

#ktistec #crystallang #activitypub #fediverse

The Mastodon-compatible API is at a stable stopping point, so I have removed the gating with_mastodon_api build flag and made it generally available.

I use the API daily via a couple different client apps. What works, works. There are known limitations, most of them rooted in architectural differences between Mastodon and Ktistec. For example, Mastodon and Ktistec manage media differently. In Ktistec, there is no simple way to upload and manage media apart from creating a post. That difference will take some work to bridge.

Here's the full changelog for this release:

Added

• Mastodon-compatible API endpoints:
  • /api/v1/accounts
  • /api/v1/accounts/lookup
  • /api/v1/accounts/:id
  • /api/v1/accounts/:id/statuses
  • /api/v1/accounts/:id/following
  • /api/v1/accounts/:id/followers
  • /api/v1/accounts/relationships
  • /api/v1/follow_requests
  • /api/v1/polls/:id/votes
  • /api/v1/preferences

Fixed

• Correctly resolve keyId from Signature header for inbox verification.

Changed

• Support both offset and cursor-based pagination in JSON collections.
• Switch following/followers pages to cursor-based pagination.

Credit goes to this thread for inspiring me to review my signature verification. There is now one more correct implementation. 😉

The next release will focus on cleaning up the internals. Ktistec is just under 100,000 lines of code, 307 source files, and 195 spec files (7030 tests). With one maintainer, the only way to stay sane is by ruthlessly refactoring and paying down the debt!

#ktistec #crystallang #activitypub #fediverse

Release v3.3.4 of Ktistec is available.

This release adds Mastodon-compatible client support for publishing posts. Just like the previous release, however, all Mastodon API support is behind a build flag (-Dwith_mastodon_api). It's still experimental, so opt in only if you're happy to work with rough edges.

Beyond that, I focused on cleanup and refactoring throughout the codebase. Here's the full changelog:

Added

• Cursor-based pagination on actor timeline and everything pages.
• Mastodon-compatible API: /api/v1/statuses endpoint for status posting.
• Mastodon-compatible API: /api/v1/timelines/public endpoint.

Fixed

• Autosave focus handling. Fixes problems introduced in v3.3.3.
• Prevent blur from creating a draft post when publishing a post.

Changed

• Integrate X-Ray Mode colors into the theming system.
• Improve CI: add npm audit, test, and caching.
• Use npm ci in Dockerfile for reproducible builds.
• Remove very old compiler bug work-around.

🏋️ Mastodon API support is coming along—more in the next release!

#ktistec #crystallang #activitypub #fediverse

I have started work on a Mastodon-compatible API layer intended to support the many Mastodon front-ends available. It is incomplete and requires an explicit build flag to enable, but what's there (the main timeline) already works with the official Mastodon app, Tusky, and Phanpy.

Here's the full changelog:

Fixed

• Editor focus now stays in the editor after the first draft is saved. (fixes #139)
• Filter settings instructions. (fixes #135)

Changed

• Improved consistency of mini button colors.

As always, check out the full diff for the complete details.

#ktistec #crystallang #activitypub #fediverse