#crystallang 17 hashtags

Release v2.4.0 of Ktistec

I just released v2.4.0 of Ktistec. This release encompasses a few things that I've been working on for a while: improved support for operating without JavaScript available/enabled and support for running scripted automations.

Except for a few items, Ktistec now works without JavaScript. Obviously, things like WYSIWYG editing of HTML don't work—I plan to add support for Markdown to compensate. Running in Lynx is a stretch, but...

lynx on osx displaying the authenticated home page

Since the early days, most controller actions supported both text/html and application/json.  I cleaned up support for the latter and have officially documented the Ktistec API in the README.

In addition, I've added support for running bots/automations (prior announcement). The Ktistec server will periodically run any executable script in the etc/scripts directory. These scripts have access to the Ktistec API and can post, follow, share, like, etc. This is experimental and obviously introduces an attack surface, though that shouldn't be a problem on correctly configured hosts.

Here's the full changelog:

Added

  • Support running scripted automations.
  • "Fetch Once" button on hashtag and thread pages. (fixes #108)
  • Support navigation to a post's threaded view. (fixes #108)
  • Add support for post name and summary.

Fixed

  • Improve support for operating without JavaScript available/enabled.
  • Only enforce CSRF protection on "simple" requests.

Changed

  • Replace use of multi-action controller with formaction. (fixes #101)

Other

  • API usability improvements.


#ktistec #fediverse #activitypub #crystallang

I just released v2.3.0 of Ktistec.  It fixes a few bugs introduced in the previous release. In particular, it handles a case I've observed a few times in production where the fetching of followed content hangs. The fix adds a monitor that periodically checks for "running" tasks without a backing fiber and sets socket timeouts to ensure requests eventually terminate (I believe the latter actually fixes the problem, but belts and suspenders...)

#ktistec #crystallanguage #crystallang #activitypub #fediverse

I just released v2.2.0 of Ktistec. A lot of code was written but the main user-visible change is streaming timeline updates to the web client using server-sent events.  The back-end pub/sub framework will also make it easier to decouple some components down the road.

I want a visual indicator that new posts are in my timeline. I appreciate that not everyone feels the same way—if there's interest I may make it switchable.

And now, off to #worldcon!

#ktistec #fediverse #crystallang

the last few days are more in line with what i expect. there is a (temporary) bump in memory, but that corresponds to a large increase in traffic.

daily charts of memory, traffic, and server restarts for epiktistes

#ktistec #crystallang

for reasons i don't understand, some restarts of epiktistes result in huge memory usage spikes—reported both by the garbage collector and the operating system. what's interesting is heap size and free memory move roughly in tandem (though the difference isn't constant), which implies that there is free memory, but perhaps it's too fragmented to be useful?

daily charts of memory, traffic, and server restarts for epiktistes

a restart typically fixes the problem...

#ktistec #crystallang

#crystallang developers… what can you count on being safe to access during a call to #finalize?

i have some class-level state that i’d like to modify via a class variable but even reading values via that variable seems to cause problems that eventually lead to crashes later in the program. (typically index out of bounds during some later garbage collection run.)

When you optimize a Crystal program, pay attention to language features that inline code. For example, pay attention to how you use blocks (consequences here and here).

Also pay attention to how you use macros. Macros, like ECR.embed and Slang.embed, inline code at the point where they are invoked. This can be powerful, because macros actually generate code—but, ten invocations later, you have ten copies of the code.

Here's a case of too many copies, but with a very happy ending...

Ktistec uses both ECR.embed and Slang.embed to generate web pages from views and partials. I wrote code to count the number of places Ktistec used embed for each view and partial it renders. There's a long tail, but here are the big ones:

| src/views/layouts/default.html.ecr           | 204 |
| src/views/partials/modals.html.slang         | 204 |
| src/views/partials/header.html.slang         | 204 |
| src/views/partials/footer.html.slang         | 204 |
| src/views/pages/generic.html.slang           | 155 |
| src/views/partials/object/label.html.slang   |  36 |
| src/views/partials/object/content.html.slang |  36 |
| src/views/partials/collection.json.ecr       |  28 |
| src/views/partials/thread.html.slang         |  12 |
| src/views/partials/detail.html.slang         |  12 |
| src/views/partials/object.html.slang         |  12 |
| src/views/partials/actor-panel.html.slang    |  11 |
| src/views/partials/object.json.ecr           |  11 |
| src/views/partials/paginator.html.slang      |  11 |
| src/views/objects/thread.json.ecr            |   8 |
| src/views/partials/activity/label.html.slang |   6 |
| src/views/mentions/index.json.ecr            |   6 |
| src/views/remote_follows/index.json.ecr      |   6 |
| src/views/settings/settings.json.ecr         |   6 |
| src/views/tags/index.json.ecr                |   6 |
| src/views/activities/activity.json.ecr       |   5 |
| src/views/partials/editor.html.slang         |   5 |
| src/views/objects/object.json.ecr            |   5 |
| src/views/actors/remote.json.ecr             |   4 |
...

The layout is part of every page and is rendered with every view, so lots of copies. Every page has a header and a footer (and some default modal dialogs) so you get those, too. The generic view is a little less obvious. It's used to render pages for which there is no more specific view—typically pages served for 400 Bad Request or 401 Unauthorized. Objects (posts) are rendered in a variety of contexts, so it's no surprise label.html.slang and content.html.slang are popular.

ECR.embed and Slang.embed inline templates at the point where they are invoked, but beyond that they don't really customize the generated code—they just duplicate it. What we want  is one function for each view or partial, which wraps embed and returns JSON or HTML.

Those changes mostly occur in commits from 399287cf to 4b025f50. To say that they made a huge difference is a gross understatement. Executable size decreased by ~13%. Build time decreased by ~50%, and the memory required to build decreased by ~30%.

#ktistec #crystallang #optimization

The Cost of Small Methods

Ktistec uses a template engine for it's views.

View templates are transformed into Crystal code that generates HTML when executed. As you'd expect, the template language allows you to use string interpolation syntax (e.g. #{expression}) for dynamic values.

To ensure expression is only evaluated once, and to limit the scope of the temporary variable holding the evaluated value of expression, I originally bound the value to the variable using Object#tap (commit 5e1bf19e). The generated code looked something like:

(expression).tap do |__value__|
   <template code that uses expression>
end

Blocks in Crystal are always inlined, so the code above should be equivalent to the following (sacrificing local scope):

__value__ = (expression)
<template code that uses expression>

Functionally, they are equivalent. But operationally, not so much! With Object#tap, the Ktistec executable is about 1% larger (36823603 bytes vs. 36526307 bytes) and build times take 20% longer (23 seconds vs. 19 seconds, generally).

In total, view templates represent about 6% of the Ktistec executable by size, so it doesn't surprise me that there's a measurable impact when I make changes to the template engine, but wow...! I can almost live with the size of the executable, but the build time...!

The cost has to be the method call.

What I'm looking for is something like let in Scheme. The following macro comes close, but doesn't limit scope quite the same way:

macro let(expr, &block)
  {{block.args.first}} = ({{expr}})
  {{block.body}}
end

I maybe have to live with the macro—I tried to implement let as a method with the annotation @[AlwaysInline] but there was no improvement over the original.

The template engine is a fork of Slang—which I've been evolving to be more Slim-compatible.

#ktistec #crystallang