Release v2.4.6 of Ktistec is out. As mentioned in an earlier post, this release focuses on database performance improvements. This means caching the results of expensive queries (like counting all posts with a particular hashtag or mention). On my instance at least, pages like the notifications page are now snappier.

There are still slow queries (queries that take more than 50msec). Most of those are requests for pages of old posts where none of the necessary database pages are in the page cache. I have increased the page cache size, and that reduces the frequency, but I don't see an immediate fix.

Fixed

• Add missing database query logging.

Changed

• Improve query performance for hashtags and mentions.
• Make less costly updates to tag statistics.
• Improve anonymous session management.
• Cache the Nodeinfo count of local posts.

Removed

• Remove support for X-Auth-Token.

Other

• Add timeout values for POST socket operations.

I don't have an immediate plan for the next release. There have been a bunch of feature requests that I think have merit. I'll probably get started on some of those.

#ktistec #fediverse #activitypub #crystallang

You can specify the SQLite database and pass options (pragmas like cache_size, journal_mode, ...) on the command line when you start the Ktistec server. The following example sets the cache size to 20,000 pages (up from the default of 2,000 pages) which improves performance on larger instances.

KTISTEC_DB=~/ktistec.db\?cache_size=-20000 ./server

You can also enable the write-ahead log (but make sure you know what that means).

KTISTEC_DB=~/ktistec.db\?journal_mode=wal\&synchronous=normal ./server

Pragmas supported are limited to those listed here.

#ktistec #fediverse #activitypub #crystallang

A new release of ktistec that improves database performance is imminent. In the past, database optimization usually meant "fixing a bunch of poorly constructed queries", and I'm sure there's more of that to do—I'm not an expert. But this time, I found most of the queries were as good as they were going to get on my watch (I'm not an expert). If you have a million records and you need to filter and count them, that's just going to take some time...

So this time, I focused on caching the results of queries like that (which really means I focused on cache invalidation, right). A case in point is commit d544b1af. Previously, the nodeinfo endpoint filtered and counted posts on every request, and it took +80msec to do that. Worse, the filtering pushed everything else out of the sqlite page cache, which made the next, unrelated database query slow!

Caching this value, and only recounting when I post something, not only dropped the service time for the request to ~1msec but actually improved database performance, generally!

More to come...

#ktistec #fediverse #activitypub #crystallang

Crystal is fast because methods are monomorphized at compile time. In simple terms, that means that at compile time, a polymorphic method is replaced by one or more type-specific instantiations of that method. The following polymorphic code...

def plus(x, y)
  x + y
end

...is effectively replaced by two methods—one that does integer addition if called with two integers, and one that does string concatenation if called with two strings.

This extends to inherited methods, which are implicitly also passed self. You can see this in action if you dump and inspect the symbols in a compiled program:

class FooBar
  def self.foo
    puts "#{self}.foo"
  end

  def bar
    puts "#{self}.bar"
  end
end

FooBar.foo
FooBar.new.bar

class Quux < FooBar
end

Quux.foo
Quux.new.bar

Dumping the symbols, you see multiple instantiations of the methods foo and bar:

...
_*FooBar#bar:Nil
_*FooBar::foo:Nil
_*FooBar@Object::to_s<String::Builder>:Nil
_*FooBar@Reference#to_s<String::Builder>:Nil
_*FooBar@Reference::new:FooBar
_*Quux@FooBar#bar:Nil
_*Quux@FooBar::foo:Nil
_*Quux@Object::to_s<String::Builder>:Nil
_*Quux@Reference#to_s<String::Builder>:Nil
_*Quux@Reference::new:Quux
...

The optimizer in release builds is pretty good at cleaning up the obvious duplication. But during my optimization work on Ktistec, I found that a lot of duplicate code shows up anyway.

Most pernicious are weighty methods that don't depend on class or instance state (don't make explicit or implicit reference to self). As I blogged about earlier, this commit replaced calls to the inherited method map on subclasses with calls to the method map defined on the base class and reduced the executable size by ~5.8%. The code was identical and the optimizer could remove the unused duplicates.

So, as a general rule, if you intend to use inheritance, put utility code that doesn't reference the state or the methods on the class or instance in an adjacent utility class—as I eventually did with this commit.

(The full thread starts here.)

#ktistec #crystallang #optimization

Ktistec release v2.4.5 rolls out the build time and executable size optimizations I've been blogging about here. It also fixes a few small bugs.

Fixed

• Handle @-mentions with hosts in new posts.
• Handle HEAD requests for pages with pretty URLs.
• Destroy session after running scripts.

Changed

• Delete old authenticated sessions.

I've started a branch full of query optimizations. My general rule—as highlighted in the server logs—is if a query takes longer than 50msec, it takes too long. It's time to address some problems...

#ktistec #fediverse #activitypub #crystallang

The Ktistec executable is now ~24.7% smaller and build times are 28% faster.

I've been blogging about optimizations here, here, and here. This is the summary of the final outcome, with links to commits for the curious. I have one more post planned with a summary of my thoughts.

Here's my approach. Use nm to dump the symbols in a release build executable and then look for things that seem redundant. The first change and associated post below is a great example of what I mean—my original implementation led to the specialization of the #== method for every pairwise combination of model classes even though the result of the comparison was just false.

This might seem like a strange approach if you come from a compiled language where you mostly write all of the code yourself or invoke generics explicitly, but Crystal takes your code and does that for you. And it's not always obvious up front (to me, at least) what the final cost will be.

I've include counts of the lines added/removed because the point of this whole post is to say if you measure first and then optimize, a small change can have a big impact.

Here are the changes:

• Specialize model #==. (+7 -5)
  I talked about this here but didn't have the commit to link to. This change results in a large reduction in executable size on regular builds (~4.0%) and a small difference on release builds (~0.2%).
• Remove conversion to Hash. (+2 -2)
  This commit eliminates specialization of methods like __for_internal_use_only that get passed both named tuples and hashes by going all in with named tuples. It also eliminates instantiations of the Hash generic type itself for these cases. Reduces executable size by ~2.2%.
• Eliminate duplicate code in the executable. (+3 -3)
  This small change reduces the size of the executable by a further ~0.4% by eliminating redundant definitions of __for_internal_use_only entirely.
• Make InstanceMethods instance methods. (+1 -5)
  This was a goofy design I picked up somewhere. It's unnecessary. Changing this saves ~0.2% on release build executable size.
• Move the code for digging through JSON-LD. (+246 -281)
  It looks like a lot of lines of code changed here, but the large numbers are the result of moving code line-by-line from an included module to a utility class. Invoking these as methods on the utility class rather than as instance methods on each including class reduces the executable size by ~0.5%.
• Use map from base ActivityPub model classes. (+10 -2)
  map is a class method defined on each ActivityPub base model class. Each definition maps JSON-LD to a hash that is used to instantiate the class. Class methods defined on a base class are available on subclasses, as well. Calling the method on the subclass results in a copy of the method. This change reduces the executable size by ~5.8%.
• Move map into helper. (+104 -88)
  The map method does not depend on class/instance state. This change ensures that the mapping code is not duplicated even if a subclass's map method is accidentally again called. It looks like a lot of changes but this commit is mostly reorganization. It reduces executable size by ~0.4%.
• Replace classes with aliases. (+62 -148)
  Implementing ActivityPub's vocabulary with discrete model classes is expensive because every model class comes with machinery for type-specific CRUD operations. Enumerate aliases on each base model class (e.g. a "Service" is an "Actor"). This change reduces executable size by ~16.9%.

I'm off to optimize some queries now...

#ktistec #crystallang

The prologue to this post, and other posts in the series, is here.

Investigating commit b65d292f was fruitful but not for obvious reasons.

Dumping the symbols (nm -j server) before and after the commit showed large number of new equality (==) methods. From the diff:

1765a1772
> _*ActivityPub::Activity::Accept#==<Translation>:Bool
1920a1928
> _*ActivityPub::Activity::Add#==<Translation>:Bool
2062a2071
> _*ActivityPub::Activity::Announce#==<Translation>:Bool
2237a2247
> _*ActivityPub::Activity::Block#==<Translation>:Bool
    ...

The use, in a controller action, of the new Translation model seemingly triggered their generation. What was going on?

A long time ago I implemented a MVC model framework in the style of ActiveRecord (2de4a4b3) and it included a method for testing for equality. Note the method signature.

# Returns true if all properties are equal.
#
def ==(other : self)
  {% begin %}
    {% vs = @type.instance_vars.select(&.annotation(Persistent)) %}
    if
      {% for v in vs %}
        self.{{v}} == other.{{v}} &&
      {% end %}
      self.id == other.id
      true
    else
      false
    end
  {% end %}
end

The Reference class—the default parent for classes—defines two base implementations of this method:  one that tests for identity (not equality), with the signature def ==(other : self), and another that returns false, with the signature def ==(other). When I implemented my method, my assumption was: redefine the former for model classes and let the latter take care of everything else. This assumption was incorrect.

In circumstances that I still don't completely understand, the compiler will generate calls to the latter (the method that just returned false) when it "should have" been calling the former, and comparisons failed when they should have succeeded. I "fixed this" with commit effeaa26 that removed the type restriction and explicitly handled the type check. Everything worked!

The problem is Crystal creates a version of this method for every possible model comparison, specialized by both self and other. Most of the time the type check fails and the method returns false. But the rest of the code is still present.

The fix (re)adds a method specialization that returns false and lets the compiler handle the type check.

# Returns `false`.
#
def ==(other)
  false
end

Because this method just returns a constant value, the compiler gets rid of the method call, as well.

Interestingly, this change reduced the size of the Ktistec server executable by 4.0% when building without the --release flag but only 0.2% when building with it, so optimization does a good job at cleaning this up even without the change. 

#ktistec #crystallang