status update on ktistec...

commits 831dce60 to aa608699 reimplement notifications and timeline management as declarative rules rather than procedural logic. this is the culmination of work announced a while back. here's a sample of what the code looks like now:

rule "create" do
  condition var("activity"), IsAddressedTo, var("actor")
  condition CreateActivity, var("activity"), object: var("object")
  any Mention, var("mention"), subject: var("object"), href: var("actor").iri
  none Notification, owner: var("actor"), activity: var("activity")
  assert Notification, owner: var("actor"), activity: var("activity")
end

rule "announce" do
  condition var("activity"), IsAddressedTo, var("actor")
  condition AnnounceActivity, var("activity"), object: var("object")
  condition Object, var("object"), attributed_to: var("actor")
  none Notification, owner: var("actor"), activity: var("activity")
  assert Notification, owner: var("actor"), activity: var("activity")
end
...

the first rule says, if an object (post) mentions the actor (you) and a notification doesn't already exist, create one. the second rule says, if someone announces (shares/boosts) an object (post) attributed to the actor (you) and a notification doesn't already exist, create one.

the implementation took three steps: 1) implement a simple, generic rules engine called school, 2) implement logic in ktistec to make it possible to expose the object model and records in the database as facts in the rules engine, 3) reimplement the rules. there was plenty of yak-shaving, too—the preceding ~15-20 commits were fixing things that were in the way. (my usual heuristic assumes 25% refactoring existing code and 75% developing new code, but in this case it was 75%/25% in the other direction.)

while not trivial, rules are easier to reason about than code. toward the end of the project i realized that i wasn't adding replies addressed to me to the timeline (they were only visible in a thread). i was able to fix that defect with a small modification to the rules.

the goal is to make rules modifiable by the user at runtime so that users can customize their ktistec instance without having to rebuild it. the next step toward that end is a simple rules definition language and parser.

#ktistec #school #crystal #nocode

if you're wondering what's going on with ktistec, i'm currently focused on creating a general purpose rules engine in crystal called school, and rewriting the internal logic using rules. ultimately adding/removing/changing rules will be something that users can do dynamically using a simple domain-specific language or user interface, and ktistec will become a highly customizable tool for connecting to the fediverse.

#ktistec #school #crystal #nocode

crystal 1.3.0 is released!

#crystal

new project sunday... the school rules engine.

i'm going to rewrite all of the logic for handling fediverse activities in kistec as rules, and then expose a simple ui for managing those rules so that users can more easily customize their instance. want to change what shows up in the timeline? no problem!

#crystal #school #rulesengine #ktistec #fediverse

lorenzo barasti does a great job here making up for the lack of official documentation on crystal’s select statement. the article’s a must-read if you are planning on making use of concurrency.

https://lbarasti.com/post/select_statement/

#crystal

what a difference one line of code makes!

commit fe18dc5 moved the line @saved_record : self | Nil = nil from the module Ktistec::Model::InstanceMethods into the macro Ktistec::Model.included.

that single change cut the build time in half and reduced the memory required to build by about a third. you can be sure the improvement came as a surprise!

but really, it wasn't just this change itself—although it introduced the biggest improvement—because after all of the subsequent refactoring, simply moving the line back didn't increase the build time again.

the build time also improved slightly after other subsequent commits. and several of those commits introduced changes that reduced the amount of macro generated code. and that's the key insight: macros reduce visible boilderplate but don't reduce actual code that needs to be compiled.

i'm still investigating, but clearly the original placement of the line resulted in the generation of a lot of redundant/unnecessary code!

TL;DR if you're planning on using macros in crystal, check your build times before and after your changes—i'm sure there's a corresponding commit somewhere in the past that introduced this problem and i didn't catch it!

#ktistec #crystal

building ktistec is still an endeavor—both in time and in space. i have work ahead of me to reduce the build footprint—since starting this project, i've learned that some combinations of the crystal language features i used to recreate rails magic (macros, named arguments and inheritance) are hard on the compiler.

in any case, having exceeded the capacity of the tiny linux vps that hosts epiktistes, i set out to build a statically linked executable on my laptop (macos mojave) that i could deploy to my vps (linux centos).

the official instructions, using docker and alpine linux, are here. to get it to build, i had to install the static libraries for sqlite, and to get it to run on my vps i had to specify the location of the openssl cert file, which is loaded at runtime and doesn't match the location in alpine linux.

tl;dr the steps

• build (i haven't automated any of this yet)
  1. docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:latest-alpine sh
  2. apk update && apk upgrade
  3. apk add sqlite-static
  4. crystal build src/ktistec/server.cr --static
  5. exit
• run (after copying the executable to the vps)
  • SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt ./server

#ktistec #crystal

This is a summary of the presentation I did today about MXNet.cr for the Crystal Language 1.0 Conference.

Machine Learning with MXNet.cr

MXNet.cr is bindings to the MXNet machine learning/deep learning library and an implementation of the MXNet Gluon deep learning library. MXNet.cr is also kind of a port of a port.

A few years ago I decided I wanted to tune up my machine learning skills, but I didn't want to use Python. At about the same time, I discovered both MXNet, with support for multiple programming languages, and mxnet.rb, a set of Ruby bindings to MXNet by Kenta Murata. For a while I was happy. But Kenta's bindings didn't include an implementation of Gluon.

Gluon is a high-level framework for building deep neural networks. I was running through the lessons in Deep Learning - The Straight Dope and bumped into lessons that required Gluon. The obvious solution was to implement Gluon. Eventually parts of my implementation made their way into mxnet.rb.

Around the time my Ruby code was being merged, I realized I'd found another language that I was really jazzed about—Crystal. You all know about Crystal—it's got Ruby-like syntax and Ruby-like ergonomics, but it's blazingly fast. So again I did the obvious thing and ported my code to Crystal.

Lessons Learned

You can almost mechanically convert Ruby code into Crystal code. Consider the following two blocks of code. Which is Ruby and which is Crystal?

def self.import(filename, inputs,
                epoch: 0, ctx: MXNet.cpu,
                allow_missing: false,
                ignore_extra: false)
  output = MXNet::Symbol.load('%s-symbol.json' % filename)
  inputs = [inputs] unless inputs.is_a?(Array)
  inputs = inputs.map { |i| MXNet::Symbol.var(i) }
  SymbolBlock.new(output, inputs).tap do |block|
    if epoch
      filename = '%s-%04d.params' % [filename, epoch]
      arg_dict = MXNet::NDArray.load(filename)
      arg_dict = arg_dict.transform_keys do |k|
        k.gsub(/^(arg:|aux:)/, '')
      end

def self.import(filename, inputs,
                epoch = 0, ctx = MXNet.cpu,
                allow_missing = false,
                ignore_extra = false)
  outputs = [MXNet::Symbol.load("%s-symbol.json" % filename)]
  inputs = [inputs] unless inputs.is_a?(Array)
  inputs = inputs.map { |i| MXNet::Symbol.var(i) }
  SymbolBlock.new(outputs, inputs).tap do |block|
    if epoch
      filename = "%s-%04d.params" % [filename, epoch]
      arg_dict = MXNet::NDArray.load(filename)
      arg_dict = arg_dict.transform_keys do |k|
        k.gsub(/^(arg:|aux:)/, "")
       end

Adding the missing but required type annotations and fixing the parts of the Ruby code that play fast and loose took the most work—for example, the Ruby (and Python, in its own way) code makes use of method_missing which is difficult to replicate in Crystal.

Crystal Language syntax for defining bindings to native libraries is first class. Seriously, I don't know who made the call but they sure got it right. In contrast, the Ruby bindings were difficult to write, although I did learn a lot about writing Ruby extensions in the process. (Ruby extensions have long been in the same boat as Elisp programming—something I am interested in but can't work up the enthusiasm to actually do.)

Crystal macros are powerful tools for removing boilerplate. Almost every operation you need in MXNet is discoverable and dynamically callable at runtime. The Ruby (and Python) way is to introspect all of the operation metadata (names, parameters, their documentation strings, etc.) at runtime and to dynamically create methods for use by application code. This is pretty slick and arguably what you want in Crystal as well, albeit at compile time.  For performance reasons I wrote a tool that dumped the metadata into a Crystal language data structure ahead of time, and then used macros at compile time to create methods.

Work on MXNet.cr has temporarily taken a back seat to work on ktistec, but as ktistec matures I look forward to spending more time on MXNet.cr

#crystalconference2021 #machinelearning #mxnet #crystal

version 1.0.0 🎇

#crystal