in ktistec news, commits 4d2f699 through 4d2f699 include several improvements and bug fixes. highlights:

  • at runtime, the KTISTEC_DB environment variable overrides the default location for the SQLite database file. i use this during testing. it may be useful for some deployments.
  • the background TaskWorker now kicks off as soon as a new task is scheduled. the previous implementation polled for new work every five seconds.
  • i collected all inbox/outbox side-effects in one location. this makes mailbox processing easier to reason about, and also made it easier to prevent a race condition that resulted in the same object being added to the timeline more than once†.
  • images are lazy loaded in browsers that support loading="lazy".

i'm also now only one feature away—blocking actors and objects—from tagging the 1.0.0 release.

† the race condition occurs when an activity, like a create, arrives that takes a long time to validate, maybe because ids need to be dereferenced. while it's being validated, an announce activity arrives for the same object. now two activities for the same object are being processed concurrently. being added to a timeline is a side-effect, and collecting all side-effects in one location made it easier to put the duplicate check after validation but before any side-effects, which saved me from having to unwind side-effects if a duplicate was detected.

#ktistec

what a difference one line of code makes!

commit fe18dc5 moved the line @saved_record : self | Nil = nil from the module Ktistec::Model::InstanceMethods into the macro Ktistec::Model.included.

that single change cut the build time in half and reduced the memory required to build by about a third. you can be sure the improvement came as a surprise!

but really, it wasn't just this change itself—although it introduced the biggest improvement—because after all of the subsequent refactoring, simply moving the line back didn't increase the build time again.

the build time also improved slightly after other subsequent commits. and several of those commits introduced changes that reduced the amount of macro generated code. and that's the key insight: macros reduce visible boilderplate but don't reduce actual code that needs to be compiled.

i'm still investigating, but clearly the original placement of the line resulted in the generation of a lot of redundant/unnecessary code!

TL;DR if you're planning on using macros in crystal, check your build times before and after your changes—i'm sure there's a corresponding commit somewhere in the past that introduced this problem and i didn't catch it!

#ktistec #crystal

building ktistec is still an endeavor—both in time and in space. i have work ahead of me to reduce the build footprint—since starting this project, i've learned that some combinations of the crystal language features i used to recreate rails magic (macros, named arguments and inheritance) are hard on the compiler.

in any case, having exceeded the capacity of the tiny linux vps that hosts epiktistes, i set out to build a statically linked executable on my laptop (macos mojave) that i could deploy to my vps (linux centos).

the official instructions, using docker and alpine linux, are here. to get it to build, i had to install the static libraries for sqlite, and to get it to run on my vps i had to specify the location of the openssl cert file, which is loaded at runtime and doesn't match the location in alpine linux.

tl;dr the steps

  • build (i haven't automated any of this yet)
    1. docker run --rm -it -v $(pwd):/workspace -w /workspace crystallang/crystal:latest-alpine sh
    2. apk update && apk upgrade
    3. apk add sqlite-static
    4. crystal build src/ktistec/server.cr --static
    5. exit
  • run (after copying the executable to the vps)
    • SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt ./server

#ktistec #crystal

commits 99a0fd6 to 70bcf3f implement metrics and charting.

inbox and outbox activity

an instance of the Point model records an occurrence of an event—for example, the arrival of an activity in my inbox. points have a timestamp and a value and can be aggregated by day, week, etc. aggregation takes into account the timezone of the account that's viewing the chart so you'll need to specify a timezone during setup or in your account settings. i use chart.js for the charts.

i'm only collecting inbox and outbox metrics right now, but this will expand to include other types of metrics—social events, errors, etc.

looking at the chart above... i clearly need to post more...

#ktistec

i spent most of the summer cleaning up the codebase of my covid project (aka ktistec).

in theory, ruthlessly refactoring should be a pleasure. truth be told, i don't enjoy it as much as building new features. nonetheless, everyone else—including my future self—should find things more consistent.

things i did include replacing ecr templates with slang templates, using form helpers to make forms behave more consistently and to reduce boilerplate, etc. etc. etc. i also moved from turbolinks to turbo, which allowed me to remove my custom code for making updates to parts of a page. i know it's not cool but i'm a fan of html over the wire. i also fixed/improved timeline handling and refactored population/management of timelines and notifications. i really need to refactor handling of all side-effects—i just found another place in the code where out-of-order messages result in duplicates in the inbox.

(interesting aside about that... it's possible to get a forwarded create activity from a source before receiving the create activity from the origin itself. it's not surprising in retrospect. the origin was busy sending to followers and i was in the queue. meanwhile, another follower who was mentioned in the related object broadcast the activity to their followers...)

#ktistec

i took time off this summer to enjoy the analog world and play board games, but i did make some small progress in a few areas.

commits 31260ad through d6b35cb fix various bugs and performance problems and improve the user experience:

  • site setup is more user-friendly
  • social details (like and announce) are visible on posts
  • clicking anywhere on a post in a feed navigates to the post
  • links open in new tabs by default

there's a bunch of work under the hood, as well—mostly deferred refactoring and cleanup.

in parallel, i'm still working on metrics collection and charts. here's a teaser:

charts!

#ktistec

i’m patiently working my way toward a 1.0 release of the ktistec server. the 1.0 release must have a stable database schema—that’s the one hard requirement. in terms of features, it’s close now. i’m adding internal metrics for the power user and block lists. then it ships!

#ktistec

and commits eb740e1 through 091a1b1 implement a materialized timeline. it replaces a large and complex query for dynamically generating the timeline from activities. the query worked well enough but was slow—over 200ms for a single page in a timeline containing over 4000 objects. the replacement is 3x better, but still builds a b-tree to get the sorted results. the solution, i suspect, is to  remove the final two joins, at which point the existing index on created_at will be used to get the sorted rows. to do that, i'll need to remove deleted actors and objects when they are deleted...

#ktistec

ktistec commits 4a32680 to 9b68508 add support for notifications—mentions, announcements/shares/boosts, likes and follows all create notifications for easy alerting and management.

#ktistec