#crystallang 11 hashtags

#crystallang developers… what can you count on being safe to access during a call to #finalize?

i have some class-level state that i’d like to modify via a class variable but even reading values via that variable seems to cause problems that eventually lead to crashes later in the program. (typically index out of bounds during some later garbage collection run.)

When you optimize a Crystal program, pay attention to language features that inline code. For example, pay attention to how you use blocks (consequences here and here).

Also pay attention to how you use macros. Macros, like ECR.embed and Slang.embed, inline code at the point where they are invoked. This can be powerful, because macros actually generate code—but, ten invocations later, you have ten copies of the code.

Here's a case of too many copies, but with a very happy ending...

Ktistec uses both ECR.embed and Slang.embed to generate web pages from views and partials. I wrote code to count the number of places Ktistec used embed for each view and partial it renders. There's a long tail, but here are the big ones:

| src/views/layouts/default.html.ecr           | 204 |
| src/views/partials/modals.html.slang         | 204 |
| src/views/partials/header.html.slang         | 204 |
| src/views/partials/footer.html.slang         | 204 |
| src/views/pages/generic.html.slang           | 155 |
| src/views/partials/object/label.html.slang   |  36 |
| src/views/partials/object/content.html.slang |  36 |
| src/views/partials/collection.json.ecr       |  28 |
| src/views/partials/thread.html.slang         |  12 |
| src/views/partials/detail.html.slang         |  12 |
| src/views/partials/object.html.slang         |  12 |
| src/views/partials/actor-panel.html.slang    |  11 |
| src/views/partials/object.json.ecr           |  11 |
| src/views/partials/paginator.html.slang      |  11 |
| src/views/objects/thread.json.ecr            |   8 |
| src/views/partials/activity/label.html.slang |   6 |
| src/views/mentions/index.json.ecr            |   6 |
| src/views/remote_follows/index.json.ecr      |   6 |
| src/views/settings/settings.json.ecr         |   6 |
| src/views/tags/index.json.ecr                |   6 |
| src/views/activities/activity.json.ecr       |   5 |
| src/views/partials/editor.html.slang         |   5 |
| src/views/objects/object.json.ecr            |   5 |
| src/views/actors/remote.json.ecr             |   4 |

The layout is part of every page and is rendered with every view, so lots of copies. Every page has a header and a footer (and some default modal dialogs) so you get those, too. The generic view is a little less obvious. It's used to render pages for which there is no more specific view—typically pages served for 400 Bad Request or 401 Unauthorized. Objects (posts) are rendered in a variety of contexts, so it's no surprise label.html.slang and content.html.slang are popular.

ECR.embed and Slang.embed inline templates at the point where they are invoked, but beyond that they don't really customize the generated code—they just duplicate it. What we want  is one function for each view or partial, which wraps embed and returns JSON or HTML.

Those changes mostly occur in commits from 399287cf to 4b025f50. To say that they made a huge difference is a gross understatement. Executable size decreased by ~13%. Build time decreased by ~50%, and the memory required to build decreased by ~30%.

#ktistec #crystallang #optimization

The Cost of Small Methods

Ktistec uses a template engine for it's views.

View templates are transformed into Crystal code that generates HTML when executed. As you'd expect, the template language allows you to use string interpolation syntax (e.g. #{expression}) for dynamic values.

To ensure expression is only evaluated once, and to limit the scope of the temporary variable holding the evaluated value of expression, I originally bound the value to the variable using Object#tap (commit 5e1bf19e). The generated code looked something like:

(expression).tap do |__value__|
   <template code that uses expression>

Blocks in Crystal are always inlined, so the code above should be equivalent to the following (sacrificing local scope):

__value__ = (expression)
<template code that uses expression>

Functionally, they are equivalent. But operationally, not so much! With Object#tap, the Ktistec executable is about 1% larger (36823603 bytes vs. 36526307 bytes) and build times take 20% longer (23 seconds vs. 19 seconds, generally).

In total, view templates represent about 6% of the Ktistec executable by size, so it doesn't surprise me that there's a measurable impact when I make changes to the template engine, but wow...! I can almost live with the size of the executable, but the build time...!

The cost has to be the method call.

What I'm looking for is something like let in Scheme. The following macro comes close, but doesn't limit scope quite the same way:

macro let(expr, &block)
  {{block.args.first}} = ({{expr}})

I maybe have to live with the macro—I tried to implement let as a method with the annotation @[AlwaysInline] but there was no improvement over the original.

The template engine is a fork of Slang—which I've been evolving to be more Slim-compatible.

#ktistec #crystallang

i've been following hashtags for a while now. i turn off shares (boosts) and replies so they don't appear in my timeline (there's too much sharing going on out there), but then follow a handful of hashtags (like #woodworking and #crystallang and #boardgame) to see more of what i like!


Epiktistes Memory Statistics

I've been tracking epiktistes inbox/outbox traffic and memory statistics (as reported by the Boehm garbage collector) for a while. There's always a consistent increase in both heap size and free memory—to the point where reported free memory is greater than the originally allocated heap—though the difference between the two doesn't appear constant over time. At the moment, heap seems to have plateaued but (pessimistically) I don't expect it to remain flat.

Given relatively flat traffic, the growth in free memory is surprising. I haven't had a chance to investigate, but based on what I understand about Boehm the increase in free memory could be due to increased fragmentation, and the growing difference between heap and free memory could be due to the conservative nature of the garbage collection algorithm.

Or there could be legitimate memory leaks. I did find one, months ago, which was the result of caching SQL prepared statements (in a Hash) and poor practice constructing queries in a couple places, which led to nearly linear growth in cached statements. The difference then was that heap growth was much more consistent, which is not what I see here, now.

#ktistec #crystallang

fdescribe/xdescribe, fcontext/xcontext, fit/xit have really been game changers during tdd. previously, i specified specs to run by line number, which works but requires me to hunt down the line number... i don't know why it took me so long to adopt this—i guess i assumed spectator didn't support them.

#crystallang #spectator

the crystal programming language always inlines blocks, which is great for performance but trades off space for speed. using blocks effectively means keeping this in mind.

somewhere along the line, i learned the habit of passing a block to a function as a means of customizing the behavior of the function. if the function that takes the block is large, it's important to remember that the body of the function is inlined where the function is called, which may not be what you are expecting. if you call the function multiple times, you even get multiple copies.

i just committed code that fixes an egregious example of this problem. in this case these ~30 lines of code replace the blocks with procs (which aren't inlined) and cut ~24mb (that's megabytes) off the executable (over a third of its size).

i regularly shoot myself in the foot trying to be clever, so i don' t know how prevalent this problem is in practice, but it's definitely something to keep in mind, especially if you see compile times and executable sizes growing!

#crystallang #ktistec

...well, for future reference, Invalid memory access (signal 11) at address 0x7ff7b19429f8 likely means the application has overrun its stack, especially if the stack trace shows even moderate recursion.

about the recursion though, this error happened after only 5 recursive calls (not 50,000) and that caught me off guard. i've been recently conditioned to think the stack only holds primitive values and references (pointers) but crystal, like c and similar languages, lets you pass around structures (instances of struct) by value and those values can be arbitrarily large.

now i need to figure out how to debug the contents of the stack...


til... the following works in #crystallang

[{1}, {2}, {3}].sort_by(&.first.-) # => [{3}, {2}, {1}]

because - is a method that returns the negative value of the number on which it is called. (the tuple in the example is obviously contrived...) this is in contrast to the following, which do the same thing but are more verbose:

puts [{1}, {2}, {3}].sort_by { |t| -t.first } 
puts [{1}, {2}, {3}].sort_by(&.first).reverse