Wheel Reinvention Jam

August 15 - 21. Happening now.
Learn more

We are a community of programmers producing quality software through deeper understanding.

Originally inspired by Casey Muratori's Handmade Hero, we have grown into a thriving community focused on building truly high-quality software. We're not low-level in the typical sense. Instead we realize that to write great software, you need to understand things on a deeper level.

Modern software is a mess. The status quo needs to change. But we're optimistic that we can change it.

Latest News

Ryan Fleury

Handmade Network is different from other software development communities. We promote philosophies and projects that care deeply about software craftsmanship. We criticize common dogma within the software industry that has produced a computing world that is far too sluggish, too bloated, too poorly-designed, and too unwilling to change. We stand for a better computing world.

In the past 4 years, we've done some big things with Handmade Network. The community has grown dramatically, with thousands of (active!) users on the Discord server (where a bulk of Handmade communication happens). We started a podcast. We've had multiple Handmade-themed jams. We transformed the Handmade Network website into a larger repository of Handmade projects and showcase content, to serve as a bastion of Handmade ideas, and concretely demonstrate how Handmade can improve us as builders of software.

Back in 2018, Handmade Network began a new chapter. I took over for Abner Coimbre as the lead of Handmade Network. I was joined by Ben Visness and Asaf Gartner, and together we replaced the original Handmade Network admin team. At that time, Handmade Network had established itself as a large and self sustaining community. It was the new team's job to continue to foster the community, and to take it to new places. The original team was ready to move on to bigger and better things, and they entrusted the keys to the community with myself, Ben, and Asaf.

Today, Handmade Network begins another new chapter. I have decided to step down from my role as Handmade Network lead, with Ben Visness taking my place. Ben has been a great staff member since the beginning. He led an initiative to rewrite and dramatically change the Handmade Network website into what it is today (writing blogs is a lot easier now). He and Asaf are responsible for all of the technology that allows the Handmade Network website to collect amazing showcase material from the Discord server, allowing the website to serve as both personal and project content repositories (in addition to the functionality that the website offered before). Handmade Network would simply not be what it is without Ben, and I know that moving forward, he'll do an outstanding job of leading the community.

I love being a part of Handmade Network, and I will remain a member. It has been an incredible resource to have as I've grown up. I joined Handmade Network in high school as a young and naive programmer, and the community was instrumental in transforming me into a much more capable and responsible engineer. I've decided to step down because I believe that it will be best for both myself and the community. I have found myself preoccupied with work, life, and some of my other personal projects and initiatives. For this reason, I think it will be most productive - not only for me personally, but for the computing world - for me to give undivided attention to those things, and hand the reins over to Ben to ensure that Handmade Network can be given the time and attention it deserves. Ben will be a strong force in continuing to grow, foster, and shape the community, and increasing its impact as a force for change in software.

It has been a thrill being a staff member, and I can't wait to see what the community will do next. Let's all keep going. Let's continue to build new projects, publish new educational material, do new research, and rethink old assumptions. Through all of those efforts, we can change the computing world into something a little better than what it was before.

-Ryan

Around the Network

Jason
synthnostate
synthnostate
Forum reply: RemedyBG 0.3.8.0
x13pixels
Macoy Madson

This post is mirrored on my blog.

It has been a while since I felt the need to add features and make some more significant changes to Cakelisp.

Two came in this past month: defer and CRC builds.

defer

The defer feature is one I had been wanting for a while, but was unsure about how I wanted to implement it cleanly.

Here's an example usage of defer:

(defun main (&return int)
  (var file (* FILE) (fopen "File.txt", "rb"))
  (unless file
    (return 1))
  (defer (fclose file))
  ;; Do file operations...
  (return 0))

By putting defer there, I am guaranteed to have the file closed if the function ever returns. This removes the need to copy-paste (fclose file) before every return, which can be very cumbersome. It also makes the program more reliable, because I might forget to paste the fclose.

This feature can be found in many other new languages, including Zig and Go. It is a simple way to have some automatic actions without needing to add C++-style constructors and destructors, which can be quite complicated.

In Cakelisp, macros and compile-time code modification make it tricky to know when the code is the final state that will be compiled.

For defer, I needed to know two things:

  • The commands that should be deferred, which can be specified in many separate blocks
  • Everywhere a scope exit occurs, so that the commands can be executed before exit

Scopes are sequences of code that will always be executed together. An if clause can have a scope executed if the condition is true, or (optionally) one that should be executed when the condition is false. Loop constructs like C's for and while enter and exit the loop body scope on each iteration. Finally, functions themselves constitute a function-body scope.

How Cakelisp code generation works

In Cakelisp, code generation happens through either a macro or a generator. Macros output tokens. There are only four kinds of tokens: strings, like "Hello, world!"; symbols, like defun; open parenthesis; and close parenthesis. Cakelisp macros can run arbitrary code, including custom validation, creating and setting compile-time variables, etc. I have written about macros many times here.

This extremely restricted world makes it simple to write the "evaluator": When the evaluator encounters an open parenthesis token, it expects the next token to be a symbol. If it isn't, it's a syntax error, otherwise, look up the symbol in the evaluator's known list of macros and generators, by name. If one is found, evaluate it immediately. If it isn't found, create a "reference" which we will hope to eventually resolve.

Generators output C or C++ code in the form of "string operations". These operations have various different flags such as "double quote" or "newline after", which are processed by the writer. The writer simply goes operation by operation, following its flags and outputting text into a file as requested.

How defer was implemented

defer consisted of three major parts.

First, the defer statement itself was implemented as a generator. The generator outputs the body of the defer into a splice.

Splices are special string operations that say, "output the array of string operations at this address". Splices accomplish a few things:

  • They create "holes" that can be later filled. This is used by invocations where Cakelisp doesn't yet know whether you are trying to call a C function or a macro/generator that has not been defined yet. Cakelisp will generate everything in that state as if it were a C function call, then if the macro/generator is later defined, it will clear the splice's operations and replace it with the macro/generator output.
  • They make it possible to change the output later. This enables code modification, which is when a function has already finished being generated, then a second pass is done at compile-time which rewrites that function with modifications. For example, GameLib has a compile-time function which rewrites every Cakelisp function to add performance profiling instrumentation.
  • They create a place to stow code for other operations. This is how defer uses them.

The defer generator outputs a single splice string operation with a flag telling the writer that it should output the contents of that splice on every scope exit.

Second, I needed to mark all the places where scopes enter and exit. I was worried this would be complex, but it turned out simpler than I expected. I had to audit all existing control flow generators (if, cond, return, break, continue, while, for, etc.) and mark up their Open and Close operations as scope-entering and scope-exiting operations. return, break, and continue statements needed special markings.

Third, the writer needed to have a stack of scopes as well as discovered defer splices. When a scope enter operation is encountered, it adds a scope to the stack. When a defer is encountered, it adds a pointer to its splice to the current scope on the stack.

Scope exits are when the defer statements need to be output. The writer has three different ways to handle scope exits:

  • If the exit is "natural", e.g. the end of an if true block is reached, the writer simply outputs all defer splices in the current scope before the if block's closing bracket.
  • If the exit is from a return, the writer must output the defer splices for all scopes currently on the stack, because return exits all scopes.
  • If the exit is from a continue or break, the writer outputs all defer splices on all scopes until it hits a "continue breakable scope", which is the start of a for or while.

Finally, the writer pops the most recently entered scope off the stack to finish the exit.

One subtle detail is that the writer always outputs separate defer splices in reverse order within the scope. This ensures that the first defer is always the last to be executed, in case subsequent defers are dependent on it.

defer did make the writer more complex, but not significantly. I implemented it in the writer because I didn't want to add an extra evaluator stage; as implemented, defer is very inexpensive in terms of performance during compile-time.

It is limited in that there is no compile-time place where the user could analyze the final code after defer has been applied, then make changes to it. This is because it happens in the writing stage, which is after any compile-time code generation or modification can occur. I will keep it implemented as is until I find I need to do that, in which case it will need to be moved into an evaluator stage.

CRC builds

My work on distributed-automation was disturbed when I had problems with stale builds. I was trying to create an auto-update build for the distributed-automation worker on Windows, but the executable wasn't being updated.

Cakelisp used file modification times to decide whether an "artifact" (an executable, object file, etc.) needed to be rebuilt. If the source (a .c file, header file, etc.) had a file modification time later than the artifact, the artifact is out of date and must be rebuilt.

The problem was that my Windows clock wasn't the correct time--it had drifted into the future.[^1] When I ran a build, all the artifacts were marked as being built at that future time. Once I set the clock to the correct time, no artifacts would be built, because they were already marked as being more recently modified than their source.

This might be obvious to someone who has already written a build system. I knew it was an issue when I wrote the timestamp system, but I figured the clocks were reliable enough that it wouldn't matter.

Now, Cakelisp takes the CRC of every source and header file and records it in a cache. On next build, Cakelisp checks the source files against the recorded CRCs. If they do not match, the artifact is rebuilt. This is slower and more cumbersome than just checking modified times, but is absolutely necessary if the modification times cannot be trusted completely.

I now invalidate artifacts if the CRC is different or the source has a newer timestamp. This lets the user e.g. touch a file without changing its contents to force a rebuild, for whatever reason. I may remove all modification time code in the future, because it's not really providing value past this.

Conclusion

Neither of these features are flashy, but defer is a big quality-of-life feature, and the CRC builds are an important fix for what was an untrustworthy build system.

I don't have anything specific planned for Cakelisp in the near future. I am still following the strategy where I only implement things when I have a pressing need for them, so I can't say what I'll do next.

[^1]: The clock problem consistently happens because I dual-boot Windows and Linux on that machine. The two operating systems don't agree on how the hardware clock should keep time, so I must manually tell Windows to reset the clock to the network time after I've booted. I know I could solve this problem by configuring one or the other, but haven't gotten to it yet. It's good to have solved the problem with timestamps either way, because time in general shouldn't be relied on for this kind of system.

Forum reply: Performance of IMGUI
Terans
Forum reply: Performance of IMGUI
Mārtiņš Možeiko
Forum reply: Performance of IMGUI
Simon Anciaux
Forum reply: Performance of IMGUI
Terans
Forum reply: Performance of IMGUI
Mārtiņš Možeiko
Forum reply: Performance of IMGUI
Benjamin Pedersen
New forum thread: Performance of IMGUI
Terans
Forum reply: RemedyBG 0.3.8.0
daMuzza

Community Showcase

This is a selection of recent work done by community members. Want to participate? Join us on Discord.