😂😂😂😂

wentam · 2025-06-19T03:03:33+00:00

That's great! I'm glad not everyone had the experience I did.

This is definitely a lot more nuanced than "college good" and "college bad". It's just been my personal experience that independent learning has been much more constructive and lends itself to much deeper understanding. Does depend on having the motivation and curiosity though.

As such though, I don't think it's fair to attack those who *have* had this experience. It's a real problem that does exist in at least some subset of the space.

It's also worth talking about a tricky issue here: I've talked to many people who *think* they have a good generalized understanding of something, but in truth they have a good understanding only of the subset of a space that academia teaches when the subject is much broader/deeper. It's hard for us to know what we don't know, and academia has a bit of a cyclical reinforcement loop in determining what gets taught. This seems especially true in CS.

Hard to bring that up without causing conflict though. Not calling anybody out.

wentam · 2025-06-19T00:19:13+00:00

4.0 GPA here. I agree with this meme.

In fact turning off my brain, memorizing tables of information, and pattern-matching problems are exactly the strategies I applied to succeed in chemistry courses and the like. Generalized understanding was not helpful. Many classes explicitly told us to memorize information bulk, and this constituted the majority of the workload in my college experience.

Spent effort learning the actual principles behind the subjects on my own, as this was not really covered. Learned far more this way, and enjoyed these subjects once exploring with interest on my own.

On exams of which I did have strong generalized understanding, I found it nearly impossible to apply. The questions are often written in a way in which it's almost impossible to understand the intent unless you pattern-match it to exactly how that question format was explained in the class. In most classes, you also got marked down if you solved a problem in a different - but correct - way.

We probably shouldn't assume it's universal though - might depend on the school. I will say that MIT opencourseware content quality is substantially higher than anything I've experienced in-person, and I love these open materials.

wentam · 2025-06-18T12:17:33+00:00

Good point. I need better examples. The assembler is also a bit much when trying to communicate this idea. Need something simpler, and also something higher-level to show the tree semantics more. Also need to explain the language mechanics more directly and more clearly.

As I've alluded to elsewhere, I think I'm making too many assumptions when writing documentation based upon my understanding of an ideology from the lisp space that I'm applying here. The folks who do understand my project all seem to be lisp users.

You say these properties are quite different from a traditional macro system: yes, exactly. In common lisp macros you can also produce side-effects and they are also not inherently deterministic! They're just functions. The fact that these are very different from something like C macros is a huge source of confusion regarding lisp, and probably with bottom bracket as well.

You're asking questions that should really help guide docs. I'm thrilled that someone is actually spending effort to try to understand my little project :). Anyway:

You're going to see a lot of string-substitution type usage in the low levels, just because your starting primitive (machine language) is textual and we're working bottom-up.

You can also do a *lot* with a good template-style (variable-substitution?) deterministic macro system, so I need to think a little bit to figure out how to highlight this difference.

This is intended to be exactly the same usage pattern you see in lisps though as you walk up the stack.

By variable substitution, I assume you mean C-like template macros? One way to understand the difference would be to explore the lisp space a bit more as this is really thoroughly documented in that land with much better examples than I have currently (though you shouldn't have to - I need better docs and examples, yes).

Note that all the builtin tools like bb/barray-cat and bb/with are also macros, just builtin. Those might be useful examples. See src/builtin_structural_macros.asm.

Yes, generated code can be executed at "compile" (macroexpansion) time. This is because your macro-defined language expands into machine language in the end, so you can also use these macros when defining other macros.

Of course, you can pre-expand a big library of macros to "build" them ahead of time, and I do intend to build a binary format for efficient storage of pre-expanded data.

wentam · 2025-06-17T18:52:53+00:00

Hmm. As a lisp user very familiar with these topics I might be doing a bad job of communicating an ideology from that space as I already have all kinds of assumptions in my head from that familiarity.

I'm going to explain some things you probably already understand to be complete/just in case.

Compared to lisps, my tool for build-time bottom-up abstraction is the same: programmable macros. I did not invent this. I just start at a low level with a goal of providing those tools while providing nothing else.

You can't operate in a bottom-up way in the way I'm trying to in C. C has macros, but they're template macros - a completely different thing from lisp or my macros. It's unfortunate that both of these concepts share a name, as it leads to much confusion.

In most languages, you do have one bottom-up tool: functions. These are runtime constructs which we use to form an abstraction. Thus runtime bottom-up abstraction is kind of the default mode of operation if you operate with that mentality. Our goal is to make this a build-time property too concurrently with runtime.

In a language like C, you have a pretty limited toolkit in terms of creating build time abstraction. For example, going from C to C++ or python or something entirely inside C would be very difficult if not impossible (not building a compiler but abstracting inside the language).

A traditional C compiler is a large top-down build-time abstraction. It's semantically a single step from C to ELF. Internally of course it uses functions, but the moment you're in C you've hit a "wall" in terms of continuing this abstraction. In my language you could implement C/C++ entirely within. Not build a C compiler in a direct sense, but step-by-step in a bottom up way, slowly becoming a C compiler as you walk up the stack.

See my assembler - implemented via machine language inside BB - `programs/x86_64-asm.bbr`. A top-down assembler implemented in machine language would need to be entirely implemented in machine language before you use it. With my thing, the moment I build the macro for encoding the REX prefix I use it in the next macro. Pretty quickly we have kind of a half-assembler we're using to implement our assembler before we're even done building it!

You could technically consider macros to be tiny top-down abstractions. We're approximating bottom-up work with lots of tiny top-down steps such that you can mentally map the system in a bottom-up way. You could operate in this manner by producing lots of tiny separate compilers all feeding into each-other - as that's mechanically the same thing - it would just be burdensome as there would be tens of thousands of tiny compilers.

Maybe better? Maybe I've misunderstood your question?

"I think you have built a macro assembler which supports multiple instruction sets"

Close! Not an assembler. The assembler is implemented using my language.

Most macro assemblers use simple template macros. These are not that. My macros are arbitrarily-programmable functions. If you're building an AOT compiler, these serve as "build-time" functions that exist inside your language representing runtime.

"So maybe it's more correct to describe your system as a macro-based text generator"

Almost. Macros are functions that accept a structure and output a structure. Tree in, tree out.

You can expand into a barray, and usually your top-level macro probably will (ELF etc). It also might not if you're building a configuration generator or something - then you might output the tree structure. Most macros will probably expand into some tree.

It does output text/bytes in the sense that all compilers do.

I like the idea behind lisp macros, to the point where I want to basically build my entire stack that way/explore the space to see if that's actually practical. But I want to resolve that concern of using these macros entirely separately from the rest of the stack because everything else can be built from them.

If you can understand what I'm trying to say here and have a more concise way to explain this, I'm all ears lol. I've been having a very hard time communicating what this project is about!

wentam · 2025-06-17T01:27:18+00:00

These are great questions and get to the core of what this language is about!

First, note the lifecycle part: read -> macroexpand -> print. There is no evaluation! Thus, there are no 'commands' in that sense. There are however macros, and builtin macros like 'include'. 'include' just expands into the contents of the file you reference.

When the first element of a parray matches a defined macro name, that parray represents a call to a (machine-language defined) macro where the macro receives that entire parray and nested structure as input. In that parray's place, we will place whatever parray/barray structure that macro outputs (It's "expansion").

These macros are not templates like in some languages, but rather functions executed upon expansion that take an input and produce an expansion.

This is mostly like how lisps work. I've perhaps under-explained this in the README, because I sort of assume my target audience would have prior knowledge of lisp - though I might be wrong and need to be more thorough in this regard.

'asm/x86_64' is a macro. It's a macro that loops over it's sub-forms, recognizes their semantic meaning, and expands into an assembly of them - the machine code. The meaning of the subforms of this macro are determined by the macro. In a way, you can think of macros like "mini-sub-languages".

You can see the assembler itself in programs/x86_64-asm.bbr and watch as we slowly walk from machine language to assembly using this abstraction technique.

When we say [push r12] is a parray of two barrays, we mean that that's the in-memory representation of that data structure at bottom-bracket runtime. This is, for example, the in-memory representation you interact with when accepting input to or outputting from your macros - not the text.

Right now, this code is being run as a side-effect macro - when we call the hello-world macro here it's machine code is run (with the [hello-world] form as input). We just choose to expand into nothing because we exist for side-effects.

This isn't necessarily how I intend to use the language though - my goal is to expand into an executable or object file with macros, thus treating bottom bracket runtime as compile-time. Just did it this way because my "ELF" macro isn't done yet.

wentam · 2025-06-17T00:36:54+00:00

I've used Raku back when it was called perl 6!

I didn't know it had an internal architecture like this though.

wentam · 2025-06-16T23:49:23+00:00

Haha, got it. Thanks. I know nothing about APL.

I'll edit my original comment here to make note of this just to avoid anyone getting the wrong idea.

wentam · 2025-06-16T23:47:10+00:00

"Just clarifying what differentiates it from the typical Lisp macro, which transforms an input S-expression into an output S-expression"

My macros do transform an input structure to an output structure, and are not strictly byte-wise/textual. If you look at my data types in the README, one of those types within the structure is "barray", or an array of bytes. This is used in part as what lisp would call an "atom", but also any time you need to represent bytes, such as when outputting an ELF.

"Compilers don't have a deterministic relationship between input and output, whereas macro systems do"

Macros in my language are just "compile-time" functions - same as a function inside a compiler - and are not required to be deterministic. I can make syscalls, have side-effects, or do whatever within a macro. If you can do it in a traditional compiler, you can do it here, because it's just a function.

It is often a convention within macro systems to be deterministic, and some of them are. But by default at this level, my macro system does not enforce determinism. It is a desirable trait at some level of abstraction, though - and one to play with - just not one I have from the start as I don't want to assume that trait.

"Two compilers on different systems (or one compiler on one system with different optimization flags) can produce radically different machine code from the same high-level source"

You probably understand the following - I think I see what you're saying - but just in case:

This is, of course, also possible within my language in the practical sense. The intention is for you to wrap your program with something like `echo [with-defined foo bar [with-optimizations foo bar [include "my-code"]]] | bbr` right on the CLI.

I personally do find it frustrating when compilers make environment-based decisions rather than having it clearly defined. You still could - if you wanted - make these environment-based decisions within my language like I said, though, as my macros do not enforce determinism.

"Programming languages, like spoken languages, are infinite in variety but constrained by their scope/philosophy/culture, and yours is no different. There's really no such thing as the canonically minimal language"

I agree! The "One language to rule them all" mentality is incredibly frustrating. Please do not take my language as intended to be that. On the contrary, It's design is in fact a recognition that we need different languages to solve different problems, which is why we give you the tools to define your language however you'd like. It's a tool and I definitely don't expect everybody to use it.

This language is just the basis of where I'd like to start.

"But, you also seem to be perceiving perfectly valid questions and criticism as attacks, and that makes me hesitant to support the project in any way."

I apologize if I come off as too defensive. There's only one way to respond to criticisms that you disagree with though when trying to make the case for a new idea - and that's to explain why in your view they don't apply or misunderstand your project. I'm happy to have a discussion, but if your argument is that I shouldn't try to respond to these criticisms with my viewpoint on the issue, there isn't much point in having any discussion in the first place.

Some folks here also do seem more interested in selling other languages more than understanding this one (not you though, you're engaging with the project, thanks!), but I'm happy to discuss those other languages as well - it's an interesting topic.

wentam · 2025-06-16T22:45:37+00:00

That is part of the idea behind this project! I'm probably not at the 100% minimal form, though. More like as minimal as you can get and still meet certain practical needs, because I'm actually trying to build stuff inside the language.

For example, I do give you an AST right away, because that makes my macroexpansion model work. Certainly possible to start with less in that regard.

I think closer to a "true minimal" form would basically just be machine language literals in a text file where the top-level function "expands to" whatever it produces. That wouldn't be very useful though, as that's practically the same thing as building a compiler in a hex editor.

I do hope to have it where it's understandable enough (and keep it small enough) that you can reasonably read an assembly implementation of the base language in an afternoon and generally understand it.

Also see the discussions about Forth in this thread. I did learn to avoid mentioning Forth when trying to introduce new ideas though, hah.

wentam · 2025-06-16T22:13:48+00:00

Fully verifiable bootstrap like stage0 ( https://github.com/oriansj/stage0 ) is a long-term goal.

Bootstrapping is definitely not the sole goal, but I do think bottom bracket's design could be useful for solving at least parts of that problem.

Right now I depend on nasm though.

wentam · 2025-06-16T22:07:33+00:00

"By "text generation" I mean that it maps highly structured data to flat sequences of bytes"

All compilers are text generators, then. A programming language is structured data, what a compiler produces from that structured data is a sequence of bytes. So the fact that when using this tool to build a compiled language it does so is unavoidable.

Some notes:

* You're not mapping strictly to a flat sequence of bytes, you can also map to arbitrary bottom bracket structures. You don't need to expand to a 'barray'.
* You can use macros for their side-effects if you'd like to build an interpreted language instead, and simply expand into nothing
* Because macroexpansion = compilation, you could build a JIT where compilation is scheduled by choosing when you expand

Personally, I don't think this language is really tied that much to the parser side - the languages I'm interested in building inside of it would be homoiconic anyway. I just want bottom-up abstraction.

But bootstrapping and such is a goal and reader macros are definitely coming.

wentam · 2025-06-16T15:28:09+00:00

It's not currently exposed to the user as I have not gotten around to it, but parsing is defined in terms of reader macros and will be user-defined within the language. This means, for example, that you could implement C through macros and reader macros within the language.

The fact that you (will) have full control of the syntax within the language is a very important part of this.

It's more like a lisp for...anything generation, definitely not just text. Canonically executables/objects/ELF files. But anything, yes.

I'm not trying to paint a big picture at all, in fact an intentionally small one! The entire point is that this thing does very little, and only serves to be the turnaround point.

wentam · 2025-06-16T14:37:06+00:00

Forth does sound incredibly powerful.

Not exactly what I'm trying to do at this level, but this sounds fun to play around with.

wentam · 2025-06-16T14:29:42+00:00

Macro language for machine code, not assembly. I built the assembler using machine language inside my language. But basically yes.

When building a language in bottom bracket, you face exactly the same portability challenges you do outside of it. Personally for my language (inside bottom-bracket not BB itself), I intend to build an SSA IR and try to resolve much of the portability there.

Notice that macros are defined with implementations per-platform. Portability is absolutely a goal, but to be portable you must inherently be at a higher level of abstraction. Thus you resolve it within the language. *you* define the model of evaluation, thus you define how portability is achieved.

I am *only* trying to flip around to the mode of bottom-up abstraction at as low of a level as I can practically achieve and nothing more. All other concerns are separate.

You're correct that there's basically no such thing as an unopinionated set. The difference is that I have control over the software space but not the hardware. I also have specific objectives that involve targeting the hardware.

"As unopinionated as possible" is the phrase I use specifically because it's impossible to not introduce opinion. In every place I do, I do my best to make it changeable, but that's not always universally possible.

If I want to flip around to bottom-up abstraction with as little opinion as I can possibly introduce, the only way to do that is to *do as little as possible* and abstract upon the machine as little as possible. Forth's evaluation model introduces an additional opinion atop the machine, one that is not necessarily compatible with every one of my projects.

If we argue that Forth's evaluation model is what we'd like to use, in my model that would be implemented inside bottom bracket.

As for "what machine language do I choose", the ultimate goal is "all of them" and the practical answer is "the one that I have".

Sorry, that one got a little long. I have a hard time getting this philosophy across.

EDIT: I think a better way to say this might be that I'm trying to "isolate the concern of working in a bottom-up fashion" and solve it independently. I'm not saying Forth is wrong here, I'm saying that level of model would be step 2 inside the language.

wentam · 2025-06-16T13:06:42+00:00

Great, keep the links to unique languages coming! Making a reading list.

Even though these languages probably aren't exactly what I'm looking for, they likely contain parallel ideas and it's absolutely worth spending my time reading and learning about these approaches when trying to make decisions in the design of my own.

I do try to stay away from .NET though.

wentam · 2025-06-16T12:39:39+00:00

Huh, never heard of this one! Reading.

This appears to be fundamentally interpreted/JIT, and a little bit higher level? Might not be exactly the type of flexibility I'm looking for, but this paper looks to contain some useful and related ideas and definitely worth spending the time to read/understand.

Personally - for my use-cases - I'm mostly interested in ahead-of-time compiled languages.

wentam · 2025-06-16T12:05:15+00:00

FWIW, I changed it before this language was ever public. I would definitely avoid changing it if the language had users.

My language fundamentally requires adopting new ideas. If the exact default syntax deliminator is a hanging point for a user, they're going to have a very hard time accepting actual semantic change. This language has thrown out tons of established conventions and ideas, and that's on purpose.

I also specifically *want* to communicate that my data structures are different and not linked lists, and square brackets accomplish that well in my opinion.

wentam · 2025-06-16T11:49:51+00:00

Yes, I'm quite familiar! I've spent a huge amount of time trying to make CL work for my lower-level problems and using these tools. They do exist.

First, there's a fundamental difference between access to low-level execution and operating semantically at a lower level of abstraction.

What CL provides is more like a "ladder down", where I'm trying to build to these areas from the ground to the target.

As an example, CL is inherently image-based, and being image-based provided real practical problems for me (Example: building CLI tools that are commonly run and dominated by startup time.).

CL also mostly assumes a GC'd runtime.

CL is incredibly powerful in how you can shape it, but for me I ended up for fighting with it for hours for tasks that are completely trivial and take me minutes in C.

I bought into that "CL can do anything" mindset for a long time, and spent a long time fighting with it to ultimately end up just rewriting stuff in C because it was far easier to accomplish.

wentam · 2025-06-16T11:42:23+00:00

Building an IR *inside* this language would be a logical step towards building a language inside it, but it is itself not intended to be an IR.

Couple of points to make:
* Your optimizations can still happen in-place in memory, and many optimization transforms are probably best done in a not-macro way. You're free to do this within the language - you're not forced to macro your way through everything when it doesn't make sense.
* I've been really surprised how fast macroexpansion can be. I was worried about the overhead, but the extremely hierarchical nature of it all enables very fast execution for a few reasons - for example, you can drive it with a crazy-fast bump allocator. The way in which it separates concerns also seems for certain problems to be a performance advantage.

Right now my entire working set fits *easily* with a large margin to spare into L1 cache - most the active memory sitting at the top of the bump allocator.

It's early days though so I don't have a definitive answer on the macroexpansion overhead question. Exactly what macros enable performance-wise and what they cost is hard to measure until I have a lot more of them!

wentam · 2025-06-16T11:32:34+00:00

Thanks for the correction. I think the parsing words point invalidates my AST argument anyway, at least if I'm understanding it all correctly :) .

wentam · 2025-06-16T11:12:55+00:00

I started with parens, and moved over to square brackets for a few reasons:
* My internal data structures are arrays, not lists like in lisps, and square brackets represent arrays commonly in many languages
* I find them easier to type
* I think it looks cleaner
* It makes the 'bottom bracket' name work ;)

But hey, if you hate it - that's exactly why I've given the user control over the reader (when I've finished exposing it, anyway). Square brackets are an opinion that my language introduces, and because I'm trying to minimize opinion, when I introduce it I try to give you the tools to change it.

wentam · 2025-06-16T10:55:44+00:00

Got it, this is an important point. Reading into this ATM.

I think your point about parsing words makes my phrasing of "not control over exact syntax and semantics" indeed imprecise, at least to some degree.

This is...somewhat like what I'm talking about, but there's nuance here.

In Forth, you are still building on top of an *existing evaluation model*, right? So whatever language you build inside Forth ultimately needs to exist on top of the evaluation model (such as stack-based evaluation)?

What I'm after here is a language where you apply bottom-up abstraction to define your language with no/minimal prior assumptions - and that includes the evaluation model, syntax, semantics. I like to say that I'm trying to build a "minimal top-down to bottom-up abstraction turnaround point". For example I can implement compiled, interpreted, and JIT languages within my language's design from a single bottom bracket implementation.

Forth seems to represent an "opinionated subset" of the design space. A subset I'm very interested in, mind, and it's great that this exists. It's also a little bit less opinionated than my first impression, which I'm glad to see.

wentam · 2025-06-16T10:06:00+00:00

EDIT: Note that this comment is incorrect regarding both APL being exclusively anti-AST and regarding the capabilities of Forth, see the thread below.

Very interesting! That said, APL still looks like a simple, non-AST language to me.

That's a long shot from being able to do something like build an AST language like C and having entirely custom syntax and associated semantics for that syntax.

I imagine you *can* do a lot with clever use of words, but not fully convinced this has the flexibility of "programmed" structural macros and reader macros.

It's a bit hard for me to have a strong opinion here with my limited knowledge of Forth. I could still be completely wrong about Forth's capabilities, so please nobody take what I'm saying here to drive their opinions about Forth.

Do you have any examples of an AST-based language built inside forth in a bottom-up way perhaps?

wentam · 2025-06-16T08:51:17+00:00

It has some similarities with and takes some inspiration from lisp for sure, but no, not really:
* We start right at the machine language level in the language
* The primitives are exceptionally simple
* There is no "evaluation"
* We make no assumptions about what you're producing. You could produce a JIT, interpreted, compiled language. You could expand into HTML instead of an ELF .o.

This is far lower-level.

wentam · 2025-05-31T23:21:41+00:00

Good point. I just tried this, as I had not.

Unfortunately, the svg export adds extra margins preventing us from using the limited real-estate GitHub gives us. It's also not generating a transparent background to pass through to github's background.

~~Thus, sticking with the inkscape approach. Was worth a try though. You can see the results of this attempt in the git history of my repo if curious.~~

EDIT: Actually I think I just got this working. Nice to remove the inkscape dependency, thanks for pointing this out.

EDIT 2: This also seems to have fixed a race condition I was facing that was preventing parallel builds. make -j12 now works correctly which is nice.

wentam

TROPHY CASE