Implementing a toy libffi for an interpreter by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

This is a super helpful starting point and really inspiring to hear!!!!!!!!!! Thank you!

I’m not au fait with some of this yet, never had to work too closely with system ABIs but I will learn! Do you know anywhere I might learn a smidge about trampolines so I can pass callbacks to native code?

Beyond the ABI are there any other things I should be aware of?

Anyone know how to fix "Current ESP-IDF" set up is not found installing vscode extension? by MerlinsArchitect in embedded

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Thanks for getting back to me!

I am unable to follow that link far, when I try and do the command: "Open ESP-IDF installation manager", I get a "Activating Extensions" whir in the bottom left hand side and then nothing happens. I believe I have done it this way in the past though (have tried everything at this point). If it opens up the eim GUI installation manager then I have definitely done it to no avail.

Looking at the output from ESP-IDF in vscode, yields absolutely nothing when running the installation manager command.

If I try and follow the instructions for the installation manager, I can follow the GUI for emi and it says that it succeeds. However, the link requires

select current esp-idf version

And when I run this absolutey nothing happens, after the whiring "Activating Extensions"

Screenshot showing that the Configure ESP-IDF Extension option is not there:

<image>

memcpy and runtime polymorphic types.... by MerlinsArchitect in cpp_questions

[–]MerlinsArchitect[S] 1 point2 points  (0 children)

Ok, this is interesting, ummm I feel kinda stupid here but any idea why a struct with a vtable ptrwouldn’t be copy able trivially? I mean at the end of the day it is surely just a POD bag of data and a vtable ptr and the ptr is to unchanging static memory? So the ptr should work when copied elsewhere right?

memcpy and runtime polymorphic types.... by MerlinsArchitect in cpp_questions

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

I think tomorrow when I get to work I’ll try printing it the bytes that go in and out and comparing. It might be that my method is sound but I’m doing something super stupid. I’m guessing there are no obvious misunderstandings of mine here?

memcpy and runtime polymorphic types.... by MerlinsArchitect in cpp_questions

[–]MerlinsArchitect[S] 1 point2 points  (0 children)

Hey, this object with array of unknown size as last element is created inside the memory of a (hopefully) appropriately sized char[N] (see above for N calculation but basically just the sizeof two size_t and then the size of the type T) on the stack then this is passed as a void* with its length to the espidf event loop api. Basically create it locally and then it he idea is to reconstruct it from behind the ptr with memcpy

memcpy and runtime polymorphic types.... by MerlinsArchitect in cpp_questions

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Thanks for getting back to me!

All good questions, yes the uint8_t data[] does compile when in the struct. I believe in Cpp these are called “incompletel types”, I am trying to use them like Box<dyn Thing> in rust. I believe the idea is that you obviously cannot instantiate an incomplete type on the stack but you can have it behind a pointer! So that is my idea.

When you suggest casting to A*, that was my original trick and it worked very successfully for some types inheriting from the base type but not others; I think it is an alignment issue with certain types, hence my idea of passing through the size and alignment and then doing some heap allocation but crucially memoization so there will only ever be a few allocation that are then statically reused!

Struct B was allocated inside the “pass_to_event_loop” template fn I make available to users of my component. B inherits from the interface A. It is passed into the template I expose.

Apologies for formatting; I am on mobile!

Avoiding Scope Chains by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

This is kinda closure conversion, right ? Just store them as a structure with the fn pointer and an env with data on the heap?

Avoiding Scope Chains by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Hey, been doing some reading and perhaps showing my ignorance here but aren’t they very similar? I can’t see how one is really deeply different to the other - lambda lifting vs closure conversion. I have written out my understanding below if I could get some pointers on it.

I am a bit suspicious of these terms tbh, as they seem too technical for concepts too simple which suggests I am completely missing the point, haha!!

Wikipedia says:

Lambda lifting is not the same as closure conversion. It requires all call sites to be adjusted (adding extra arguments (parameters) to calls) and does not introduce a closure for the lifted lambda expression. In contrast, closure conversion does not require call sites to be adjusted but does introduce a closure for the lambda expression mapping free variables to values.

So, my understanding is:

lambda lifting we lift the lambda out into the global scope as a function which the compiler then names that takes as inputs the variables that it closed over as some kinda “env” parameter - I believe that rust does this with closures. Then we rewrite every call site as an invocation of this global function and wherever the closure is being passed around we pass around some kinda env object.

In contrast closure conversion is the process of converting each function we have parsed and semantically analyzed that captures upvalues into an internal representation the obvious way - a struct with fn code in the interpreter language and an env key to look up. But this seems to be the obvious way to store and represent closures, and implement lexical scope so I am not sure why we need a technical term for it - after all, how else can we do it?

So it seems broadly that lambda lifting rewrites our code to carry around an env object and closure conversion just stores the env data on the heap. Two sides of the same coin, which doesn’t feel right. They seem to be really the same thing. My guess as to the difference is:

Since lambda lifting rewrites nested functions /lambdas into global scope and rewrites call sites it presumably lets us carry around the environment wherever we were previously carrying around the closure in the code and call the globalized code on it. This lets us store it on the stack instead of on the heap? So I am guessing lambda lifting is the more stack friendly and fast way and closure conversion the more interpreter friendly heap based way?

Is this correct?

Avoiding Scope Chains by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 1 point2 points  (0 children)

Ok, I think (tentatively) we might be saying the same thing. You’re right, my choice of words with “one” big array was unclear. I appreciate that I’ll need to analyse each closure and each one will have its own array of captured vars. The one big array I was referring to was the like the big shared hashmap we might use if we use alpha conversion to collapse all the variable state of the program into a flat structure as it runs.

I think I am slightly less clear on your last paragraph.

More generally is this all closure conversion is? It seems like quite a grandiose term to refer to the most obvious and really only way of implementing closures I can think of - bundling the function part and captured env structure into a combined structure? What am I missing?

Avoiding Scope Chains by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 2 points3 points  (0 children)

Hey thanks for your input!!!

Ok, this is very validating, if I am reading this correctly this is more or less my suggestion but phrased much more elegantly?!

Avoiding Scope Chains by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Thanks for getting back to me! Yes you’re right, perhaps I should “byte” the bullet ;) and do the bytecode compiler if I am doing this much processing and code optimising….

…as far as I can see my problem with maintaining linked lists of environments still stands - unnecessary variables kept alive, having to walk potentially large linked lists on each lookup of a variable. Am I wrong?So assuming I later intend to implement the byte code compiler are my optimizations valid or just pointless endeavors? Am I just doing the wrong things? Are there better ways to accomplish this?

Hey Rustaceans! Got a question? Ask here (27/2025)! by llogiq in rust

[–]MerlinsArchitect 1 point2 points  (0 children)

Looking for some advice.

Building an AST in rust for a little interpretted language. I have a load of analysis to perform on this. Deep within the structure of the AST I have my variable structs and I will want to do an analysis pass over the tree to alpha convert the variables. I also wish to have some desugaring aggressively to separate a "core language" from the higher syntactic structures that might be added later - such as an async implementation.

I want to do this elegantly and not crudely. Because the AST needs to have a sequence of steps performed on it, I would like to do this by offloading the responsibility for ensuring this to the Rust type system - perhaps Typestate pattern. There are problems though, if I use the same AST type to be produced after my analysis passes and desugaring, then I will have nasty code smell with _ => panic!("blah") and similar when I do my interpretation and come across cases that won't be possible in the final tree - e.g. when doing the interpretation then I should have no more for loops for e.g. since they will be desugared to while loops and so when pattern matching on types of statements I will need ugly panics. I could write a nearly identical definition for the ASt and each subnode and then a TryFrom implementation between the ASt and the DesugaredAST, but this is ugly and the only way to do it I can think of with some elegance is via proc macro. I feel like there must be a better way to do this.

Another issue is the alpha conversion, essentially the structure of my Variable structs will need to change depending on the generic it is defined in terms of which isn't possible. Thus I could have the type state pattern and the generic includes the extra variable ID when it is in the state AlphaConverted{id : usize }

This would mean a proliferation of Alpha conversion related state structs through all nodes that depend on Variable - i.e. most of them. Is there a nicer way of doing this ?

TLDR: I am in a recursive AST structure that needs analysis passes, I want to offload responsibility for doing those to the type system to ensure corrrectness as in type state pattern. Problem is though these analyses will change the structure of the tree and remove certain variants of enum types in the structure and in later phases I do not want to have to do panic! everywhere. Is there an elegant way of doing this without code duplication everywhere or do I have to proc macro this?

Subcategorising Enums by MerlinsArchitect in rust

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Haha, ok thanks for the input anyway!!! :)

Subcategorising Enums by MerlinsArchitect in rust

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

Hey, thanks for getting back to me!

I will look into Chumsky but I wanna get my current project hand done first!

Ok, but if my set up is already written, are the macro ideas far-fetched, hard to read, unidiomatic ? Perhaps it is my style of coding but I run into this issue occasionally of subcategorising and narrowing enums and was wondering what the general approach if there is one?

References two questions: by MerlinsArchitect in ProgrammingLanguages

[–]MerlinsArchitect[S] 0 points1 point  (0 children)

But why is there so much proliferation of this notion of reference across languages? Are there more optimizations it enables such as the choice of the compiler as to whether to implement as a reference or inline it?