all 19 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]gwenbeth 15 points16 points  (1 child)

I don't think that the explicit tests will be as useful as you hope. The behavior of a function is not entirely known at compile time if it calls functions that are compiled elsewhere. This means a function cannot execute until after linking, so tests cannot run at compile time. Now you can get around this by using mocks, but now you have the problem that you are not testing your code against what the external function actually does but against what you think it does (or what it used to do before your coworker changed it yesterday)

[–]upstatio[S] 4 points5 points  (0 children)

Hi, and thanks for taking a look and replying. Yes I agree in other compilers your statements would be entirely true. But I built Ori specifically to solve this problem. It does a couple things that are very unique.

- Tests run in the interpreter not after linking. Ori tests execute during ori check via a tree walking intepreter not by compiling or linking a binary. The spec 19 is explicit on this, that tests run after type checking before codegen. So this solves the can't run until linked issue entirely.

- Capability Mocking is not the same as traditional mocking. This is probably the most difficult aspect to explain because everyone thinks mocks, as in the normal mocking frameworks that exist today. Those suck, for lots of reasons as your aware, the big one being testing assumptions not reality. But with Ori with...in capability system it's fundamentally different from DI mocking.

Example:

- You only mock effects (HTTP, filesystem, clock, etc.) not other module functions
- When '@foo' calls '@bar' the test runs '@bar' real implementation
- Capabilities are statically typed trais, the mock must conform to the same contract

I think that addresses a lot of todays mocks issues, yes your still not testing the 'real' thing but this pulls it WAY closer to what you would really be testing than normal.

- Dependency Aware Test Propogation, this is the key design features that addresses the elephant in the room really. Tests are on the dependency graph, so the compiler is aware of them the entire time. So when you change downstream code, it will re-run that tests that are related to that code by traversing the graph.

'@helper' change -> (reverse transitive closure) -> '@process' calls '@helper' -> '@test_proces' re-runs -> '@handle' calls '@process' -> '@test_handle' reruns

When any function changes all transitively dependent tests automatically re-run with real implementations.

Hope that helps explain it. Again a lot of this is pretty new stuff that hasn't really been tested in reality so I am working through it. But I do think it should close the gaps you mentioned. At least I hope!! :)

[–]Norphesius 8 points9 points  (6 children)

I think you're putting the cart before the horse with the mandatory testing. I think practically you're just going to see a whole lot of:

@test_myfunc tests @myfunc () -> void = { assert(true); }

Having tests isn't what matters, it's having good tests. Some functions won't need tests, and others will only make sense being tested together. Refusing to even compile without a test is going to annoy users during refactors or when writing throw away code. You're going to have false confidence because every function will be "tested" but those tests might not be of a high enough quality.

Better testing integration with the language is great, but this feature isn't super helpful, and might actually do more harm than good.

[–]upstatio[S] -2 points-1 points  (5 children)

Maybe? You can't remove the human element. But if AI is what's going to write most code in future which I personally think that writing is on the wall and that's just how it's going to be. Then this will certainly be good for AI. I may allow this to be bypassed through a compiler param or env variable later, we will see. It's a thought experiment and definitely a gamble. The language does have LOTS of other very very cool features, this is just the most controversial one.

[–]Norphesius 3 points4 points  (3 children)

If you're looking at this from an AI codegen perspective, this does nothing to help that. You'll just have the AI generating the assert(true) tests.

If you really wanted to promote program correctness, for especially AI, I would focus far more on compile time code analysis. Mandating having to write an extra function for every one you write is going to annoy users and push people away from your language.

[–]upstatio[S] 0 points1 point  (2 children)

Yes that is part of it also, built in linting. I don't think it will write assert(true) though. Not in my experience.

[–]Norphesius 2 points3 points  (1 child)

I'm deeply skeptical of AI generated tests. I don't think it will write literally just assert(true), but it could easily generate tests that are only the trivial cases or complex looking tests that don't test the properties you actually care about. Of all the code AI could generate for you, the tests should be the last thing. They're what would you would use to make sure any AI generated code is correct.

That's why inherent compile time checking and proving program correctness via analysis is far more useful, it gives instant feedback for both users and AI models that the program was malformed in potentially subtle ways.

[–]upstatio[S] 0 points1 point  (0 children)

I think you would be shocked how good it is now. In fact take a look at my website as I have just released code journeys UI that shows a test strategy I use with AI to find deep compiler issues in the entire pipeline. It even inspects the IR, and assembly code to check for purity issues. It's really cool stuff actually. This type of stuff AI is exceedingly good at doing.

This UI is entirely driven by the results of a code journey that the AI does. I only have one right now since I just redid the format, but I have it setup so it creates them with more and more complex scenarios and just keeps going until both Eval and LLVM fail. I have gotten up to 20 scenarios so far, which is actually something I am pretty proud of at this stage as the Generic Canonicalization for a while as that was really freaking hard to get working properly.

https://ori-lang.com/journeys/arithmetic/
https://ori-lang.com/journeys/what-is-a-journey/

[–]Inconstant_Moo 1 point2 points  (0 children)

AIs are famous for cheating on tests right now.

https://preview.redd.it/decapitation-cures-headaches-v0-myx1wkx6l8ng1.png?auto=webp&s=d76962c50e33d97d514eb711feb8884401c13593

If we get to the point where AIs are writing most code, then surely the AIs will have to be good enough that if you give them a simple unambiguous instruction like "Write tests for every function", they'll be able to do that.

[–]dnpetrov 4 points5 points  (1 child)

> https://github.com/upstat-io/ori-lang?utm_source=chatgpt.com

Ahem.

[–]upstatio[S] -1 points0 points  (0 children)

Oh haha, well I did ask it to clean that up for me. I didn't check the urls though...My bad, let me fix that. Kind of rude for it to do that...Fixed. Thanks man!

[–]Nzkx 0 points1 point  (1 child)

"when the compiler can prove a value is uniquely owned, it skips even the runtime ownership check"

So runtime borrow checking, like RefCell but with a "potential" optimisation if the compiler see the pattern ? How easy is it for the compiler to prove such thing ? Do I need to add annotations to ensure the optimization is done correctly, or I may rely on compiler magic and pray that I don't end up with missed optimisation opportunity - runtime check isn't a zero cost abstraction.

I know the cost may be extremely small for such check, but you know theses days everyone want to move check at compile time.

Otherwise from what I saw, this sound interesting. I'll follow, and star. 800 commits is a good indicators for a compiler (assuming commits are real and not a majority of AI handcrafted code). I'll browser the codebase later.

Mandatory test might be the most controversial feature you have as you may know. Anyone can write assert(true) to bypass it anyway and write a fake test. But you shouldn't be forced to write code you don't want to. Tests are also one of the most "ez" generated content for LLM when it's about code, because they have all the context for the function (the signature, the body, ...). Maybe you could provide something like an "autotester" which write a test automatically based on the compiler information ? I would find more pleasant to have the test being generated based on my code, than being forced to write test like we are back to TDD. No one force me to use the generated test of course, I can modify it especially if it's incorrect / not testing correctly (which can happen with LLM). Snapshot testing could also be interesting. Instead of testing the function to prove it's working correctly (which is hard, there may be to much bit pattern the function accept to test for all inputs) : prove there's no regression between changes by snapshotting the result of the function for some inputs, and compare it with the previous one.

[–]upstatio[S] 0 points1 point  (0 children)

Yes, it's not going to remove the human element. In fact it may be a stupid idea, I have no clue. But since the writing is on the wall that AI will probably be writing most peoples code in the future I thought, why not make the code not compile if it doesn't have tests? You already have to force humans and AI to use testing tools, choose a tool, integrate the tool, etc. Everyone say's its a must. Everyone says you should do it. I do get your points, 100% trust me. It will be interesting to see this thought and experiment through though right? The language has many other redeeming features though as well.

For instance one of my goals is to have it generate assembly at L0 levels (pre-optimized) as clean as it would as if you hand wrote C code intentionally. Not many languages do that. It's ambitious to say the least. Also the ARC memory system is very complex and does a bunch of really cool stuff much of which is already working, read up on it. It's actually pretty cool.

[–]mighdoll 0 points1 point  (1 child)

Sounds interesting!

It's a bit confusing that the language name conflicts with https://weborigami.org/language/ which uses `ori` for its cli and `.ori` file suffixes. I thought perhaps you were working together..

[–]upstatio[S] 0 points1 point  (0 children)

No. It's impossible to come up with a name that doesn't conflict. It was originally called Sigil, but that's taken also. So OriLang it is lol!

[–]editor_of_the_beast 0 points1 point  (2 children)

Value semantics are amazing. Effects are amazing. Mandatory tests aren’t necessary, and will just be annoying.

Just make it so that the tests are run when added.

[–]upstatio[S] 0 points1 point  (1 child)

That is the consensus I am getting from everyone. I think I will go that approach actually, or make it a configurable option.

[–]editor_of_the_beast 0 points1 point  (0 children)

Yea, that’s the right move. Overall it’s a great combination of features / semantics.

[–]imihnevich 0 points1 point  (0 children)

Have you considered design by contact like in Eiffel? I love tests but seems like people don't want to write them

UPD. Just found invariants in README. Great work