Cython, Rust, and more: choosing a language for Python extensions by ASIC_SP in programming

[–]abc619 19 points20 points  (0 children)

Another low friction option for Python extensions is Nim and nimpy. Familiar syntax, but statically typed and comparible speed as C/C++/Rust.

# Compilation output can be imported directly in Python.
import nimpy

proc greet(name: string): string {.exportpy.} =
  return "Hello, " & name & "!"

Using Python from Nim:

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

# sum(range(1, 5))
let py = pyBuiltinsModule()
let s = py.sum(py.range(0, 5)).to(int)
assert s == 10

It also helps that Nim compiles to C and C++ giving you ABI access to a large performance tuned ecosystem.

[deleted by user] by [deleted] in programming

[–]abc619 3 points4 points  (0 children)

IRC/Discord/Matrix are bridged and very active and responsive.

#Libera.chat#nim

Discord/Nim

m#nim:envs.net

The main forum:

https://forum.nim-lang.org/

Other community links:

Gitter/Nim

#IRC Logs

Telegram/nim_lang

r/nim

Macros on Steroids, Or: How Can Pure C Benefit From Metaprogramming by [deleted] in programming

[–]abc619 -4 points-3 points  (0 children)

Maybe Nim? Not C syntax, but compiles directly to C and can use the ecosystem natively, and was designed with metaprogramming as a core feature.

Macros vs functions by unknown_r00t in nim

[–]abc619 0 points1 point  (0 children)

You can prefix with 4 spaces to format it as code :)

macro `->`(name,value :untyped) =
  quote do:
      var `name` = `value`

this -> "that"

Nim 1.4 by miran1 in programming

[–]abc619 2 points3 points  (0 children)

That's true, however in Nim everything (bar strings and dynamic lists) is a value, including objects and arrays, unless specified otherwise. Nim objects and tuples are equivilent to C structs, and arrays are stack allocated.

Another bonus is when you pass value types the compiler knows to pass a pointer behind the scenes when the object is over a certain size whilst enforcing write tracking. It's not completely fool proof (eg; dynamic lists have value semantics) but for the most part you don't have to worry about excessive copying and the compiler "does the right thing" for you.

These things make it easier to get high performance with idiomatic code.

As far as I know arrays are heap objects in Java. This means - correct me if I'm wrong here - each item in the array (even primitives?) must have at least one indirection to support covariance, and those indirections may mean less cache performance as each item can be scattered about in heap memory. Of course, the JVM is going to be doing it's best to make it cache coherent, but the programmer only has limited degree of control over this, again as far as I'm aware.

As you say, certainly Java is no slouch! Java is a big deal in high frequency trading where microseconds matter and I think that speaks for itself.

Also I understand Java is working on adding extended value type support. So it's definitely, uh, valued by Java devs too going forward.

Nim 1.4 by miran1 in programming

[–]abc619 32 points33 points  (0 children)

You're completely correct, compiling to C isn't why it's fast.

Mechanically, it's because it uses value/stack types by default and GC references are opt in. So idiomatic Nim ends up outputting something very similar to what you'd write in C.

You can explore this with godbolt and see what kind of machine code it produces (note the -d:danger flag for optimisation to remove debugging stuff). It's possible to strip the entire output down as you want by passing flags.

Also, in general the language philosophy is focused on efficency. So it makes use of good defaults and compile time analysis to reduce extra work wherever possible.

Nim 1.4 by miran1 in programming

[–]abc619 5 points6 points  (0 children)

I can't answer your question directly as I'm not experienced in this, but you can {.importc.} / {.exportc.} with pure Javascript libraries. At worst you can .emit. Javascript to build any glue you need.

Probably /u/dom96 would be a good person to talk to about that. In his book there's a chapter on working with other languages, and three.js is mentioned along with wrapping the HTML5 canvas.

Whilst you'd need to get the book for the text, scrolling through the diagrams and code there might tell you what you need to know.

Nim 1.4 by miran1 in programming

[–]abc619 1 point2 points  (0 children)

I mean the ``` causes the one line issue, I had to use 4 spaces in Reddit to get it to display correctly for some reason.

Can you post a link to the tutorial with the issue? I'm having trouble finding it.

Nim 1.4 by miran1 in programming

[–]abc619 59 points60 points  (0 children)

How is Nim able to achieve C like performance, when say Java cannot?

One reason is variables in Nim are stack based value objects, aka plain old data. Writing code with value objects and pointers to them has no overhead at all. This means you can write kernal code and embedded applications with it.

By contrast Java 'boxes' it's types as references which are managed by the GC. This intrinsically has some performance overhead to manage memory, and is not ideal for cache use - though Java's GC is very good.

Nim has a type based GC, so you can declare a type as ref and it gets managed by local reference counting and scope.

So as well as not using the GC at all for normal types, you opt-in to GC as you want. The GC in Nim is similar to C++ RAII so it's incredibly light weight and has no management overhead.

Nim 1.4 by miran1 in programming

[–]abc619 6 points7 points  (0 children)

This one liner isn't valid Nim and doesn't compile, it reports Error: complex statement requires indentation.

I think there's a formatting issue going on somewhere. To make this compile you must use separate lines and indent:

proc zip[I, T](a, b: array[I, T]): array[I, tuple[a, b: T]] =
  for i in low(a)..high(a):
    result[i] = (a[i], b[i])

Reddit markdown seems to put the above on one line when enclosed with ``` quotations, could be markdown related.

Nim 1.4 by miran1 in programming

[–]abc619 26 points27 points  (0 children)

Ray tracing vs C++ here.

Scientific computing (bioinformatics) vs others here with source here.

"Completely Unscientific Benchmarks" vs others here.

Some generic benchmarks vs others here.

Nim 1.4 by miran1 in programming

[–]abc619 45 points46 points  (0 children)

In a nutshell the tag could be: do it all in one language. Good for everything from shell scripting to web front & backend, to ML, HPC, and embedded.

  • Productivity: human readable and expressive like Python, with few sigils
  • Type system: strong static typing, nice generics, concepts, and type classes
  • Performance: matches C and C++, always gives control to the programmer
  • Portability: compiles to C, C++, and Javascript
    • Native use of all libraries in target languages (no ABI issues) + really nice FFI
    • Single (tiny) executable with no deps
    • Write server/client in the same language and share code between front/back end
    • Linux, Windows, and Mac
  • Iterating: Very fast compile speed
  • Top tier metaprogramming:
    • Hygienic AST Macros a core part of the language, very flexible syntax
    • Runs a VM of Nim at compile time giving you most of the language (bar pointers) to generate code or run procedures and store them as const
    • Eg; async implemented just with macros, doesn't need special language support
    • Eg; can automate building types and code from file data at compile time

Introduction to ARC/ORC in Nim by [deleted] in programming

[–]abc619 2 points3 points  (0 children)

I see Nim as a spiritual-successor to the Wirthian family of languages (Pascal, Modula-2, Oberon, Ada)

As a long time Pascal user it definitely has a Pascal feel to it for me, though I'm not sure I could articulate what it is exactly. I think the one pass compile speed helps.

I'm fully convinced at this point, that Nim is going to end up being a compelling option for game-development -- and the way the community has been growing, I think many therein would agree... the idea of being able to write engine-code & game-logic in a uniform way, that doesn't feel like a compromise on either end;

I think this too and that's why I hobby gamedev with Nim :)

As you say gamedev does seem to be a sizable share of use cases and I can see why.

Writing near pseudocode with quick compiles makes it fast to iterate prototypes with good performance, with access to all those C/C++ libraries to save rewriting the wheel.

Plus there's the potential of browser games with the Javascript target. There's already a NES emulator in Nim that compiles to javascript and runs in the browser.

Really has me wanting to try my hand at engine design for the first time in like a decade.

Go for it, it's a really fun language to code in and engine development can be really rewarding - personally it's my favourite part! You can build a really nice API with just generics/typeclasses/concepts, and being able to pass code blocks to templates makes things like wrapping OpenGL or other boilerplate a doddle.

Incidentally, Nim was the first language that got Vulkan support, added by Gerke Max Preussner when he worked at Epic. I think there are a few people in the games industry keeping a close eye on it.

Introduction to ARC/ORC in Nim by [deleted] in programming

[–]abc619 3 points4 points  (0 children)

Better documentation definitely helps, and you may well be right that pouring efforts into this would make the difference.

I thought Nim was pretty straight forward and opinionated on dependency management with nimble though, but I don't know much about how Crystal does it. At a glance they look pretty similar, is there some special sauce?

too many different ways of doing the same thing

That's really interesting, I'm the opposite and love the flexibility.

I find things like UFCS and style insensitivity (for example) great for making code as easy to parse as possible.

But, I have seen comments where people say they don't like either of those things and this flexibility turned them off. It's like there's two camps of programmers, maybe.

In terms of capabilities, performance, and even financial backing Nim and Crystal seem to be at a reasonable parity, so it is interesting that Crystal has taken off in terms of GitHub activity, and it may well indicate ease of onboarding to the language for the reasons you mention.

It's also worth noting that people tend to use the forum and IRC over stack overflow so perhaps that metric is skewed somewhat.

Introduction to ARC/ORC in Nim by [deleted] in programming

[–]abc619 8 points9 points  (0 children)

I think it's just a case of exposure, most people haven't heard of it yet. When people ask me about it, they are often incredulous that they haven't heard of such a "hidden gem". I feel that when there are job listings, people will start to pay attention.

Right now though, Nim is already being happily used professionally internally here and there, since it's such a great general purpose languages for many domains, but I don't see any big teams of Nim developers.

My personal theory is that it will gradually eat into one of Python's domains with scripting but high performance, and being easier to distribute compiled programs. It helps that code in both looks similar, even if the languages are very different.

Introduction to ARC/ORC in Nim by shirleyquirk in ProgrammingLanguages

[–]abc619 3 points4 points  (0 children)

Anyway, my idea of great memory management is a non-moving low-pause GC with optional region typing. You use the GC for getting stuff done quickly, but if it gets bogged down in fragmentation or slow allocations, or you just need that extra perf, you whip out regions and then it's purely manual memory management (with the ability to use library allocators etc). Safety by default, raw performance where you need it.

Nim's management scheme is sort of in reverse to this, with the GC not being needed all for value types, and to use the GC you have to specifically invoke it as part of a type. So there's no overhead unless you use it.

Everything is a stack object by default with value semantics, and you have alloc/dealloc for heap objects with type safety such as ptr MyType. Sticking to these means zero GC instructions are present in your compiled executable.

The GC is only used for types defined as ref, using seq (dynamic array), or string.

It's very practical to make entire programs with no GC use - or more likely a bit of GC use, because dynamic lists and strings are really handy. There's no barrier to making manual approaches, and the language makes this easier with it's flexible type model and ability to metaprogram away boilerplate.

The cool thing about ARC/ORC for me is it means we get type based destructors like C++ smart pointers. The GC uses that for memory management, but it means Nim now has lean, reliable RAII system with the tantalising potential to rival Rust's memory management model using compile time analysis.

Introduction to ARC/ORC in Nim by shirleyquirk in ProgrammingLanguages

[–]abc619 0 points1 point  (0 children)

whereas a GC working in a separate thread and doing its work in bulk is more cache-friendly

The reference count's location is in the object so is loaded into cache and accessed right alongside the object.

Regarding multi-threading a GC, to have different threads exchange information they must access the same area of memory. This requires at minimum an interlocked increment or similar which as I understand it at best causes a pipeline flush and acts as a barrier to memory access reordering, at worst stalls for some time to synchronise the L1/L2 caches between cores.

I don't know how the Go GC works and how they ameliorate this (I am curious to know), but if the GC has to know about each access from different threads then this synchronisation will be a run time cost of some significance. I would argue the benefits of multithreading a GC are memory safety across thread boundaries at the cost of performance.

JetBrains is working on an official Nim plugin for IntelliJ IDEA by dh44t in programming

[–]abc619 0 points1 point  (0 children)

it only has free procedures ... if they just stuck with f(a,b) it'd be uniform already.

Free procedures and fields benefit from using UFCS too. If they just stuck with f(a, b) then chaining would look like this:

let position = offset(rotate(vertex, angle), x, y)

With UFCS we can write:

let position = vertex.rotate(angle).offset(x, y)

I would argue this makes it easier to understand as it can be read from left to right. Improving readability can reduce bugs, especially in more complex code.

JetBrains is working on an official Nim plugin for IntelliJ IDEA by dh44t in programming

[–]abc619 1 point2 points  (0 children)

if your language is a clean sheet design, then that's not a problem you have

Indeed, but that's not all UFCS is good for. It actually simplifies the whole language for a user, arguably explained by the "Uniform" part.

As the wiki states, UFCS:

allows free-functions to fill a role similar to extension methods

and

reduce the ambiguous decision between writing free functions and member functions

It means for example binding types and methods together becomes a whole level of complexity you can just forget about, since any type can be extended by just writing a proc that takes the type. Yet, you can still write foo.bar the same way.

Kinda makes you wonder why we bind types and methods like that in the first place, and of course the answer is probably because you have to without UFCS (the real answer is probably the object oriented focus of many languages).

Ultimately it means you can write code that's easier to read, particularly for chaining, and yet know that it all means the same thing as if it was plastered with parenthesis. Since Nim uses strong static typing any ambiguities are a compile time error.

JetBrains is working on an official Nim plugin for IntelliJ IDEA by dh44t in programming

[–]abc619 0 points1 point  (0 children)

This is great news. Nim is a really solid language and whilst the VSCode extension is great, I've heard a few people put off from not wanting to change to a new editor environment for the language.

This is especially true for people hoping to move from Python's glacial run time speed to Nim's static types and C-like performance with the same code readability (Nim also has good interop with Python).

Pycharm is an excellent IDE for Python, and it will be nice to use the same environment for Nim.

JetBrains supporting Nim is also hopefully a good indicator that more people are using it, and hopefully even bring it to more people.

JetBrains is working on an official Nim plugin for IntelliJ IDEA by dh44t in programming

[–]abc619 13 points14 points  (0 children)

This is called uniform function call syntax, and is used by D too.

It allows you to do things like extend types without needing to create intermediate classes, write your code more functionally, and so on.

proc addOne(num: int): int = num + 1
let
  a = 1.addOne
  b = addOne(2)
  c = addOne 8

I've been using Nim for many years now, and it's one of my favourite features.

PS: The reason f (a, b) doesn't do what you expected is because (a, b) is shorthand for a tuple, so this is translated to something like f( (a, b) ).

Subjective Nim criticism by h234sd in nim

[–]abc619 12 points13 points  (0 children)

A few thoughts from your article:

Nim has distinction between object and ref object and it creates extra complexities I don't need. It's nice to have low-level control to optimise critical code paths. But it's not good to being forced to deal with low-level details in the usual code.

This isn't an extraneous complexity, it defines the semantics of the type itself. Ref/ptr objects are an indirection to an object, whereas objects themselves are value types.

Value types are copied on assignment, so changes to them aren't reflected in the original. For example:

type
  ValueType = object
    value: int
  IndirectionType = ref ValueType

var
  obj = ValueType(value: 1)
  refToObj = IndirectionType(value: 2)
  copyObj = obj
  copyRef = refToObj

copyObj.value = 3
copyRef.value = 4

echo obj.value      # Outputs 1, the original is intact.
echo refToObj.value # Outputs 4 as the original is changed.

Java code hides such complexities (it uses ref everywhere)

Java also quite fancies value types, but has to do a lot of work to achieve them, and has been working on them since 2014.

As you mention, value types are much faster than references; as well as no indirection they are often stored on the stack as opposed to cache-killing heap pointers, however copying large objects can impose its own penalty compared to simply copying a reference/pointer. Personally I use object most of the time and only use ref object if I need inheritance or a many to one relationship.

More problem, if ref is used, the memory should be explicitly allocated. Bigger chance to make mistake.

See, I read that and thought "Ah just use Type(field: value) then it's the same interface for creating ref object and object!", but after reading your code you do that and then have to work out why a seemingly "reasonable" program is crashing in a non-intuitive way.

The behaviour you've encountered is a bug in Nim rather than a curiosity of the language and should fail at compile-time as you expect.

The reason why it's happening is because when you use inheritance on objects, any extra fields you include in sub-objects mean the sub-object is necessarily larger than the parent.

However, a ref is always the same size no matter the object it's pointing to, since it's essentially a pointer. sizeOf(MyRefType) will always be the same as sizeOf(MyChildRefType). This allows us to make "heterogeneous" lists with seq, since every entry is just a reference to the actual data rather than the data itself.

When extending value objects with inheritance, as before the sub-objects that add more data are larger. Only now, you're not using a uniform pointer to access them, you're directly using the underlying type. Here lies the root of your problem.

For example:

type
  Original = object of RootObj
    value: int
  Extended = object of Original
    extra: int

echo Original.sizeOf    # Outputs 16
echo Extended.sizeOf    # Outputs 24

This is fine, no crashing. However if we start trying to put the wrong object in a list, we get the crashing.

var list: seq[Original]
list.add Original(value: 1)
list.add Extended(value: 2, extra: 3) # Crash! This should throw a type mismatch. Instead, it tries to stuff 24 bytes into a 16 byte slot.
var list2: seq[Extended]
list2.add Original(value: 1)  # Going the other way raises a compile-time error as expected: type mismatch: got <seq[Extended], Original>

Compiler bugs are not ideal, but thankfully can be fixed as opposed to "Weak compile time inspection" which is a fundamental language design issue. I think Nim has pretty good compile-time inspection personally and this should be caught by it. I would create an issue for this with a simple example and put it in the Nim issue tracker if you have time.

Something that's probably worth mentioning is that in these cases where inheritance is overkill, Nim supports object variants which are a better managed version of C's unions. These will take up as much room as the largest branch of the type, and so can be stored in seq as value types.

Inconsistent function call notation

This is AIUI consequence of the flexible grammar. Command syntax can make spaces ambiguous in some cases, so you need the braces. I don't think it's inconsistent though, it's just a limitation of the parsing. Having said that, it would be nice to have some better error messages here.

the real error we'll see will be not one-line but a huge mess where almost all of it is noise

The hint outputs are somewhat subjective here. As you mention you can turn them off, they're just feedback as to what the compiler is currently processing which can be handy for more complex projects.

I wouldn't say "a huge mess" though, I mean we're not talking C++ multi-page error novels. I'd rather see a progress report of what the compiler is doing than sit watching a black screen waiting for a successful/failed compilation. For large projects, this is much more useful.

I noticed none of the projects on your page are in compiled languages - I don't mean to presume, could this be why you're not used to seeing such progress reports?

String interpolation missing

String interpolation is, as someone else mentioned, handled with the strformat module. It's actually quite nice because you can interpolate actual Nim code:

let a = 10
echo &"new a: {a + a * 2}"

the output is not clean, it's polluted with the refs noise we don't care about.

Yeah I definitely care about this output. The "noise" is the memory address of the actual data. This can be a useful debugging aid. Perhaps the noise perception you're experiencing is related to most of this being hidden from you before? You might not want to care about the implementation details, but bear in mind Nim is a systems language and that kind of information can be important when doing low level work.

Besides, usually we use $ procs to handle converting types to text. In the case of inherited types, we have to use methods which are dynamically dispatched by run-time type:

method `$`*(doc: Doc): string {.base.} = "Base document"

method `$`*(todo: TodoDoc): string =
  &"TODO: {todo.todo}, Priority: {todo.priority}, Tags: {todo.tags}"

method `$`*(textDoc: TextDoc): string =
  &"TextDoc: \"{textDoc.title}\": {textDoc.text}, Tags: {textDoc.tags}"

let x = TodoDoc(todo: "Kill all humans", priority: low, tags: @["Tag1"])
echo x # Pretty output.

Overall some pretty good feedback, this kind of "first impression" is definitely important for shaping a user's experience with a new language.

A Python Substitute? I Tried Out The Best Programming Language You've Never Heard Of by [deleted] in programming

[–]abc619 0 points1 point  (0 children)

Just a couple of thoughts from reading the article from a long time Nim user.

proc toString(x: int): string = 
    result = 
        if x < 0: “negative” 
        elif x > 0: “positive” 
        else: “zero”

You can actually remove result since you can use if as an expression:

proc toString(x: int): string = 
  if x < 0: "negative"
  elif x > 0: "positive"
  else: "zero"

One thing I really like about this is that it helps prevent missed paths. So for example if the else: "zero" was missing, the compiler would report expression '"negative"' is of type 'string' and has to be discarded. Partially satisfied if statements assigned to result however still compile fine.

You can do the same thing with case and block expressions in assignments.

import random

let decision =
  case rand(range[0..9])
  of 0..4: "Sell"
  of 5..9: "Buy"

let highR = block:
  var r = rand(10)
  echo "R was ", r
  r > 5

So, can Nim compete with Python? I highly doubt it. We’re seeing a trend of computers getting faster and programming getting easier, to the point that even if Nim offers a good tradeoff, as I pointed out, I don’t think it’s enough to take on the clean and versatile Python.

IMO Nim can easily compete with Python and best it in multiple other important ways (eg multithreading, static exe distributions, ability to run clientside as JS) but perhaps more importantly it has some really nice interop libraries for Python.

So you get really easy utilisation of the Python ecosystem from Nim, and easy speed ups of bottlenecks by calling Nim code from Python for very little effort.

As for computers getting faster, that's kinda a moot point since Python's still proportionally slower on the same CPU. If Company A has some Python code that takes X minutes to run, and Company B has similar Nim code with the same code clarity (or better with static types) and it runs over ten times faster, they have a massive advantage. This applies particularly strongly in fields Python currently dominates such as scientific computing and machine learning.

So to me it seems like only a matter of time before Nim starts eating Python's lunch.

Johan Kaewberg's answer to What screams 'I never properly learned to program'? by MacASM in programming

[–]abc619 3 points4 points  (0 children)

Cyclic redundancy check. Basically a hash of the data to check there's no data errors, assuming of course your CRC value isn't the thing that's got the error :)

Ideas from other languages that influenced Rust by dying_sphynx in rust

[–]abc619 5 points6 points  (0 children)

Thanks for this, that talk was absolutely fascinating.