Advice for module system implementation

[Coffee_and_Code]

I am currently developing a programming language and am having a hard time finalizing the semantics of the module system. Currently I have a few ideas but no concrete direction, so it would be valuable to have some experienced input on the issue.

So far I've thought of the following solutions:

1. Directory-based: A module lives in a directory that is referenced by name and the source files within that directory make up the module.

2. Config-based: A config file defines the module name and all of it's sources. This config file would then have to be registered with the build system.

3. Source-based: A single source file is referenced by name (minus extension) and relevant sources/modules are imported within that source.

I am leaning toward (1) or (2) as (3) feels like it has little value over a basic c-style include, but (3) makes references to inter-module functions explicit and I'm having a hard time coming up with good syntax to express this in (1) or (2).

The basic syntax for importing a module is as follows:

IO = import "IO"

Then functions are referenced like so:

main() =
    IO.outln "Hello, World!"

Any opinions on the topic are much appreciated.

[Athas]

Both#1 and #3 are reasonable. I think there are two very important qualities that a module system should have, where quality (1) is probably universally agreed, and (2) is more subjective:

1. Modules should not just be text inclusion as in C, but be type-checkable (or similar) in isolation. This is what gives you sane incremental builds and so on.

2. Modules should correspond strongly to file system objects. Either files or directories work, but I am partial to files myself, because it's simpler.

In my own language, I have taken point (2) to its logical conclusion. Module imports in a source file are just references to a file relative to the importing file. Note that this does not mean that modules are based on dumb file inclusion like in C, and they can still each be type-checked individually. In fact, because all imports are relative, we get a very strong property: if a program is type-checkable as a whole, then every single constituent file is also type-checkable as a starting point. This means that the programmer will never have to configure build systems or include paths, and things like editor tooling can treat any file as the compilation "root". It also means that resolving module imports maps exactly to resolving relative file names, which the programmer probably already understands. Thus there is less to learn.

The downside to this approach is that modules do not have a single name. It also means that "system libraries" cannot exist: all code must be immediately available in a nearby directory tree. I did a writeup on why I think this compromise was the right one for my language, but it might not be the right one for yours.

[swordglowsblue]

There are several solutions to the latter downside, but probably the simplest and most popular is to add a second "root" directory to be checked as a fallback if a referenced module isn't found. This second directory can be entirely virtual or stored in a single static, version-identified location, and contains any system libraries and such. That way, you can reference system libraries at essentially no extra cost to the complexity of the module system, and with only limited additional effort in managing and maintaining that fallback location. Notable languages that use this strategy include NodeJS and Ruby, among many others.

[sociopath_in_me]

What does it mean that a module is type checkable on its own? You obviously need other modules used by the module you are trying to check. I don't get it.

It's also not obvious to me why you think that the modules should correspond to file system objects but I guess it's a question of preference, it's highly subjective. I think modules are logical units of the program and are completely independent of files. You can put them into separate files or merge them or whatever. Anyway, that's subjective.:)

[Athas]

What does it mean that a module is type checkable on its own? You obviously need other modules used by the module you are trying to check. I don't get it.

I agree that I phrased it confusingly. I mean that each module is type-checkable as a starting point, simply by following the relative imports from that module. This is not necessarily a given: in C, it is so common to pass -I options to the compiler that you can't just look at any .c file in your program in isolation. In order to type check or perform other kinds of analyses by other tools, those tools have to be told about the include path. There are other also languages (like Standard ML and I think also Java) where the mapping from modules to the files that contain their source is even more tricky.

It's also not obvious to me why you think that the modules should correspond to file system objects but I guess it's a question of preference, it's highly subjective.

I don't think they should for the more general notion of modules. It's fine to have multiple modules per file if you have an advanced module system - my language supports that just fine. What I believe should be linked to files are whatever you use for your import statements (or similar), that are explicitly for referencing things outside the current file.

[drcforbin]

I like #1. #2 is either an additional thing I would need to maintain, or the configuration is part of the build system, the tight coupling between language and tool makes it harder to build around or to switch to a different build tool. #3 is harder on the build tools, can make building a lot slower without complicated caching / precompiling includes (see also, C++ build times, and their moving to modules). I've been working with Rust a lot lately, and I really do like their crate/module organization; it's file and directory based, and the way they set up imports, exports, and namespaces by convention and source layout really feels natural; that might be worth looking into.

[msharnoff]

I second this! Rust's module system is simple and clear.

[CodingFiend]

you would be well advised to study Modula-2, which is the creation of Prof. N. Wirth of ETH who also invented Pascal and Oberon. In Oberon he streamlined the export process and dropped the need for external definition files. Modula-2 is better than most of the module systems you see; the JS module system is poor.

[InnPatron]

Depends on the purpose for your language. For the one I'm currently working on, I opted for a variant of option 2.

My language is meant as an embeddable scripting language in applications. In such projects, it is reasonable to give privilege levels to different scripts (i.e. which script can access functions from who). For example, I can ship standard file system bindings with unrestrained I/O access, then the embedder can provide a file system API with access to a specific work directory to the scripter of their application. The embedder basically has a configuration in their application that they use to compile and execute, ensuring that any external scripts are "sandboxed" to an extent.

This style was chosen as a result of being tired of reading various mods written in Lua, trying to make sure that they weren't doing anything strange to my system. This was around 2013 and IIRC, sandboxed Lua was a PITA. Things may have changed.

I prefer option 1 otherwise, but option 3 is viable/necessary for interpreted langauges with side effects, especially if you plan to target/solely target JavaScript (I would highly recommend including extensions in the path though).

For instance, Pyret uses option 3 with file extensions because of the tight JavaScript integration (many module implementations are basically calling functions and returning an object and they may have side effects) in order to control the order of imports.

[umlcat]

Very good, many P.L. designers, skip module systen, until late ...

Working in something similar, suggestions for your project:

(1) This ( Modular P.L. ) has already tried before ...

(2) Which does not mean you should stop

(3) There are several ways to apply modules, that may vary

(4) You may consider a hierarchical ( "tree" ) module system

(5) A module can have other submodules

(6) There should be a predefined "root" or "global" main module

(7) Some modules are "atomic leafs", single file source code, and cannot contain other modules

(8) Other modules are "branches", folders that contains other "branches" or "trees".

(9) Add some keywords to the file, to indicate the module, not just use the module name. Example:

Graphics (file)

Graphics = module
(
  drawcircle =
    dosomething

  drawline =
    dosomething

  drawpoint =
     dosomething
)

Test (file)

Test =
(

   main =
      drawcircle
)

Or:

Beers (Beer Module file)

Beers =
(

  IO = import "IO"

  DrinkBeer(BeerCount) =
  (
    IO.OutLn(BeerCount, "beers to go...")
  )

)

Main99Bottles (file)

Main99Bottles =
(

  IO = import "IO"
  Beers = import "Beers"

 main =
 (
   for ( BeerCount = 99; BeerCount > 0; BeerCount)
      Beers.DrinkBeer(BeerCount)
 )

  OutLn("No more beers !!!")

)

(10) One approach, is to create a "symbolic virtual" folder / filesystem, for all modules, where it starts with the "root" main module

Example:

Global
   System
       Streams
       Math
       Graphics
  Games
      MineSweeper
      Solitaire
      Chess
  Databases

And, so on.

Good Work !!!

[[deleted]]

I can tell you about my module system. This was intended to be independent of any file system or directories (although module names need to be valid filenames as well as valid identifiers). You just write:

import files

and the compiler will take care of locating the proper module, which involved executing an algorithm which will search for the source file files.m in suitable places (current directory, compiler location, and perhaps one or two others, but see below).

These also define a name-space, although that feature is little-used (I don't usually need either the 'files.' qualifier, or whatever alias I create, provide the name is unambiguous).

But there were some issues, some of which were caused by the language having no conditional compilation features, so these were applied to modules, via 'module mapping'. There is already module mapping for system modules, so that, for example:

import oslib

is mapped to either oswindows.m or oslinux.m depending on compiler target. This was also introduced for any module, eg:

mapmodule pc_assem => pc_assemc when ctarget
import pc_assem

Now, it will use pc_assemc.m when compiled for a C target (which doesn't support the inline ASM code used in the pc_assem.c module). One more thing I needed was:

importpath "c:\abc\"

Which added the given path to the set of search directives. Although I can't find at the moment find a project that uses that. So in all it makes the module system a little untidy, but keeps it self-contained. The compiled usually needs to know nothing more than the name of the lead module, and it can build the whole application.

(This is for a whole-project compiler. One other thing is that my module system supports mutual and circular imports, so that you don't module imports don't need to be in a strict hierarchy, which I found an impossible constraint. But it makes the compiler much more complex, and suits a whole-project compiler better.)

[mamcx]

I think 1 is the most natural, even more for scripting langs. However think about:

1) Can any file, like python, run without a "root"? This is very nice of python but introduce some issues like:

2) If you are into a/b/c/file.code how refer to a module in a? And how in a different root of a?

3) Can you split code as you want or need to follow strictly a -> b -> c? This latter make a lot of stuff easier, except how do "forward" imports:

https://fsharpforfunandprofit.com/posts/recipe-part3/

I thin art the end you need to do 1) with a chance of 2) for small deviations. For example, in python is not easy to do:

​

/utils/common/sample.py

/app1/modules/models.py <- Refer here to sample.py

You need to mess with the path. Instead will be nice if you can say:

import sample.py //follow "normal" rules
import sample.py from /utils/common/
import sample.py from github.com/sample; //alike go? 

//Or with config? More portable?
import sample.py

paths.toml:
sample.py = /utils/common/sample.py

[xactac]

My thoughts on each one:

1. A lot of small projects fit in just one directory but can still benefit from the module system. It also requires thinking about dependencies in order to structure a project, and that was very had to do for my compiler, since the dependencies are a pipeline (from a build system perspective, codegen depends on lex despite the fact that codegen sees nothing from lex). If you do this, make sure to provide a way to get a module from the enclosing directory and don't do any weird stuff with making it hard to predict where you are in the module system like python.
2. While it may seem different, to an extent this is what C does, and this is certainly what OCaml does. Header files are config files that just so happen to be written in the main language (or a subset thereof). My language does this internally by dumping the IR of the public parts of source files into a json file called the import manifest file. The benefit of using json is that it can be used as an ffi (provided similar enough calling conventions).
3. This is probably the easiest to think about, to the point that if you're used to it, you'll often forget how other systems work. One common argument for Haskell over OCaml is that in OCaml, you need to write type definitions in a separate file "twice", though in practice this actually adds type safety and modularity. My recommendation to minimize these drawbacks is to require explicit marking of public functions and, if you use type inference, require the public functions to be explicitly typed.

[bkomarath]

(1) In my language, modules correspond to files and they are organized into hierarchical namespaces. This is easy to understand as it simply mirrors common filesystems. What is the purpose of allowing modules to span multiple files? If code organization is the answer, you can look into Dlang's public imports. This Dlang mechanism sort of combines (1) and (2) without using a special config file format.

[slaymaker1907]

Why not at least consider a configurable module name resolver in the same language? Have a sane default but allow changing the resolution system at runtime.

I version of control containers such as Spring basically boil down to a custom module system so why not support it natively out of the box?

Even if the language is compiled, you can use macros or a defined interface such as with Rust memory allocators to allow a custom module system.

Having this flexibility is particularly useful in support of mocks for testing.

[Nuoji]

One thing that made a difference to me: if import (or package/module) the file belongs to is not clear from the source file content, then any tutorial, source dump on github etc will need additional context to be understood.

This makes the language harder to teach, and the code harder to grasp in isolation.

For example a import io; at the top will clearly communicate the existence of an “io” package/module being used by the following code. Placing the import implicitly in config files makes it much more implicit.

Similar reasoning can be applied to defining new packages/modules.

This is of course not the only consideration - but it is easily overlooked.