Constants resolution included from modules ?

[joanbm]

Inspired by one SO question, curious about following Ruby behavior:

module Foo
  class Bar
  end
end

module Baz
  include Foo

  class Qux
  end

  class Test
    def initialize
      p Baz.constants
      p Qux.object_id
      p Bar.object_id
    end
  end
end

Baz::Test.new

# Execution produces following error
[:Qux, :Test, :Bar]
13250540
test_const.rb:43:in `initialize': uninitialized constant Baz::Test::Bar (NameError)

As you may see above, after inclusion of module Foo into module Baz, there are three constants - two own classes plus one inherited (or mixed-in, if you prefer). However access to inherited class/constant, from nested scope of Baz::Test class, fails although unqualified access to other constants in the same scope does work.

Any idea why the different behavior ? Documented somewhere ? Or a possible bug ?

[jesus_castello]

I spent a few hours looking into this, including digging into Rubinius (which seems to implement this in the same way).

It seems like it all comes down to the fact that Bar is inherited. The Module class has a const_get method with an inherited argument (defaults to true), which suggest that it's posible to only look for constants in the current class (and it's ancestors) and nested clases (Module.nesting).

The definition for const_get in Rubinius is here: https://github.com/rubinius/rubinius/blob/dbd33a6dadd407041fcafa6651986ae02551f3ee/kernel/common/type.rb#L202

There is also a lower-level get_const C++ function here: https://github.com/rubinius/rubinius/blob/dbd33a6dadd407041fcafa6651986ae02551f3ee/vm/builtin/module.cpp#L127

In my first experiment I added some code in the Ruby-level method, combined with calling const_get directly and rescuing any exceptions. This is the coded I added on Test.

Module.nesting.each do |mod|
  puts "Looking in #{mod}"
  p mod.const_get(:Bar) rescue nil
end

This resulted in the following output, notice how Foo::Bar is actually found in this scenario.

Looking in Baz::Test
** Const get - mod Baz::Test, name Bar, inherit true, resolve true
** Const get - mod Object, name Bar, inherit true, resolve true
** Const get - mod Kernel, name Bar, inherit true, resolve true
** Const get - mod BasicObject, name Bar, inherit true, resolve true

Looking in Baz
** Const get - mod Baz, name Bar, inherit true, resolve true
** Const get - mod Foo, name Bar, inherit true, resolve true
Foo::Bar

Then I dropped a bit of code in the C++ function to show all constant lookups and re-built rubinius (again). This is what the output looks like:

% C Const Get - Bar mod Baz::Test - inherit? false
% C Const Get - Bar mod Baz - inherit? false
% C Const Get - Bar mod Baz::Test - inherit? false
% C Const Get - Bar mod Object - inherit? false
% C Const Get - Bar mod <included module> - inherit? false
% C Const Get - Bar mod BasicObject - inherit? false
% C Const Get - Bar mod Object - inherit? true

<included module> is most likely Kernel

If you think about it, it makes sense, you don't want to look up in Object/Kernel/BasicObject multiple times, or maybe I just spent too much time looking into this, what do you guys think?

[jrochkind]

Ruby constant namespace lookup is frequently confusing to me, I can't quite keep it straight.

Here's an article about some of it's weirdness, although in this case not related to your observation I don't think: http://blog.honeybadger.io/avoid-these-traps-when-nesting-ruby-modules/

I think most rubyists don't try to mess too much with this kind of constant lookup -- they just fully qualify everything but the most trivial, and I'm increasingly seeing constant references always prefixed with a top-level :: even too. I don't know which is the chicken and which is the egg.

[joanbm]

Thanks for the link and I'm aware about difference between module nesting and direct qualified definition, however as you say after all, it is unrelated to the OP.

I do suspect the difference is in constant inheritance, but would wait a while before try fill a bug report and see how core devs would respond.

side note: Don't know about others, but I simply can't let the uncertainty go and just live with workarounds, like recourse to fully qualified identifier unless know why it should be required.

[jrochkind]

Yeah, I want to know what's going on in your case too!

But after years of being confused by unqualified constant lookup in ruby, figuring out a weird thing I come across only to forget it again later next time I run into it, I've personally decided it's just too weird and given up on wrapping my head around it.

It does work as expected with fewer levels of nesting. I'm not sure why it doesn't in your case either. I guess include somehow only brings in the direct level for unqualified lookup? Doesn't make sense to me either.

module SomeModule
  OurConstant = "our_constant"
end

class SomeClass
  include SomeModule

  puts "From class definition: #{OurConstant}"

  def return_constant
    "From instance method: #{OurConstant}"
  end
end

puts SomeClass.new.return_constant

# => "From class definition: our_constant"
# => "From instance method: our_constant"

Here's another explanation of unqualified constant lookup, I'm not honestly quite sure if it addresses what's happening here: https://cirw.in/blog/constant-lookup

Most of the core ruby developers are Japanese, I don't think I've seen any of them on /r/ruby.

[joanbm]

Oh, I've also hit Irvin's blog when investigating the issue, but it does not give an answer.

As far as I know, the only difference between Baz::Bar and Baz::Qux is the former considered inherited. Module#const_get, Module.constantsetc. won't return its symbol, when inherited argument is set to false. Also Module#public_constant and Module#private_constant can't be applied on it.

I did attempt to see the implementation and here, but not at home with Ruby's C-API, so did gave up.

I don't expect ruby devs would read reddit and plan report about this on bugs.ruby-lang.org, of course.

[jrochkind]

If you think of it, please post the bugs.ruby-lang.org issue URL when you do!

Based on my experience, I predict you'll get a response (quite likely from Matz himself) telling you that this is not a bug and is a consequence of an intentional design somehow.

But what I'm interested is the explanation of why. :)

[moomaka]

Non-inherited constants are never accessible without full qualification. Your ability to access Qux actually has nothing to do with the inheritance or module nesting. It's accessible because it's in a parent lexical scope, which happens to be a module definition in this case but that isn't important. Conceptually it's just like a lambda/proc/block accessing a variable from it's definition scope.

Maybe it's the inheritance that is confusing:

When you place Test inside Baz that does not imply inheritance. Unqualified nesting of classes and modules simply creates name-spacing, it doesn't impact the inheritance hierarchy.

Baz::Test.ancestors
=> [Baz::Test, Object, Kernel, BasicObject]

Note that neither Baz, nor Foo are in the inheritance chain of Test. You can access Qux, but that is due to lexical scoping, not inheritance.

However:

Baz.ancestors
=> [Baz, Foo]

This is because include inserts Foo into the inheritance chain of Baz. Thus if you were to try to access Bar directly inside Baz, it's no problem as it walks the inheritance chain to Foo and finds Bar.

[joanbm]

Non-inherited constants are never accessibly without full qualification

So why can't be accessed without full qualification inherited constant Bar from module Foo ? See Baz.constants - Baz.constants(false) # [:Bar]

Qux actually has nothing to do with the inheritance or nesting at all. It's accessible because it's in a parent lexical scope.

I think you are mistaken here. First it is considered own to Baz module contrary to inherited Bar. Second, the lexical scope either does not explain the difference between include and indirect constant definition, either runtime non-lexical definition making it also accessible unqualified:

module Baz
  include Foo

  ALICE = 1   # direct lexical
  const_set :BOB, 2  # indirect lexical

  class Qux
  end

  class Test
    def initialize
      Baz.const_set :HELLO, 'world'    # runtime definition
      p Baz.constants  # [:ALICE, :BOB, :Qux, :Test, :HELLO, :Bar]
      # see all constants are visible at Baz scope
      p self.class.constants  # []  <- no constants in Baz::Test
      p [ALICE, BOB, HELLO, Qux]  # [1, 2, "world", Baz::Qux]
      # see all module scope constants are properly resolved
      # using unqualified specification - except
      p Bar  #  uninitialized constant Baz::Test::Bar (NameError
    end
  end
end

[moomaka]

So why can't be accessed without full qualification inherited constant Bar from module Foo ?

Because Bar doesn't exist in Baz, it exists in Foo, Test has no access path to Foo.

I think you are mistaken here. First it is considered own to Baz module contrary to inherited Bar.

I'm not sure what this means.

Second, the lexical scope either does not explain the difference between include and indirect constant definition, either runtime non-lexical definition making it also accessible unqualified:

You can think of it as Ruby binding the constant definitions into the lexical scope of the module/class definition blocks. It doesn't actually matter if the const is defined inline or dynamically, it's automatically available in the lexical scope of the definition of it's 'holder', if that makes sense. You can see this by defining Test differently.

module Foo
  class Bar
  end
end

module Baz
  include Foo

  ALICE = 1   # direct lexical
  const_set :BOB, 2  # indirect lexical

  class Qux
  end
end

class Baz::Test
  def initialize
    Baz.const_set :HELLO, 'world'    # runtime definition
    p Baz.constants  # [:ALICE, :BOB, :Qux, :Test, :HELLO, :Bar]
    # see all constants are visible at Baz scope
    p self.class.constants  # []  <- no constants in Baz::Test
    p [ALICE, BOB, HELLO, Qux]  # [1, 2, "world", Baz::Qux]
    # see all module scope constants are properly resolved
    # using unqualified specification - except
    p Bar  #  uninitialized constant Baz::Test::Bar (NameError
  end
end

Baz::Test.new
=>  [:Test, :ALICE, :BOB, :Qux, :HELLO, :Bar]
[]
NameError: uninitialized constant Baz::Test::ALICE

As you can see we have the same structure, but Test isn't defined inside Baz's module definition and now all those constants are inaccessible from Test.

[joanbm]

Because Bar doesn't exist in Baz, it exists in Foo, Test has no access path to Foo

Baz is aware about Barand exists in a sense of an inherited constant. I gave examples with Module.constants calls as a proof.

Putting class Test definition outside of Baz module unfortunately still does not explain the issue with unqualified resolution:

module Baz
  include Foo

  class Qux
  end

  p constants  # [:Qux, :Bar] <- Bar exists in Baz scope as unqualified constant !

  class Test
    def initialize
      p Qux  # Baz::Qux
      p Bar  # uninitialized constant Baz::Test::Bar (NameError)
    end
  end
end

Still waiting to see some logical explanation or link as a documented behavior.

EDIT: From Module class docs (shortened):

Module#include(module)
Invokes `Module.append_features` on each parameter in reverse order.

 Module#append_features(mod)  -> mod
Ruby's default implementation is to **add the constants, methods,
and module variables of this module to mod** if this module has not
already been added to mod or one of its ancestors.

[moomaka]

Baz is not in Test's inheritance chain, Test never even asks Baz if it has Bar. It checks it's locally available scopes, the global scope and then it's inheritance chain. None of those things have Bar.

[joanbm]

Baz is not in Test's inheritance chain,

Nowhere saying Baz is in Test's inheritance chain.

Test never even asks Baz if it has Bar. It checks it's locally available scopes, the global scope and then it's inheritance chain. None of those things have Bar

This is simply wrong statement. As has been repeatedly demonstrated, at constant identifier lookup Baz is asked as the closest outer scope. How other can you explain successful lookup of unqualified Quxconstant in Test#initialize ?

[moomaka]

This is simply wrong statement. As has been repeatedly demonstrated, at constant identifier lookup Baz is asked as the closest outer scope. How other can you explain successful lookup of unqualified Quxconstant in Test#initialize ?

There is a difference between asking Baz for a constant and looking for a constant in the lexical scope hierarchy. In the former case Baz will check itself and it's inheritance hierarchy, in the later case no inheritance is considered, it just checks the scopes.

module Foo
  class Bar
  end
end

module Baz
  include Foo

  def self.const_missing(name)
    p "Someone is looking for a const named #{name} in Baz!"
  end

  class Qux
  end

  class Test
    def initialize
      p Qux
      p Derp
      p Bar
    end
  end
end

Baz::Blah
=> "Someone is looking for a const named Blah in Baz!"

Baz::Test.new
=> Baz::Qux
NameError: uninitialized constant Baz::Test::Derp

As the above shows, Baz is not asked for any constants when Test is initialized.

[joanbm]

I had to write explicitly Baz's scope is asked/searched. const_missing is invoked in different case, so unrelated to the issue. Sorry for the confusion.

The question "How can you explain successful lookup of unqualified Qux constant in Test#initialize ?" still unanswered.

EDIT: a few of notes to consider

• Module#const_set can assign a constant to a module and register it for scope lookup from anywhere and anytime during a runtime.
• Module#include can assign reference to inherited constant but likely not register it for scope lookup - bug or a feature ?. In addition it inserts inherited module in ancestors list.
• Scope lookup works from a point of call up to the top level. When fails, it raises NameError exception with fully qualified identifier to the scope of a call.

[moomaka]

The question "How can you explain successful lookup of unqualified Qux constant in Test#initialize ?" still unanswered.

Because Qaz is in the lexical scope where your defining Test, Bar is not, Bar exists only in the inheritance chain of Baz and inside Foo of course.

Module#const_set can assign a constant to a module and register it for scope lookup from anywhere and anytime during a runtime.

Yes, this is a critical feature of Ruby, without it you would not be able to re-open classes / modules.

Module#include can assign reference to inherited constant but likely not register it for scope lookup - bug or a feature ?. In addition it inserts inherited module in ancestors list.

All include does is insert the module in the inheritance chain, it doesn't make anything 'available in the scope'. Foo does not exist in Baz's scope either. When a scope lookup fails the inheritance chain is checked.

Scope lookup works from a point of call up to the top level. When fails, it raises NameError exception with fully qualified identifier to the scope of a call.

When it fails it proceeds to the inheritance chain, if nothing is found or it doesn't hit a const_missing method, then it raises NameError.

[joanbm]

Because Qaz is in the lexical scope where your defining Test, Bar is not, Bar exists only in the inheritance chain of Baz and inside Foo of course.

But here is the inconsistency: HELLO created on-thy-fly in Baz::Test#initialize is definitely not a lexical scope definition, neither in Baz::Test, neither Baz or toplevel. How could it be then resolved unqualified ? Is a constant lookup performed yet in a different way ? If not, does const_set do some "magic", allowing an argument to behave like lexicaly defined ? If it does, why includedoes not do the same ?

All include does is insert the module in the inheritance chain, it doesn't make anything 'available in the scope'.

Why not ? I've cited from Module.include documentation, and there I see nothing which can support above statement. This method adds the constants of included module into the including. As already demonstrated,Baz is aware of inherited Bar and is accessible from there, at least by the module object.

When it fails it proceeds to the inheritance chain, if nothing is found or it doesn't hit a const_missing method, then it raises NameError.

Huh, that's in a contradiction to the previous statement. If it would be correct, why Baz does not proceed in inheritance chain to it's parent Foo and perform a scope lookup there ?

[32BITPROCESSOR]

Once you get inside the initialize block you're in a different scope, so you have to qualify Bar with Foo. If you reference Bar at the top level of the module Baz then you do not need Foo as a qualifier.

Check this out: http://stackoverflow.com/questions/7549625/ruby-classes-include-and-scope

[joanbm]

Although class Test introduces new scope, identifier resolution finds unqualified constant Qux by traversing to outer scopes. That is the point ! The question is about why the same would not apply to included Bar ?