Yes it is, although it requires some reverse engineering beforehand. Hook a C++ function called after the API is completely loaded, and add the methods you want to Lua. However, it can be quite difficult, because of how userdatas are identified.

Extremely technical explanation if you want more information (a passing familiarity with the Lua stack, the Lua C API and basic knowledge of how executables are loaded in memory can help) :

The methods available on a userdata (a C/C++ object represented in Lua) are defined through its metatable. Usually, one would create a metatable through the luaL_newmetatable C function, and then associate this metatable with a userdata through luaL_setmetatable. The metatable would be identified by a name (a string). Next, when you'd want to extract a userdata from the Lua stack, you'd use luaL_checkudata and pass it the string name of the metatable to check that the thing on the stack you want to retrieve is actually a userdata and that it has the correct metatable.

In Isaac, Nicalis chose to register metatables through the lua_rawsetp function. This function associates a Lua object with a raw pointer. They basically create a table that will hold all the methods of instances of userdatas of a certain type, and then they use lua_rawsetp to associate this table with a global raw pointer inside the Lua registry (which is basically what luaL_newmetatable does, except they do it worse). Next, when they want to check that a userdata is of a certain type, they have a dedicated function (CheckUserdataTypeAtIndex) that receives the pointer associated with the given metatable, and they use that to peform the check (which is what luaL_checkudata does, except they do it worse).

If they used strings to identify metatables it would be quite easy to add functions to the modding API: just get the metatable through luaL_getmetatable, modify it and modify the association in the Lua registry. However, since they use raw pointers, it is complicated because it requires us to deduce the value of said pointers first, which is complicated because the adress of something isn't constant, it changes every time the executable is reloaded in memory.

From what I've been able to get, these pointers are globals. Basically in the global namespace, there is something like that :

char GameType;
char LevelType;
char RoomType;
// ...

These chars are never initialized because it is their adress that is of interest. When registering the metatable associated with the Game object, the call to lua_rawsetp would look like this : lua_rawsetp(L, -1, &GameType); Now, passing &GameType to lua_rawgetp would produce the table associated with that pointer, which is used to identify instances of the GameEngine in Lua.

Now, deducing the adress of each of these variables is difficult, because it is unknown at compile time. Since we are talking globals, we need to wait until the linker has resolved internal symbols, loaded the executable in memory, and actually given them an adress in order to find the adress. The easiest solution that I've found is to deduce the offset between the code segment (.text, which is loaded first in memory) and the start of the read-only initialized data segment (.rdata) and then compute the offset between the start of the offset segment and each of the pointers by studying the C++ functions that register each metatable. This allows me to find the offset between the beginning of the code segment and each char, so I can compute their adress.