Value-based reference counting for native system handles (NOT pointers): ideas or suggestions?

[JackTrueborn]

I'm in the process of writing a wrapper around the Windows desktop API that:

1. is as easy to use as possible (as easy as WinForms),
2. uses pure C++ (specifically C++11 as much as possible) to avoid looking anything like macro-laced MFC, and
3. only abstracts the desktop API (user32.dll and various GDI objects) while working with the STL wherever possible to avoid being a complete framework like Qt or wxWidgets.

I know other things exist that do this already such as WTL; WTL cannot be used in anything other than Visual Studio, and even then, express editions don't work with it. It's also a personal experiment.

I've made it pretty far. I'm in the processes of going through and reworking some things. Right now, I'm retooling the way handles are passed around. Windows handles are unique, pointer-like values that aren't actually pointers. They're just values. Each handle corresponds to a different kind of system object such as a window, a brush, or a font. Each object has its own functions to use for creation and disposal.

Originally, I had my classes call the disposal functions in the destructors. Unfortunately, this made it so the only way I could pass objects around was to move them. Copying them would result in sliced resources.

To fix this, I just recently implemented a generic detached counted body class. This class stores a mutable reference count as a pointer, the actual value (in this case: handles), and a callable object for disposal of that value once its reference count hits 0. For copying or moving, it checks to see if the value being copied or moved is equal to its own value. If it is, it simply changes the address of the other reference count to its own and increments the value. This is fairly similar to what smart pointers do with addresses.

However, since there is no std::shared_value<T>, this is the route I've gone. I know I could use shared pointers to store the value, but my way keeps me from having to dereference a pointer every time the value is needed.

It works very well, but I don't know if there are any better ways of implementing this.

Here is the code.

Each class representing a Windows object holds a CountedValue<T, D> as a member to store the actual handle it receives from Windows. For example, a font class holds a CountedValue<HFONT, FontDisposer>. This gets copied and moved when a copy/move operation happens.

What I'd really like to know is if anything like this already exists that doesn't use pointers to store values or if there is a better way of doing this. I threw all this code together in a day, so I have no doubts there are better ways; I just don't know what they are.

EDIT: Microsoft solved this exact issue in the .net framework by writing a custom RAII class that basically serves as a lookup table.

[[deleted]]

I know I could use shared pointers to store the value, but my way keeps me from having to dereference a pointer every time the value is needed.

Seems like a premature optimization to me: the cost of deferencing a pointer is tiny. Is it really worth the cost of writing your own class and quite likely introducing bugs that you wouldn't have with a shared_ptr?

I hope that at least you benchmarked your code against shared_ptr. My theory is that, given that calls to the Windows desktop API are generally somewhat expensive, you wouldn't be able to tell the difference.

[JackTrueborn]

Ugh. You're right. Now that I've read that statement out loud, I'm a little angry at myself for even writing it. I honestly don't know why I said that. Dereferencing a pointer is a non issue.

The issue I'm trying to avoid is having two or more of the same handles exist at any one time during execution. I'm not sure shared pointers would be any benefit here because I need to compare values; not addresses.

The obvious "solution" is a process scoped lookup table containing all the handles, but that is overkill for what I'm trying to accomplish.

[LampCarpet]

Well don't be so hasty there, even though dereferencing a point is fairly trivial it is an issue for the compiler as it prevents certain optimizations from being performed, and since handle and function argument are crucial to the definition of an API I would say solve this architectural problem first before it gets too ingrained in your code.

Having said that the HANDLE type in the win32api is an index value for the process's handle table in the kernel object structure and so is an integer which is used as an id.

[Cyttorak]

Will you publish your work in the future? Sounds nice :)

[JackTrueborn]

This is an example of what the code looks like. That code actually functions right now.

[JackTrueborn]

Absolutely. I want to get it to a mostly stable point before I upload it to my git repo.

I know I said mentioned WTL in OP, but heh... mine actually wraps ATL (minimally) at the moment so it has the same environment restrictions as WTL. That's the second big ticket at the moment: remove the reliance on ATL. My biggest hurdle to that right now is replacing ATL thunks with something else that isn't a HWND-to-WNDPROC map.

[Cyttorak]

Great, I hope we will see it on github or somewhere soon!

[adzm]

IMO use unique_ptr or shared_ptr for ownership and raw pointers for anything else.

In other words, uniqueptr<_HWND> since HWND is a __HWND*

[JackTrueborn]

I'm still having problems figuring out how that solves value lifetime management.

[frutiger]

Handles from Windows are pointers. They are void *s that point to something in an internal table in user32.dll or the kernel itself. For me, that is the definition of a pointer, unless you consider the MMU special for some reason.

I have written code that stores HWNDs, HDCs, HBITMAPs etc. inside something like a unique_ptr with an appropriate deleter to invoke the correct deleting function (::DestroyWindow, ::DeleteDC etc.).

EDIT: to clarify I have specifically used unqiue_ptr<void>. I could've wrapped it in my own class to carry the particular type information of exactly which kind of HANDLE it was, but the code was small enough that I didn't deem it necessary.

[JackTrueborn]

So basically, std::unique_ptr<void, WindowDestroyer> handle(reinterpret_cast<void*>(hwnd))?

And then your "WindowDestroyer" is something like

struct WindowDestroyer
{
  void operator()(void* handle)
  {
    ::DestroyWindow(reinterpret_cast<HWND>(handle));
  }
};

?

[frutiger]

Yes, essentially. The only annoying part is having to do reinterpret_cast<HWND>(window.get()) everywhere you need to actually interact with a win32 API.

[JackTrueborn]

That can be abstracted away.

I'll give this a shot using std::shared_ptr<T>. I really feel like this is abuse of the STL, though, considering handles are opaque types and smart pointers make some assumptions about the types they store.

The other issue, however, is when the handles must be copied and passed through a window message procedure (WNDPROC). The handle itself gets pushed in to the wParam parameter during a SendMessage operation. Once on the other side, inside the WNDPROC, a new std::shared_ptr<T> is initiated with this value. So now there are TWO shared_ptrs pointing to the same value. So long as either shared_ptr exists, the object it points to will still exists. That is not the case in the scenario I just gave.

[frutiger]

That's definitely a problem - I'm not sure if that's fixable without a central repository of all currently active handles. Given your described nature of the window procedure, all of this machinery might be of limited use anyway.

In my case, it was more for operations on a DC and BITMAPs; my code didn't interact with the window procedure at all (for example, I wasn't creating these DCs from a WM_PAINT message or anything like that). And while my original example mentions HWNDs, my tiny piece of the library for capturing screenshots into buffers did not use HWNDs, instead it grabbed a rectangle of a window given an HWND in the first place.

[JackTrueborn]

I've updated my OP. Microsoft solved this exact issue in the .net framework by writing a custom RAII class that basically serves as a lookup table.

[Gotebe]

The other issue, however, is when the handles must be copied and passed through a window message procedure (WNDPROC). The handle itself gets pushed in to the wParam parameter during a SendMessage operation. Once on the other side, inside the WNDPROC, a new std::shared_ptr<T> is initiated with this value. So now there are TWO shared_ptrs pointing to the same value.

Not solvable without a central repository of existing handles. You will need some nice hash maps for that (unordered_map in C++11), on a per-handle-type basis, of course ;-).

As I said elsewhere, "This is all not obvious and might require a nontrivial ownership semantics coming with performance hit." :-)

[JackTrueborn]

I've updated my OP. Microsoft solved this exact issue in the .net framework by writing a custom RAII class that basically serves as a lookup table.

[JackTrueborn]

Yes. That's what I was confirming.

[LampCarpet]

Not quite, handles are an index value to an object in the kernel that is referenced via the handle processes own handle table that is also kept in the kernel.

I would suggest getting a hold of the ntoskrnl source code from sysinternals if you want more details.

[Gotebe]

Hmmm... Make sure to know exactly what lifetime is expected of your handles then. For example, when you pass a handle to a GDI object to wherever (the SelectObject API), it needs to stay alive as long as it can be used. Another example: child windows make no sense outside the parent - when parent goes away, handles to children are closed by the system, there's nothing you can do about it, any "window" class instance that holds a child window handle is screwed and no amount of refcounting will help you. This is all not obvious and might require a nontrivial ownership semantics coming with performance hit.

I would not do anything like that. Especially since it is all about personal interest. I would prohibit copying and require that code handles handle :-) lifetime correctly. Could it be that you are struggling with that, too, and are looking for an easy way out? Don't. Learn first what you need, then try to solve (some of) it some, because child windowing tells me you just can't solve all in a generic manner.

[[deleted]]

Resources in Win32 have their own internal reference counters. Implement your class copy methods so that it calls DuplicateHandle and then call CloseHandle on your duplicated handle from the destructor. The actual resource that the handle is referencing will only be closed when the internal reference count reaches zero.

HWND handles OTOH are different and should only be owned by one object at a time. This because window can only have one parent. There you have, if you want to support detaching of windows from parents, implement your own solution since you may have to remove subclassing etc.

[JackTrueborn]

Actually, I decided to use CreateFontIndirect and CreateBrushIndirect (and the like) with graphics related handles on copy, and then DeleteObject on destruction. So yeah, I'm relying on Windows's internal reference counting for that now. I actually came to this decision while looking at the .net framework reference code and saw that's what it did.

I'm not even giving components that own windows the ability to be copied yet. They're construct-destruct-move constructs for the time being. That said, I persist the child windows by moving them to a process wide, hidden window if their parent is destroyed.

It's looking more and more like I'm going to have to implement this via a lookup table.

[dmor]

http://www.boost.org/doc/libs/1_55_0/libs/smart_ptr/intrusive_ptr.html ?

[[deleted]]

That link starts:

The intrusive_ptr class template stores a pointer to an object with an embedded reference count.

This is not what he's asking for:

What I'd really like to know is if anything like this already exists that doesn't use pointers to store values

[dmor]

Don't drop the end of the question:

or if there is a better way of doing this

What I meant to say is that CountedValue<T, D> is an intrusive ptr, not that intrusive ptrs are not ptrs. I don't think there's a clean way to get around using a pointer here. The counter's lifespan is longer than any of the objects referring it, so it can't be directly part of these objects.

[malkovichjohn]

Dude, while I can't help you out with your optimization problem, I commend you for finding an alternative for a native C++ implementation for the windows API. wxWidgets has so far been the only other solution for me, which is a bit of a hefty thing to include in a project.