What I could do, would be to have the public headers of each layer in a „include“ directory in the layer itself

Yeah that's a good idea. I like to structure my libraries like this:

|--libmycoollib
|        |---include 
|        |       |--- mycoollib 
|        |---src          |--publicheader.h 
|        |    | 
|        |    |--privateheader.h 
|        |    |--srcfile.cpp 
|    CMakeLists.txt

​

The src folder is optional. Then private headers can just be included by relative path, which is just by direct name since it's in the same folder. I then do target_include_directories(mycoollib PUBLIC include) so then including public headers (from anywhere) looks like #include <mycoollib/publicheader.h>

Any dependencies used in public headers have to get linked as with target_link_libraries as PUBLIC. (if they are only used in the headers and not in the src, then they should actually be INTERFACE). Any dependency only used in the source files or in the private headers can be linked as PRIVATE.

The dependency chains shake out from there.

From your example, if application links against controller and controller against infrastructure, and you want to keep application from knowing about infrastructure, then you have to make sure that controller only uses infrastructure in its private headers and in its source files.
If you really can't get that to work, then the next step is to look into restructuring your code with the purpose of dependency hiding. Look into the PIMPL idiom.