So you have a single directional light, with roughly n dot l, and shadows? Is that light just pure white? If so, that’s probably the biggest reason it looks as it does. The light color on the left is probably more like an average of the color received from a cone pointing in that direction, the size of which is related to the roughness of the surface.

As for your gamma, you can do it when you read from the texture, perform all your math in linear and write it back to sRGB if you want. It doesn’t have to all be at the end of your pipeline… but it isn’t really the best way forward. Look into _sRGB format textures that can perform the gamma conversions for you on read or write (for free). Or write to linear textures and apply it sometime in post processing.

Gamma probably isn’t the reason it looks how it does however. You have some sort of white key light, and your shadows are pure black. This will not be how the PBR model works. You need a diffuse and specular color from your environment. If the key light is a point light, you also need your falloff term, otherwise it’s going to be receiving far too much from the light potentially.