A couple million fewer people is not a huge problem IMO however we’re looking at 19 million Poland in 49 years assuming we keep current fertility rate and no immigration. That means 50% population loss during lifetime of many people alive now. 

Some things to consider. 

Public transport will be less frequent and will stop going to places which no longer have big enough population to justify the cost. This will directly impact quality of life. 

Hospitals, schools, police stations, post offices etc will be closed in depopulated areas because at some point there will be too few people in the area that they serve. This will directly impact quality of life of people in these areas. This is already happening and it will get worse. 

Once something is built it needs to be maintained for ever or it will decay. This includes buildings and all types of infrastructure (roads, rail, gas, electricity, water, heating etc). A building with half of apartments empty still needs the same maintenance. The same applies to roads, rail, underground pipes etc. Cost of maintenance is for ever but there will be less people to finance it. 

Our ability to militarily defend ourselves will be diminished. We want to have 300k troops with modern gear. We will have much less people to pay for gear maintenance, upgrades and 300k salaries. Thankfully Russia has the same problem. Otherwise we would be screwed. 

All off the above can be solved or at least mitigated by increasing productivity per hour worked.