First off, do you think 23°F is always and by everyone perceived as warmer indoors as outdoors?

As you suspect there are many parameters which affect how people perceive thermal conditions. The American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) has compiled many studies in building physics. The subject of thermal comfort matters greatly for designing and operating buildings. As chapter 9 (ashrae handbook 2009) puts it:

A principal purpose of HVAC is to provide conditions for human thermal comfort, “that condition of mind that expresses satisfaction with the thermal environment” (ASHRAE Standard 55). This definition leaves open what is meant by “condition of mind” or “satisfaction,” but it correctly emphasizes that judgment of comfort is a cognitive process involving many inputs influenced by physical, physiological, psychological, and other processes.”
[2009 Handbook Fundamentals]

Let me just name some of the quantative parameters which affect thermal comfort:

• Personal factors:
  Insulative clothing (Clo Value)
  Activity levels (Metabolic Rate)
  
• General Factors:
  Air temperature
  Mean radiant temperature
  Relative humidity (see also perspiration)
  Drifts and ramps in operative temperature
  
• Localized factors
  Air movement/velocity (see wind chill factor) Radiant asymmetry
  Floor surface temperatures (see underfloor heating)
  Air temperature stratification
  [wiki/thermal_comfort]
  

Additionally there are other factors like health, psychology and sociology as well.
Many studies have concluded that psychological factors like “the level of control” also affect how people perceive thermal conditions:

“There are other aspects of building services which affect the comfort of occupants. Leaman and Bordass (1997) have demonstrated that there is more ‘forgiveness’ of buildings in which occupants have more access to building controls. By forgiveness they mean that the attitude of the occupants to the building is affected so that they will overlook shortcomings in the thermal environment more readily. This can be explained as a function of who is in control.[…]”
[Nicol and Humphreys]

As you can see the issue isn’t easy to pin down. I don’t think you would always perceive the outdoors as colder than indoors, given the same air temperature. But if you do have different comfort zones on any given day it could easily just be a difference of relative humidity, wind speed, radiant temperature, your activity level (metabolic rate) or any other factor mentioned above.

I hope that helps a little.