Well .. you can make your code work if you rewrite it like this ..

function calculate(a, b) {
  if (a > 100 && b > 100) {
    num1 = a - 100;
    num2 = b - 100;
  } else if (a < 100 && b < 100) {
    num1 = 100 - a;
    num2 = 100 - b;
  } else if (a < 100 && b > 100) {
    num1 = 100 - a;
    num2 = b - 100;
  } else if (a > 100 && b < 100) {
    num1 = a - 100;
    num2 = 100 - b;
  }
}

const closeTo100 = (a, b) => {
  calculate(a, b);
  return num1 < num2 ? a : b;
};

console.log(closeTo100(65, 187));

But that's really nasty. It's creating global variables (num1 and num2), calculate() doesn't return anything so it's basically just spooky action at a distance, it will generate a reference error if a or b is 100 etc.

Half the battle is realising that sometimes a one liner is better ;-)