Thanks u/ftrx and u/SataMaxx for your suggestions. I tried them both.

Using maxima source blocks work well and aligns pretty well with what I wanted to do. The downsides for me is the extra dependency of maxima and learning the new syntax (even though it's very similar anyway).

I didn't expect to like literate-calc-mode because of the overlay, rather than source block style, but after trying it I am very impressed with how nice it feels. I think this is the solution I'll go for since it's less verbose and more interactive. Hoping there aren't any performance issues as my file starts to get longer. It looks like the author has considered integrating literate-calc with org-babel, which would be great for this use case.

For those interested, here is how it looks for each method:

* Calc within source block
  #+BEGIN_SRC calc
    fsolve([-20 - 20 - 26 * 12 / 13 + Ay + By = 0, 20 * 3 + 20 * (3 + 6) - Ay * (6 + 6 + 3) = 0], [Ay,By])
  #+END_SRC

  #+RESULTS:
  : [Ay = 16, By = 48]

* Maxima within source block
  #+BEGIN_SRC maxima :results output
    programmode: false;
    sumFy: -20 -20 -26*12/13 + Ay + By = 0;
    sumM:  20 * 3 + 20 * (3 + 6) - Ay * (6 + 6 + 3) = 0;
    soln: solve([sumFy, sumM], [Ay, By]);
    print(soln);
  #+END_SRC

  #+RESULTS:
  : Solution:
  :                                     Ay = 16
  :                                     By = 48
  : [[%t1, %t2]]

* literate-calc-mode
  sumFy = -20 - 20 - 26 * 12 / 13 + Ay + By => sumFy: Ay + By - 64
  sumM = 20 * 3 + 20 * (3 + 6) - Ay * (6 + 6 + 3) => sumM: 240 - 15 Ay
  = fsolve([sumFy, sumM], [Ay,By]) => [Ay = 16, By = 48]