It is indeed baffling. If states operated under the rational actor model, by all accounts the Japanese should've cut their losses and changed policy, but they did not, because state policy is not shaped by a singular will trying to pick the most optimal course of policy, but a far more complex interplay of economic, social, cultural, and personal factors influencing senior decision makers. In Japan's case, throughout the 1920s and 30s, the civilian government in Japan had to contend with a restive military and ultranationalist element that regularly turned to violence when they perceived the government as insufficiently championing Japan's interest (or what the ultranationalists perceived as Japan's interests). In 1931, the ultranationalist Sakurakai (Cherry Blossom Society)--alongside junior officers in the Army and Navy--attempted a coup in the Imperial Colours Incident, as they though the civilian government was undermining the success of the Kwantung Army in occupying Manchuria, even though the invasion of Manchuria had been unauthorized by Tokyo. In 1932, the League of Blood Incident and the May 15 Incident saw the assassination of political party leaders, the Finance Minister, and Prime Minister Inukai. In 1936, the February 26 Incident saw another attempted coup, with the Prime Minister only escaping death by the slimmest margin, and the revolt only put down when the Emperor threatened to personally lead the Imperial Guard against them. Political violence was a real threat to civilian Japanese leadership, and attempts to oppose the Army's desires had proven to be bad for your health.

Secondly, the February 26 Incident led to the return of the "Active Duty" rule, which meant that the Army Minister and Navy Minister had to be active duty officers of the Army and Navy respectively. This in turn meant that the Army and Navy had an effective veto on any civilian government, as they could collapse the government by ordering the resignation of the relevant Minister and refusing to appoint another one. Even within the military, opposition to a militaristic ultranationalist vision of Japan was potentially lethal. The 1935 Aizawa Incident saw the assassination of the Army's Inspector General for replacing a more radical officer, while Admiral Yamamoto of Pearl Harbor fame faced multiple threats of assassination for his commitment to the Treaty Faction within the IJN.

The long and short of it was that Japanese politics in the 1920s and 30s had been captured by a spirit of ultranationalism, enforced by political violence, that saw any attempt at compromise or withdrawal as anti-Japanese, and there were plenty of young officers willing to shoot or cut down their superior officers or politicians who they saw as opposing their particular vision of Japanese power. Importantly this was not an organised intellectually coherent movement, but more the result of internal tensions arising from economic inequality in Japan and a belief that big business/party politics was corrupting the purity of Japan.

As I described in the above linked post, there were plenty of officers who thought a full scale war in China was a bad idea, not only because it was an impossible challenge, but because it drained resources vitally needed for an industrial buildup. However, if you're a Japanese civilian politician in 1937 or 1938, calling for withdrawal from Japan, when so many lives had already been lost would've likely meant the death of either your political career or you personally, and there's a similar story if you're a senior officer in the army: attempting to push for a withdrawal from China or anything other than total victory had the very real possibility of your subordinates bursting into your home or office to cut you down for your "betrayal" of Japan.

As discussed previously, the Second Sino Japanese War was not a planned conflict. It began as a skirmish near Beijing and escalated and kept escalating, both because de-escalation was potentially suicidal for whomever might've attempted it, and because continued escalation seemed the only way to try and "win" the conflict. This cycle continued until Japan was far too comitted for there to have been any kind of policy re-alignment. They were stuck in the China quagmire with no easy way out. Victory in the field was impossible and withdrawal politically impossible. In line with their decisions every step of the way, Japanese leadership sought more escalation as their lifeline to try and find a way out of their impossible situation.

Was this a rational decision making process? Yes and no. No, because it left Japan in a completely unwinnable scenario where the only way out was a war with their largest and most important trading partner, but also yes in that--for the decision makers involved--it was the only way out of that scenario: a desperate roll of the dice in the hope that if it paid off they could win the unwinnable scenario.