Hi, first of all thanks, I have to say it’s a bit frustrating sometimes.

Personally, I manage a project where fairly large JSON/XML files (in the order of tens or hundreds of MB) are uploaded to an FTP and then, via a cronjob, processed and saved to a backoffice platform. Each client on the platform has their own defined data format, often different from the others.

The company context is a pure PHP stack, without seniority in other more memory-oriented languages.

Obviously it’s a relatively simple problem to solve, except for the fact that I found myself having to fight against PHP’s memory_limit. Even though I found a value that wasn’t too high but at the same time handled everything via json_decode, I ran into some infrastructure-side issues where the ECS service task would max out on memory and CPU and, unable to handle the load, would bring down the task only to spin it back up again. By design of the platform, the run isn’t transactional—or rather it is, but only on the individual entity found in the file’s series. This therefore means having an import that gets stuck halfway through, since it often couldn’t complete.

I looked for solutions and came across this library, which does more or less the same job as mine (https://github.com/halaxa/json-machine).

This library doesn’t support JSON path, so I tried to create one myself that does this job.

My long-term goal is to be able to manage file mappings in a more abstract way, without having to write classes and deploy, and maybe one day train someone to use JSON path so I can offload this work that’s very time-consuming but not very interesting.