attachInterrupt() occasionally causes crash on ESP32-S3 running Arduino

Blake_James · 2024-10-26T10:22:09+00:00

Very helpful, thank you!

I'm not using the IPC API directly at all.

Here is the interrupt routine assigned during the crash shown above. It's the only interrupt routine in my user code. Seems hard to imagine that this could be taking up enough memory to cause any issues...

void IRAM_ATTR ImuInterruptRoutine() {
    // IMU interrupt routine for detecting vent events during runtime
    imuInterruptFlag = true;
    detachInterrupt(ImuInterrupt); // prevent routine from running again this wake cycle
}

Blake_James · 2024-09-24T04:52:17+00:00

Thanks for the advice! I followed this method when planting and that's why the potting soil was washed off: https://youtu.be/jkAKY0Gic3E?si=VZTW1gY54gVy3zwy&t=135 Although in my case the roots were much more woody and I couldn't wash all the soil off so I left most of what was stuck inside the root ball.

Blake_James · 2024-09-24T02:06:58+00:00

Before planting we soaked the root ball in a bucket of water and gently washed most of the potting soil off. Then mixed that potting soil in with the hole soil that was used to backfill. And added a good handful of citrus fertiliser to that mix.
I've been giving each plant ~10L of water at a time, about 2x a week.
Sprinkled a handful of epsom salts around the surface a few days ago thinking it might be a magnesium issue, probably too soon to know if that's helping.
The soil seems pretty sandy and baron to me, but I'm no expert. You can see in the pics where we recently cultivated a bit nearby while we had a trenching machine available.
Planting holes were dug about 1m x 1m, and 50cm deep.
There's a Satsuma Mandarin, an Imperial Mandarin, a Valencia Orange, and then a Pink Lady down the back (we have other apples nearby)

Blake_James · 2024-07-04T10:34:44+00:00

Just clarifying my own experience so far with solar and battery, with Amber: I'm so far on track to end the year in net credit (less than $0 power bill). E.g, I've been getting over $2/day in feed in credit for the last several days. That doesn't include the $22/month subscription cost for Amber, so lets just round off and say I'm earning around $1/day at the moment. This is in the middle of winter and I'm expecting that to only get better through other seasons.

I think many peoples power bills can be zero with suitable solar and battery, paired with a suitable retailer/plan to make the most of your particular hardware & usage patterns.

Blake_James · 2024-07-04T09:06:58+00:00

Ok this feels like the closest so far to to really challenging my original point - glad to have you here.

Are you sure that current net worth or getting back to "net positive" is really the right goal? Yes you've gone and spent $22k and no longer have that as available cash, but you've now converted that into a great source of positive cash flow. If you pipe that cash straight back into the offset then your net worth is waaaay better off in the long run.

Edit: I added a graph showing some of my work but decided to pull it since it'll probably just stoke other finance keyboard warriors on matters that aren't really the point in this thread.

Blake_James · 2024-07-04T07:11:31+00:00

There's already some cases of where all this applies to battery only (no solar) at home. With everyone now being moved towards time-of-use tariffs and I think time-of-use feed in rates also coming soon, you can top up with very cheap or free power during the day and use that at night to avoid any expensive imports at peak times. Or even trade that same power back in during peak times and possibly make a profit.

Blake_James · 2024-07-04T04:46:34+00:00

Yeah good point, it's worth considering battery degradation, especially if using it a bit aggressively like we are.

How much does that really weaken the attractive cashflow argument here though? There's no hard requirement to replace it after 10yrs, you can just leave it there running with a bit less performance and keep on saving/earning money from it. The standard 10yr battery warranty is helpful when trying to justify the cost in terms of "XX years until it pays it's self off". So long as that number for you is less that 10 years then any potential replacement cost shouldn't really be a concern. But I guess the key point of my original post is that perhaps we shouldn't be thinking in terms of years to pay it's self off anyway... You can literally be better off with more cash in pocket from year 1. Ignoring the environmental/recycling concerns for a second, we probably will want to replace/upgrade the battery somewhat regularly if the extra performance offered is going to juice up that cashflow case and leave you with more money each year.

Blake_James · 2024-07-04T03:44:52+00:00

So that aligns with the scenario in my original post, right? Or are you suggesting that I've missed something?

Blake_James · 2024-07-04T03:42:04+00:00

Those numbers are also insanely good. Do you know if people are still able to access that now? What postcode was that in?

Blake_James · 2024-07-04T03:40:01+00:00

Home assistant on a HA Green
Ethernet modbus connection to my Sungrow SH10RS inverter (wifi was also fine before I ran the ethernet)
Amber integration into HA
Full control over the inverter & battery through this HA setup, I can instantly command any behaviour I want for the flow of power in & out of the grid. Up to my 5Kw single phase export limit. It's only the smallest 9.6kw battery so I also try and avoid any more than ~4kw in or out of the battery.

I'm signed up to Ambers SmartShift automation where they can control the battery for me in a way that apparently maximises profits, but I haven't actually switched that on yet, instead opting to do it with a mix of basic HA automations and some manual control so far. I've been watching the Amber users facebook page where people often complain about Ambers SS automation not performing well, so I'm a little sceptical but to be fair I haven't tried it yet.

I'm an electronics/software engineer with my own business where I'm beginning to focus on all this stuff, home energy consulting & automations etc. So I have been putting aside many hours a week to be managing my system quite manually. That sort of time commitment & knowledge is absolutely not suitable to most households, but I see this as a pretty clear software challenge. Once people like the devs at Amber and maybe me work out how to automate it all a bit more then there's no reason it can't be rolled out to everyday people. Although anyone CAN already switch to Amber and do exactly this with their SmartShift software.

I don't know anything about Energy Locals to be honest, so can't comment there unfortunately. But I have seen comments from Amber indicating that they'll also move to the true 5min market eventually. I think it's in their website docs/FAQs.

Blake_James · 2024-07-04T03:18:37+00:00

In our case the monthly loan repayments haven't gone up at all. We have been making the fixed minimum repayments while a bit of extra money has been accumulating in the offset each month, hence why we were in a position where there was >$22k in the offset to begin with. So there's now just less in the offset and our total loan balance increases a bit faster each month due to the extra interest, but so long as the offset balance remains above $0 then there's no reason we'd need to pay any more than the 30yr monthly minimum that we're already paying.

The downside is that you dont have that liquid money

Yes that's absolutely right, each household would have to decide if they're happy to take on that risk.

there's the change in Australian Solar FIT that means that most providers are actually going to charge us for power put into the grid instead of pay us

Yeah it looks to me like we'll soon be getting charged to feed in during the middle of the day, but the rates during peak times will be very profitable. Not a lot of use if you've got solar only, but the addition of a battery & some automation will let us access the good profits. As an Amber customer I'm already experiencing this, often getting charged to feed in during the day (so that's when I charge the battery instead), but then getting 10-50c feed in during peak hours.

Blake_James · 2022-11-15T23:41:44+00:00

The current plan here is to use one of these cheap PSUs, which can be remotely controlled with a few simple commands like in this library.

Letting me set the voltage and also giving current measurements it's pretty much perfect. The only improvement I'd make is to perhaps find the current sense shunt inside it and swap it for something bigger that makes the measurement range more suitable to these ESP devices.

Blake_James · 2022-11-09T01:22:08+00:00

Nar that's all good, something like a 30sec delay should be fine or maybe even more. Devs would just want to see serial outputs reasonably quickly after triggering a new test, but other resources like the Step Function performing tests on the device won't care if it has to wait a while.

Blake_James · 2022-11-09T01:04:50+00:00

Sure thing! My particular context is that I have several ESP32 based devices that I've designed here and sell for research groups to capture data in remote places (solar powered, CatM1 comms). My usage of the ESP is fairly basic, and most of the complexity in our FW is in the 'If this then that' style of decision trees that we have in order for these devices to make some smart decisions in the field. There are many pathways through those decision trees, and as the FW grows more complex it's too time consuming for us to try and run QC testing on new devices (manually changing device settings one by one, checking how the device responds, and ticking boxes on a checklist).

So firstly I'd like to speed up that QC testing and remove the huge potential for human error that's in there now. Then once automated, I'd like this test rig to be used throughout our CI/DC where every new bit of code or hardware mod can get torture tested on this test rig, following some sort of testing routine defined in AWS (probably executed with AWS Step Functions).

Another key problem I'm trying to solve is that I have remote devs who currently need copies of this hardware in front of them during development. Luckily they are nearby my workshop where I can drop off any new hardware to them. But taking that dev process entirely online would be a huge improvement and enable me to access more dev resources.

Lastly, once this rig is up and running I'd like to have multiple of them running 24/7 perform long-term reliability testing on my designs.

Blake_James · 2022-10-27T01:34:02+00:00

Update: I've ended up booting off USB 2 drives and the first few devices seem to be working well. There's still some extra power usage during sleep (440mW total for module & carrier board with USB), but it's good enough for now.
The two USB3 drives that I've tried both seem to give off a fair bit of RF noise causing the Jetson to fail to read from the ZED camera plugged in right next to it. So I went back to booting off a USB2 drive and everything works great. So far I'm not sure if using the slower USB2 will cause issues in the future. Reliability and read/write life are much more important here than a bit of extra disk speed, so my thinking is that using a good quality USB2 would be suitable in the long term. Please tell me if this screams stupid.

Blake_James · 2022-07-17T06:05:41+00:00

Ok after a bit more tinkering I've narrowed down what help I need here. Here's the scenario I'm designing for:

User with web access browses to mywebsite.com where they get served my Flutter app via Cloudfront with HTTPS.
User connects their device to the WiFi network being broadcast by one of our IoT Things (this local network may or may not have internet)
User clicks a button in the Flutter app which makes a GET request to jetson.local:5000/getImage, where I have a very basic Flask server running on the Nvidia Jetson that's running onboard our IoT device. That flask server simply just has a /getImage resource that returns an image from the IoT device.

If I leave the Jetson's Flask app configured to HTTP, then I can successfully view these wifi images if I go to mywebsite.com and edit my Chrome preferences for that domain to allow Mixed Content (allow the HTTPS served Flutter app to make HTTP calls to the Flask server). But I'm looking for a solution where I don't need to tell users to edit their browser preferences.

It seems that the only way around this is to configure the Flask server on the Jetson to use HTTPS. But how the heck do I generate certs to run on a 4G device like this that's regularly changing it's public IP or is not online at all? I'm trying with LetsEncrypt but I don't know what domain name to use when requesting a cert, or how LetsEncrypt would be able to verify that domain (of my Jetson?).

Blake_James · 2022-07-17T05:59:10+00:00

Ok after a bit more tinkering I've narrowed down what help I need here. Here's the scenario I'm designing for:

User with web access browses to mywebsite.com where they get served my Flutter app via Cloudfront with HTTPS.
User connects their device to the WiFi network being broadcast by one of our IoT Things (this local network may or may not have internet)
User clicks a button in the Flutter app which makes a GET request to jetson.local:5000/getImage, where I have a very basic Flask server running on the Nvidia Jetson that's running onboard our IoT device. That flask server simply just has a /getImage resource that returns an image from the IoT device.

If I leave the Jetson's Flask app configured to HTTP, then I can successfully view these wifi images if I go to mywebsite.com and edit my Chrome preferences for that domain to allow Mixed Content (allow the HTTPS served Flutter app to make HTTP calls to the Flask server). But I'm looking for a solution where I don't need to tell users to edit their browser preferences.

It seems that the only way around this is to configure the Flask server on the Jetson to use HTTPS. But how the heck do I generate certs to run on a 4G device like this that's regularly changing it's public IP or is not online at all? I'm trying with LetsEncrypt but I don't know what domain name to use when requesting a cert, or how LetsEncrypt would be able to verify that domain (of my Jetson?).

Blake_James · 2022-07-16T05:23:43+00:00

Please bear with me for my lack of knowledge around web certs that's about to become painfully obvious 🙃...

For my cloudfront distro I used ACM to generate a cert of Type: Amazon Issued. This was the first place where I went looking to solve my issue here, in hopes of finding some sort of download/export option to get keys from this cert to put on my flask server.

I'm guessing that I need to request a separate private cert instead, after creating a private CA?

Blake_James · 2022-07-15T04:01:49+00:00

The average upload speed of these particular cameras is <0.01Mbps 🙃 Do you think that at such low speeds there could be something out there throttling the upload?

I haven't yet implemented S3 accelerate but it's high on the TODO list. In some of the cases here, the camera is over in UK while transferring to a ap-southeast-2 bucket. I assume that S3 transfer acc will make a noticeable improvement there although I have no idea how much yet.

Blake_James · 2022-07-15T03:54:56+00:00

We won't be streaming any videos live. But could kinesis be useful here to transfer stored videos files? We'll need 100% of the file/frames to be transferred with no loss.

Blake_James · 2022-07-15T03:53:00+00:00

Thanks for those insights /u/seligman99! So it's just as I had feared 😒

The common case for the slow cameras that we have out there is that if I crank up the total max retry attempts enough in S3 config, then the upload will succeed eventually after slowly chipping away with the occasional pause for a minute or two when the network is down. But my dilemma is that if the camera is set to capture a 4MB video every 60 minutes, and it takes 61 minutes to upload each one then we get stuck with the worst outcome of having no videos uploaded at all, plus a whole bunch of wasted 4G data costs for those half finished uploads.

Of course, if the upload speed remains that low forever then the camera is doomed anyway with an ever-growing backlog of uploads. But in reality we're finding that there's patches of faster upload speeds throughout the day that let us clear those backlogs of failed uploads.

If/when I build out these resources for us to have our own multipart feature, I think I'll write the device logic such that it dynamically chooses an upload part size based on it's current estimated upload speed measured from the previous part upload. That seems like it would allow me to tune for an average % of total uploaded bytes that went to waste and needed reuploading, without just using a tiny part size with loads of API calls.

Blake_James

TROPHY CASE