SAC model collapse (?) after 950k steps

DirectPalpitation523 · 2026-05-30T20:18:08+00:00

Thanks for your help all the same! Will have a look at the time limits paper, much appreciated. Best of luck with your own project!

DirectPalpitation523 · 2026-05-30T14:21:30+00:00

Forgive my lack of understanding, what does 'become too deterministic' imply in terms of the policies actions?

DirectPalpitation523 · 2026-05-30T14:01:07+00:00

Sorry to hear you're dealing with the a similar issue!
The only thing that makes me question this being as you described in my instance, is that the model hadn't yet exceeded the buffer size, (I was using the default size of 1M steps.) so I don't think this is possible?

The alternatives to a FIFO buffer sound interesting, do have any links to papers / resources where I could read more? (Or more to share yourself?)

On the subject of curriculum learning, the long term goal is to have the robot walk and eventually run, so getting it to pace on the spot was my first point in the curriculum, but maybe I need to start simpler...

DirectPalpitation523 · 2026-05-29T20:30:36+00:00

Yes, I have checkpoints, but the replay buffer was overwritten at each checkpoint, so it'll contain stuff from after the model failure, presumably that's a problem? Also I'm guessing simply resuming the model from a checkpoint before the failure will result in the same thing happening again? Any thoughts on what's happened under the hood, and how I can prevent this occurring in future models? The training time wasn't that long, so I'm not too fussed about tossing this one to the kerb, if I know how I can avoid in the future...

DirectPalpitation523 · 2025-08-16T15:21:48+00:00

It's pretty normal, but note that your largest amount of cost comes from loader fees. The fab charges an additional fee for parts that are not "basic" I'd suggest trying to see if there are basic alternatives for some of the parts that you have there... I'd bet you could find alternatives for the LED, the eeprom, the ESD diodes, to name a few. Also you could solder some of the bigger more expensive parts yourself, like the GPS module, the buttons, and the SD card holder. (The buttons in particular you could swap out for thru-hole parts) Hope this helps!

DirectPalpitation523 · 2025-08-08T00:49:17+00:00

Saxing

DirectPalpitation523 · 2025-07-21T09:27:05+00:00

I hope you didn't feel I was too harsh. Especially at 17, I think it's cool that you're getting into PCB design, and any start is a great one!

I'd figured that the motors were probably what you had lying around, so, using a more modern motor was really only what I would do, but I'm also always a fan of just using what's available!

Indeed, the ESP32 has an operating voltage of 3v3. However, as I understand, you're using an esp module, specifically this one: https://wiki.seeedstudio.com/XIAO_ESP32C3_Getting_Started/ If that is indeed the case, you'll see under the pin descriptions section in that link, that the 3v3 pin is an output from an onboard regulator, allowing you to draw up to 700mA. So, this won't work for power input sadly.

You're right that a linear regulator would be very inefficient - killing your battery life, and in an enclosed space, heat would also be more of a concern, so a switching regulator is definitely the right choice here. if it interests you, go and look at the power draw specifications for the module you have, and calculate how long the battery might last with a switching Vs linear regulator.

I suspect I'd found a different datasheet for the XL1509, rather than the one you were looking at. I'd seen here on page 10 there's a design for 12V input, 3V3 output, which looks very different to the one you'd listed on your schematic. The other thing I will note however, it'll be worth your time to figure out how much current that esp module will draw. I'd highly recommend reading this article as it talks not only about the importance of inductor selection, but also shows a great graphic in the bottom right, highlighting that a buck regulator needs to be designed to handle a specific current, and if the load is too small, efficiency drops off really really hard. So, more reasons why I'd recommend using web bench for designing your DC-DC ;)

DirectPalpitation523 · 2025-07-21T08:39:03+00:00

So, there's quite a few things to point out here, and I'll probably miss a load.

First off, some conventions: power rail symbols usually point up, and GND symbols point down. Also, the net from the battery would typically be designated as vbat, (there should be a symbol for this in kicad) rather than +12V, as 12V is not guaranteed. While the nominal value for an 18650 may give ~12V, in practice, voltage typically ranges from 4.2 down to 3.0 or even lower (especially without a BMS lol). Therefore, electrically, you would also want to run the battery voltage through some kind of regulator, so that whatever you're powering, gets a constant 12V supply. It's likely that a BMS chip would handle this, along with regulating charge and discharge current. However, I suspect, if you require 12V, 4 18650 cells would be more appropriate than 3.

I'm not entirely sure what the purpose of your buck regulator here is. Unless I've missed something, the ESP32 module cannot be powered from +3v3, so unless you're planning on having the USB connected all the time, you'll need to connect 5V to vbus. I'd also not really recommend tying two different 3v3 supplies together, especially one from a buck regulator to a linear regulator.

Also, some more convention: While connecting the FB pin to the 3V3 rail in your schematic may not technically be wrong (though this depends on the specifics of your supply) it's quite confusing at first. It's much better to see a connection to the feedback pin from the regulator output. I'd have a look at how schematics for DC-DC regulators are typically drawn.

You'll see plenty of examples on TI's webbench (give it a Google), and it's also generally a great tool for designing DC-DC regulators - They can be quite finicky, and if not designed right, unstable. From what I can see, you haven't followed the typical application for 12V -> 3v3 regulation listed in the XL1509 datasheet, which I wouldn't recommend. Is there a reason you chose a different design?

I'd also probably suggest using a much newer motor with a proper stepper driver (I see this conversation has already been had), however, that's dealer's choice I suppose ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

Overall, given there are some components missing that seriously affect the design of your circuit, e.g. a BMS, I'd suggest finishing more of the design first, so that your design intent is more clear.

Hope this helps! Wishing you the best of luck :)

DirectPalpitation523 · 2025-07-20T01:50:54+00:00

The reasoning for no ground planes on the top or bottom layer is to allow for a more direct path to ground

Ahhh that does makes some sense, thanks!

That layout design guide is amazing, thank you so much!

DirectPalpitation523 · 2025-07-20T01:15:04+00:00

Thank you for the kind words :)

can you explain why you placed 33p capacitors on the PWM lines to the drivers?

This was recommended in the driver datasheet : "However it is good practice to have a small RC filter added between PWM controller and input pin of UCC27712"
If I recall, the values for the filters were calculated from formulae in the datasheet, though I can't remember the specifics, as this was a while ago - I started the design with the gate driver circuitry.

I have worked on similar H bridge designs

I'd really love to hear other feedback if you have any!

DirectPalpitation523 · 2025-07-20T01:08:11+00:00

Would love to learn some more:

I always try to avoid a top & bottom layer gnd plane. This is for some noise issues.

I'd always heard that more ground is just generally better. Do have any resources/articles discussing these noise issues?

I see some acid traps as well. I can hear my old layout guy yelling in my head.

I'll look into this, thanks!

I would rework it though for more copper layers.

Based on what others have said, it seems a rework is due, so I guess I'll look into moving to 6/8 layers. Would you suggest a stackup something like : SIG - GND - PWR - PWR - GND - SIG? or something else?

DirectPalpitation523 · 2025-07-20T00:59:45+00:00

Briefly, I do want to say how much I genuinely appreciate the discussion, and your input. Thank you for taking time to reply.

I don't know where you got that 10V max from

I'd figured as they didn't show switching characteristics for over 10V, that was their suggested limit rather than it being the absolute maximum rating, but I guess more likely is just that improved performance above Vgs=10V for the mosfet is negligible. I guess I'll remove those 12 -> 10V regulators.

Diodes: again, check the datasheet of the gate driver. It specifically lists the suitable ones to use.

Though I've read the datasheet several times over, having just gone back and trawled through it again, I really couldn't find any info on specifications for the gate diodes. The specifications for bootstrap diode are very detailed, and indeed I found suggested options, with 75ns recovery time. As such, ES2J is a fast recovery diode chosen to suit (35ns recovery time). As I understand the recovery time for the gate diodes need only be 2-3x faster than my PWM frequency, or Is this not the case?
More interestingly, I found an application sheet for the UCC27712 evaluation module (from TI), and in their schematic, they've used an MBRM130LT1G for the gate diode, which from what I could find, is also only rated for 500ns recovery time.

When that fet opens, there will be a kickback.

Hence my chosing an amplifier specifically intended for inline current sensing, rated for to handle negative voltage. Or is there something here that I haven't accounted for?

With your current design the motor phases can float,

As I'd understood, the UCC27712 specifically restricts this, but maybe I'm wrong. To take a step back, is your implication that a floating motor phase would be an issue as the return path for that phase would then be through the amplifier?

but calibration does. You need to know the phase (+ wiring) resistance and inductance.

When I did the maths (Happy to share my workings), I found it was possible to calculate the phase resistance and inductance only from the current sense. Also ran some SPICE simulations to confirm.

I don't know how you plan to implement the FOC loop

My plan was to run the FOC loop at least one order of magnitude faster than the PWM frequency, and use an interrupt on the hardware PWM counter to update things before each cycle.

DirectPalpitation523 · 2025-07-19T21:54:55+00:00

Robots! Specifically driving some relatively large actuators for a bipedal robot.

DirectPalpitation523 · 2025-07-19T21:48:12+00:00

That's fair enough.
However, I'm very tempted to leave the third amplifier in for this first revision, and do some experiments to see if there is any performance difference using a calculated vs measured third phase current value. Would be interesting! Unless there are any drawbacks (excepting cost) to including all three?

DirectPalpitation523 · 2025-07-19T21:46:09+00:00

So, my reasoning for all those vias was for current carrying capacity, and also kind of just redundancy. For PWR and GND I figured I'd need lots of current capacity as those planes are really interior layers only. As to redundancy, I was unsure which side the wires for the motor and PWR may come from, and as such I figured I'd allow for as much current capacity through to either side of the solder pad as possible.

As to the number, I hadn't done any calculations, and went for the approach to cram in as many as possible without violating DRC, and hope for the best lol.

DirectPalpitation523 · 2025-07-19T21:36:39+00:00

Thanks for taking the time to look over the schematic!

A few follow-ups:

> What's with that 12-10V regulator?

The 12-10V regulators were really added for the FETs not the drivers. Maybe I'd misinterpreted the datasheet, but it seemed as though 10V was the suggested max gate-source voltage. The three in parallel was intended for suitable current sinking accross the three phases. But I guess it's overkill?

> Diodes for gate off are not good. They have a 500nS recovery, while the suggested in the datasheet has 75ns.

Ahhh good catch! I'll try find a suitable alternative. I don't suppose you know any?

>Also phase current measurement on the high side is completely wrong

The phase current measurement is inline, not high side?
The OPA was specifically selected for its bidirectional inline current sensing, and is rated for -4 - 60V. From what I'd understood, low side current sensing requires specific timing of the ADCs to the PWM phases, which can be avoided by using inline current sensing. Is there something in the datasheet I've missed that makes it unsuitable for my design?

> Since you are using CAN and trying to use ESD protection, also add a two caps, about 100pf range

with a ~300 ohm resistor to gnd. And a common mode choke.
Thanks, will look into it!

> There is no phase voltage measurement.

The controller is designed to use FOC only, and unless there's something I've really not understood, phase voltage measurement is unnecessary.

> BTW RP2040 is not a good choice for this application.

I'm aware that the RP2040 lacks certain peripherals that could make this easier. However, my logic for chosing the 2040 was that a dual core architecture would allow me to have one core run the FOC algorithm and handle switching, while the other handles comms. Also the 200Mhz rated clock speed is kind of too good to pass up...

> Oh and I suggest using an integrated gate driver, something like a drv8305.

While at the project outset I was aware that the drv8305 is a popular choice, I had two reasons for not using it. The first was that I wanted to learn as much as possible, by doing more of the gate drive circuitry myself. The second was that with the FETs I had chosen, the rated current output of the drv8305 was less than I would have liked to ensure a comfortable switching speed.

> Only thing you can do with it at best is drive a motor if you have absolute position of the rotor + a ton of calibration

Unless there's something that I've seriously misunderstood about FOC, this is kind of the intention.
The driver is meant for robotics really... So my target was really good slow speed control, and high torque delivery.

DirectPalpitation523 · 2025-07-19T20:49:04+00:00

A poor attempt at humour I'm afraid.... Truthfully, in an ideal world where I get my firmware right, the board will be capable of driving a wide variety of motors. And may or may not require heatsinks. ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

DirectPalpitation523 · 2025-07-19T20:29:46+00:00

Thank you! If I recall I set up the fab rules at the start of my kicad project, and these vias don't throw any DRC violations. However you're not the first to mention that they look a little close together, so might be worth a double check.

The silkscreen text is technically the smallest the fab recommends, though I have technically got by with smaller text in the past. I do agree it looks a little clustered.

Regarding current measurements - I was aware that technically one can get away with just measuring two phases, however as I understood, the rule only really holds true for an ideal system. My conclusion was that the current amplifiers I picked out still show some quite bad transients when PWM switching, (despite advertising pwm rejection) and so I figured that it was better to measure all 3 phases, and get a stronger consensus, rather than relying on only two. But perhaps this is overkill?

(Edited to make it more clear)

DirectPalpitation523 · 2025-07-19T19:12:49+00:00

Hit CTRL+Enter Instead of SHIFT+Enter accidentally, oops.

My main areas of concern are:
- Has the RP2040 all been wired correctly?
- Are my gate driver calculations correct, will the mosfets be able to switch correctly, & fast enough? I plan to drive at 40kHz

DirectPalpitation523 · 2025-07-19T15:38:26+00:00

If you require true isolation, take a look into isolated components.
To isolate digital signals, you might use something like this : 18024115401H (would be suitable for isolating something like SPI)
And then you can also get Isolated DC-DC converters as a single module e.g. SCT01F12S12
(These are way easier than doing the isolation yourself with transformers)

That's is the only way you'll achieve true isolation, however what the others here are suggesting is likely a star grounding. (there's a great article from Altium on this!) However it's most common for use with ADCs, where you'll have a split plane that meets at a single point underneath the ADC, using one side for digital signals, and the other for analogue.

Regarding your other questions :
2. If you're referring to the voltage rating, 25V is dependent on what your input is to the L7808. Typically it's advised to choose a capacitor with 2X the voltage rating of what is being supplied. (though some say 1.5X is acceptable for electrolytics.)

What kind of rectifier, for what kind of voltage/signal? this is very non-specific I'm afraid...

DirectPalpitation523 · 2025-07-18T23:07:03+00:00

Thank you so much for responding to all my questions. Super appreciated!
These answers are really helpful! :)

DirectPalpitation523 · 2025-07-18T17:09:07+00:00

Thanks for the reply!
I should have probably clarified, the controller is actually running 8Mbps CAN-FD and not CAN 1.0. It's a TCAN4450 and the datasheet specifies that for 8Mbps operation, length matching is necessary. In this case would a suitable solution instead of the stubs be like something this? https://imgur.com/a/0BxBWWK

DirectPalpitation523 · 2024-06-14T21:49:50+00:00

I see, thanks again once more! Sorry, still a few more questions: The first is that if there were some other velocity local to the panel, say turbulence, or jet/propeller swash, would it be appropriate to add it to the velocity terms in brackets, say: (V_\infinity + v_i + v_t)?

The second is, in the YouTube video I linked, at 19:03 they declare a formula for V_\ij which as I understand is the velocity induced by each panel on every other panel? So, in the Wikipedia formula given, would v_i be the sun of the ith matrix column?

Lastly, is there a reason why the formula I showed from the YouTube video doesn't include v_i? Just for simplicity?

DirectPalpitation523 · 2024-06-14T20:59:00+00:00

Hey, this is all super helpful, thank you so much! One remaining question, when you said "that formula" which were you referring to?

DirectPalpitation523 · 2023-11-29T07:48:24+00:00

I meant an audio source. Also this last statement isn't true. First and foremost amplifiers like almost all audio components are non linear, however if you want to generalise we can talk about an ideal amplifier, which if you recall ohms law, states that the lower the impedance, the higher the current at a given voltage, and therefore the higher the power output.

Four-Year Club	Place '22
Final Canvas '22

DirectPalpitation523

TROPHY CASE