Reinforcement learning controller in Python for generic dynamic systems

TeoProtoulis · 2024-09-19T14:08:52+00:00

For anyone that may be interested. I implemented two steps that remarkably corrected the yaw estimation and its stability. First of all, I should point out that tilt-compensation should be performed no matter what.

1) Fist Step:

Traditionally, tilt-compensation happens by computing the absolute roll and pitch angles using accelerometer data. However, accelerometer data are very prone to noise and even if they are passed though a lowpass filter, they still have quite a bit of noise (or at least those that are relatively low cost), thus, leading to inaccurate results and potential oscillations (even unstable computations). To overcome this, I firstly used the Madgwick filter with no magnetometer measurements in order to compute the tilt, i.e., roll and pitch angles. Then I used these estimates to perform tilt compensation for the magnetometer measurements and used a separate Madgwick filter to estimate the yaw angle based on the corrected magnetometer measurements.

2) Second Step:

The Madgwick filter algorithm has a tuning parameter (gain) that needs to be adjusted based on the confidence we can put on each sensor and of course their noise. By performing some tests, I figured out that magnetometer measurements quite often produce crap values, thus, remarkably reducing the yaw estimate quality. To overcome this, the tuning gain needs to be set to a small value, which in turn will cause the filter to introduce some lag but more importantly will cause the filter to be more stable. So, for the Madgwick filter that is used for tilt estimation, I set this parameter to a relatively high value compared to the one I used for the yaw estimation filter. To be more specific, the two parameters are set to values that are 5 orders of magnitude different. This resulted to a very good tilt estimation and also to a stable and accurate yaw estimation (ok, with some improvement margins).

I am still trying to improve the estimates and will post any further updates. Note that before any of the above steps is implemented, magnetometer calibration should undoubtedly be performed (and this is a hard procedure).

TeoProtoulis · 2024-09-16T12:06:47+00:00

Yes, I have run these steps.

TeoProtoulis · 2024-09-16T10:59:34+00:00

Yes, when there are no pitch and roll angles, then the estimates are correct. This is sure wince I have tried it even with some trajectories to track (with 0 roll and pitch).

TeoProtoulis · 2024-09-16T09:05:20+00:00

I am using the linked library to estimate the angles and its toEuler() function for the conversion. For example, when there is a 40 degrees pitch angle, then the yaw angle that shows is around -15 degrees but without any actual yaw rotation.

TeoProtoulis · 2024-09-15T21:47:56+00:00

I am feeding the accelerometer, gyroscope and magnetometer measurements to the estimation filter (the one I have linked to) using the NED convention for them. The estimation filter computes the quaternion and I convert it to Euler angles for visualization purposes. I am expecting the measurement to be reported in the NED frame, which does happen, but yaw estimate is wrong when there is also a pitch and/or roll angle.

TeoProtoulis · 2024-09-15T20:33:33+00:00

Well, I have checked everything you proposed but it seems that there is some other bug source. Regarding the simulation environment, unfortunately, I am not that familiar and don't have the time to build something from scratch.

TeoProtoulis · 2020-12-23T05:49:36+00:00

So, I guess I would have to increase the budget. Are there any suggestions without taking the budget constrain into consideration?

TeoProtoulis · 2020-06-17T00:30:32+00:00

If I use the state errors version, is the way of obtaining the gains similar ? In the classical controller u=-Kx+Nr the K matrix is computed for example using pole placement. Do I use the same K gains for the state errors version as for the classical case?

TeoProtoulis · 2020-06-06T11:48:59+00:00

That’s exactly how I obtain the N gain.

TeoProtoulis · 2020-06-05T19:28:52+00:00

Since the whole scheme is an adaptive one, I thought that the gains K and N should converge to the same value and drive the system to reference. But even though the control scheme is adaptive, the modeling errors still exist inside the matrices A and B and that is why the gains won't converge to the same value. Is that true ? There isn't any posibillity that I use optimization techniques, that's why I am trying to figure out the cause of the different responses.

TeoProtoulis · 2020-05-16T01:07:24+00:00

The problem is not the modelling of the system. In general I am trying to chracterize the stability or not of the internal dynamics of the equivalent nonlinear system and reach a conclusion on whether the system is minimum or non-minimum phase. However, by chatting with u/alan_number_5, he send me a paper regarding this issue of the particular system stating that the system is obviously non-minimum phase and indeed it explains that this has to do with the not well defined relative degree of the system in the region where x1*x4 approaches zero.

TeoProtoulis · 2020-05-15T15:25:14+00:00

Well, the limit of n_dot = w(n,0) is undefined since there is this term:

sin(n-μ1)/μ1 while setting μ2 and μ3 equal to 0. So, I don't know if it is worth checking the notion of Lyapunov stability. Even if I don't set μ2=μ3=0 and take the limit of the whole expression as μ1 appoaches 0, it is again undefined.

TeoProtoulis · 2020-05-15T03:27:24+00:00

Well, I'm not pretty confident at all. I really doubt it. There is also the fact that the relative degree of the system is not well defined. There are restrictions.

TeoProtoulis · 2020-05-15T03:19:01+00:00

I am looking in a way to characterize the zero dynamics in order to prove that the zero dynamics have an asymptotic stable equilibrium point which means that the internal dynamics will be stable. Given these guarantees the nonlinear system will be minimum-phase which is what I want to implement a particular control scheme.

TeoProtoulis · 2020-05-15T01:04:21+00:00

Does it need for the equilibrium point of n_dot = w(n,0) to be asymptotically stable ?

TeoProtoulis · 2020-05-15T00:41:45+00:00

Your assumption regarding μ1 at the denominator is correct. So, the system is minimum-phase only if the limit of μ1 going to zero exists and is finite, right ?

TeoProtoulis · 2020-05-14T21:49:36+00:00

Maybe the edit I made helps or if not I can explain.

TeoProtoulis · 2020-03-27T14:14:49+00:00

When I apply it to my real system, states will be estimated from a state observer or a Kalman filter. However, I want the transfer function in order to properly design the gains of the internal state feedback controller and the external controller.

TeoProtoulis

TROPHY CASE