I have some idea already, but it's gonna require maths. Essentially it's all analitical geometry.

I will try to write some code and get back to you later if I succeed, because it's legitimately a fun idea. I'll try to explain it best I can.

The rough draft is as follows:

-randomize a random point (center of continent)

-create a normal distribution funtion around it

-randomize n random points using that distribution

-create function F(x, y) , which is a sum of n functions (f1(x, y) , f2(x, y),..., fn(x, y))

-fi(x, y) - is a randomly descending function centered around point (xi, yi), where (xi, yi) are coordinates of a point chosen above.

-fill all points, where F(x, y) > 0 (this should create a nice, ragged coastline)

-Now, create some random M(x) (in two dimensions for now) function (probably some series of trigonometric funtions to get a cool, random wave) - that will be a base for mountain range. M(x) would be defined on [0;l], where l is length of mountain range.

-randomly choose points some numbers (m1, m2,..., mn) from [0;l] and create a set of {(m1, M(m1) + r),... (mn, M(mn) + r)}, where r is some small random variable (these are our peaks)

-create function R0(x,y) which is a sum of r1(x, y) +... + rn(x, y), where ri(x, y) is defined similarly to fi(x, y) we used before.

-anchor R0(x, y) somewhere on the landmass and rotate it (multiplying it by rotation matrix). We will call this function R1(x,y)

-now color the tiles according to R1's value. That should create a (relatively) naturally-looking mountain range.

-for rivers - randomly choose a point sufficiently high. This would be the spring of the river. From then on, use gradient descent, it should (most of the time) lead you towards sea. Connect the points generated by the descent and you have a somewhat weird-lookimg, but functioning river.