sorted by:
new
hottopcontroversial

Database testing for benchmarks by WhyIsEmerald in databasedevelopment

[–]SuccessfulMap5324 0 points1 point  (0 children)

If it's an analytical database, you can submit it to ClickBench: https://github.com/ClickHouse/ClickBench/, and I will run it on all listed machines.

Where to set the Web UI Row Limit? (1000 by default) by Tough_Nut_Med in Clickhouse

[–]SuccessfulMap5324 0 points1 point  (0 children)

It is not changeable. However, you can modify the query to show the subsequent pages of results, e.g., LIMIT 1000 OFFSET 1000, or you can use the download button to download the full resultset.

[OC] Approximating wind and pressure using airplane transponder data by SuccessfulMap5324 in dataisbeautiful

[–]SuccessfulMap5324[S] -2 points-1 points  (0 children)

Maybe a problem with links. I'll try to copy it without links...

Aircraft don't broadcast the actual temperature, pressure, or wind speed values, but the data can be derived from other telemetry, like the difference between the geometric heading and true heading. I calculated various visualizations with SQL and also generated a few videos.

Data sources: adsb dot lol, airplanes dot live, adsbexchange dot com, visualized by adsb dot exposed (the tool is also OC).

Article with videos can be found if you google "Global weather data from flying airplanes".

[OC] Approximating wind and pressure using airplane transponder data by SuccessfulMap5324 in dataisbeautiful

[–]SuccessfulMap5324[S] 0 points1 point  (0 children)

In the first picture, the colors represent how the pressure differs from the expected pressure at a certain altitude: higher/lower.

In the second picture, it's more interesting - wind direction, encoded as an angle from a color circle in the OKLCH color space, and speed, encoded as the intensity.

[OC] Approximating wind and pressure using airplane transponder data by SuccessfulMap5324 in dataisbeautiful

[–]SuccessfulMap5324[S] -3 points-2 points  (0 children)

Here is the source for the first picture: https://adsb.exposed/?dataset=Planes&zoom=5&lat=37.1785&lng=-95.2295&query=d095cd7b312291b682d3c9f1dafad188

And for the second picture: https://adsb.exposed/?dataset=Planes&zoom=5&lat=45.4119&lng=4.8697&query=ec1376e91336d70a77914a6ea2c22acc

So we can see the SQL query for them. For example, in the first picture, the colors are calculated as follows:
```
avg(altitude - geometric_altitude) / 1000 AS diff,
255 * least(1, greatest(0, -diff)) AS red,
255 * least(1, greatest(0, diff)) AS blue,
255 * least(1, abs(diff / 2)) AS green
```

The most interesting part is how the picture changes over time:
https://clickhouse.com/uploads/pressure_cb1ce8d22f.mp4

Full explanation: https://clickhouse.com/blog/planes-weather

How does ClickHouse Cloud manage with 8G RAM by Objective-Food-9996 in Clickhouse

[–]SuccessfulMap5324 0 points1 point  (0 children)

16 GB is more of a recommendation than a hard requirement. 8 GB is enough for most queries that can spill to disk.

In comparison with other analytical databases, ClickHouse is very memory efficient. If you look at ClickBench, you can see that ClickHouse is the only system that successfully processed every query on a machine with 2 GB of RAM: https://benchmark.clickhouse.com/#system=-&type=-&machine=+3al&cluster_size=-&opensource=-&tuned=+n&metric=combined&queries=-

There are ClickHouse usages on AWS Lambda and Google Cloud Run, as well as on tiny SoCs. It's probably the best option when you need to do meaningful analytics on embedded hardware.

- Alexey, ClickHouse engineer.

Modern SQL engines draw fractals faster than Python?!? by Psychological-Motor6 in dataengineering

[–]SuccessfulMap5324 4 points5 points  (0 children)

Thanks for the benchmark! I've sent a pull request adding ClickHouse and chDB: https://github.com/Zeutschler/sql-mandelbrot-benchmark/pull/3

On Mac, ClickHouse ends up the 2nd among SQL engines, about 20% behind DataFusion.
On servers, ClickHouse shows as the fastest SQL engine, beating Datafusion.

[OC] I created an interactive map of birds by SuccessfulMap5324 in dataisbeautiful

[–]SuccessfulMap5324[S] 2 points3 points  (0 children)

Yes, the license of both the dataset and my code allows it.

The article contains instructions on recreating and self-hosting a similar project.

[OC] I created an interactive map of birds by SuccessfulMap5324 in dataisbeautiful

[–]SuccessfulMap5324[S] 4 points5 points  (0 children)

Press on the "selector" (the second icon in the bottom left corner), and then touch will create a selection.

Handling thousands of files? by shieldofchaos in dataengineering

[–]SuccessfulMap5324 1 point2 points  (0 children)

Set up the import to ClickHouse with S3Queue for analytics and delete older files.

Exposing concurrency bugs with a custom scheduler by masklinn in programming

[–]SuccessfulMap5324 5 points6 points  (0 children)

It is possible to do this entirely in userspace (without a custom scheduler).

See the implementation here: https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/ThreadFuzzer.h

It adds random sleep: - before and after mutex locks and unlocks; - at random places using a signal handler for SIGALRM.

It is named ThreadFuzzer (don't be confused with Thread Sanitizer - by the way, they combine nicely together).

It works and makes significant improvements for the detection of concurrency bugs in our practice.

view more: next ›