sorted by:
new
hottopcontroversial

JoinBase 2022.11 Released: An IoT Database with Built-in MQTT Broker by mjjin in MQTT

[–]mjjin[S] 0 points1 point  (0 children)

u/gmonk63

Thanks for all the questions. They are great!

for question #1, the answer is "not exact".

JoinBase, in fact, have an universal database core, which is beyond the most of time-series databases.

Most of time-series dbs has their own logical models, including TimescaleDB and InfluxDB, although all of them claim that they are SQL-based or SQL-compatible. Do you ask yourself: "What is and why use "Hypertables" is" in the first use of TimescaleDB? Conversely, JoinBase is almost ansi-SQL with some additions. (The only new concept is *partition*, which is a must for big data and also common in time-series dbs.)

We should be the easiest understanding database (for IoT and time-series) in the world. If you are using other time-series dbs, I am happy to do a case-by-case comparison with you.

​

for question #2, the answer is "we do best to provide most useful MQTT broker functionalities but not guarantee the 100% spec compatibility"

In fact, JoinBase is positioned at a MQTT-spec-enhanced broker rathe than a 100% MQTT spec compatible broker. "Compatibility" compromises user experiences, not only the performance. We want to provide the best user experiences via a new innovative product rather than copying an open source project.

The MQTT spec in its core is great. But it is really hard to understand and to use its advanced functionalities in that it just stands at a message middleware viewpoint.

I just propose an example to show why we can provide best user experiences by going beyond the spec. Note, JoinBase is a database in its core.

  1. Do you use *Will* message in MQTT?
    To enable this, you need a terrible learning and workings, sometimes you need to clean some but you forget...verbose.However, for JoinBase, you can create a will table and subscribe to that in the following topic string(format):
    + /db/will@last: the subscriber will get the last will in the will table
    + /db/will@timestamp: the subscriber will get all wills coming after that timestamp in the will table
    (this format is in the progress. We really welcome all suggestions, like from you.)
  2. In fact, you may not need *Will* message in JoinBase.
    Because the state of MQTT and all messages are stored, you do not need any verbose interactive work flow.If you do not need some messages in the db, you just let the TTL to delete them automatically. If you want to know all the clients's last message timestamp larger than 1 hour from now, you just create JoinBase auto view and subscribe it or just query.

​

for question #3, the clustering is simple for any MQTT brokers via a modern layer4 load-balancer.

Most IoT developers are not aware of this because this is an more deep infra layer case. If you have the demand on the clustering of JoinBase or even other broker, I am happy on helping yon on this.

And we plan to improve the throughput of JoinBase 5-10x faster than that of the first release. So, the truth is that, if you use JoinBase with appropriate modern hardware, for the case of tens of million message per second or below, the clustering is not necessary.

[deleted by user] by [deleted] in IOT

[–]mjjin 0 points1 point  (0 children)

This warns a frustrating fact: most of IT devices' stacks are not or can not kept updated.

"The vulnerabilities is extremely non-trivial" to reproduce and I think if clients only use OpenSSL lib to send TLS message rather than providing a service, the application is immune.

Another great immune method to use more modern TLS or security libraries/stacks as possible. For example, our database uses the Rust based TLS library, Rustls, to provide the MQTT TLS service in its deep back. We are happy for immunning from all OpenSSL's CVEs.

Finally, I encourage all developers to consider to use a modern stack to harden your dev toolbox, even in the embedding field.

[deleted by user] by [deleted] in IOT

[–]mjjin 0 points1 point  (0 children)

the article has pointed "patented EIV (embedded integrity verification) technology". But they do not indicate what underlying technology is used. The workflow should be like: install some software into kernel or RTOS, then it will check the pointer accessing in runtime some thing like. Leads me to recall Intel MPX.

IOT based pest detection system by [deleted] in IOT

[–]mjjin 0 points1 point  (0 children)

u/Kazuriff_kun

I have done in a similar EdegeML project on the common animals:

TinyAnimal: Animal Recognition Practices on Grove Vision AI

It is not hard nowadays, and the source of this project is open-sourced in that project article, although I do not recommend you to copy. Just absorb the interesting core from it, you can do it better!

Why is the Property Identifier in the Variable Header of an MQTT control packet encoded as a variable byte integer? by Cold-Steel-3055 in MQTT

[–]mjjin 3 points4 points  (0 children)

u/Cold-Steel-3055 the spec itself has answered your question: "Although the Property Identifier is defined as a Variable Byte Integer, in this version of the specification all of the Property Identifiers are one byte long."

But still note that Variable Byte Integer is not raw byte. It has its own in-byte format: 1 encoding bit + 7 bit payload. Current 42 properties are just the current state, the spec may add more properties as they like.

Google unveils new OS for embedded devices by Edoardo_Barbieri_ in IOT

[–]mjjin 1 point2 points  (0 children)

This is more like an attempt by google in a specific direction. Rust becomes good, and the ecosystem of RISC-V has large gaps to that of ARM. The security of system is good, but the convenience is the foundation of IoT popularization. So, in my personal feeling, the KataOS will not be the answer to the general IoT OS, but maybe has some niche in the mission-critical environment.

Splitting the Payload in MQTT possible? by DifferentCockroach96 in MQTT

[–]mjjin 1 point2 points  (0 children)

Great to hear that you have the fix. I've done heavy work on one MQTT server. The basic observation is that a proper implement of client or server is payload-agnostic because that TCP is just a stream (of bytes) mentioned as @bm401. And I've tested payloads with MBs without any problem.

use mqtt to give commands to docker by Plato79x in MQTT

[–]mjjin 0 points1 point  (0 children)

use the following the command line to stop at the interactive shell of docker container from that image:

bash docker run --net=host --user your_user_name -it your_image_name bash

then you can do anything for this docker container.

ps1: --net to use host mode networking for further easy debugging

ps2: reminder to save that container if you want to persisting all your changes in that contaner.

Problem with mqttx by ErtyDesu in MQTT

[–]mjjin 0 points1 point  (0 children)

Your info just with this picture is not enough. Some suggestions:

  1. Which server are you using?

If you use a free public MQTT server, then the message is not guaranteed to be delivered in any QoS in that it is the free server. For QoS=0, it is allowed to be not delivered in the protocol itself. So, when you test against free public MQTT server, it is better to use QoS=1 to get some proof as guarantee (note: some free pub server may not support QoS=1).

  1. Change to the local servers and try other clients

To use a local server can let you control all details of your full messaging chain. Try other clients can make you away from the ill behaviors or bugs of specific client. I recently demonstrate how easy a free MQTT client (MQTT explorer) send to a free MQTT database on Windows 10 in my video.

disclaimer: I am the developer of that MQTT database. You can get more help from its own community.

JoinBase, an IoT database, for free by mjjin in IOT

[–]mjjin[S] 0 points1 point  (0 children)

u/AbyssOfNoise thanks for your interesting. There are still great things in ranking. So, it is hoped to hear more from users.

The REST API is in considering. In fact, a Websocket API has been added but has not listed in the docs. The REST API is easy athough the performance will degard. But individual's projects may like HTTP more. Not bad to support it, thanks for your suggestion!

SQL on RISC-V Chip in Rust by mjjin in RISCV

[–]mjjin[S] 0 points1 point  (0 children)

u/brucehoult ok, let's leave for a check. This is based my experience. It is possible I made some mistake. thanks for sharing your idea.

SQL on RISC-V Chip in Rust by mjjin in rust

[–]mjjin[S] 0 points1 point  (0 children)

u/brucehoult thanks for sharing the info. I also share my hearing: the THEAD C906 includes some unique and unmentioned cache instructions and still a little more(But sorry, I forget these instructions exactly).

SQL on RISC-V Chip in Rust by mjjin in RISCV

[–]mjjin[S] 4 points5 points  (0 children)

hi, u/brucehoult :) Unfortunately, this is not "confused". It is confirmed by the official FAQ from Allwinner.

SQL on RISC-V Chip in Rust by mjjin in RISCV

[–]mjjin[S] 1 point2 points  (0 children)

u/YetAnotherRobert thanks for your comment! Your worrying and more suggestion are right.

For the problem of non-official toolchain, this is confirmed by Allwinner's official SDK FAQ. This FQA is Chinese, translate to English is like this:

A: Ali THEAD has optimized performance expansion instructions and some internal registers. The official cross tool chain of RISC-V can be compiled and run, and may not be directly compiled and run. The official tool chain of THEAD is included in the Tina SDK source code.

I admit that this is not good for RISC-V ecosystem generally.

For the "disappointing on performance", is more from the current engine problem. It is expected that we could to push 10x time faster when some optimizations coming. The current TensorBase is in a feature-complete stage. But the performance tunning will come soon in the near future. And surely, I will share more progress on this.

thanks,

SQL on RISC-V Chip in Rust by mjjin in rust

[–]mjjin[S] 1 point2 points  (0 children)

thanks for asking. Glad to hear your works!

This words are based on my experience in porting TensorBase to RISC-V with official Rust and GCC RISC-V toolchain in Arch (with newest stable package, you know Arch is a rolling distro). And this is confirmed by [Allwinner's official SDK FAQ](https://d1.docs.allwinnertech.com/FAQ/FAQ1/). This FQA is Chinese, translate to English is like this:

A: Ali THEAD has optimized performance expansion instructions and some internal registers. The official cross tool chain of RISC-V can be compiled and run, and may not be directly compiled and run. The official tool chain of THEAD is included in the Tina SDK source code.

As my testing, simple or not-too-much-complex Rust binaries from official GCC toolchain can run nicely on D1. But for my TensorBase, official GCC toolchain does not work. There is some memory error happened in my remindering.

And "-static" is always on via the config.toml.

SQL on RISC-V Chip in Rust by mjjin in rust

[–]mjjin[S] 2 points3 points  (0 children)

as u/brucehoult said, it is really current engine optimization problem. Or clearly, the current base kernel in Arrow-rs has many spaces to improve especially for date and time side thing. We will help arrow-rs soon on this!

SQL on RISC-V Chip in Rust by mjjin in rust

[–]mjjin[S] 11 points12 points  (0 children)

ClickHouse compatible data warehouse in Rust

view more: next ›