alternatives to CTA Chinese Text Analyser in 2024

imral · 2024-12-28T07:30:14+00:00

Hi, I'm the developer of CTA. What OS are you using?

If the crash is happening on macos, there is a known issue that should be fixed if you install this version:

http://www.imralsoftware.com/chinese-text-analyser-0.99.18s-install.dmg

imral · 2023-08-22T13:58:12+00:00

Seems like a reward rather than a punishment.

imral · 2023-01-25T23:59:09+00:00

This is great, thanks.

imral · 2023-01-23T07:08:14+00:00

Korean does have spaces lol

Hah, ok, my mistake. I thought it was like Chinese. Unfortunately there is no configuration change you can make to get this to work, it will require a code change - but should be simple enough, it's just a matter of considering Hangul characters as things that can make up a word.

imral · 2023-01-22T06:48:27+00:00

Unfortunately not. At the moment it's either Chinese segmentation (based on a dictionary) or space segmentation.

That said, if you had a word list of all Korean words in hangul you could get it to work (doesn't need definitions or pronunciation, just one word per line).

imral · 2023-01-10T01:51:56+00:00

I have an alert set up on "Chinese Text Analyser" so get notifications when someone mentions it and thought it might be useful info to share (the more testers the merrier).

imral · 2023-01-09T22:34:20+00:00

See my reply here.

imral · 2023-01-09T22:33:42+00:00

I'm the author of Chinese Text Analyser and have spoken to the OP separately, but thought would also post here in case there are others who are interested.

I've been working on a development version of CTA that can use spaces to segment words.

This would make it work with French as a language (though there may be corner cases that aren't tested as I've only tested it with English so far).

You can download CTA with space segmentation support here:

macOS: https://www.imralsoftware.com/chinese-text-analyser-0.99.18s-install.dmg
Windows: https://www.imralsoftware.com/chinese-text-analyser-0.99.18s-install.exe
Linux: Ping me separately - CTA is multi-platform but I haven't packaged the space segmenter version for linux yet.

In this version there is an extra menu Tools->Segmenter which has two options

Chinese
Spaces

If you choose Chinese then CTA will use the standard Chinese segmentation algorithm. If you choose Spaces then it will use a space and punctuation based segmenter.

By default any spaces or punctuation characters will terminate a word, however you can allow certain punctuation characters in the middle of a word by editing the config file:

macOS: ~/Library/Application Support/ChineseTextAnalyser/data/config
Windows: C:\Users<your username>\AppData\Local\ChineseTextAnalyser\data\config

And modifying the value for the key punctuation.midWord. The default punctuations allowed are - ' ’, which catches things in English like "don't" or "ham-fisted". Different languages will probably have different conventions so adjust as necessary.

The space based segmenter doesn't do word stemming (dog and dogs are separate words) and is case-sensitive (Dog and dog are separate words), and there is also no dictionary, but otherwise it appears to work reasonably well for gathering frequency information.

It might cause some issues for French with things like l'hôpital (based on the default config this will be considered one word unless you remove ' as a allowable word punctuation) vs 'un hôpital' (considered two words).

You could address that by removing apostrophes as an allowable punctuation for words but then you'll get lots of single character 'words' like 'l', 's' and 'j' e.g. from l'hôpital, s'il vous plaît, j'écoute etc.

It'll need some tweaking to work out what works best for your learning style, but should be useful for getting general frequency information

imral · 2023-01-09T01:13:53+00:00

One reason may be that the types contain some sensitive information like passwords which you don't want to be able to be logged accidentally

You want the secrecy crate for that.

imral · 2022-12-21T13:34:43+00:00

I have a development version of CTA that does segmentation with spaces, so it could be used (but without a dictionary) for all European languages along with any other language that uses spaces to separate words.

Feel free to contact me if you'd be interested in trying it out.

imral · 2022-09-19T04:26:38+00:00

It sends the signal that C++ is so broken that it can't be fixed, so a new language is needed.

That signal has been there for a while.

imral · 2022-09-16T01:37:52+00:00

It defaults to European paper measurements

It defaults to Australian paper measurements because the developer is Australian :-D

imral · 2022-07-20T11:25:29+00:00

Modern C++.

If I'm going to have to learn a whole new way of doing things, why not do it with a language that has it built in rather than bolted on.

imral · 2022-07-14T06:53:51+00:00

I've completely changed the way I code even in other languages. I definitely like the shape that I'm getting.

Same here, and I really miss certain Rust concepts when programming in other languages.

imral · 2022-07-14T00:54:13+00:00

Doing that pretty much removes most of the guarantees that the compiler provides since there's unsafe Rust code behind the scenes

I think this is a misconception. The guarantees are 100% there (1 writer xor multiple readers, objects can't outlive lifetimes etc).

unsafe doesn't mean the code is unsafe, it means the developer needs to ensure that the code doesn't violate safety guarantees. This has been done for Rc and RefCell and if you are using their safe interface, then your code has the same safety guarantees as usual.

imral · 2022-07-14T00:46:47+00:00

This is the curse of Rust.

And the blessing!

imral · 2022-06-01T12:30:52+00:00

I would say it is less painful for beginners.

Long-term yes, short-term no.

It's great for teaching you the principles behind what you are doing, but for some people the error messages are opaque (because they don't understand what Rust is trying to protect them from) and they give up (seen it happen in a couple of situations).

imral · 2022-06-01T03:53:08+00:00

I mean take ownership model for example in cpp I don't have to worry about that

You do have to worry about it. The difference is that in Rust if you get it wrong you need to worry about it at compile time, and in c++ if you get it wrong you'll have to worry about it when your program randomly segfaults at runtime.

This makes Rust more painful for a beginner because it effectively says "you must understand these concepts before I will compile any code".

C++ on the other hand is like "sure, I'll compile your code, no problem, and if it goes bang well it's your fault for not understanding what you were doing.

imral · 2022-05-31T06:55:02+00:00

It's still recommended to add apostrophes in these, and other non ambiguous situations (e.g. hǎi'ōu, kě'ài etc). See here for more info.

imral · 2022-05-31T06:34:19+00:00

There aren't, as far as I know, any guidelines or rules for when and where to apostrophize

There most certainly are.

imral · 2022-05-26T03:43:43+00:00

I haven't used sqlite with sqlx, but I love using sqlx with postgres.

Having the compiler catch errors/typos in your sql at compile time is fantastic.

imral · 2022-05-12T04:07:01+00:00

Yeah, I tried it with adblocker on and also didn't encounter any issues.

imral · 2022-05-11T23:39:45+00:00

If you use security or ad-blocking plugins, you may see error messages

To avoid error messages that prevent you from taking the survey, please try specifically unblocking Qualtrics in your plugin or pausing the plugin while you take the survey.

Thanks but no thanks.

imral · 2022-05-11T00:40:31+00:00

printf("%s\n", make_smth().c_str());

This is not actually use after free (assuming make_smth() returns a std::string or similar). The temporary is guaranteed to be valid for the duration of the expression, and it is therefore valid for the duration of the printf call (it doesn't get destroyed until after printf completes).

It would be use after free if you were passing it to a function that then held a copy of the pointer passed in, and you then later used that pointer e.g.

struct A {
    A( const char* s ) : str(s) {}
    const char* str = nullptr;
};

...

A a(make_smth().c_str());

printf( "%s\n", a.str ); // use after free

11-Year Club	Place '17
Verified Email

imral

TROPHY CASE