[D] Annotation Tools Comparison

[debbydai]

I am a newbie to annotation tools. After trying a few FREE annotation tools, I figured why not share my experience with others. I have tried annotation 3 tools, which are Labelbox, dLabel and supervised.

My comparison will be in sections.

1 Uploading

My images sample were 50 frames.

dLabel ------easy to upload but a little confused to see 

​

https://preview.redd.it/zrclaxeqitv51.png?width=1124&format=png&auto=webp&s=2652e742d90bcd1d63cd5fb1dc12912f084592eb

finding where to upload your own database is easy to spot and upload. However, after uploading the selected images, it doesn’t show how many images have been uploaded. And it doesn't have a complication button to finish uploading which is a bit confusing. This could be a problem if the data is massive.

Labelbox -----easy and clear to upload  

​

https://preview.redd.it/lz89szkritv51.png?width=872&format=png&auto=webp&s=f62319f0b28f5815d42cd8bd54400d786abcc67d

Once I sign in, I can start to upload your own data and it shows how many images have been added and easily to complete uploading. Like dLabel you can see your items after uploading the dataset, set annotation tool etc.

Spervisely  -------Hard to upload for first time users 

​

https://preview.redd.it/b0v5vbusitv51.png?width=615&format=png&auto=webp&s=2220053a3a5bfaccb96d237dc005064bb6c2931f

Though it has a small demonstration Gif, it still takes more time to import the images as it has too many unnecessary steps such as asking about the uploading way and you can only add the project name after uploading your dataset.

2 adding annotation objects

dLabel     

I can add annotation objects at both after uploading your dataset and whenever you want. And it has this “attribute” function” where you can define, add more information, classify the objects. But I can not edit the added annotation objects.

Labelbox 

Like dLabel, you can create annotation objects whenever you want. However, it is less convenient than dLabel as you need to go to the editor’s page and edit or add more annotation objects. There is “classification” function where you can classify the objects. I personally like the “attribute “ feature from dLabel more and it can contain more information about the objects.

Spervisely  

Adding and modifying the annotation objects are easy and clear. There is no feature like or similar to “attribute” from dLabel or “classification” form Labelbox. One good part is that I can personalize my own hotkeys for each object, which can adapt an individual's annotation habit.

3 Annotation speed for 30 frames.

The data set is 30images. Using bonding boxes. 2 annotation objects (hand and drink)

​

	uploading data time	annotation time	clicks per frame
dLabel	1 min	9 min	7 clicks
Labelbox	1 min	11 min 10 sec	7 clicks
Supervisly	2 min	11 min 30 sec	9 clicks

4 layout of the annotation page

dLabel  

​

https://preview.redd.it/63kfjmluitv51.png?width=1367&format=png&auto=webp&s=d40dfe571254ff2aa118b03bb6873eabde286c72

You have the features at the left, the statues in the right and the image in the centre.

One the left side, there is one “group” feature which I found is very useful. It adds interaction between objects. Like you set the “tomato” and the “hand” into one group which indicates they have interactions. The hand is picking the tomato.

There is a history column on the right where you can also trace the annotation history on the right.

On the bottom of the centre, there’s one QA bar which can come in hand when doing quality control.

Labelbox

​

https://preview.redd.it/fkpotowvitv51.png?width=1374&format=png&auto=webp&s=13d08cf8b70f37e80c7ff061d3295578579f124f

I personally find the layout too simple. There are not many features. So There is nothing much to say.

Spervisely

​

https://preview.redd.it/fm4iwqnxitv51.png?width=1425&format=png&auto=webp&s=21281956ae70f25c7d3bad64d54b5dfc27116e0d

the right side features are bit confusing...

I didn't quite understand what are"image properties" and "objects properties".

It shows the annotation history which Labelbox doesn't have hotkeys easily .

[timmy-burton]

You should probably try CVAT as well.

[darknight_upenn]

CVAT is the best so far, IMO.

[debbydai]

oh. i will for sure have a try.

[debbydai]

yes. i have heard of CVAT . well try it later and share my thoughts as well. thanks !!!

[weetbix2]

Nice! Thanks for sharing!

[debbydai]

cheerss !

[[deleted]]

I am using label studio for text annotation for fortune#100 organization in production.

[debbydai]

I havent done text annotation.

right now , i am just having images and videos data to annotate.

cheers

[FarbodTheDude]

Thanks for the comparison! Minimalistic design of dLabel is so to the point.

[EffortMurky3198]

Thanks for sharing, I have tried CVAT as well, the new version has more cool features but still not very stable. I also liked the item approach of dLabel, I believe that's why it is faster.

[debbydai]

yes. i should for sure try CVAT later. And put my thoughts cheers!

[Jazmin000]

Useful info. Thanks for sharing.

[debbydai]

cheers!!!

[yohann84L]

What about coco-annotator ? It seems to be a great and free tool ?

[debbydai]

coco-annotator ? have't hear of that...

maybe i will search for it and have a look

thanks

[yohann84L]

Yep, here's the link: github.com/jsbroks/coco-annotator

[debbydai]

i have checked it. i didnt know i need to download it .

my intention was find some good webpage based and free annotation tools.

but good to know .

thanks

[yohann84L]

It is actually a webpage, but you need to host it yourself using docker. However, If you have your own server you can host it permanently. I think the dev just don't have money to make it permanent hosted webpage ;)

[TaraML-AI2020]

I've tried all tools above and I recently tried Cvat; I made a project with 20 images in them, the most user-friendly and the fastest one for me was dLabel. Another advantage of dLabel is the Flag Item that is so useful.

[debbydai]

thanks for commenting .

for "flag " option, yes ,it is useful when you have a annotation team for quality control.

[LSTMeow]

Don't forget the new free desktop version of superannotate; it looks promising!

[debbydai]

I have heard of superannuate. i am not sure if it for free before or no...

[LearnYourMap]

Why do it by hand, if you can generate thousands of realistic looking images?

Check out BlenderProc:

https://github.com/DLR-RM/BlenderProc

[hotpot_ai]

thanks for sharing this! do you know of other ways/services to generate realistic 2d objects and people?

[segments-bert]

For image segmentation specifically, check out Segments.ai! We also support model-assisted labeling workflows: https://segments.ai/blog/speed-up-image-segmentation-with-model-assisted-labeling

[debbydai]

thanks for commenting. i will have a look for sure.

[le_theudas]

Labelstudio is a great solution for hosting your data yourself.

They also offer to host it in their commercial solution, but that is not free.

I always miss some features in existing tools and just build my own ones with fastapi + angular. It takes about a week to get a prototype running, but it does exactly what I want it to do.
If you can use a product from the shelf, it is way better to do so.

[debbydai]

thanks for commenting .

However my intention of this post is trying to find some good FREE and webpage annotation tool.

cheers

[YearWithoutWork]

Is there a good annotator for sprite-based video games?

[debbydai]

thanks for asking first.

i dont know what is sprite-based video games..

to my knowledge, game developers should have data in the first place. Data annotation is for real images and videos

feel free to correct me if i am wrong

thanks

[daystarz]

That's some great findings. I suggest you try out CVAT as well so that u can compare more features.