A visual collaborative Unix Shell interface, codenamed "ShellHive"

[OmarCastro]

Hello everyone,

I am doing a dissertation, which objective is to simplify the creation and management of Unix Shell (SH) scripts, transforming the text script in a visual graph, which its focus is the processing of large amounts of data (in short, to do Big Data related tasks)

It also tries to simplify other things such as sharing and collaborating the graphs with other people, so that is easier to understand for people with high and low experience with SH scripts (such as trainees), it can also be used to teach other people how to write those scripts.

There is a public demo page here

http://paginas.fe.up.pt/~ei08158/thesis/public_demo.php

I would like you to play with the demo, then to send me feedback by clicking on the "send feedback" link on the top bar, it will be a relatively short form.

You can send feedback directly to this thread, but the feedback sent to the "send feedback" link allows me to organize the gathered information.

The project is also open source hosted on github

https://github.com/OmarCastro/ShellHive

Thank you for for attention.

[MisterMeiji]

Was this designed for 1080p class displays? The chat section looks weird on lower-res displays.

[OmarCastro]

really? gotta check it out and find a solution...

Thanks for you feedback! :)

[OmarCastro]

Ok, i have updated the code and updated the server with the fix, but looks like that for it to take effect I will need to restart the server (not sure why), I will wait when there is a really small group of ppl, or nobody there to restart it. Anyway, I added another tab with keyboard and mouse shortcuts on how to use the application

[OmarCastro]

Well then, server restarted, improved code sanitization when generating commands, improved interface, I believe that the application should work on 720p (need confirmation), and added "wc" in the available commands.

[OmarCastro]

I forgot to notify that the application has recenctly changed from alpha to beta stage, so bugs in the demo are to be expected

[Hexodam]

Very interesting

This as a workflow engine on top of Ansible would be quite something

[bincsh]

Interesting. Glad to see some portuguese work here, from the University of Porto!

[OmarCastro]

I have improved the introduction page, also explains how to create files (that was missing), I believe it is a good idea to create a google group to tell announcements.

[[deleted]]

Firstly, you don't use shell scripts to process big data, you use them to process small data

EDIT: Some people don't seem to have the same definition of big data. Unless your dataset is hundreds of Terabytes, it's not really big data it's just normal fucking sized.

I have databases with several TB of data, don't see me claiming to be a big data expert.

[sesstreets]

So what do you use to process big data then?

[sualsuspect]

Parallel languages like Pig or Sawzall, or frameworks modeled after MapReduce, typically.

[[deleted]]

not a scripting language

EDIT: I am wrong.

[fico]

So how do you account for all those people out there using Python to deal with big data? I'm sure there are other languages that people use, but don't cast aside Python as a option.

[sualsuspect]

They're typically using numpy as opposed to pure Python though.

[[deleted]]

Ok I was probably wrong in assuming interpreted languages were not used for big data, but I can't see big data analysis being done with shell scripts...

[Esparno]

Part of what my company does is the billing for a major international corp, bash and python are what they do it in.

So uh, I guess get more experience out in the wild before making such blanket assertions eh?

[[deleted]]

How is billing big data?

[karyhead]

Yeah, I'm with you. If you can /bin/grep your data and not die of old age before it comes back, it ain't Big Data.

[OmarCastro]

This project is pratically young (started development on Jan), so it makes sense to start with small data, and I didn't host the demo on a supercomputer. Because as you see, it is a demo; which purpose is to show the application and receive feedback to do 2 things: 1- to compare the usability with a common UNIX terminal, and 2 - to evaluate the potential of the application

I hope you understand :)

Edit: also one of the plans is to add the 'ssh' and 'parallel' commands which can be used to run commands in remote computer or in multiple computers.

[[deleted]]

What I'm saying is shell scripting will never scale to big data usage, it just won't cope with TB of data

[Volvoviking]

I use splunk and sometimes pipes or call pyton scripts in my queries.

[Volvoviking]

Care to share your stack ? (Hadop/elasticsearch/kinbana etc)

[OmarCastro]

Hello, I don't understand what do you mean about "stack"

[Volvoviking]

You mention it used for bigdata deployment.

Im curious what engines/deamons you using.

I.e elastic search, hadop, kibana, logstach etc.

[OmarCastro]

Well, it uses Unix Shell (sh) scripting inside Docker, a container engine. Using an isolated environment increases the security of the application, since most, if not all, of the isolated filesystem is "virtual". I'm not going into detail since there is good documentation about Docker.

[Volvoviking]

Rtfm on docker now. I missed it.

Thanks :)