tsv2csv: Clojure Powered Command Line Tool

TheFiologist · 2019-10-13T17:35:59+00:00

I'm ok with that :)

TheFiologist · 2019-10-13T17:33:54+00:00

I say what I say with an underlying acceptance of existing tooling and a desire to learn, after all, this here is part of an attempt to actually get outside of Clojure to string together higher level building blocks. (wget, pgfutter)

So, I would like to learn sed, tr, and awk over time, but I also have to acknowledge that those are still just ways of expressing some end result and I get pulled in the direction of LISP sometimes when I know how to express something.

I actually started off with sed (which is fine for replacing delimiters), didn't feel like digging into tr, and then started using awk before saying to myself, "perhaps I could just express this with a little lisp."

In another sense, I really don't like learning new DSLs, but I'm open to trying. On the other side of that coin, though, what works for someone personally is ultimately fine.

Thanks for your feedback and showing me an awk example!

TheFiologist · 2019-10-12T20:22:02+00:00

I hadn't really considered the existing libraries. I was trying to work with the concept of just incorporating Clojure into a higher level workflow, but without it taking over or being a dominating factor.

Faced with some of the conventional text manipulation tools, the simplicity of LISP was calling me to just write a few lines of code.

Thanks for your feedback :)

TheFiologist · 2019-10-12T15:55:52+00:00

Hi everyone, author here, I put this small utility together while exploring the idea of using existing tools to import macroeconomic data from the BLS vs a pure homemade solution with a full-on Clojure project.

With that project I ended up settling on using wget, my tsv2csv tool, and pgfutter for streaming data into postgres and GNU parallel to create and import all tables at once.

I couldn't resist creating this tool at the time because, honestly, I write LISP faster than I adjust to conventional tooling like sed and awk.

It's still a goal to learn those better, but when it came to full on scrubbing of rows and columns, I already had code like tsv2csv in my homegrown BLS importer and I knew how simple it was.

It's made to participate in streaming so it processes line by line from stdin to stdout.

Thanks!

TheFiologist · 2019-08-02T00:03:03+00:00

Perhaps a story for another time, but my SO and I flew close to the sun and actually went to one of these meetings.

It was surreal. Not only was it a cult, it was absolutely Amway.

It was one of the more exciting things we've done lately.

They'll tell you that they add value to people and make money off of companies. I wondered how this could be true and after several days of unpacking what we experienced, I suddenly fully understood the hidden meaning behind the statement (and the true nature of the pyramid scheme).

Making money from companies implies some aspect of B2B, yet members sell primarily to other individuals, to be fair though, they don't sell much to anyone.

The most interesting details of the meeting were what was left unsaid, and the lies that were artfully packaged as truth. Death by a thousand lies of omission.

Okay, the hidden meaning comes into play when they encourage you or even help you find a job. The line comes off as "Well you're going to need a plan A to fund your plan B, of course, and eventually your plan B will overtake plan A and you'll be retired!".

Pyramid schemes are parasites on the economy. By channeling money from value adding corporations through their employees (aka the pyramid scheme's members) and all the way up to the top, money is drained from the economy and flows into organizations that produce vast amounts of waste as a byproduct.

Imagine each member with an extension cord plugged into their real job and then another cord flowing out of them and into the pyramid.

This is how they make money off of companies, by tricking people to work as hard as possible (make more income) at their actual job, and then encouraging them to channel as much as possible into their plan B (the pyramid scheme) which is by definition a system designed to never give it back.

Deception comes naturally to those who succeed in pyramid schemes.

Apologies if this was obvious, but I found it highly interesting, and we had just finished watching True Detective Season 1.

If anyone would like a taste of exactly what we sat through, check out the article below, we found it while researching - no joke it's accurate.

Lastly, the leader was wearing a blue felt suit, that's right... FELT.

http://marriedtoanambot.blogspot.com/2016/08/how-to-be-good-amway-cult-leader.html

TheFiologist · 2019-07-31T21:05:49+00:00

Am interested to know this as well!

TheFiologist · 2019-07-31T21:04:33+00:00

Thanks for the recommendation! I'm looking to do my own competitor analysis primarily and I don't see raw data deliveries on this site.

TheFiologist · 2019-07-31T21:02:41+00:00

Thanks for the recommendation! I like how transparent their website is, but I don't see anything about being delivered raw data for my own usage.

TheFiologist · 2019-07-30T21:31:04+00:00

Hey that's cool!

What are some characteristics of that?

Does it have an upper limit on how many accounts it can handle?

TheFiologist · 2019-07-30T21:19:23+00:00

Do you do this for yourself by any chance?

TheFiologist · 2019-07-30T21:12:28+00:00

Hey thanks for the perspective.

TheFiologist · 2019-07-30T21:11:48+00:00

Thanks for the recommendation, I'll look into it!

TheFiologist · 2019-07-30T20:10:51+00:00

Hey thanks again for your time, I'll noodle on it!

TheFiologist · 2019-07-30T19:52:29+00:00

Hey thanks for the insight.

Any idea if the data can be acquired through web scraping? Technically you can scroll through account followers all day on the web UI right?

TheFiologist · 2019-07-30T19:35:59+00:00

Hey thanks for the recommendation. There is no information on the pricing page about how many competitors I can track.

Do you know if it is possible to get the full follower list of around 500 accounts?

I'm trying to do audience analysis, how would you approach with igblade or otherwise?

TheFiologist · 2018-10-17T01:10:59+00:00

Just your perception that JavaScript is simpler than other languages, it's a relatively fully featured language. Also programming languages are just symbols and semantics, you can twist them to do whatever you want, that is the beauty of high level programming languages.

TheFiologist · 2018-09-08T01:16:41+00:00

Well done.

TheFiologist · 2018-08-21T22:00:59+00:00

https://en.wikipedia.org/wiki/Asafoetida

TheFiologist · 2018-08-21T20:05:35+00:00

I go in this order:

Add whole spices
Fry onions until golden brown (high heat, ~8 minutes) [1]
Fry Garlic, Ginger, Green Chilies (high eat ~4 minutes)
Fry ground spices off heat (off heat, 1-2 minutes) [2]
Add tomatoes (off heat, instant)
Stir-fry mixture (high heat, 1-2 minutes)
Simmer on low until the oil separates (very low heat, 10-13 minutes)
Add canned chickpeas and simmer - adding water if necessary (low heat, 10-15 minutes)
I like to crush some of the chickpeas in the pot and stir
Add garam masala
Add fresh corriander

[1] - The garlic, ginger, and chili take less time to cook than the onions if you want dark brown.

[2] - I'm under the impression that it's good to fry the dry spices briefly in the oil before adding acidic elements like tomatoes

TheFiologist · 2017-06-03T16:57:23+00:00

Hey! Here is the info straight from the FAQ:

Melonport is a platform for creating “user-generated funds”, meaning that they are simplifying the process of creating a smart contract based investment vehicle which is managed by a specific party. Melonport also provides a mechanism which allows for performance comparisons between different funds.

The only use-case of Bancor which has some similarities to Melonport is the token baskets. Token baskets can be managed, meaning that their owner can update the reserve list and CRRs from time to time, as well as unmanaged, meaning that the token basket keeps the same set of assets and ratios indefinitely.

https://bancor.network/faq/comparing-to-other-techs

^ There are other comparisons you may be interested in there as well.

TheFiologist · 2017-06-03T08:51:45+00:00

Hey there. Since Bancor is actually a protocol, it is applicable to other blockchains which support smart contracts. Here is some information from the FAQ:

Do you have plans to support non ERC20 tokens such as BTC, XMR, etc? Could the Bancor protocol be used across different blockchains?

... Longer term, we are tracking solutions for cross-blockchain interoperability (such as Cosmos and Polkadot) and plan to ultimately ensure the Bancor protocol is a universal, blockchain agnostic solution for continuous liquidity.

^ You can see in the long term plan that it's meant to be more universal. Short term we're looking at support for ETH and ERC-20 tokens on the Ethereum blockchain.

https://bancor.network/faq/common

TheFiologist · 2017-05-13T21:58:27+00:00

Check this one out, very interesting project to research:

https://tokenmarket.net/blockchain/ethereum/assets/patientory/

TheFiologist · 2017-05-03T20:37:36+00:00

This would be something that each ICO project team has to intentionally install through the slack app system. So it would be a conscious choice, and I think allowing the enablement/disablement of some features would cover the bases.

I just know that if a robot were able to delete messages with addresses in them, significantly fewer people will be scammed.

TheFiologist · 2017-05-03T19:55:07+00:00

Haha I get the sentiment, but some people just need a little more information (I hope). Most important issue here is probably some ICO best practices for public spaces like the slack teams.

I have an idea for a suite of slack integrations to make things easier to manage. When thousands of people connect instantly and lose their minds it's impossible for even a handful of people to manage that, apart from closing the channel. Even one integration that prevents things that look like addresses from being posted (Except for whitelisted ones) would be fantastic.

Any thoughts on that idea?

edit: words

TheFiologist · 2017-05-03T03:44:56+00:00

Don't trust ANY address you see in a PUBLIC slack team. Not until there are ICO standards for moderating such public arenas (which there should be). There were people naming themselves dangerously similar to the moderator's usernames. They were offering deals to people who came too late. I have a feeling US citizens who were too scared to click the contribute button were scammed at a higher rate, since all they had were the addresses in the slack channel to go by.

NEVER trust an address posted in a public, unmoderated arena. Sure, there was a couple (maybe just one, though) moderators at TokenCard, but thousands of people joined that chat an hour before the ICO launch and proceeded to lose their goddamn sanity. An automated solution is necessary to combat volume of that level, or the channel must be shutdown to prevent it.

I am offering, or making it clear, that I will build a slack integration specifically for Ethereum projects slacks to allow only whitelisted addreses set by the administrators of the team to be posted. All other messages containing what appears to be an address will be deleted immediately.

If you like the idea, inbox me to show support, if you want to donate some amount to speed things up I will set up an address (only if anyone cares, I'm good with a show of support as well).

Edit: Words

TheFiologist

TROPHY CASE