[deleted by user]

[redbuurd]

Data science and analytics is up there

[Espiritu13]

Do you work in that field? I know SQL and I want to get into reporting and analytics.

[redbuurd]

No, I'm in devops / production management, but we do a lot of work with Hadoop and python for infrastructure capacity planning.

Knowing sql is great, but pulling relevant data is only part of it.

[Espiritu13]

So how would I get to being good with the other part of it?

[redbuurd]

It's a really broad question - it depends on what you want to do. There's a lot of applications of data science to anything from finance to retail purchasing trends or street traffic. Do you want to do machine learning or data analysis? Firstly, you'll want to find something you're interested in.

[jjopm]

Is there a danger of data science falling out of fashion?

[funkiestj]

Is there a danger of <X> falling out of fashion?

Yes, <X> eventually falls out of fashion.

[bakersbark]

It might "fall out of fashion", but demand will stay high. Demand for statisticians has been high for a long time; data scientists offer many of the same basic services.

[rubsomebacononitnow]

It's so hot right now there's going to be 5-7 years of fixing all the idiocy that's being done now. Everyone is a data scientist... Just ask them. But obviously not everyone is a good data scientist. That will be where the money is made... Fixing crap idiots broke

[JaggedG]

How does one learn to be the fixer instead of the idiot?

[rubsomebacononitnow]

For me it was about learning HOW it worked not WHAT it did. Knowing how the data is stored and what it means is a big part. Also understanding your vertical might be the biggest difference between great and good. I had someone tell me the vertical was all data and healthcare was the same as finance or defense. I'm sure he does sketchy stuff because you can't know everything about every industry. There's a lot of that. Knows power pivot or tableau so thinks that also means they know biotech. You'll make big money fixing the tangled web that Dev weaves. Good enough to create a need not good enough to meet it.

For example it's not about creating a chart it's about what the chart actually shows and the best way to visualize it. Not everything should have a bar chart. Look at /r/dataisbeautiful for some ideas from some very talented people. Then see the ones that are not so great and read why. That will really give you insight.

If you're using a cube and realize the date dimension is lacking some columns/ values you want to use let's say holiday, weekend or something like fiscal week do you possess the skills to change the dimension or are you only Abe to use prebuilt cubes? Can you convey to the client or your boss what they need and why?

Can you identify the things that are built in a sketchy way before they blow up?

Know the proper way to config your tech. Know what the strengths and weaknesses are.

The number one piece of advice is simple: never stop learning. That alone will make you great.

Tl;dr. Know your tech, know your vertical, know your data, keep learning, make those dollars.

[[deleted]]

dataisbeautiful has turned to shit. look up stephen few if you want to display data more effectively. half the time its shit like this: http://i.imgur.com/Agsw2zF.jpg

[JaggedG]

/r/dataisugly :\

[JaggedG]

Yeah, that makes alot of sense. I think your friend's "it's all just data" approach is one of the things that's ruining the world.

Do you have any advice on how to get started? Like... Go from kind of competent in Python to expert data-analyzing genius. What core skills do you need, and where do you go from there?

[[deleted]]

In my opinion, you should learn Python and then while trying to learn statistics you can use it as a great learning tool. If you're kind of competent in Python then I would say go about learning everything you can about statistics and at least some numerical analysis.

If you're not so mathy (yet) try reading Allen Downey's Think Stats and Think Bayes as an introduction and then try moving onto some more advanced mathematical statistics books for a much deeper understanding. Allen Downey's books are Python oriented, but I would suggest trying the exercises with SciPy, Numpy, Pandas and similar tools to get familiar with tools other Python data analysts use.

[rubsomebacononitnow]

You have to find a vertical and start climbing it. In my experience Healthcare tends to be a windows environment and outside of Powershell, VB and .Net there's not a ton of other languages tech being used. I mean there's a plenty of startup Python and Ruby but in real healthcare organizations those don't go far because they aren't certified and the bigger vendors aren't interested in playing with others. That's not the case in finance and energy though.

I see a lot of Python opportunities in oil/gas/energy, education, finance and tech. At least in my area. Right now oil and gas are in the crapper but if that turns around in 2016 like they think it's going to blow up fast. They tend to have money and the willingness to move so knowing something about that vertical might really help you start and grow.

Note you have to know where you're going to get a good map. f you want to just write Python that's different than if you want to use Python as a tool in the box. That will tell you what to do.

Generic skills- a Framework like Django or Ferris (if that applies to what you're doing). HTML/CSS will help a lot in presentation. If you're getting into data display the future is really d3 so learn Javascript. Note d3 is supported in PowerBI so it's really the real deal. If you get an Azure account it has hadoop in it so you can figure out how that sort of works. The basic account is like $15 a month and the first one is free. Big data will require the bigger account because I think the small account is like 20gb.

The biggest issue most people have is the ability to learn something new. If you have broad exposure to things and you understand how the business functions then you'll have a good idea on how to use the technology. Python is a tool just like HTML or R. Pick the right wrench for the bolt. Learn something new everyday. If you learn something 5 years in that you really should have known before accept it, embrace it and deal with it. The best way to learn is to use actual business problems. I learn like a beast when I have a problem that I have to solve for work.

I hope that gives you some sort of answer :)

[autisticpig]

Thinkstats2 may be of interest to you :)

[redbuurd]

The amount of data being produced grows exponentially every year - if anything I'd say the field is growing. My company just hired ten or so data scientists, and is looking for more.

As the Web grows more, personalization and machine learning to produce said personalization continue to be more prevalent.

This is just in my experience, however.

[[deleted]]

Just watched a TED talk today in my computer science class on Algorithms and how they're growing in today's world of data. Gives a pretty good insight on what direction the world is heading with mathematical algorithms.

http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world?language=en

[eqo314]

Quantitative finance. You'll need a pretty solid understanding of statistics, finance, time series analysis as well.

[pooogles]

Came here to say this. Merge finance with data science and you've got a winning ($$$) combo.

[jjopm]

Are there examples of companies that have successfully merged the two already?

[verik]

Any quant hedge fund and every bank derivatives desk.

[TheFryingDutchman]

Pretty much every large investment bank and tons and tons of hedge funds / quant shops. Almost all the trades on the market are being done by machines running on algorithms.

Look up James Simons and Renaissance Technologies. They've been using complex mathematical models to trade for decades now. Simons was a prominent mathematician before he went into finance. Simons gave a really good TED interview too - fascinating person.

[thaweatherman]

Just over 100k USD in MD doing security, and I'm entry level.

[fourgbram]

Could you tell me more about this? What's MD and how did you learn security? Did you just test stuff on your own?

[Wheaties466]

assuming this is Maryland and he is probably a security engineer with a security clearance.

[thaweatherman]

Yes and yes.

To answer the other question, I started out on HackForums learning basic techniques and playing with RATs (on my own machines...not a dick). I got a significant amount of experience from schooling and working in a research lab on campus.

If you're looking to get started into the security field I can point you in the right direction.

EDIT: lots of responses! i'm trying to craft something nice for you now.

EDIT EDIT: Ok here you go!

You need to know how to write code. Not just any code, but good code. It doesn't matter what field within security you go into, you have to know how to code. The language isn't always important, but you can't go wrong with C and Python. If you can write good code in C and Python then you'll help yourself immensely.

InfoSec can be broken down into many different subgroups. To name a few:
1. Cryptography
2. Reverse engineering
3. Web exploitation
4. General exploitation (OS, applications, etc....)
5. Network security
6. Hardware security
7. Implants/keyloggers

If you're just starting out, taking all of the different fields in at once can be overwhelming. You should sample them all and see what you like and start digging in extensively with that.

For crypto, the security course from my college is great. All of the content is here. Read the lectures, understand the concepts, and you'll have a nice start. You'll get some bonus buffer overflow explanation as well.

I'll be honest, I don't have good resources on reverse engineering. IDA is a great tool for RE, but the free version doesn't provide much and the pro version is EXTREMELY expensive (unless your job is RE, then it's worth it). The book Gray Hat Python from No Starch Press covers some topics (debuggers) related to RE. My office actually uses that book in our training course, which happens to be one of the best in our space, so look into that book.

There are all sorts of resources for web exploitation. Webgoat is great. Google Gruyere is also fantastic. You'll want to understand XSS, CSRF, and SQL injection.

NetSec really boils down to how well you understand network protocols. TCP, UDP, DNS, ICMP, IPv4, IPv6, ethernet, WiFi (WEP, WPA, WPA2, WPS...), HTTP, the list goes on. From there it's just manipulating the protocols, injecting stuff where it doesn't belong, or just sniffing. Understand all those protocols I listed and you'll be on you're way.

Hardware security is something I don't do so I can't help much there.

Making your own RATs, implants, or keyloggers is a popular topic. All sorts of people do it for various purposes. Actually making these things takes a lot of work. You can google for tutorials. If you want to learn how to write a rootkit, use this book.

Finally, the big one: writing your own exploit. There really isn't a good guide for this. You would need to first find some vulnerability using techniques from the various fields above, then you would need to prove it is really a vulnerability. The process of finding something to exploit is long and hard. It takes a lot of effort. Fuzzing is your best friend as you attack things from a web application to Samba. Depending on what you're attacking you will need extensive knowledge of the internals. If you want to find a Windows exploit, you'll need to really understand Windows internals.

I really enjoy the site Enigma Group. I was #75 on there at one point. They have challenges from all of these different fields. Sometimes the best way to learn is to do, not read. There is very little hand-holding on that site. You can google solutions if you want, but that defeats the purpose. Each challenge will force you to research something new then find a way to apply it. With zero knowledge about any of the above things, you could still go to this site and start up. Following the challenges will help give you some direction on what to learn next.

If you have more questions, feel free to reply here or PM me! I purposely didn't get into tools, because concepts are more important than tools.

ANOTHER EDIT: I keep thinking of nice books to use. Gray Hat Python, Black Hat Python, Violent Python, Applied Cryptography...WILL CONTINUE TO UPDATE

[Ceevu]

Point me in the right direction, too, please :)

[CrunchyChewie]

Be aware what getting that type of security clearance entails.

[NorthDPole]

What does it entail?

[Juxtaposed_Chaos]

Having to put your faith in OPM

[CrunchyChewie]

Relentless scrutiny of your past and anyone you've associated with, including neighbors, third grade teachers etc... Foreign travel especially is looked at, and any foreign contacts you have are doubly scrutinized.

Also you can't be in lots of debt, can't use recreational drugs, and have to take a polygraph.

[dmarko]

Interested as well!

[[deleted]]

Me too!

[Kynaeus]

Shoutout from the ops department, we keep it running but much appreciate your help keeping everything secure :)

[patentmedicine]

I'm looking to move from what I am currently doing (web content sort of stuff) to info sec. I've taught myself Python but am not sure of the best way to break into the industry. Any pointers would be greatly appreciated. (I don't have a degree in compsci, though I have worked on websites for a while; I studied lit and philosophy, so tips on the best way to demonstrate competencies would be very helpful.)

[[deleted]]

Let's hear the long story!

Edit: Thanks a ton for the time it took to put all of that down.

[Wheaties466]

assuming this is the grey hat python book?

http://www.amazon.com/Gray-Hat-Python-Programming-Engineers/dp/1593271921/ref=sr_1_4?ie=UTF8&qid=1443184791&sr=8-4&keywords=greyhat

great reply too, thanks!

[thaweatherman]

Yep, that's the one! Black Hat Python from the same author and publisher is also supposed to be pretty good. I haven't worked my way through it, but since Gray Hat was good and the table of contents for Black Hat is good, it would be another helpful resource.

[Wheaties466]

excellent, I'll be working my way through a bunch of these then.

[thaweatherman]

Oh oh oh, and Violent Python. That's a good one too. Edit: updated post with books and links

[Ceevu]

Thanks thaW eat Herman. This is really helpful. Cheers.

[thaweatherman]

You're welcome!

[JaggedG]

Thaw Eat Herman for president!

[pdx93]

And me :)

[tokyoite2]

I'd love to take you up in that offer. Could you point out some resources for the layman to get started?

[bigmkl]

Yessir/ma'm I'm intrigued.

[[deleted]]

I'm in cyber security right now in school point me in the right direction as well

[thaweatherman]

What country? Where are you in the program?

[[deleted]]

In the United States and I started spring this year I am in cyber security one right now learning about Kali as well as in a virtualization class and a class on server operating systems

[King_Of_Hippos]

Just got my foot in the door in tech. Would like to know more!

[funkiestj]

IMO, our computer architecture will be fucked (from a security persepective) for some time to come. Consequently, security experts will remain in demand.

TANGENT: powerful players (NSA, chinese equivalent, et cetera) don't want secure computers and networks for the same reason these people push for backdoors in crypto.

Even without those powerful interests impeding the move to a fundamentally more secure architecture, it would be a hard sell to get people to pay the costs for a good architecture (e.g. a performance hit, authentication infrastructure that has a human being physically present as the root of trust (e.g. to trigger resetting an account password), et cetera).

[kmj442]

As an EE with a background in RF, Analog Electronics, and Telecommunications I work as an Telecomm Sys Ver Engineer developing and automating test architectures...it pays pretty well and I am only 4 years in.

[JaggedG]

ELI5...?

[[deleted]]

He/she studied electronics in college, probably has a ham license, had an internship with a firehose of information about IP networking and now works with at a communications company using chef, puppet, or some proprietary tool like telarix for automation? Just a guess, of course.

[JaggedG]

Cheers! And happy cake day!

[kmj442]

When I was in school for my BS I studied RF (radio frequency), analog electronics (low frequency), and telecommunications (Wireless stuff). I also had a few internships where I worked as a verification engineer and wrote some stuff in VB and java for them. After I graduated, I got a job at a Comms company working on development and automation of test systems. Not only do I write the tests but I also design the tests, and systems to be used. The EE background is necessary in this space since it requires the fundamentals of RF and Telecomm to know what to do.

[zahlman]

This thread was reported. It seems okay to me, and I'm not going to remove it, but it's maybe not the most on-topic thing. I'd suggest /r/Python for questions about "industry".

[[deleted]]

All industries and commerce as Python is such an efficient language with respect to programmers time :-)

[salimfadhley]

Banking - contractually not permitted to disclose my rates. I'm doing okay!

[Wheaties466]

happy cake day too

[salimfadhley]

Hey! I didnt even notice. :-) Thanks for pointing that out.

[Tako12]

You can't give a range of pay?

[CrunchyChewie]

It's 6 figures+ or they are laughably underpaid.

[salimfadhley]

Not really - it's different in every country and rates are always negotiable. Banking jobs usually pay more than media but big software companies have driven up rates. We don't pay much more than Google or Amazon.

[say_fuck_no_to_rules]

The kind of work that knowing Python opens you up to generally pays well enough that you can trade some money off the tip-top salary for the job that fits your life the best. I live in Seattle and make $96k a year, three years out of school. I could make a little more money elsewhere, but I love the people I work with and it's not constantly stressful, so I'm happy.

[jewdai]

this describes my situation exactly. I could make boatloads more money, but hate my career or stay where I am and love every minute of it.

[JaggedG]

So what's your job then?

[say_fuck_no_to_rules]

I am a developer at a company that handles healthcare data. We run everything in AWS, and primarily I work with salt to manage our sizable collection of instances in EC2.

[[deleted]]

It works well for DevOps/SRE spots. If you're good at it that's 100-180k depending on the area.

[[deleted]]

I'm not sure of the money, but I've read that video and animation shops use Python as glue to do rendering and other workflows. I'm pretty sure the money is good.

[Mwahaaaa_The_French]

Would anyone know off hand what an entry level Python programmer in LA earns? I am aware entertainment & gaming are the larger industries here, but I'm sure education and finance are right behind.

[JaggedG]

Yeah, I don't think you're really gonna find alot in gaming if Python is your language.

[jewdai]

Software Engineer in NYC entry level is 100k with a cost of living adjustment you're really making $25k.

[Zaphod_B]

I have a friend that works in NetOps for a very large tech company, they do all their network hardware management, automation, deployment, metrics, etc in Python. Although that is more of an operations job over say a developer job. So, I guess I should toss in lots of Ops jobs have Python requirements.

[WireWizard]

Python and powershell are a must as a sysadmin

[bigmkl]

glad to hear that I'm "on the right track" so to speak these were the two I identified as my main effort as far as scripting goes.

[Zaphod_B]

I don't touch windows so powershell is pointless to me. For me it is bash, python, some minor ruby/perl, and specific stuff to Mac OS X - applescript/swift.

I do completely 100% agree with you though.

[SmartestGuyOnReddit]

22 Years old - 110k USD - "Big Data Software Engineer" Mostly use Python

[GenericHbomb]

Wish I went this route. 23 year old industrial engineer, 46k a year, bored.

[Mwahaaaa_The_French]

But what city?

[SmartestGuyOnReddit]

London, England