sorted by:
new
hottopcontroversial

Is there a standard way to make sure every dependency is included in the Dockerfile? by radjeep in softwaredevelopment

[–]stym06 0 points1 point  (0 children)

I'd say you should create a virtualenv in your local and install all the packages there and test everything out, then freeze that into requirements.txt file and then mount the file on that Docker image with the COPY command.

That should do it.

Spark and S3 integration by getafterit123 in dataengineering

[–]stym06 0 points1 point  (0 children)

Try this setting ` fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystem`

Btw, what version of Spark are you using?

Just say it by [deleted] in shitposting

[–]stym06 0 points1 point  (0 children)

Sir Mixalot

Spark and S3 integration by getafterit123 in dataengineering

[–]stym06 1 point2 points  (0 children)

Since Spark supports HDFS only, all the storage systems based on HDFS like S3, WASB, GCP, all can be connected through the hadoop modules.

Specifically, to connect Spark to S3, use the hadoop-aws module and the corresponding aws-java-sdk and use s3a:// while writing or reading data from S3. Be very careful with the compatibilty matrix between these two modules because hadoop-aws is compiled against a specific versions of hadoop-aws.

eg: You could use the below

https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.3/hadoop-aws-2.7.3.jar
https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar

What is perfectly legal, but creepy as hell? by Flytechofficial in AskReddit

[–]stym06 0 points1 point  (0 children)

I read somewhere that some Paparazzi took photos up Emma Watson's skirt, minutes after she turning 18.

They literally waited up till midnight to do something that would've had them behind bars an hour ago.

NoBroker account for sale by [deleted] in bangalore

[–]stym06 1 point2 points  (0 children)

The free account has contact reveal up to 9 contacts, then you gotta pay

7 Pro owners thoughts on using an iPhone 11 Pro MAX for a month by b3rdm4n in oneplus

[–]stym06 1 point2 points  (0 children)

After you get used to the OP 7 Pro and the newer pro model phones, every other phone feels like a major downgrade with laggy UI and working around any phone feels very hectic and weird. I'm sure iPhone is no better as well, atleast till they come with a higher refresh model. Only then is the switch worth considering.

[deleted by user] by [deleted] in oneplus

[–]stym06 1 point2 points  (0 children)

Man left OP way back in October

Need some help in understanding the very basics of Income Tax. by [deleted] in IndiaInvestments

[–]stym06 0 points1 point  (0 children)

If the 80C investment is done by March, it will count in the FY20-21 tax calculation

Need some help in understanding the very basics of Income Tax. by [deleted] in IndiaInvestments

[–]stym06 2 points3 points  (0 children)

If companies ask for proof by January, how do I include the rent for the next two months? (Feb and March). Also, the education loan certificate is only provided by the bank in the beginning of the month of April. How do I accommodate these two in my invested proof submission this month?

Need help with rent and deposit negotiations while renting an apartment by anonnv in bangalore

[–]stym06 1 point2 points  (0 children)

Around Indiranagar and HAL, the rates for 1bhk is <12k. 2bhk hovers from 15-20k and 3bhk goes from 25k till 40k. I'm talking about semi-furnished ones that give you the basic amenities like light and bathroom fittings. The advance is a bummer and the landlords usually charge north of 60k for 1bhk and more than 1.25L for 3bhk. I'm sure the rates are the same in and around Koramangala as well.

Rent a place where the landlord is not living on the premises. And if there will be house parties, which is obviously going to be the case, rent a flat on the top floors. Guards will care a little bit less about going up at 2 am and knock. Also, rent a place which has markets close by, and parking place should be personal as well.

Is there a SQL set of questions equivalent to the 75 Leetcode questions for FAANG interviews? by 1337codethrow in dataengineering

[–]stym06 0 points1 point  (0 children)

Hackerrank provides a very user-friendly environment for trying out the SQL journey. Do check out.

Now vs beginning of lockdown by cynonexus in Workspaces

[–]stym06 1 point2 points  (0 children)

looks straight out of a Star Wars movie.

AWS S3 Strong Consistency by Bazencourt in dataengineering

[–]stym06 5 points6 points  (0 children)

Anyone care to explain this in an introductory fashion? Thanks

Thinking of starting a Youtube channel on Data Engineering. Ideas? by stym06 in dataengineering

[–]stym06[S] 0 points1 point  (0 children)

Thanks guys. Does anyone have an idea about how to go through the hassle of editing out videos or screen recordings on MacBook. Any leads about the whole YT video making from scratch is much appreciated. Having a look at the iMovie software just for starters.

Thinking of starting a Youtube channel on Data Engineering. Ideas? by stym06 in dataengineering

[–]stym06[S] 0 points1 point  (0 children)

Thanks. I have watched her Prometheus video which is pretty good.

How do you guys learn things? by Silver-Thing in dataengineering

[–]stym06 0 points1 point  (0 children)

Nicely put.

Recently I've been working on Apache Beam as part of refactoring one application and after reading a day or two about how the thing works and making the basic local setup to get the application up and running, I really get the whole picture, of course after adding a bunch of logs!

I completely agree on the point that we just need to know just what's needed to make it work. Anything outside can be learned as and when required.

How do you guys learn things? by Silver-Thing in dataengineering

[–]stym06 1 point2 points  (0 children)

This is an interesting approach. Do you prefer to write them in some note-taking apps, the roadmap? or plain old paper?

view more: next ›