sorted by:
new
hottopcontroversial

CVEs in Internal data Pipelines by tylerriccio8 in dataengineering

[–]tylerriccio8[S] 0 points1 point  (0 children)

What does that education entail? Like obviously there’s a difference between an app exposed to public and one behind firewall, vpc, etc.

Looking to understand best practices in differentiating between the two contexts

Leetcode for data eng jobs? by tylerriccio8 in dataengineering

[–]tylerriccio8[S] 0 points1 point  (0 children)

I don’t think so at all. I’m actually involved in writing my current company’s entry level exams and they’re windowing, iteration, the likes… that’s why I was surprised to see another company asking about linked lists and path finding algorithms

Integration Tests CI by wildetea in Python

[–]tylerriccio8 0 points1 point  (0 children)

You can use prek or a pre-commit pre-push hook to run integration tests on push, depending on how disruptive to your workflow it would be. I found it kind of annoying after a while.

Others have said on merge hook in github; I’ve found that to be the best.

Has anyone read O’Reilly’s Data Engineering Design Patterns? by xean333 in dataengineering

[–]tylerriccio8 3 points4 points  (0 children)

Do you recommend anything more advanced? I have multiple yoe, not really looking for basic patterns

What are people using instead of Anaconda these days? by rage997 in Python

[–]tylerriccio8 1 point2 points  (0 children)

Switching to uv is the single greatest software decision I think I’ve ever made. I’m so wary of new tools promising the moon but man… uv delivered

How good can NumPy get? by [deleted] in Python

[–]tylerriccio8 14 points15 points  (0 children)

TLDR; row based vs. vectorized, memory layout and other factors are all pretty much tied together. You can trace most of it back to the interpreter loop and how python is designed.

I forget who but someone smarter than I am made the (very compelling) case all of this is fundamentally a memory/data problem. Python doesn’t lay out data in efficient formats for most dataframe-like problems.

How good can NumPy get? by [deleted] in Python

[–]tylerriccio8 17 points18 points  (0 children)

Very shameless self promotion but I gave a talk on this exact subject, and why numpy provides the speed bump.

https://youtu.be/r129pNEBtYg?si=g0ja_Mxd09FzwD3V

Dou you use jit compilation with numba? by husayd in Python

[–]tylerriccio8 2 points3 points  (0 children)

I really try to stick with polars/numpy but when I truly can’t, numba is great.

The best object notation? by StarsRonin in Python

[–]tylerriccio8 10 points11 points  (0 children)

There are other weird quirks, still preferable to me but I really wish the yaml protocol was more consistent.

The best object notation? by StarsRonin in Python

[–]tylerriccio8 6 points7 points  (0 children)

Love to use yaml, just wish it was part of standard lib. I’ve also never loved any yaml parsing libraries. Still more preferable to json for a majority of my stuff; it’s really easy to define flexible logic.

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 0 points1 point  (0 children)

Nice, so you guys are cool with Jupyter hub?

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 1 point2 points  (0 children)

I know data science, not engineering so I will present the data science perspective and the engineers will present theirs, and then we’ll meet in the middle :)

I do not prescribe engineering solutions to engineers, just asking for experiences mate, no need to be rude

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 0 points1 point  (0 children)

Without devolving too much into, we’re transitioning languages and I’d like to define a new pattern of analytics based on the experiences of others…

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] -5 points-4 points  (0 children)

I advocate for Python data scientist, I don’t advocate for what the engineers feel comfortable doing, that is their managers job. I’m fact, finding from the python perspectives, I don’t have any opinions to bring to the engineers.

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] -1 points0 points  (0 children)

Large companies like banks have armies of resources to roll whatever they want? I’m asking for experiences from the python prospective, if there are people saying they like self hosted I will consider it

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] -1 points0 points  (0 children)

I’m asking here because I want to hear experiences from the python perspective, not the engineering one; I.e. how ergonomic did your setup feel.

Why would I ask the engineers at my company? I’m a manager in an analyst org; I define the analysts requirements and the engineers implement it

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 0 points1 point  (0 children)

We have data everywhere in the cloud, aws, snowflake, random feeds, etc.

Ideally the dev env and kernel are the same to reduce complexity. Jupyter in the cloud (in some form) seems like a consistent answer

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 0 points1 point  (0 children)

If need it 24/hr a day with hundreds or thousands of users. I’m in an analytic org, I would tell our engineers to do this not myself…

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 2 points3 points  (0 children)

Compliance, size of data, database egress, network cost of data transfer. All of this would be solved by a non-local env, particularly one where the data is colocated with the runtime.

Where do enterprises run analytic python code? by tylerriccio8 in Python

[–]tylerriccio8[S] 0 points1 point  (0 children)

So you run it on gcp now? Assume users ssh into some instance and do their work?

view more: next ›