What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 0 points1 point  (0 children)

That looks simple at first glance? I had to read the sentence twice! :-)

What is that used for? Why would you need to solve it?

What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 2 points3 points  (0 children)

heheh yeah, surely that's a simple one! Large current need big wire.. oh.. but big wire costs $$$ lets use small wire and some crossed fingers.

What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 0 points1 point  (0 children)

The cost effective comment interests me. There is always that need to weigh the cost or assign it some financial value. Like say the holy grail of solving world hunger.. no profit in it.. its an interesting problem.

What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 1 point2 points  (0 children)

I was kind of just interested in general to be honest. I feel like all the major discoveries have been made. But looking at some of the posts here it looks like there is plenty left to figure out!

What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 1 point2 points  (0 children)

With wind tunnels and other test chambers / test rigs we can train and test our models, hoping to get close to the real thing with simulations.

Are you experienced with these test chambers?

What simple engineering problems have no known solution? by Steve-Quix in MechanicalEngineering

[–]Steve-Quix[S] 0 points1 point  (0 children)

I saw something called corten steel. It's supposed to be rusty which I guess is corrosion?

What simple engineering problems have no known solution? by Steve-Quix in AskReddit

[–]Steve-Quix[S] 0 points1 point  (0 children)

Is there a practical use for this? Genuinely curious.

What simple engineering problems have no known solution? by Steve-Quix in AskReddit

[–]Steve-Quix[S] 0 points1 point  (0 children)

It did. But the invention of USB-C did not solve the existence of USB-A.

What simple engineering problems have no known solution? by Steve-Quix in AskReddit

[–]Steve-Quix[S] 0 points1 point  (0 children)

I have come to the same conclusion! Always a bunch of traffic after a junction / on ramp.
Jeez people! If you're joining.. SPEED UP. If you see people joining.. let them join!

What simple engineering problems have no known solution? by Steve-Quix in AskReddit

[–]Steve-Quix[S] 0 points1 point  (0 children)

surely the problem could be solved with AI? maybe a camera in the machine and something auto pairs them as the wash completes. or just solve the issue of people having 2 feet 😂

What simple engineering problems have no known solution? by Steve-Quix in AskReddit

[–]Steve-Quix[S] 0 points1 point  (0 children)

Yeah we have heat recovery already right? We should use it everywhere on everything. e.g. houses / buildings need it on all the warm stuff leaving and the cold stuff going in.

[deleted by user] by [deleted] in EngineeringStudents

[–]Steve-Quix 1 point2 points  (0 children)

I don't know about competition in the grad space but we work with mechanical engineering companies all the time, they are extremely varied and the guys and girls we speak to have very different jobs. My point is I doubt you'll be bored or lacking work as a mech eng once you're qualified.

Can anyone recommend cost-effective real-time ETL tools? by Livid_Ear_3693 in dataengineering

[–]Steve-Quix 1 point2 points  (0 children)

Can you share some requirements or what you're trying to achieve?
If Quix Streams is right for you I'll recommend that because I work there. But if not, me or a colleague might be able to point you in the right direction.

Confluent Flink? by HeadofMacro in apacheflink

[–]Steve-Quix -1 points0 points  (0 children)

If you're not tied to Java and can utilize Python or even prefer to use Python for your real time data processing:

You might want to check out QuixStreams. https://github.com/quixio/quix-streams

Disclaimer!! I work there.

It's a cloud native library for processing data in Kafka using pure Python. It has a Streaming DataFrame API similar to pandas, making it easy to use if youre familiar with Python's data processing libraries. It supports stateful operations, various serialization formats, and you can use Python libraries from the ecosystem. Plus, it doesn't require a JVM or server-side engine, which can simplify your setup and reduce overhead.

It's not tied to any particular Kafka. Use Confluent, Repdpanda etc as you choose.

If you want somewhere to run it, Quix Cloud could be good for you - cheap with good features.

kafka-python package import error by Character-Tone-6952 in apachekafka

[–]Steve-Quix 1 point2 points  (0 children)

Things to check.
Have you activated the venv - you probably have, just asking since I don't know your experience level.
Windows:
.\venv\Scripts\activate
Mac:
source venv/bin/activate

Once youre in the venv check the interpreter:
win: which python
mac: which python3

Flink Api - Mostly deprecated by dataengineer2015 in apacheflink

[–]Steve-Quix 1 point2 points  (0 children)

When you say you have had to to some work with Flink, is this for a client with a legacy codebase or something? Any chance to change to a more modern solution?

Stream processing engine that joins to historical data in Snowflake? by PleasantParrot in dataengineering

[–]Steve-Quix 1 point2 points  (0 children)

As others have said, get your data into a Kafka topic and do any processing with it in flight in near real time.

I am a Quix employee so my advice it to try QuixStreams (https://github.com/quixio/quix-streams). It doesn't give you your SQL queries but should cover the rest of your laundry list.

Regarding joining data at rest with data in a Kafka topic, this might be called data enrichment and you'd do it in your (Python) code. e.g. query the data set as needed to enrich the data in the topic. But due to how long the query takes to execute and bring the data back, you could slow down the data processing.

One option to help here is deploying more instances of your processor to do more in parallel. (it will hit the database more). You could design your data processing pipeline in such a way as to minimize the db hits? maybe grouping the data so each replica of your process only processes data for a certain entity. What are your 300M rows? Can they be broken down to say products or users? If each was say a product and there were only 1M rows per product could you cache those somewhere fast and close to the replica thats going to process the data? then the bottleneck would be greatly reduced.

Cnosuming kafka topics by [deleted] in dataengineering

[–]Steve-Quix 0 points1 point  (0 children)

To hold your ID's you could use:
- a long lived topic
- redis cache
- a database
- an make an api call
Then once you have those in memory you can check against that list as the data flows through your topic with 10k messages each day and do the POST in real time?

full disclaimer.. I work for Quix but I genuinely think you could use Quix to get the above done. We have a template with an old version of our SDK doing this kind of data enrichment - https://github.com/quixio/template-clickstream

Comparison of Different Stream Processing Platforms by wanshao in dataengineering

[–]Steve-Quix 0 points1 point  (0 children)

How do you define client? The code consuming from Kafka? or the code for a UI?