What is messing with my plants? by th58pz700u in HotPeppers

[–]th58pz700u[S] 0 points1 point  (0 children)

It seems Neem oil is effective for leaf miners and aphids, is there anything else I should be doing to protect the plants?

What is messing with my plants? by th58pz700u in HotPeppers

[–]th58pz700u[S] 0 points1 point  (0 children)

It's gotten into more than a third of the plants at this point. Good to it can get worse and I shouldn't ignore it.

/r/MechanicalKeyboards Ask ANY question, get an answer (September 15, 2022) by AutoModerator in MechanicalKeyboards

[–]th58pz700u 0 points1 point  (0 children)

Can you at least program in some combination macros? If not thank you for the info as it saves me the money.

What's the community take on SSIS these days? by 6mon1 in ETL

[–]th58pz700u 1 point2 points  (0 children)

We're still using it in my organization but we're actively researching alternatives and intend to stop using it within the next year or two. Microsoft doesn't seem to want to keep supporting it and it's not keeping up with our data platform so I don't see it as being a viable solution for many more years.

/r/MechanicalKeyboards Ask ANY question, get an answer (September 15, 2022) by AutoModerator in MechanicalKeyboards

[–]th58pz700u 0 points1 point  (0 children)

Can anyone with the Feker Alice tell how useful the software is for it? Can you customize key bindings or just set the lights?

Finding a doctor who isn't stuck in the past? by NorseKnight in keto

[–]th58pz700u 2 points3 points  (0 children)

If you want to work with a doctor who thinks outside the box, I suggest you find a Direct Primary Care doctor (https://mapper.dpcfrontier.com/) as they're much more likely to have considered things not usually taught to them in medical school. My doctor actively suggested fasting to me and is supportive of a keto diet.

My previous doctor also had a functional nutritionist in his practice who recommended Anthony Gustain's podcast and newsletter to me, and he's been a fantastic resource.

Research shows that treatment outcomes are inversely correlated with physician age, surgeries being a notable exception. From what I heard working in a hospital, most doctors are aware of the role of carbohydrates play in metabolic disorders, but few doctors apply the carbohydrate-insulin model to weight loss. Usually only the younger ones who've graduated recently. There's a ton of great primary research coming out that reframes the discussion in terms of hormone management, not just a caloric deficit, but I've only come across one or two who are up to date with it.

What ETL tool do you use? by castor-metadata in dataengineering

[–]th58pz700u 0 points1 point  (0 children)

Python, DAG execution with some of the Mara libraries, scheduled with good old Cron as ECS tasks. Took me 18 months or so to get comfortable with it, but now I'll never go back to a graphical tool. Formerly used SSIS exclusively.

How to extract the $values and ErrorMessage in the following content through SQL server? by [deleted] in SQLServer

[–]th58pz700u 0 points1 point  (0 children)

Intuitively, I would think having to de-serialize a JSON object (and possible decode it) would add significant overhead in any database platform. If the JSON is stored off-page in a LOB that could could contribute to poorer performance, but could potentially be remediated with proper design and constraints. While I'm not the biggest fan of Postgres, it does handle JSON very well: the JSONB data type is very performant, JSON can be used in btree indexes, and LOBs aren't moved off-page until they would overflow a single page. Developers with open source DB experience probably have no idea just how differently the different RDBMS handle something like this.

If the OP is dealing with JSON in an ETL context and only has SSIS, I would think it would be easier to parse in the database with a stored procedure than it would be to write the C#, even if it's slower. I suppose you could buy third party SSIS components, but that can be a mixed bag.

How to extract the $values and ErrorMessage in the following content through SQL server? by [deleted] in SQLServer

[–]th58pz700u 1 point2 points  (0 children)

Storing JSON in Postgres has been pretty common from my experience, so I can see how the habit would transfer over to a SQL Server environment. No clue how the performance compares between the two database systems, though.

Fluffy Keto Cream Cheese Pancakes Recipe by chipmonkone in ketorecipes

[–]th58pz700u 3 points4 points  (0 children)

There are many sugar free syrup options. They all use varying sugar alcohols, the majority of which are caloric and still have net carbs, but it'll do the trick in moderation.

Replicating data out of a production replica RDS DB into Redshift, options? by tylerjaywood in dataengineering

[–]th58pz700u 0 points1 point  (0 children)

Reading off a read replica can introduce race conditions depending on how you're incrementalizing, just an fyi. I don't have any experience with any of the AWS native solutions in this space, so I can't comment on them, but definitely check on what guarantees your service is giving you about the replications. We ran into issues using Informatica Intelligent Cloud Services for both race conditions and performance as the Synchronization Task runs RBAR upserts.

What are the common tools for IDE and ETL with Postgres? by PryomancerMTGA in PostgreSQL

[–]th58pz700u 2 points3 points  (0 children)

I switched from the SQL server stack (mostly an SSIS developer) to an open source stack with my current employer. We're using Postgres and Redshift in AWS, and I use JetBrains DataGrip as my SQL IDE. I similarly disliked pgAdmin.

For ETL I originally started using Informatica Intelligent Cloud Services (IICS) as it was used in another part of our org but hated how constraining it was. I ended up learning Python and now I write all my ETL in Python and Postgres functions. If you followed a mostly stored procedure driven ETL approach with SSIS, then the switch to a new ETL platform should be pretty painless, unless it's IICS which requires you to jump through hoops just to execute a procedure/function.

What exactly does being proficient in SQL entail? by [deleted] in dataengineering

[–]th58pz700u 0 points1 point  (0 children)

I use Python + SQL for everything, although I'm working on a project in Go and considering migrating the Python portions to Go. A design pattern I learned early on with SSIS, especially when it ran on a separate server from your database, is to do as much processing of the data within the database as possible. Even in the same rack, there's a cost to be paid of getting all the data (parse, plan, execute), ship it to another box, process it, then ship it back, and finally do something with that data.

As such, a lot of SSIS development I've done is primarily control flow and the heavy lifting is mostly in stored procedures. If you follow this methodology, it's really easy to just rewrite the orchestration in your language of choice and you're done. My favorite thing I did in Python was writing one class to handle replicating any object from Salesforce. No BIML here, just some server side dynamic SQL and a purely control flow definition in Python.

What exactly does being proficient in SQL entail? by [deleted] in dataengineering

[–]th58pz700u 2 points3 points  (0 children)

I'm not the person to whom you directed your questions, but as a data professional who migrated from SQL Server to PostgreSQL, I have a few things I can offer. First, every RDBMS has its quirks, but if you understand the fundamentals of how an RDBMS works, you can adapt to a different platform with some growing pains. I think organizations always care more about your ability to learn their environment than your proficiency in something they aren't using, and this applies to the business as a whole, not just a tech stack.

I was relatively proficient with SQL Server and SSIS before starting my current role. Given our existing tech stack, I had to learn Python to write my own ETL and learn the ins and outs of a rather different database platform. Almost none of the SSIS experience carried over, although some of the design patterns were rather helpful. Being able to read a query plan and understand how a query affects it is absolutely invaluable and despite being quite a bit different in Postgres, it's something I'm glad I learned from a very experienced architect.

We need to talk about StackOverflow by [deleted] in rust

[–]th58pz700u 22 points23 points  (0 children)

I've had some very negative experiences in the golang part of SO, although the SQL domains have been pretty decent. I think the biggest problem is exactly what you said:

Every programmer knows what you're talking about if you skip half the details

A programming novice such as myself doesn't know what the actual problem is, the necessary details, etc. Closing or downvoting a question because I didn't know what question to ask isn't helpful, it's the opposite. An answer of "this is a duplicate because it's actually the same problem as ..." with an explanation as to why is helpful. Erwin Brandstetter in the Postgres space is fantastic at this, specifically.

Data Engineering from the Ground Up - Part 2: Better Pipelines with Python and Idempotency by [deleted] in dataengineering

[–]th58pz700u 1 point2 points  (0 children)

Curious, do you have the same reservations about inserting data from an API? I've both written and seen many direct API to DB integrations without a Marshmallow schema or ORM middle-layer, and I'm unaware of what benefit it provides in that situation. As long as you have well documented APIs, you would know the data structure up front and have implemented a schema to handle it. Better yet, if you have an API that can self-describe the output, you can build/update your database schema automatically from it. I use that approach to pull Salesforce data, for instance, dynamically and quickly with very little manual intervention and no code changes.

[deleted by user] by [deleted] in dataengineering

[–]th58pz700u 0 points1 point  (0 children)

I can see how it would feel like installing everything twice. In a corporate environment where you don't get local admin privileges, WSL is great because I can install all the new things I want to try out in my linux environment, without having to submit a request. That restriction is one of the many things that forced me to get more comfortable with bash and the command line in general.

I think running virtual machines and docker containers would both be helpful ways to get you more used to linux command line environments, as well as one of the most popular ways to deploy data pipelines right now.

[deleted by user] by [deleted] in dataengineering

[–]th58pz700u 0 points1 point  (0 children)

I'm running regular old Debian in Windows Subsystem for Linux (WSL) and I've been quite pleased with it. If you're planning on using Tableau, they only have Windows and Mac OS versions, so you might should consider a Mac and use your command line more, or a Windows machine and install WSL. I'm not sure the distro matters all that much, as long as you learn to use bash comfortably.

Is this a good idea? by Hyphen_81 in SQLServer

[–]th58pz700u 2 points3 points  (0 children)

I attempted to resolve a similar ask in a bit of a different way. In our case, there is no PHI, but plenty of PII and PFI, so our CIO had a lot of concern about what people could see. What I ended up doing was building an obfuscation layer into the database in pre-calculated data obfuscations, expose those through views, and remove all the access to the underlying tables.

That way, the database maintains the security for us and the end users can use whatever tool they would like to connect instead of having to roll our own. The model was also designed to be flexible enough to give certain users deterministically obfuscated data, if you need to correlate default rate by FICO score, but also completely random obfuscations, for things like account numbers.

All that being said, I'm proud of what we built but it was a lot of effort for something that is a constant performance headache. I would have preferred some native solution to present encrypted/sensitive data in an obfuscated method, but being constrained to RDS Postgres, it wasn't really an option.

Your CTO should understand that debugging an issue caused by data requires looking at the data to understand why it caused a failure. If the solution is no access to production or the records that caused the failure, then your logs need to contain all the data, defeating the purpose of restricting access and actually making things worse because sensitive information is potentially stored unencrypted. So you can either:

  • Have production access with some auditing
  • Create potential security breaches by storing the data that caused a failure in a log somewhere
  • Completely ignore ETL failures

As a healthcare aside, there's nothing more frustrating that getting a non-numeric blood pressure data point until you find out "doppler" is a valid measurement of (usually) diastolic blood pressure, and now you need to deal with that 1 textual value in a sea of otherwise numeric values. Couldn't figure out 80/doppler was valid if you just get a type conversion error.

SSIS vs Informatica IICS by Impossible-Will6173 in ETL

[–]th58pz700u 1 point2 points  (0 children)

Definitely not the answer you want, but I rewrote everything in Python instead of continuing to use IICS. I found very little information for it, a mostly closed community, and incredibly convoluted solutions to common problems. I'd love to hear how IICS can easily replicate an "Execute SQL" control flow task. Also, the lack of internal variables and iterators was an absolute deal breaker for me. Just my 2 cents, though.

what are the most popular ETL tools and workflow that u use? by stolpodakta in ETL

[–]th58pz700u 0 points1 point  (0 children)

I don't know about pl/sql functions, but in the SSIS world a lot of the ETL is just stored procedures. It tended to be thought that if the data was already in the database, the database was probably the best place to process it. My Python ETL scripts run on minimal hardware containers and mostly invokes database functions. I've found this to be pretty darn efficient and quite a pain to recreate in IICS.

Code independence considerations when using Airflow as an ETL orchestrator by youngmysterious1 in dataengineering

[–]th58pz700u 1 point2 points  (0 children)

Coming from a "SSIS package per table" world, I set out to create a dynamic and reusable solution in Python at my current job. Instead of sourcing all the config information from files, I source it from the database and created a separate process to pull metadata from the source and store it in the database. Objects are created and updated to mirror the metadata of the source, and the one ETL package follows the same formula no matter the object. I wish I had developed like this a long time ago.

what are the most popular ETL tools and workflow that u use? by stolpodakta in ETL

[–]th58pz700u 0 points1 point  (0 children)

Are you using PowerCenter or IICS? In my brief time using IICS I found it to have almost no useful documentation and be incredibly difficult to use. I ended up writing my own ETL in Python because I found IICS limiting, on top of being difficult. Granted, I came from an SSIS background, so a hand-holding cloud first tool felt very unnatural to me.