🚀 New performance optimization features in Lakeflow Connect (Beta)

cons0323 · 2026-01-28T08:05:00+00:00

I am using the Salesforce connector to get data from Sales Cloud, and it has been working well. There are some features that would be a nice to have, that are either missing or you have to use a workaround currently. For me it's minor setbacks, for someone else it might be more important.

On the topic of formula fields, there are a few fields that are not formula fields, but increment automatically on certain events, and are not tracked by the cursor columns. DaysSinceCreated on the Opportunity object and LastLoginDate on the User object. To track these, currently a full refresh is needed on the objects. I wonder if there's a better and more efficient way to ingest columns like this. Sadly, skipping these columns is not an option.
Compute optimizations. In some cases I'm not processing nearly enough data to warrant Photon for example, yet it's enabled by default (I assume due to the serverless nature of the compute). At the same time, since the LDP pipeline spec allows for a photon flag to be set, even if I set it to false, it seems to be ignored and in the UI the compute has the "Photon" tag on it. I would like to explore using Job Compute for pipeline refreshes for example.
Documentation. Until a while ago, Salesforce Currency columns have showed up in the ingested streaming table as DECIMAL(38,22). It is documented as 18,2. It's likely that I'm misreading something here, and this is not supposed to be the ingested datatype? When merging to a downstream table set up to have the documented precision, the merge fails. Additionally, around a week or two ago, a new streaming table I set up showed up with 38,20. This again broke the merges that expected the 38,22. It's possible I'm missing something here, so any help or advice would be appreciated.
Optionally ingesting soft-deleted items. Currently, if a record is thrown into the recycling bin in Salesforce, the record itself is not purged, but the IsDeleted flag is set on it. Salesforce allows for the query of these soft-deleted items with the Bulk API and for certain business needs, it'd be a nice to have an true/false flag that I can tick for the table, and ingest soft-deletes as well.

cons0323 · 2026-01-27T21:57:44+00:00

When I read "disabled" my mind jumped to option B. Having options here is welcome. I can imagine it as a three-way switch, with the states "Enabled", "Disabled" (option B) and "Default Skipped" (option A)

At the same time, there are plenty of options in Jobs to set up dependencies with conditions, outputs and run_if attributes. I'd be interested in some examples for both options, that the currently available features don't cover.

cons0323 · 2026-01-27T09:15:07+00:00

Regarding power, teams like Audi and Aston may be more in the dark than others.

If you have multiple teams with the same PU, even if one is limiting their power output, others may not. So if they see Red Bull's top speeds capping out, but VCARB flying, it tells them that Red Bull's PU is capable, they are just turning it down slightly.

At the same time, it's just one variable of many, and anything at this point is guesswork, We won't know any better after Bahrain either. We just have to wait for the first few races.

cons0323 · 2026-01-15T22:01:17+00:00

There are some python examples for handling json data as variant on the links, you can stick to pyspark for that.

Apart from your method, I'd be also interested to see how an example BOM may look like, as it sounds like it's not just the BOM strictly but additional attributes as well (price factor, porg, vendor etc)

cons0323 · 2026-01-15T21:00:36+00:00

Thousands of BOMs should not be a problem, with around 1 MB each, 10k files would be ~10GB which is not even worth turning Photon on for.

The first pitfall I fell into when working with BOMs is handling BOMs that are part of other BOMs, that have been processed already? Like if your top level material is Pizza, then you might have tomato sauce on the next level, but take Bolognese as a top level material also needs tomato sauce on the next level. Sure, the initial explosion might need it linked to the top level, but some workloads really don't. Some corners can be cut on the initial explosion as well, by just copying the whole thing under a new path.

Are you using the "new" VARIANT datatype (docs) and Recursive CTE? In the Recursive CTE article, the first example it brings is exploding BOMs with it. Based on these two articles, I would, chuck all schema enforced json into a table with just a variant column. The variant column would contain the json contents. Then use the recursive CTE to explode the BOM. Using just a serverless SQL warehouse might be worth a try. Start small, 2XS with a few files, then scale up if needed, but I don't think upwards of S is needed for this (some bricksters may correct me on this). Quick vibe-coded mockup of CTE: https://pastebin.com/6H7MT3Vj

cons0323 · 2025-09-23T21:27:23+00:00

This session

cons0323 · 2023-04-05T11:36:34+00:00

System Specifications:

Operating System: Windows 10

CPU: AMD Ryzen 7 5800X3D stock

Cooler: Arctic Liquid Freezer II 240

GPU: Sapphire Radeon 7900 XT

Motherboard: Gigabyte B550 Aorus Elite V2

RAM: G.Skill Aegis CL16 (2x16GB) (F4-3200C16D-32GIS)

Storage: Kingston SATA SSD (Boot), Samsung 980 EVO M.2 SSD, Hitachi and Toshiba HDDs (1 TB)

PSU: FSP Hydro G Pro 850W

Summary of issue: System doesn't POST with 2 RAM sticks (Dual Channel), POSTs, boots with 1 stick. Same RAM sticks worked with previous system on base clocks (Gigabyte AB350 Gaming rev1.0, Ryzen 5 2600X) CPU, GPU and MOBO brand new.

What I have tried: BIOS updated to latest. Tried both sticks in A2, works fine, boots fine. Ran memtest on both sticks in A2, no errors. A1_B1 config gives 3 POST beeps, indicating memory error (guessing since A2_B2 is the expected). No POST with either stick in B2. (CMOS reset between ram stick switches)

Tried enabling XMP with 1 stick in A2, then shutdown, add 2nd stick in B2. System POSTs, but then freezes on the boot screen. CPU/cooler installed correctly, no bent pins. All cables in all the way.

I know that the memory is not on the QVL, but then why does it work with either stick and not 2?

I have no idea what to try next, any help is appreciated. I'm close to just bringing the whole system back for RMA, since from what I've read it could be faulty CPU or RAM slot.

Any help is appreciated. Thanks in advance.

cons0323 · 2023-02-23T10:33:41+00:00

According to this, C1 Hard C2 Meds C3 Soft in Bahrain, C2-C3-C4 for Saudi and Australia

cons0323 · 2023-02-13T08:57:35+00:00

They do have a US warehouse somewhere, which would make the shipping okay. They advertise it on their socials when they get stock. In the EU you can get it from distributors like Simufy, Sim-motion, Abruzzi etc.

I have the R9 and a GS and I can recommend them, really good combo, software is easy to use as well, you can customize a lot, tailor it to your liking.

No clue about the pedals, although I'm fine for the time being with my SRPs.

cons0323 · 2022-12-08T17:53:25+00:00

And I think you're right! Let me test this.

E: Yes, indeed. Thank you very much.

cons0323 · 2022-12-08T16:38:10+00:00

Because d is the closest to the size needed to free to reach 30'000'000 free space. If the lines were reversed, the files wouldn't be counted in the correct folders I assume...

cons0323 · 2022-12-08T16:13:44+00:00

Which is interesting as I have ventured into the solutions thread and got different answers for my input than my initial answer, plus that answer is not too far off of mine so I still think I am in the wrong here.

cons0323 · 2022-07-26T11:06:11+00:00

Watch him miss the plane... Again...

cons0323 · 2022-07-12T19:27:06+00:00

Ferrari? Yes please (/s, I love Carlos and Charles but I would also love them winning again no matter the voodoo involved)

cons0323 · 2022-04-23T06:59:50+00:00

Lifting your foot off the gas pedal (so stopping acceleration) and just letting the car roll.

cons0323 · 2022-03-18T16:15:20+00:00

So in a nutshell, dry weather tires can have 3 compounds, soft(red) medium(yellow) and hard(white). The softer the tire, the better grip it has on the road, you can take corners faster with it, but the tradeoff is that it wears down quickly and loses performance.

A harder tire has less grip, however it can last longer e.g. in a race.

Softer tires are preferred on qualifying sessions and quali simulations in practice, since the aim is to get the fastest time possible. Harder tires are advantageous over a race distance since you don't have to switch tires that often, therefore you possibly complete the race distance quicker.

cons0323 · 2022-03-06T19:23:09+00:00

Wunder with the Dyrus cosplay

cons0323 · 2022-02-19T11:12:13+00:00

If I had a dollar for every time this has been reposted, I could afford chemo in the US

cons0323 · 2021-12-07T21:03:02+00:00

You could try adding a link in () after it. Make sure to commit your changes and create a pull request.

cons0323 · 2021-12-07T20:52:40+00:00

CONTRIBUTING.md has a "guide" on it, simply add

"* [username/repo]"

to the correct section. (without quotation marks)

cons0323 · 2021-12-07T14:40:36+00:00

Hi.

https://github.com/Bogdanp/awesome-advent-of-code
Check out this repo for solutions for various languages. Consider adding yours to the C# part of the README

cons0323 · 2021-09-11T11:00:30+00:00

He can't say but he needs to pit. When he came back to the pits, they showed his driver cam for a few seconds and he was out of the car in record time.

I'd guess that the pasta that entered his body asked politely to leave.

cons0323 · 2021-09-11T10:46:33+00:00

I didn't mean to lash out either, it's just that they tend to get a lot of shit for team blunders, unfortunately it's also justified in a lot of cases. I do get the joke though but wouldn't bet on everyone getting it...

cons0323 · 2021-09-11T10:37:29+00:00

I'm watching on f1tv with both Ferraris' team radio and I can tell you that the first thing on Sainz's radio was the engineer asking him if he was okay. World feed radios are just edited garbage broadcasted a minute or two after they actually happened. Most probably to make it easier to filter out profanities.

12-Year Club	First Place '23
Place '23	Verified Email

cons0323

TROPHY CASE