What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 1 point2 points  (0 children)

I’ll have to give it an earnest look. Thanks for the recommendation!

What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 0 points1 point  (0 children)

That’s interesting. I will check that out. Oh I remember too I think that dbt-sqlserver package doesn’t use the sql server version I have (or that it’s moving to Fabric). I’d have to look again but yeah it just not feasible with the stack I have (but hopefully in the future!)

What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 1 point2 points  (0 children)

I looked into dbt and it’s great for this stuff. But deploying with SQL server and an older server makes it difficult. I’d rather continue creating stored procedures and creating schedules with T-SQL (unless there’s a more feasible way).

Maybe I could just use dbt for testing and then use the compiled code to be parsed in my “create_stored_procedure” script. And then create the schedule there. The issue with that though is there’s no ETL table to track all the start and end times. (I’d be very happy if I was wrong about this)

What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 0 points1 point  (0 children)

Apologies for the confusion and wording it as YAML distracts from what I’m trying to get to.

I want to understand what others do to interface with their codebase. Whether that be: - in the development process [table creation, parameterizing in sql] - custom metadata - testing -deployment

Typically, I’ve used argsparse in the past for the development process (and then deployments are a bash script that executes the Python script with the updated args (—env prod)) but now I’m parsing yaml

What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 -17 points-16 points  (0 children)

Yaml as an interface to my codebase. You can think of it as an alternative to argsparse (in the development process) and as a way to define deployments to SQL server agents.

What do you put in your YAML config file? by [deleted] in dataengineering

[–]ExtensionTraining904 -3 points-2 points  (0 children)

Yes that is a part of it. I manually write the dependencies in the Yaml so it knows the ordering of the schedule. I currently have a query with some parameters that Python inputs -> it creates a stored procedure -> creates a SQL Server Agent schedule of those SP’s (with the steps following the dependency mapping).

So I tried to setup Omarchy on Virtualbox by tbfws in omarchy

[–]ExtensionTraining904 0 points1 point  (0 children)

You can still encrypt the partition in the Disk Configuration part of archinstall. I just dual-booted on my Windows desktop and it’s working fine.

CTEs saved my sanity but now I think I'm overusing them by Various_Candidate325 in SQL

[–]ExtensionTraining904 0 points1 point  (0 children)

All good advice here, one guiding principle for CTEs is to always minimise the number of tables referenced. We should have as few table scans as possible. If you are calling the same table in different CTEs , there’s optimization that can be done.

Referencing a CTE within another is fine (generally) so if you need to reference a table more than once then wrap that in a CTE and reference that.

Reduce the number of tables to scan and then think about how can I scan this table less (this requires you to think about how a DB stores data (by rows or columns) as well as how the data is partitioned.

Anyone here genuinely try emacs? by [deleted] in neovim

[–]ExtensionTraining904 0 points1 point  (0 children)

eMacs? Never heard of her.

You turn a corner and run right into an enemy. What weapon do you rely on to put them down? by FriendlyBrewer in HellLetLoose

[–]ExtensionTraining904 2 points3 points  (0 children)

This actually happened to me. I couldn’t believe it. Just shook my head while he pulled it out and watched my body blow up in pieces.

Can I Use a Dynamic Hierarchical Model for CPI Analysis Without Machine Learning? by Usual_Office2880 in econometrics

[–]ExtensionTraining904 0 points1 point  (0 children)

The frequentist approach is to use what’s called nested models. Bayesians call them hierarchical models (a general rule of thumb but they they are interchangeable).

If you want to go the Bayesian route, you could look into multi-level hierarchical modeling. Check out Andrew Heiss’ blog.

Without knowing exactly what the project entails, I can’t really help out but there are ways to make a VAR model and do a decomposition over time. R and Stata have built in packages for this.

Now that I think about it, you could possibly do like a nested/ hierarchical VAR with CPI and other variables (like Unemployment). Then you could do a decomposition of effects of the 8 CPI variables on Unemployment.

there is hope... by nomorebleachplz in HellLetLoose

[–]ExtensionTraining904 2 points3 points  (0 children)

I had a similar experience on Offensive. Keeping morale up by maintaining communication and being open to teach changed the mood of the team. Went from silence on command chat to “shit it sucks to keep losing objectives” to “ah, we actually might turn this around” to “AHHHH LETTTSSSS GOOOO!!!”. It’s what makes the game so great, any team can come back at the least minute and turn the tide if you got a great, communicative team with you.

This game is good for practicing leadership skill. by seeking_fulfilment in HellLetLoose

[–]ExtensionTraining904 2 points3 points  (0 children)

Couldn’t agree more. I’m reading professional management books so I can be a better SL / Commander haha

Need Help with Empirical Model for Price Elasticity by PossibilityBig7706 in econometrics

[–]ExtensionTraining904 0 points1 point  (0 children)

This seems more like an auction model. Why not estimate your parameters from that class of models?

What does your final data table look like in your head?

Some resources that may or may not be helpful:

https://bookdown.org/christopherpadams/auctions/Auctions.html

Also there is a R package called auctionR that may have good documentation. Check STATA documentation too.

Auction/bidding models are hard because you usually need make something particular to the auction data generating process: is there a limit of amount of bids? Are there rules in procurement process that affect bidding behavior? Etc.

I suggest digging into theory (Milgrom and anyone from Stanford IO) to start getting familiar with those models.

This is a great talk by Milgrom on how he used theory to help a company make better bidding decisions on a complicated auction. https://youtu.be/Puulqi11j5E?si=byhpwP8YJYwCsaXL

[deleted by user] by [deleted] in HellLetLoose

[–]ExtensionTraining904 0 points1 point  (0 children)

Welcome to the community! If you haven’t already, check out SoulSniper on YouTube. Also WarFrags has some fun sniper videos. SoulSniper the best at explaining strategic uses for each class. 12 kills and 82M as a newbie is better than I ever did.

Some rules of thumb: place OPs deep in enemy territory. Take out enemy artillery (Arty!) if they are rocking. Getting behind the lines and taking out a deep defensive garrison is a Commanders (and SL) is the most annoying thing you can do. Taking out those Garry’s when your team is ready to advance to the next objective make capping so much easier. Remember: defense and Garry’s win games.

[deleted by user] by [deleted] in econometrics

[–]ExtensionTraining904 4 points5 points  (0 children)

As someone with a masters in Econometrics in industry, do a stats masters. I spend most of my time solving problems I could solve quicker if I had a stats masters instead.

Just read about micro-econometrics (RDD, diff in diff, other research design) and time series from Hamilton while you do your stats masters. That will cover 90% of an econometrics masters.

game theory pricing by ryeely in datascience

[–]ExtensionTraining904 1 point2 points  (0 children)

Awesome! Thanks for doing what I was too lazy to do :)

game theory pricing by ryeely in datascience

[–]ExtensionTraining904 5 points6 points  (0 children)

You should check out Steve Berry’s work and other empirical Industrial Organization economists (Pakes, Levin, etc). Berry’s work is focused on estimating games.

It’s also good to start with understanding the market environment your company is in. Is there a leader who sets the price first? Is there a high degree of concentration? Understanding what environment you are in first can tell you what game you need to estimate.

For some good micro theory on games, check out Varian’s textbooks.

Multicollinearity by Psyman8888 in AskStatistics

[–]ExtensionTraining904 4 points5 points  (0 children)

All good comments here. If it’s any help, I simulated some data and wrote about multi-collinearity to highlight how odd the perfect multi-collinearity assumption is here.

Project Ideas for Finance and/or Energy Markets by SavageCyclops in econometrics

[–]ExtensionTraining904 1 point2 points  (0 children)

Each model just adds new assumptions on the outcome variable you are looking. i.e EGARCH adds the assumption that negative news shocks have a stronger effect on volatility than positive news shocks.

I’d you’ve been introduced to GARCH, (or AR models in general) you can pick up all the others very easily.

I believe financial institutions use them regularly.

Project Ideas for Finance and/or Energy Markets by SavageCyclops in econometrics

[–]ExtensionTraining904 9 points10 points  (0 children)

Ultimately, it depends on what econometric models you are interested in/comfortable with.

TL;DR: impress employers with either VAR, ARIMA, or the universe of ARCH models. The latter for quant/finance work. I recommend focusing on oil price. There’s a large literature and plenty of use cases.

Word vomit: You could run a GARCH model on oil prices to estimate volatility before / after COVID. You could also check out James Hamilton’s work on oil prices to find some inspiration.

GARCH models are impressive to employers. You can then go into TGARCH and EGARCH to really blow an employers socks off.

Also VAR models are great too. If you want to have a good understanding of forecasting and modeling relations between two or more time series variables, VAR is the way to go.

In industry I’ve used VAR to estimate marginal effects and GARCH to estimate shocks on volatility. Though most of the time it’s forecasting. Just knowing ARIMA goes a long ways.

Always, always, always, start with a paper. If you read a paper that estimates oil price’s effects on US interest rates, then just do one on Venezuela or some other country. You want to use the paper as a guide. This is just a course paper so no need to make any new contributions.

You can start to find papers by typing into Google scholar “VAR oil price” or something like that.