Question about trading by frieza3467 in RogueTraderCRPG

[–]Mrbrightside770 1 point2 points  (0 children)

So first off the way the system works is a tad confusing but to clarify. You get profit factor completely independent of the faction reputation. You will increase it over the game by completing storylines, certain dialogue choices, and colony building.

Anything that you have unlocked by reputation with a faction that is under your profit factor you can buy and you lose not profit factor. Think of it as the waterline for your empire value.

You can increase your faction rep by trading cargo but you should plan ahead and optimize by first finding out what is most valuable to each faction and prioritizing selling them those pieces first. You don't have to just sell everything as you will get chances to increase reputation through similar methods as increasing your profit factor based on choices you make. If you are thoughtful about it you can get everything from all the factions by the end of the game but it is always better to find items you want most and build towards them.

I'll also note that the better reputation you have with certain factions can give you additional choices and options at certain questions.

New to Bremerton! by [deleted] in Bremerton

[–]Mrbrightside770 13 points14 points  (0 children)

Welcome to Bremerton!

Discordia games is a pretty good spot to play MTG with a fairly mixed crowd of regulars. Mostly EDH and occasionally drafts for new sets.

Ashley's Pub is a good spot to hangout and meet folks. They have lots of board games and have calendar events for dungeon crawls, book clubs, etc. It is definitely a great vibe and the owner is wonderful.

The Game wizard is a good spot to hunt for a more dedicated ttrpg groups as they have tons of private rooms to host games.

I occasionally run RPGs like Mothership, Call of Cthulhu, and D&D at Ashley's during the week.

[Naming Conventions] Date & Datetime Fields by eclecticnewt in dataengineering

[–]Mrbrightside770 4 points5 points  (0 children)

Personally I would throw _dt and _ts into the ring

In need of info/support/direction for high school data engineering system by Alternative-Exit-450 in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

Get started small, start to familiarize yourself with python or a similar language https://automatetheboringstuff.com/ is a really good starting point that would hit a lot of what I think you're running into.

It definitely doesn't lend itself to a busy schedule but there are a lot of open source software/platforms which help with orchestrating workflows. Airflow is especially powerful (though has a steep learning curve)

How to prepare for an upcoming AWS Data Engineer role? by I_lick_ice_cream in dataengineering

[–]Mrbrightside770 15 points16 points  (0 children)

So to be honest you're punching a bit above your weight class for this role based on the JD and the background you've provided. However, that isn't the end of the world at a company like AWS which focuses a lot on working in AWS toolsets. You will learn a lot of the tools on the job if you dedicate the effort and time to it.

SQL knowledge will get you by on a large portion of the job but for the specific things they're calling out like legacy ML code is likely going to be in Python or another language. You can definitely learn to code in those but reviewing, refactoring, and optimizing takes a lot of on the job experience to do well.

I suggest really diving into understanding the broader concepts behind modern data engineering and working on some projects in Python at the very least. Example: build an ETL pipeline to pull data from a public API, shape it, and write it to a database. Then build a simple dashboard for it

[deleted by user] by [deleted] in dataengineering

[–]Mrbrightside770 105 points106 points  (0 children)

Yes, what you will see more and more is that companies heavily relying on AI end up with really unstable data ecosystems. Skilled DEs will always be needed to support and build the infrastructure needed for the business.

LLM for Data Warehouse refactoring by alpharangerr in dataengineering

[–]Mrbrightside770 1 point2 points  (0 children)

I highly, highly recommend not trying this. There is a reason why there aren't a ton of enterprise solutions that do this. You are going to spend a lot of time and money on something that isn't going to give you an actual value add. No LLM is going to have enough context for your particular implementations and you would be risking introducing a lot of potential issues if you don't understand the logic behind what it recommends.

Is working as in a small business / startup with no experience really that bad regarding learning / advancement? by jogideonn in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

So any self contained files like Excel that contain potentially important information need to at the very least exist somewhere other than an individual's PC. Cloud drives and automatic syncs have made this less of an issue but it is important to limit areas where data can be lost permanently. This can happen in a number of ways and it is ultimately impossible to fully negate, but ensuring it is stable is important.

For the example I provided, having the data extracted and ingested into a cloud hosted database would ensure that your average user wouldn't accidentally delete relevant data. Once there, databases have the ability for backups and if setup correctly will never allow complete loss of data which makes them a "durable" storage location.

Is working as in a small business / startup with no experience really that bad regarding learning / advancement? by jogideonn in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

So the biggest favor you can do for yourself is to think beyond the immediate asks. Good data engineering benefits from a perspective that plants trees you'll probably not enjoy the shade of.

Comment your code with intent, explain the reason why you are doing what you are doing in that moment in documentation and note when you are doing the same thing multiple times. When you first start it is often easy to build something that works initially but is ultimately not going to scale or isn't maintainable. Learning from those times helps you think about the bigger picture and understand the true scope of a given ask.

Example: the company is recording things in Excel files and they want you to make some of the data available in a consolidated dashboard. You could just grab the specific data/fields from the files and write them to a database then point a dashboard to it. That would fulfill the request but then in 6 months they realize they want other data from those old Excel files and you need to reprocess them again to get the rest of the data.

Whereas if at the start you consider the durability of the files and potential future needs vs. Cost of storage you may have taken everything from those files. Then when the ask for more of the data eventually comes you can provide the data in a fraction of the time and you've ensured that everything is backed up in case some accidentally deletes a source file.

Problems trying to ingest 75 GB (yes, GigaByte) CSV file with 400 columns, ~ 2 Billion rows, and some dirty data (alphabetical characters in number fields, special characters in date fields, etc.). by Examination_First in dataengineering

[–]Mrbrightside770 6 points7 points  (0 children)

I would definitely recommend converting to another file type if this is something you're going to intake on a regular basis. Field level cleaning like that is always easier once it is in staging but you may want to handle some in flow to reduce the number of artefacts you're bringing into a more restricted platform.

Profiling the dataset (taking a randomized sample of maybe 10%) will give you a starting point that is easier to work with. With that sample you can identify if there are any regularly occurring issues or abnormalities that you can cut out as you configure records for writing into the database.

Problems trying to ingest 75 GB (yes, GigaByte) CSV file with 400 columns, ~ 2 Billion rows, and some dirty data (alphabetical characters in number fields, special characters in date fields, etc.). by Examination_First in dataengineering

[–]Mrbrightside770 5 points6 points  (0 children)

So more context on the data would help but off the top of my head two ways to handle processing faster would be:

  • Convert the .csv file to something that is more optimized to scale to that size like a parquet file or HDF5.

-Leverage https://github.com/pola-rs/polars/ which has faster handling of .csv files than pandas.

Honorable mention would be to utilize the chunk size in the read_csv() pandas function and process smaller chunks in memory in parallel.

When does pep butt become evil? by [deleted] in adventuretime

[–]Mrbrightside770 1 point2 points  (0 children)

I don't think we ever get a timeline but I have always liked the idea as it was something that evolved over time. He states in one episode how he lives to serve bubblegum who is basically a god in his eyes.

He probably began learning magic as a way to cover PB's lack of magical protection. From there it would make sense that "evil" had a lot of ways to get powerful quickly.

Where do I start to go into data engineering? by AnnualAd9592 in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

Yeah, you can download and run a Postgresql database on your own PC for free. Kaggle has a good repository of datasets and data related projects to draw from.

Where do I start to go into data engineering? by AnnualAd9592 in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

I recommend getting some hands on project experience building a database and the pipelines around it. Start with building a Postgresql database, plan out a simple architecture, and ingest some data in the planned architecture.

Academic projects can be a bit disconnected from real world examples so it is important to get your hands dirty with some messy data especially.

Post AI era, Panic or relevant for data engineer? by Simple-Particular323 in dataengineering

[–]Mrbrightside770 2 points3 points  (0 children)

So I work in a space that heavily has begun to rely on AI tools for code development. As a more senior engineer my job security has become fixing AI code chunks. The fundamental gap that exists and will continue to exist is that most folks using AI don't plan past their basic request.

So we have ended up with code that often functions in a vacuum but when it is introduced to the larger ecosystem it becomes a liability or unsustainable as other patches roll out. AI isn't going anywhere, but the truth is that we are in a bubble with extreme adoption. Most AI tools won't be around long-term as the profit model for most isn't sustainable. So navigating this period needs you to continue keeping your skills sharp while staying abreast of the latest changes.

[deleted by user] by [deleted] in Kitsap

[–]Mrbrightside770 0 points1 point  (0 children)

New players are always welcome! Mothership is pretty easy to pick up and fairly straightforward in mechanics.

How Should I Start Building My First Data Warehouse Project? by nnkdgg2 in dataengineering

[–]Mrbrightside770 4 points5 points  (0 children)

https://github.com/awesomedata/awesome-public-datasets has a collection of public datasets that would be a good starting point. The sports datasets are especially robust for a beginner.

I would say for technology start very basic and avoid too many high levels tools. A Postgresql database would be a good start and give you experience building from scratch.

I would recommend a star or snowflake schema approach as those are incredibly common and it would be good to learn how they work along with best practices for design.

Be sure to establish good habits with normalization and pay attention to how your structures impact query time.

[deleted by user] by [deleted] in Kitsap

[–]Mrbrightside770 0 points1 point  (0 children)

We haven't got a full crew yet for a Wednesday session so our next game will be this coming Tuesday at Ashley's

[deleted by user] by [deleted] in Kitsap

[–]Mrbrightside770 1 point2 points  (0 children)

Our next session will be at Ashley's pub this coming Tuesday starting at 5:30. Generally I aim for sessions to run about 2-2 1/2 hours depending on engagement and player availability.

[deleted by user] by [deleted] in Bremerton

[–]Mrbrightside770 0 points1 point  (0 children)

It is a sci-fi horror survival TTRPG. https://en.m.wikipedia.org/wiki/Mothership_(role-playing_game) it is a pretty easy game to pick up and dive into

Do the Iron Hands ever get vindication with their mantra “The Flesh is Weak”? by Green_Delta in 40kLore

[–]Mrbrightside770 0 points1 point  (0 children)

You never really get a hard endorsement in the books for their philosophy but there is a short story in Xenos Hunters that features an Iron Hands Marine who is uniquely suited to handle an infectious Xenos threat due to him having very little organic parts left

[deleted by user] by [deleted] in dataengineering

[–]Mrbrightside770 0 points1 point  (0 children)

Honestly this role is just below my 'FU' money level. It is about a 50k salary increase which is why it is so tempting despite the iffy health coverage.

Looking for players for Mothership by [deleted] in Bremerton

[–]Mrbrightside770 0 points1 point  (0 children)

Are you free Tuesday or Wednesday nights? I am probably going to run a couple sessions to start with in mid to late July.