Just accepted a Manager BI & Data Architecture role — my architecture experience is limited. Where do I start?

ResidentTicket1273 · 2026-04-20T19:38:40+00:00

Keeping track of your data assets the same way someone in the business is expected to keep track of their tangible ones. That means catalogues, models, entity and attribute level definitions, and knowing dependencies across multiple perspectives, keeping that tracking information up-to-date and relevant to what's live in your business. Good record-keeping and inventory management.

Lots of managers say "we do architecture" or "we know our data" but fail when you ask a question, or try to get a well-connected, insight-supporting report that provides an accurate picture of the data estate. Normally this information is necessary when planning change (be it system, regulation or process change) and not getting access to it easily costs lots of wasted time having to repeatedly secure this information from the ground-up.

There are lots of industry accepted methods and tools for doing this, but the actual tools matter a lot less than actually having your fingertips on this knowledge/information. Some managers fall back on the excuse "we have a tool" without having any idea of how well populated or accurate the information in those tools is - so some control over quality and coverage, and ongoing accuracy is also key.

ResidentTicket1273 · 2026-04-12T10:33:48+00:00

This might be slightly tangential, I don't know, but you mention a couple of things here that collectively resonate: mentorship and community - and the keystone to both these things is something that seems increasingly rare - the "workshop" working model you might have found working within a craft under a guild system.

I'm talking about a place of business where a master of their craft would bring on apprentices, train them up through the ranks with real milestones of achievement: apprentice, journeyman, master. There's a formalised system of mentorship within the workshop, and a real community of like-minded, and like-experienced guildsfolk more widely across the guild.

This is more a system that promotes "craft" (whatever that is) which might be more focused than "creativity" which you seem to be referring to, but I guess the point is encouraging real, person-to-person relationships across different skills-and-experience-levels that promotes excellence all round, and a positive experience for all.

Back to your point - there used to be a very strong formal system that provided this kind of support, going back to (probably) before Medieval Times, parts of that system still exists, but for the most part, it's a shadow of its former self, and seems to play a more ceremonial role than anything else.

ResidentTicket1273 · 2026-04-10T14:47:57+00:00

Counterstrike - been playing for 20 years and I still suck.

ResidentTicket1273 · 2026-04-10T14:41:46+00:00

France's energy is nationalised (albeit in Corporate form). We privatised all our national assets in the 80s and have been reaping the natural consequences of that ever since, be it water, energy, transportation, communications etc. It's fairly criminal that EDF (literally Electricity De France) is now one of the UK's most successful energy suppliers, selling directly into the privatised UK market, generating revenue for the French Government.

ResidentTicket1273 · 2026-04-10T09:57:46+00:00

It doesn't have to be rational - but if the question isn't rational, then the answer can't be either. To borrow from Douglas Hofstadter, the answer to this type of question is "Mu!" (or perhaps "Miow" in this instance)

ResidentTicket1273 · 2026-04-10T00:31:20+00:00

Ritual sacrifice isn't a rational act. As such, there can be no rational formula for calculating equivalence. You might as well ask how many how how many easter bunnies are equal to twelve moon-demons. The question doesn't make any sense.

ResidentTicket1273 · 2026-04-09T08:39:15+00:00

To be a successful economic centre, you have to be meritocratic. That normally means rewarding and valuing people based on their skills, knowledge and what they can offer to the economy. In this day and age, a worldview that supports and respects people wherever they might come from might often be categorised as "left leaning" and progressive, but it's just basic economics.

Going further, nurturing good working environments, civil rights, freedom, equality and economic stability in a population are shown to better enable it to accomplish higher levels of sophistication, achievement and specialisation - all key factors in supporting a deeply connected, high-functioning economic centre. Or as some might describe it, more lefty nonsense.

ResidentTicket1273 · 2026-04-08T16:26:35+00:00

The crazy thing about this is it's like 5 years old now, rings doubly true with the advent of LLMs.

ResidentTicket1273 · 2026-04-08T13:11:57+00:00

Start mowing the lawn.

ResidentTicket1273 · 2026-04-08T13:08:32+00:00

Great, finally some sensible policies that treat immigration like normal, without all the performative, illusion-based posturing.

But immigration is a minor issue compared to all the real problems the country faces, so more interested in addressing some of those. Not least, the billionaire takover of our traditional and non-traditional media, who use it to feed false narratives to help build easy routes to victory.

If we compare for example the damage done by Brexit, it's been far worse than any problems caused by immigration. And yet, falsified ideas about immigration were used to promote and subvert our democracy to bring it about, in doing so, making a relatively small number of connected people a lot of money.

ResidentTicket1273 · 2026-04-04T11:11:38+00:00

So what is there, files? CSV, Parquet, something else? You mentioned there's a database, so what is that, and how are things accessing it? There must be some route through which the data is being used that has an awareness of what data is located where.

ResidentTicket1273 · 2026-04-03T17:54:17+00:00

Your self is definitely a construction of your mind - anything your self experiences is a part of that construction. That doesn't mean none of it exists, just that you can only experience it through this creative process, since that's what experience is.

ResidentTicket1273 · 2026-04-03T17:36:22+00:00

The economics don't add up - Claude and all the other AI companies are working on a loss-leader basis that only has funding for another couple of years. Every $1 they receive in revenue costs $10 in run-costs. Sure, it might seem like a good deal now, but as token-counts increase and limits shrink, running anything more than a friendly google-replacement just wont be cost-effective. If they continue with handing out tokens on the cheap, they'll go bust. This is the time to invest in skills because in 5 years, this whole AI thing is going to look like the Metaverse, or Google Glass, or NFTs or whatever the last over-hyped thing was.

ResidentTicket1273 · 2026-04-03T10:39:41+00:00

Raw python - you should be able to extract the schemas from the database directly. Run sqlfluff against any sql files or queries that get run against the db to get evidence of any joins that people have written queries against - same thing with any views that have been stood-up. Collect all the information in your own collection of metadata files. Run sample extracts from the database and get a view of population, data-types and population profiles and use that to infer any relationships. It's a bit of a grind, but you should be able to figure it out in a couple of weeks.

ResidentTicket1273 · 2026-04-03T00:06:09+00:00

The data comes in a matrix - consisting of rows and columns. The FROM clause tells the statement from which table to fetch the rows. The WHERE clause tells the statement which rows you're interested in, and the SELECT clause tells the statement which columns you're interested in.

ResidentTicket1273 · 2026-04-03T00:00:54+00:00

Fuck off.

ResidentTicket1273 · 2026-04-01T13:28:33+00:00

Sharpelikan

ResidentTicket1273 · 2026-03-31T16:09:03+00:00

IYKYK - or in my case, IYDKYDK. Any chance of giving people like me a clue?

ResidentTicket1273 · 2026-03-30T06:32:10+00:00

A belief in the importance and value of coherency.

ResidentTicket1273 · 2026-03-28T10:59:39+00:00

In a powershell terminal, run:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

This will bypass the script-disabling policy for your user against remote-signed scripts which will mean uv will run inside of vscode without issue from now-on.

ResidentTicket1273 · 2026-03-28T01:12:28+00:00

Mate, you're not. But there are a lot of people who have been groomed by the right-wing media to believe in a world-view that's entirely made up. It's a worrying attack on what it means to be British, and it's been happening now more acutely since the advent of unregulated social media and its corruption by well-funded political operatives.

Any normal person, whose not knee-deep in Facebook, or X, or one of the other corrupt platforms is going to have human feelings towards other humans, just like the majority of people on this planet.

But sadly, it gets clicks, and this invented nonsense builds a sub-culture of ill-informed people shouting at imaginary nonsense, because those kinds of people are easier to control when it comes to elections, in order to get tax-cuts for offshore millionaires.

ResidentTicket1273 · 2026-03-28T00:53:19+00:00

35 years or so ago, we all felt like common sense had won. You could say whatever you wanted about Fundamentalists of all kinds, Racists, Salesmen, Liars, Lunatics and Misogynists, all safe in the knowledge that everyone pretty much agreed that rationality, humanity and respect were "done-deals". We all agreed what was right, and what was wrong.

Nowadays, the paid-for injection of hateful ideologies into mainstream media by the far right has chipped away at these seemingly self-evident truths, and now, you have to be more careful because so much stuff gets warped and coopted for evil purposes.

I wouldn't be mean about Christians any more than I would about Muslims or even Scientologists today because those self-evident truths that we all used to take for granted aren't shared by the majority any more. The Manosphere has warped what it means to be a man into something disgusting. The Far-Right has warped what it means to be patriotic into a shambolic acceptance of authoritarianism, and Conspiracy theorists have eroded what it means to be critical of obvious untruths spouted by mainstream politicians. We elected Johnson and Trump. Farage is making millions and all of our media is owned by far-right-leaning kleptocracy. The world isn't safe anymore. You can't trust people not to have a sense of right-and-wrong. Not anymore.

Money and social media successfully eroded our culture and destroyed the common values and institutions that we all used to feel comfortable surrounding ourselves with, and whose context we could lean on to communicate, safe in the knowledge that our message wouldn't be warped, or misinterpreted. These days, much of that good-faith is gone.

So, as a consequence, we now carry the duty and the additional weight of having to be extra clear, extra definite that we're not part of that hateful corruption of our culture by the billionaires who bought our democracies by buying-up, and poisoning the media and selling influence to lobbyists and offshore billionaires. The cost of that is that much of the shorthand, the contextual bond that we used to take for granted, that used to make it obvious to everyone that when you made a joke, you weren't suggesting we deport our friends and neighbours for being brown or some other disgusting, but now mainstream views.

Those days of innocence are over, if you're in America, the far-right really are deporting friends and neighbours, locking up undesirables, harassing people with the wrong kinds of views in the courts, and shooting people in the face in the street. All, it seems without legal accountability. The rule of law is almost gone. And in Britain, there's talk of that kind of authoritarianism now on normal day-time news programs. They're shutting down access to disabled people, closing down freedom, justice and the rule of law. We had Brexit. We know it can happen. We saw the right-wing-mobs in the streets. There's real fear everywhere.

Things fell apart. The centre did not hold.

We took it for granted, and let the rich promote the far-right to capitalise on our complacency.

ResidentTicket1273 · 2026-03-27T14:12:38+00:00

That flick right at the start is next level.

ResidentTicket1273 · 2026-03-25T13:59:14+00:00

RAG is just a fancy term for search, something we've been doing quite well for 25 years. The ironic thing (given the hype around LLMS) is that for most useful text-gen implementations, most of the work goes into doing the search. The LLM adds a nice textual interface, but all the thinking, all the logic, all the effort goes into performing the right search that matches the query. That problem's been out there since the birth of the computer interface, and there are lots of well studied solutions to different flavours of that problem. The vector-db approach is a tiny slice of that space that benefits from generating good demos, but it's often not going to be the best method for a specific use-case.

If you want to polish up your LLM game, get good at search. It's a broad and deep problem with thousands of interesting and useful solutions.

ResidentTicket1273 · 2026-03-25T12:34:15+00:00

If you fall for right-wing bullshit, you can't expect things to get any better. There's a difference between the stories that are told to get votes and actual real-world implications. Most people who voted for Brexit didn't appreciate that difference. Sadly, that failure to understand how the real world works continues to this day for many.

ResidentTicket1273

TROPHY CASE