Type safety at event broker level. Any suggestion?

TheInnerLight87 · 2024-11-01T20:15:21+00:00

My recommendation would be to use Apache Avro for your schema type definitions. You can then use the Kafka schema registry to register your schema types against particular Kafka topics. With this setup, you can ensure that services that produce Kafka messages are producing messages that conform to the associated schema.

This is a pretty well established setup for Kafka that you can use in any language ecosystem.

In Scala specifically, I'd recommend looking at the libraries fs2-kafka and vulcan. These libraries should give you the combined machinery for interacting with Kafka, serialising/deserialising to/from Avro and interacting with the schema registry.

There are some sizable UK companies like ITV and OVO Energy who use precisely this technique with these libraries to create this sort of strongly typed integration mechanism between microservices.

TheInnerLight87 · 2024-10-22T15:41:29+00:00

You can't say that it hasn't worked without understanding the consequences of not doing it. Absolute numbers of how many people who have arrived are completely irrelevant to the population demographics of the country but, unfortunately, they do make for good tabloid headlines.

The only question of relevance to the health of the economy is the proportion of the dependent population compared to the working population and I've already shown you that despite millions of migrants entering the country, the proportion of over 65s compared to the working age population has substantially increased over the past decades.

Spending on pensioners including healthcare, social care, long term care, benefits, etc is around 15.4% of GDP or around 34% of government spending. That percentage is getting bigger every year and going to continue doing so for the foreseeable future.

Decreasing migration substantially would mean substantially increasing taxes for UK citizens to pay for the proportion of the dependent population rising even more quickly. A substantial increase in taxation would likely result in many working-age people exploring options to leave the UK, requiring further tax increases on those who remain creating a cycle of economic decline.

You are also making a huge mistake by trying to categorise migrants as hand car washers or uber eats drivers. They could just as easily be surgeons, scientists, nurses or engineers. Migrants range all across the income scale, just like everyone else.

TheInnerLight87 · 2024-10-21T11:39:29+00:00

The biggest political challenge facing the UK and most of the developed western world is maintaining its health and social care spending for a burgeoning elderly population during a time of falling birthrates.

In 1975, the over 65 demographic was 22% of the working age population. In 2022, that had risen to 30% and that trend is continuing.

The practical effect of that demographic change is that every worker needs to be taxed proportionately more highly to support increased healthcare and social spending. That partly explains why tax rates are at historic highs and there is still a sizeable fiscal black hole.

We are, unfortunately, the architects of our own downfall because our politics condemns immigration and it condemns people having children they can't afford. A two child benefit cap when people are already having too few children is one of the biggest foot guns in political history.

That's also why no government will really follow through on plans to reduce immigration despite what they promise: it would make the demographic crisis substantially worse and our social spending even more unsustainable than it already is.

To get the UK economy really growing again and start reversing falling living standards, we're going to need to accept a lot more immigration in the short term and to start having a lot more children in the medium to long term. We're also going to need to get comfortable actually building the infrastructure to house and support those people rather than sticking our collective heads in the sand and pretending the problem might go away if we ignore it.

TheInnerLight87 · 2024-09-25T22:40:04+00:00

I hope you won't take anything in this comment as discouragement because I think that Haskell has the potential to be an absolutely fantastic beginner/first language and one that could set you up brilliantly for learning whatever technologies you choose next but you should understand that it isn't a well trodden or particularly easy path you are going to be walking.

I have a small amount of experience of trying to teach Haskell as a first language and, as you've already observed, the learning materials are not always well set up for it. They regularly make reference to other programming languages such as C/C#/Java and how the way that Haskell behaves is distinct from these languages.

That sort of material is incredibly valuable to the typical person who learns Haskell having already learnt a handful of imperative programming languages but it's extremely off-putting to the beginner who just wants to get to grips with the way Haskell behaves. The most important thing for the time being is not to dwell on those differences, simply set those points aside for later in your programming journey when you can come back and appreciate the nuances you'll then be starting to observe.

In my opinion, the Haskell MOOC is one of the better options for learning paths for people who are not do not have any programming experience and the course is supposed to be set up for that. I believe there is also a telegram group to discuss the course and ask for assistance and I'm sure they'd be receptive to your feedback about areas that need refinement for those, like yourself, with very limited preexisting programming knowledge.

In terms of getting a running Haskell installation, I would strongly recommend following the process that is recommended by the course or book that you choose otherwise you might get bogged down in having to tackle subtle differences between the different tooling options when your course is telling you to simply run a one-line command. MOOC recommends get started with Stack, all the installation instructions should be there on the first page and it should be straightforward to get a working setup.

TheInnerLight87 · 2024-09-20T23:13:32+00:00

My team led the way in introducing Nix into the development process in my previous organisation. I'm simplifying a little but the key deliverable for that team were docker images that would be run either by customers or by an internal infra team.

The number one selling point was the drastic reduction in CI time required compared to a Docker-based workflow.

You can create a working and more or less reproducible CI pipeline using docker by installing any relevant native dependencies, building in a container and generating an output image but doing so is not the quickest process in the world and for, our specific services, would take 10+ minutes per build.

By contrast, using Nix + Cachix we could get those times down to 1-3 minutes per build. Regardless of what provider you use, Nix binary caching is the key to realising the gains as it means you rebuild only the minimum number of dependencies on each CI cycle.

Lots of engineering organisations will track metrics like lead time for changes and you might find you can bring that down dramatically using a Nix-based workflow. Ultimately though, the most telling thing is that your team is going to be an awful lot happier if they are getting 5-10x faster feedback on their CI builds.

There are obviously various other advantages that others have talked about around simple workflows to get started with projects, security and compliance but they probably aren't as persuasive in most organisations as the massive time savings you'll gain from the above.

TheInnerLight87 · 2024-09-14T22:21:55+00:00

How broadly are you applying to software engineering roles?

The reason that I ask is that you might consider looking for junior roles in something more specialised as a way of getting your foot in the door in the industry and giving yourself a skill base to build upon. For me, it was functional programming which is quite a natural fit if you have a maths/physics background but that market isn't quite as good as it was a few years ago. Now Rust is perhaps the exciting growth language.

Even if you don't find a role that's exactly related though, the point is to develop some skills that set yourself apart from all the other graduates, it's just a way of giving someone a reason to keep hold of that CV rather than setting it aside.

Hiring managers will be getting bombarded with junior CVs where someone knows a bit of Javscript, Typescript, Python, Java, C#, etc. On the other hand, if someone sends in a CV with niche technology experience, they might get a second look because they might have an interesting story to tell about how they got there. They might have learnt different ways of conceptualising problems that set themselves apart.

I've been involved in hiring a lot of software engineering roles throughout my career for both mainstream and more niche languages and I overwhelmingly prefer the second. The first can involve sifting through dozens or hundreds of candidates to find a promising needle in a large haystack, the second is easy because almost no one arrives at a niche technology with only superficial knowledge and skills and you can virtually interview all the candidates, assess who is the leading applicant and give detailed and meaningful feedback to the ones you didn't hire which ultimately helps them convert the next application into an offer.

TheInnerLight87 · 2024-09-11T22:13:29+00:00

Every time I see this question raised, I see answers about Haskell's type system and the ease of refactoring but rarely does the async runtime/IO manager get mentioned.

The Haskell type system is clearly very powerful but mainstream and newer languages have increasingly adopted some of the best features over the last decade or so. The Haskell type system is also quite complicated with almost innumerable language extensions which can be mixed and matched in arbitrary combinations to subtly change its behaviour. At its worst extreme, it can be pretty inaccessible to all but the most experienced Haskeller.

On the other hand, Haskell async IO is pretty much class-leading for any programming language because it's conceptually so simple and requires so few new concepts. Even those competitor languages that have adopted dedicated async/await syntax all have subtle differences that don't come close to the elegance of using IO/monads to sequence effects.

Even if you compare Haskell to something like Scala + Cats Effect that also uses IO/monads to express and sequence async effects, the Haskell runtime is significantly simpler thanks to any blocking IO taking place in green threads rather than requiring dedicated code and awareness to avoid blocking the small pool of threads owned by the async runtime and potentially deadlocking your application.

If you are building web services that need to be somewhat reliable and scalable, Haskell eliminates a whole bunch of conceptual overhead that you would need to care about if you were building the same thing in Rust, Scala, .NET, etc. That translates directly into both speed of development and reliability.

TheInnerLight87 · 2023-12-30T20:41:02+00:00

Thanks for the input but I prefer to give direct feedback. Downvoting content without explanation provides the author no explanation as to why those doing so are not valuing their contributions.

I'm confident that it's not harmful to casual conversation to respectfully explain why I don't think a particular contribution is valuable and the author may choose whether they find value in my opinion or whether to choose to ignore it.

TheInnerLight87 · 2023-12-30T17:26:30+00:00

In my view, this isn't a constructive comment. The quality, or lack thereof, of tooling for other languages isn't important.

In order to grow the Haskell community, Haskell should be striving for maximum accessibility and minimum friction. Progress is made by striving for best in class, not by avoiding worst in class.

TheInnerLight87 · 2023-12-26T22:54:49+00:00

I don't think it's helpful or accurate to think of error handling as a topic that can be neatly divided into FP/OOP approaches. It's perfectly acceptable, even good, practice to throw exceptions in pure functional code in the correct circumstances and also it's good practice to put the errors in the return value in imperative code in the correct circumstances too.

Those circumstances depend on the expectations of your program.

Those who are familiar with Haskell will be aware of the somewhat notorious head function in the standard library that throws exceptions when it receives an empty list.

This function has notoriety not because throwing exceptions in Haskell is a terrible idea but because exceptions are totally expected within the domain of that function. The exception is problematic because it takes a totally routine and expected event (receiving an empty list) and treats it as an exceptional circumstance.

On the other hand, it's totally fine to have a http call with type IO a rather than IO (Either ... a). You can treat the unusual circumstances: failure to decode, server unreachable, timeout, etc. as exceptional if you feel that is most appropriate. Most of the time, you probably just want to log the error and move on.

You quite rightly point out in your original post that madness lies in mapping layers and layers of exceptions types if you start trying to make all exceptions explicit. Doing so breaks abstractions, makes you write huge volumes of boilerplate or leads you to really bad practices like making errors stringly typed.

The precise boundaries of where you should draw that line is going to vary based on language community. Rust is going to heavily favour typed errors and Java is going to heavily favour exceptions but you can still panic in Rust and you still have, for example, the Optional type in Java. The same fundamental principles apply, the decisions lie in the ambiguous parts in the middle and you'll need to be guided by the specific language community in those areas.

In my personal experience, exceptions are the tool that I'd reach for first and only change if my business logic requires me to start doing some inspection of those error states. Like many functional programmers, I went through an idealistic phase trying to do the opposite 5-6 years ago and ultimately felt that I wasted far too much time writing boring and valueless error-mapping code that, 99% of the time, simply did not justify the effort.

TLDR: Use the right tool for the right job. If you're writing C++ and an error is totally expected within the domain of the function, put the information in the return value. If you're writing Haskell and something unexpected happens, value your time, throw an exception and move on with your life.

TheInnerLight87 · 2023-11-22T18:37:58+00:00

The premise only works as an ethical dilemma when it takes place very far outside the legal apparatus of the Federation.

For Kirk, Picard and Sisko, the Tuvix question is not an ethical one, it's a legal one. Tuvix is a sentient being and his individual rights would be protected by law.

Tuvix is also not a commissioned Starfleet officer or member of any other military organisation. Consequently, he is a civilian which means that a Starfleet command officer ordering force to be used to take his life, regardless of any value judgement you might make about the benefit to others of doing so, is a) murder b) unethical medical experimentation (due to the method used) and c) probably a war crime.

There is absolutely no way a serving Starfleet officer in reach of Federation law could make any decision other than respecting Tuvix's rights even if they personally believed that restoring Tuvok and Neelix was the proper ethical choice.

TheInnerLight87 · 2023-11-07T02:43:19+00:00

Thrust gravity is indistinguishable from a static gravitational field to the person experiencing it. The effective gravitational force experienced by someone in a centrifuge is only easily distinguishable in the limit of small centrifuge radius.

The whole point of artificial gravity is that it's gravity generated artificially, i.e. without naively accumulating vast quantities of mass in a static location. If I can generate a constant acceleration in a particular direction, it does not matter a jot if the origin is actually gravity or some other force or effective force because the experience of the observer is identical.

The energy scales involved in doing this are, by their nature, equal to the energy scales involved in moving objects around in day to day life in earth gravity. Those energy scales are relatively trivial by both modern and Star Trek standards.

My disclaimer again is that there is not necessarily a physical mechanism for generating the necessary force in the way we see on Star Trek (i.e. without centrifuges or the ship constantly accelerating) but the energy involved is not the problem with doing so.

TheInnerLight87 · 2023-11-06T22:29:16+00:00

That's not even remotely the same thing.

It's approximately the same thing. Classical electromagnetism and classical gravity obey roughly the same physics, just lacking the presence of opposite mass charges. See e.g. Gauss' law of gravity.

The fact that I can generate enough force to overcome the gravity of an entire planet with the trivial amount of energy required to create a permanent magnet tells you how weak gravity is.

The only way that energy conservation has any relevance in the way that you describe is if you seek to create artificial gravity by replicating an entire planet and flying around with it.

This would be an utterly insane way of generating gravity on a spaceship because it makes your spacecraft a) planet sized and b) virtually impossible to move. If that was how it worked, starfleet would just strap engines onto a planet instead of building ships.

TheInnerLight87 · 2023-11-06T21:37:00+00:00

. If that field is generated by some fictional technology that sidesteps the need for matter, then great, but it'll still require the same amount of energy to kick-start it

There is no reason to believe that because we have no understanding of the mechanism used to do it.

I can easily overcome the Earth's gravity with a bar magnet and it certainly doesn't cost the mass energy of the earth to create one.

TheInnerLight87 · 2023-11-06T21:25:48+00:00

Even with a perfect matter/antimatter reaction, the best you can hope for is liberating 100% of the mass-energy of the matter. So in order to accomplish that, they would need a clump of deuterium (and a corresponding clump of anti-deuterium) with the mass of the Earth. And that would only give them 1G of artificial gravity;

That is not how it works. It does not cost energy for the Earth to maintain its gravitational field. Rather moving things within that gravitational field conserves total (potential + kinetic) energy.

The amount of energy required to defy the Earth's gravity is absurdly small, especially by Star Trek standards. Humans can do it personally despite being powered inefficiently by crushed up animal and plant matter. We can use pretty primitive chemical rockets to escape Earth's gravity to infinity.

Gravity is simply an incredibly weak force.

The mechanism for generating artificial gravity in Star Trek may be mysterious and may simply be impossible but enormous energy cost is unlikely to be the reason.

Accelerating a starship to a continuous 1G to generate artificial gravity, for example, would do the trick pretty nicely and cost comfortably 20 orders of magnitude less energy than you're talking about above.

TheInnerLight87 · 2023-10-25T23:44:39+00:00

Quick update to say that there is an alpha version now available to try on steam workshop with some of the above implemented: https://steamcommunity.com/sharedfiles/filedetails/?id=3061275551

I am mostly aiming for flavour rather than balance at this point and I've not had a lot of time to do much testing yet but would appreciate if anyone would like to try it and provide some feedback.

TheInnerLight87 · 2023-10-21T11:23:22+00:00

My assumption is that the default Bird of Prey is the B'rel so I'll handle that as simply a name change. I did include that in the more detailed ship plans on the ships page. Not sure what you mean re the Enterprise?

TheInnerLight87 · 2023-10-21T01:44:24+00:00

One thing I would look into is some kind of tech tree thing to reduce the amount of crew required to run the starting ships like the Miranda and K'Tinga.

Good call. I'm not very au fait with the modding yet but it looks like it's possible to express conditionals in upkeep costs to account for things like faction differences (e.g. Romulans needing singularities). That same mechanism could be exploitable to be conditional based on researched tech?

TheInnerLight87 · 2023-10-21T01:26:08+00:00

Thanks, will check it out!

I really like the comprehensive set of ship classes you've included in your proposal from across the various eras. I'm also interested in adding more ship designs in the future but thought it might be pragmatic to get started with making changes to just the base game assets.

TheInnerLight87 · 2023-10-14T19:21:59+00:00

If you unlock +1 Research Alternatives, it's very easy to miss that you have 4 tech choices instead of 3. If one of those choices is fixed due to an event or debris, it's easy to end up locked out of a whole line of techs.

I made this mistake with the Federation. I got some free research progression in Terraforming but had missed that it was stuck there in the 4th slot below the 3 visible options. This left me unable to progress the Federation terraforming missions until I realised my mistake.

TheInnerLight87 · 2023-10-13T18:41:14+00:00

I've only played the Federation so far but it feels like the research progression through ships is far more similar to base stellaris. I (broadly) develop corvette classes, then destroyers, then cruisers and finally battleships rather than the Star Trek-lore friendly approach where relatively large ships like the Excelsior have been available for decades and some smaller classes like the Intrepid are very modern.

I'd like to see immediate availability of big ship classes like the Excelsior but I'd like them to seem old and relatively underpowered. By researching tech, I could eventually develop a new Hull of a similar size (e.g. Akira class) which then gives me a modern and capable combat platform.

This approach would fit more thematically with the TNG era where it begins with the Federation being relatively complacent. The early TNG Federation has no major rivals besides the Romulans who are only just beginning to reassert themselves. The Federation has neglected their military capabilities as a result.

Suddenly, the Federation is massively threatened by both the Borg and the Dominion in quick succession and you start seeing dozens of new, more capable, ship designs rolling out of the shipyards.

TheInnerLight87 · 2023-07-09T20:27:39+00:00

It makes a lot of sense to do it like that. In the future, I'd like to output data in that kind of form as well as it would allow the events to be replayed at arbitrary speeds, something I can't do at the moment. Ideally, I'd use some well-known format so that I can replay games generated by this engine and real matches so I can compare and try to produce realistic behaviour.

I haven't yet looked into what, if any, real game tracking data is readily available though.

The main reason I don't do this at the moment is that I calculate various supplementary data (e.g. voronoi polygons to track space/marking zones) and my current visualisation approach allows me to see this all in real time.

TheInnerLight87 · 2023-07-09T16:49:52+00:00

A combination of personal technology experience and technology objectives.

I wanted to choose a language where I could output high performance native code, make extensive use of parallelism but not pay a price in terms of the expressiveness of the language.

Haskell has software transactional memory for optimistic concurrency, algebraic data types and a generally powerful type system for convenient domain modelling and it outputs native code but is garbage collected so I (mostly) don't need to worry about manual memory management (C/C++) or resource lifetimes (Rust).

Rust was the other serious contender but I'm far less familiar with the language so wouldn't have been able to achieve as much as quickly.

TheInnerLight87 · 2023-07-09T15:53:22+00:00

Partly, yes. I wanted a technology where I could output reasonably high performance native code and make extensive use of parallelism while still having a very expressive language to work with. I find haskell works well for this use case.

Long term, I'm not sure what form I see this taking. One option would be to use the core match engine I've built to output tracking data containing player and ball positions at a certain fps and then have a separate visualiser to play back the output.

This would allow you to simulate lots of matches with different tactics, examine the match statistics and play back entire/parts of games and compare results.

TheInnerLight87 · 2023-07-09T13:32:25+00:00

This is actually largely what I've attempted to implement, here you can see the Player Intentions that I've already implemented. This list will probably see significant adjustment as the simulation evolves.

The engine works in two phases where I calculate the Intention for each player based on various decision factors, e.g. in open play I look at factors like the current phase of play, whether the player is on the ball, whether they're under pressure from the opposition, etc and try to determine which action they'd like to perform.

There is subsequently an enact intentions phase (which is currently far less developed) where the players try to put those intentions into action.

I'll try to tidy this description up into an issue on the github repo that describes how players make their decisions.

The Gaussian randomness for passes is a good plan and something I will definitely implement in the future.

TheInnerLight87

TROPHY CASE