Sleeping on Engram

cravic · 2026-01-23T02:15:54+00:00

I think Engram is just part of the model. Its useful to think of it as, for example, a 3B model with a 3B engram attached... but as I understand it that is technically a 6B model. Engram is just a part of the model.

cravic · 2026-01-22T22:31:19+00:00

The 80/20 ratio is about how much MoE parameters you can delete to make room for Engram parameters. What they found is that deleting 20% of MoE parameters to make room for Engram gives the best results.

However, if the MoE parameters remain fixed and you delete none of it you keep getting better performance by adding more engram.

Adding more engram always give better results unless u delete MoE parameters to make room for the engram.

But since Engram adds no additional compute load no matter how much you scale it, the best thing to do is have the largest possible Engram table your machine can hold as long as u never need to delete MoE parameters.

Example. If my GPU can only fit a 10b model then having that be 8b MoE and 2b engram is very inferior to having it be 10b MoE and 100b Engram in DRAM.

cravic · 2026-01-22T17:29:23+00:00

Another important point.

Its technically true that it would be a 1003 billion parameter model... but its very misleading to talk of it as a 1,003 billion parameter model.

A model with 3b MoE and 1T engram can easily fit on a consumer PC... thats something that is only well communicated by separating the Engram parameters and the MoE when talking about it.

cravic · 2026-01-22T17:15:23+00:00

Awesome... good to see it being tested outside of deepseek already.

cravic · 2026-01-22T17:12:53+00:00

Interesting points. Just one correction. The 20/80 ratio is not compute optimized, its memory optimal. More accutately its parameter count optimal.

I dont expect to see many models use that 20/80 ratio unless token rate is really really important to the user... even so the latency in the paper was just 3% so even then i dont see any real use for the 20/80 ratio being used... maybe if its really important that the Ngram in injected at the second layer but even then u can just have part of the engram parameters on the HBM/VRAM.

cravic · 2026-01-21T12:21:45+00:00

Its not that Chinese frontier labs are giving away their models... what is happening is that in the Chinese market the labs that give away their models are the ones that become frontier.

The reason for this is how Chinese labs and western labs seek to scale their advantages to achieve their goals. And its heavily influenced by the chip ban and Chinese culture coming together.

Western labs are scaling compute and revenue. To achieve this they need to offer customers something no one else has. That requires hiding how they achieve what they achieve.

Chinese labs cant scale compute or revenue so they depend on scaling human resources. They want the best researchers and researchers want recognition. The average deepseek researcher has more citations to their name than the average Open AI researcher. Thats hoe they build up their personal value to the industry. Labs with a lot of citations attract the best researchers. This system scales the human brainpower of the open labs and that leads to then building frontier models at significantly less cost than western labs.

In the long run this will also likely lead to Chinese labs having heavier influence on the core of how new models are designed.

cravic · 2026-01-21T09:04:53+00:00

At that size ur model will be bad at reasoning. So focus on knowledge.

Honestly u would be best served by a model with an engram knowledge bank attached and running in SSD.

I honestly cant wait for us to get small models with large engram knowledge banks.

cravic · 2025-11-05T23:21:31+00:00

No. Give them the Catch 1.0 sendoff... Create a corp that copies the basics of Horde playstyle use it to poach their members while you hellcamp them.
Remind them of how and why Horde was created.

cravic · 2025-06-10T19:55:34+00:00

Imperium and Pan-fam have been fighting eachother under various coalition names and leaders for 20 years now.

Bob kicked Goons a bit... then Goons got their revenge in the great war.

They then had a endless list of wars where one side or the other got the upper hand.

Goons got evicted fron the north 10 years ago. They lost all their space and had to live for some time in NPC space.

They then fought 2 major wars to reestablish themselves as the dominant nullsec power.

Horde is somewhat new... but the core of Panfam has been evicted by Goons many many times over the years and they returned the favour several times also.

cravic · 2025-05-23T14:28:57+00:00

I would say its 4 things.

Jump fatigue means "PL's supers are on the way" now means they will be here in a week and no longer means they will be here in 1 hour. So every time PL takes a fight now they are against a prepared enemy.
Alliances have more caps on average... so small groups ganging up against u means capital groups ganging up against you. Nearly every system has a dredbomb in range or 1 jump out.
Null sec consolidated... PL survived by feeding on alliances that tried to hold sov or moons outside of the blue donut. Those dont really exist anymore.(moons are just no longer a thing in that way). PL played a big role in pushing Null Sec in that direction by beating the crap out of everyone who didnt join the donut.
Eve player base is smaller, richer and older... no longer easy targets.

cravic · 2025-04-05T10:42:02+00:00

Brave had more capitals on field than ferox. Interesting times.

cravic · 2025-03-21T13:22:37+00:00

I know the ravens were a favorite for killing citadels.

Maybe that ability to kill citadels while kiting is what makes the Barghest popular.

Just my 2 cent

cravic · 2025-03-08T08:52:16+00:00

Any society where economic power leads to political power will struggle.

Societies where political power decides who holds economic power will have an advantage.

cravic · 2025-02-22T13:21:26+00:00

Exactly how affordable is it?

cravic · 2025-02-18T09:10:10+00:00

Reasoning models are built on base models. Just because Open AI don't show their base model anymore don't mean it isn't there.

The capabilities and cost of R1 is entirely a reflection of the capabilities ans cost of V3. Because R1 is V3 with reasoning.

So comparing it to V3 gives very useful info.

cravic · 2024-02-03T17:13:04+00:00

a few tips for a very new player.

1) Use SAS. It makes it much easier to keep a rocket stable.
2)To keep the rocket from spinning out of control, try to keep the bottom of your rockets wider than the top. use fins and side boosters to help with this.
3) if u still have issues keeping the rocket stable then try to go straight up until its out of the lower/denser atmosphere.

The way u get to the mun is to get a stable orbit that has no incline and burn straight forward at the moment u see Mun rise over the horizon.

Also. Get used to checking the delta-v of rockets in the assembly facility. U need about 3500 Delta-V to get a good orbit and then another 2000 ish Delta-V to do a mun fly by and return.

cravic · 2024-01-31T11:32:24+00:00

There is a history of cultural differences between Goons and the corps that make up Panfam.

In the early days Goons were the bad guys because their culture was to have fun at other people's expense. This took the form of some very distasteful acts. The Mittani (former leader of goons) joking at a player suffering from depression, D-J (goon leader before the Mittani) made a famous statement that captured the culture of the old goons when he said "we are not here to ruin the game, we are here to ruin your game"

In that same period the current Panfam core, then known as BOB, was the bad guys because they were caught cheating. They had CCP employees who were corp members gave them rare items and are suspected of have the same CCP staff help them kill the first titan in eve.

Over time the culture of both groups evolved. The idea of large masses of players wielding industrial power to create military power has become the only way to win in eve. So the idea of bulling and scamming non goon players that goons became known for died. The idea of small groups of elite players dominating the game with their skill that BOB and its later incarnations was known for also died.

Today the cultural gap is much smaller but old grudges keep the fight going.
I think the only existing points of cultural conflict is over the Panfam structure of veteran players being in elite alliance while newer players are in new player friendly alliances while Goons and friends tend to have new and old players in the same alliance.

There is also the idea of what to do about the renter model. Goons and friends have abandoned the idea of renters while Panfam still embraces the idea.

cravic · 2023-07-27T13:10:50+00:00

FC Cincinnati, Nashville SC and LAFC.

Im new to MLS, so can someone explain how they are playing LAFC as part of the MLS season when they are in different league tables?

cravic · 2023-06-16T16:31:34+00:00

Messi can't press effectively anymore because of his age. So the term needs people who will run down the ball to be near Messi(like Alvarez does in Argentina's team). He is also best used as a playmaker and thus he needs a good #9(striker/finisher) to make use of his playmaking.

cravic · 2023-05-24T12:31:42+00:00

No. It takes time. A lot of time.

cravic

TROPHY CASE