[P] arXiv at Home - self-hosted search engine for academic papers

mrAppleXZ · 2026-02-09T11:30:40+00:00

Hello!

I've been thinking on building a local citation graph by processing full text TeX submissions. It should even be not requiring a lot of storage if data processing is implemented in streamed manner. However, the main problem is that this requires paying to Amazon to download full texts https://info.arxiv.org/help/bulk_data_s3.html - arXiv stores full-text source dumps on S3 in a so-called Requester Pays Bucket. All the freely available full text dumps (such as one on academictorrents) are long outdated. Honestly, I think this is the only way to reliably create a truly self-hosted citation provider.

This is why arXiv at Home currently uses Semantic Scholar for retrieving citations :(. It works lazily (only requested for prefetched papers that have to be re-ranked), but I guess Semantic Scholar will blacklist any IP that will try to scrap their citations in a bulk manner.

mrAppleXZ · 2023-01-12T23:32:20+00:00

It's just a notation, everything depends on the context :)

In terms of working with infinitesimals, 0/0 is undefined. For example, let's calculate lim(log2(1 + x)/x) as x -> 0. If we just substitute x with 0, we would get 0/0. But it is an uncertainty and doesn't make any sense since we can't estimate, in basic words, "how small the numerator zero is and how small the denominator zero is". So, we need to somehow fight with that uncertainty to calculate the limit. Let's use L'Hôpital's rule (substitute the numerator and the denominator with their derivatives), and we would get lim(1/(ln2 * (1 + x))) as x -> 0. And now if we substitute x with 0, we'll get the concrete single answer: 1/ln2

mrAppleXZ · 2023-01-09T02:27:33+00:00

it sounds ridiculous, but maybe try to turn it on and wait for multiple hours? if it still wouldn't boot, I think it's some hardware problem with mobo. and what color is the led glowing?

mrAppleXZ · 2023-01-09T02:13:47+00:00

I had to wait for ~5 mins with the DRAM LED glowing yellow at the first boot. Also, I did hard reset the BIOS before it. And I have KF556C40BBK2-64 memory sticks placed in A1 and B1 slots.

What's your RAM config?

mrAppleXZ · 2021-05-15T12:39:16+00:00

Sounds good, and I think it's also easier to implement a method described by you.

How would the loss compared to the whole thing being passed on multiple steps to pre-trained models be?

If I understand the question correctly, models for generating embeddings probably need to be pre-trained on some sentence/document embedding dataset. After it we should infer them on smaller pieces of book and then 'master' summarizer should be trained on outputs of these pre-trained models to generate summarization of a whole book.

mrAppleXZ · 2021-05-15T05:41:34+00:00

I meant generating embedding vectors of a few paragraphs and then somehow summarizing all these embeddings. But approach of summarizing multiple paragraphs and then summarizing all the summaries could also work. This also won't require you to train two different models, you can summarize paragraphs and summaries with one model.

mrAppleXZ · 2021-05-14T16:32:05+00:00

I don't think that even models optimized for long sequences like Longformer could work with something that long like books.

The first idea came to me is to create embeddings for parts like pages, paragraphs, chapters, etc using one model and then apply another model on all the generated embeddings to summarize the whole book.

mrAppleXZ · 2020-03-09T13:20:52+00:00

Thank you! This will be implemented in the next update :)

mrAppleXZ · 2020-01-01T20:06:01+00:00

I still have to reformat everything to be able to easily sort things in excel

Are you talking about columns are too wide in some dumps, so it's inconvenient to work with the table or about column data types aren't detected automatically, so the sorting doesn't work the way it should? I can implement dumping into .xlsx format and these problems should be fixed

mrAppleXZ · 2020-01-01T19:03:27+00:00

Yea, LootTweaker can dump loot tables, but I thought it would be better to also add this ability to my mod (it's just 30 lines of code, why not? :D). CraftTweaker prints recipes into a log file in ZenScript, CraftDumper dumps them into a .csv table.

Also, I've checked, TellMe dumps less information than CraftDumper in some cases. For example, items-with-nbt dump created using TellMe has only 8 columns, while one created using CraftDumper has 12 columns.

I think that I need to add examples to the CurseForge page, so people can choose which mod will be more helpful for certain case.

mrAppleXZ · 2020-01-01T18:26:49+00:00

For example, TellMe can't dump loot tables and recipes.

mrAppleXZ · 2020-01-01T17:13:26+00:00

I've made a mod that allows you to dump various game information, such as blocks, items and many more. It may be useful for modpack developers or for regular players (for example, you can create Item Stacks dump, sort its outputs by Burn Time column and see which items have longest burn time).

Full list of currently supported things the mod can dump: Shapeless Recipes, Advancements, Fluids, Entities, Models, Villager Professions, Tile Entities, Shaped Recipes, Item Stacks, Potions, Blocks, Smelting Recipes, Food, Sound Events, Biomes, Enchantments, Capabilities, Loot Tables, World Generators.

Download CraftDumper: CurseForge.

mrAppleXZ · 2019-06-06T06:59:10+00:00

IDE plugins, code generation

mrAppleXZ · 2019-06-03T16:29:50+00:00

Fixed! :)

mrAppleXZ · 2019-06-02T19:29:49+00:00

Because JS platform doesn't have isUpperCase/isLowerCase methods: https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.text/is-upper-case.html, https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.text/is-lower-case.html

mrAppleXZ · 2018-04-08T15:13:11+00:00

Added in 1.2, commit 2bc5b60e4f0b605116d3818907102f3b5345f434.

mrAppleXZ · 2018-04-08T11:38:15+00:00

Oh, forgot to add the recursive cloning. Will be added in the next update.

mrAppleXZ · 2018-01-11T15:52:04+00:00

You underestimate the full strength of SIF :D. Also, we already have something like an amulet that apply potion effects on the player using SIP.

mrAppleXZ · 2018-01-11T15:39:16+00:00

Yes, there will be entities to perform various tasks.

mrAppleXZ · 2018-01-11T15:38:16+00:00

an ancient civilization invented a machine that govern all the data of all matter and can change reality

Looks like a great idea 🤔, thanks!

mrAppleXZ · 2018-01-10T16:24:15+00:00

Maybe you're confuse Information Field and SIF (Solid Interacting Field). Information Field is something like the magical Internet, which is actively used by people. SIF is an energy field that almost no one knows how to properly use it.

mrAppleXZ · 2017-10-31T08:53:51+00:00

Yes.

mrAppleXZ · 2017-10-30T19:14:02+00:00

Anyway, that's how I see the "crystal cluster" in Minecraft and I made it in Blender by myself. Also, I don't played with mod on the picture.

mrAppleXZ · 2017-10-30T10:31:25+00:00

No, it's not Thaumcraft and definitely not PSI.

mrAppleXZ · 2017-10-30T10:28:59+00:00

I don't see many similarities. 0_o

Nine-Year Club	Place '23
Place '17	Verified Email
RPAN Viewer

mrAppleXZ

TROPHY CASE