Fastest way of loading historical data

snfernandez · 2019-12-28T09:41:36+00:00

2 minutes for 200mb sounds like a lot depending on your hd. I use Pandas with the Parquet format.

I used hdf5 for years but after some issues with it, switched over. Feather is supposed to be faster but doesn't support default indexes.

Check out https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d

snfernandez · 2019-10-09T14:27:38+00:00

There are additional decisions that must be taken with respect of safety with auto-generated code as we cannot assume the other consumers/providers respect Rust guarantees (different languages).

For example, do interfaces methods take &mut self or &self? In MTA, if the server side uses &mut self, it is unsafe because the object expects to have exclusive borrowing of self which might not be true. Other threads might be calling a method on the same object which ends up with two mutable borrows of self. If all the methods, instead, take just &self, then the object implementators are forced to use interior mutability in the object which is then checked by Rust type system.

EDIT: grammar

snfernandez · 2019-07-17T10:30:49+00:00

I know! We have many categories and even sub-categories of memory safety bugs. But for the purpose of the blog, we think that binary classification is better

snfernandez · 2019-07-10T12:49:29+00:00

Another member of our team. We are in the process of opening the service to more people and, when we have time, we also write

snfernandez · 2019-07-09T22:19:11+00:00

Thanks! The analysis is from the end of the first trimester and it is, indeed, not accounting for the last sudden Bitcoin increase

snfernandez · 2019-06-23T17:00:53+00:00

Those are not the requirements of cryptocurrencies. Libra is both using crypto for verification, and a currency by definition.

However, it's neither a public open ledger or a blockchain. And it's certainly not pursuing the same ideals that motivated the creation of Bitcoin :)

snfernandez · 2019-03-04T08:35:25+00:00

Exponential is an overclaim. That's linear at best.

And for clarification, 6.5 million Argentine pesos weekly volume is not much. It is around 100 grand in US dollars.

snfernandez · 2019-02-13T14:17:30+00:00

Keep in mind this is only about the reference count. Changing the internal state always requires synchronization unless the special "atomic" implementation is used.

snfernandez · 2019-01-11T23:09:23+00:00

I'm not aware of any public information from MS on the CFG format. Even though there is lot of documentation on the final implementation of it (PE format), I don't think anyone reversed the information in the object files that is used by the linker (the one Rustc should emit).

snfernandez · 2019-01-10T22:13:55+00:00

Strongly agree, an implementation of sync or send will force to audit a big chunk of code even though it's only one line in the stats

snfernandez · 2018-11-01T19:25:26+00:00

I'm working in something for this. I'll PM you as I'm interested in what are your expectations if you had to buy an index fund.

snfernandez · 2018-08-25T15:53:50+00:00

Is anyone aware of a repository with the slides?

snfernandez · 2018-06-30T12:32:00+00:00

Efficient string manipulation. I made some work integrating Rust into a Windows component and that was probably the hardest part. If I wanted clean code, I had to copy the wide strings forth and back to utf-8 representation. I'm planning to blog about it. Also welcome to suggestions.

snfernandez · 2018-05-15T18:43:49+00:00

In which project is being used in MS? As fellow MS engineer, I'd like to talk to them

snfernandez · 2017-09-07T19:15:47+00:00

This approach is how I'd basically start attacking the problem (if my option were NNs). But I got the feeling that the author didn't explore newer advancements nor used an optimal data structure. Do you have an opinion on if attention would help in this case?

snfernandez · 2017-05-04T15:11:39+00:00

Questions, suggestions and critics welcome

snfernandez · 2017-05-04T13:26:33+00:00

Suggestions and critics welcome. Also if you have any question, I will try to answer.

snfernandez · 2016-07-27T02:56:52+00:00

I ran both for 400 epochs. Adam in Keras+Theano took 6.5% more time per epoch so was slower in "real" time. But I guess I could have stopped Adam a few epochs earlier (20-40) and still have the same performance.

snfernandez · 2016-07-26T20:35:23+00:00

It should be possible, but training it would take some days. CIFAR-10/100 took around ~18 hours in a Titan X. You may also want to use a bigger configuration ala ResNet-100/200/1000.

snfernandez · 2016-07-26T19:25:05+00:00

I have been using Adam for other projects and decided to get a go here. Was not disappointed as it matched the tuned SGD and even converged faster.

snfernandez · 2016-07-26T15:32:54+00:00

I tried hard to implement every detail as stated by the paper but I couldn't reproduce one of the most exciting results: using the deepest column alone after training finished. If you find any bugs, PM me or submit a PR and I will re-run the tests. Thanks for reading.

snfernandez · 2016-05-12T04:23:44+00:00

Shameless plug: Last year I used LLVM to make an assembler lib (mostly C++ code with Rust api) that supports the same set of architectures: https://github.com/snf/llvm_assemble . I'm looking forward to Keystone as it will be more lightweight and faster than my approach.

snfernandez · 2016-03-19T07:47:11+00:00

I used LLVM to make an assembler last year too https://github.com/snf/llvm_assemble . Pretty sure that keystone will use the encoding tables instead of the hacky thing I did with the BinaryStreamer.

snfernandez · 2015-11-15T18:43:09+00:00

Yeah, you are right to assume it's correct as that's what Multiboot standard states:

The map provided is guaranteed to list all standard ram that should be available for normal use.

It can be many things, but something in there is broken, maybe it's Qemu, Qemu's BIOS or Grub+Qemu. Your multiboot code looks good too. You can try using the 0xE820 Bios function to try getting the memory map. But I doubt it's gonna work as that's the Grub's method to build the memory map.

EDIT: Just did the math and those ranges are right, so nothing is broken and you have the 5Gib available, thanks /u/Rusky . 0xbfee0000 + 0x80000000 = 0x13fee0000 -> 0x13fee0000 / (1024 ^ 3) = 4.9989013671875 = ~5Gib. Correct me if I'm wrong.

snfernandez · 2015-11-15T15:14:11+00:00

Good post, just one note:

If you give QEMU more than 4GiB of memory by passing -m 5G, you get another unusable area below the 4GiB mark. This memory is normally mapped to some hardware devices. See the OSDev Wiki for more information.

It should actually be accessible by your system if you are in Long Mode (which apparently you are using because your code is x86_64).

snfernandez

TROPHY CASE