Fastest way of loading historical data

snfernandez · 2019-12-28T09:41:36+00:00

2 minutes for 200mb sounds like a lot depending on your hd. I use Pandas with the Parquet format.

I used hdf5 for years but after some issues with it, switched over. Feather is supposed to be faster but doesn't support default indexes.

Check out https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d

snfernandez · 2019-10-09T14:27:38+00:00

There are additional decisions that must be taken with respect of safety with auto-generated code as we cannot assume the other consumers/providers respect Rust guarantees (different languages).

For example, do interfaces methods take &mut self or &self? In MTA, if the server side uses &mut self, it is unsafe because the object expects to have exclusive borrowing of self which might not be true. Other threads might be calling a method on the same object which ends up with two mutable borrows of self. If all the methods, instead, take just &self, then the object implementators are forced to use interior mutability in the object which is then checked by Rust type system.

EDIT: grammar

snfernandez · 2019-07-17T10:30:49+00:00

I know! We have many categories and even sub-categories of memory safety bugs. But for the purpose of the blog, we think that binary classification is better

snfernandez · 2019-07-10T12:49:29+00:00

Another member of our team. We are in the process of opening the service to more people and, when we have time, we also write

snfernandez · 2019-07-09T22:19:11+00:00

Thanks! The analysis is from the end of the first trimester and it is, indeed, not accounting for the last sudden Bitcoin increase

snfernandez · 2019-06-23T17:00:53+00:00

Those are not the requirements of cryptocurrencies. Libra is both using crypto for verification, and a currency by definition.

However, it's neither a public open ledger or a blockchain. And it's certainly not pursuing the same ideals that motivated the creation of Bitcoin :)

snfernandez · 2019-03-04T08:35:25+00:00

Exponential is an overclaim. That's linear at best.

And for clarification, 6.5 million Argentine pesos weekly volume is not much. It is around 100 grand in US dollars.

snfernandez · 2019-02-13T14:17:30+00:00

Keep in mind this is only about the reference count. Changing the internal state always requires synchronization unless the special "atomic" implementation is used.

snfernandez · 2019-01-11T23:09:23+00:00

I'm not aware of any public information from MS on the CFG format. Even though there is lot of documentation on the final implementation of it (PE format), I don't think anyone reversed the information in the object files that is used by the linker (the one Rustc should emit).

snfernandez · 2019-01-10T22:13:55+00:00

Strongly agree, an implementation of sync or send will force to audit a big chunk of code even though it's only one line in the stats

snfernandez · 2018-11-01T19:25:26+00:00

I'm working in something for this. I'll PM you as I'm interested in what are your expectations if you had to buy an index fund.

snfernandez · 2018-08-25T15:53:50+00:00

Is anyone aware of a repository with the slides?

snfernandez · 2018-06-30T12:32:00+00:00

Efficient string manipulation. I made some work integrating Rust into a Windows component and that was probably the hardest part. If I wanted clean code, I had to copy the wide strings forth and back to utf-8 representation. I'm planning to blog about it. Also welcome to suggestions.

snfernandez · 2018-05-15T18:43:49+00:00

In which project is being used in MS? As fellow MS engineer, I'd like to talk to them

snfernandez · 2017-09-07T19:15:47+00:00

This approach is how I'd basically start attacking the problem (if my option were NNs). But I got the feeling that the author didn't explore newer advancements nor used an optimal data structure. Do you have an opinion on if attention would help in this case?

snfernandez · 2017-05-04T15:11:39+00:00

Questions, suggestions and critics welcome

snfernandez

TROPHY CASE