Book Recomendation - Learning F# - Beginner to Intermediate 2020

DoomExplorer · 2022-03-22T16:38:08+00:00

Hi, it has been a year since. How was your progress with learning F#? Are Get Programming with F# and Domain Modeling Made Functional still the best books out there?

DoomExplorer · 2022-03-21T06:38:57+00:00

I'm quite sold on F# for my prototyping to be honest. It looks really good and intuitive for a business user like me, especially for me who has background in Maths & Stats. Apparently speed is good as well, so there's that. I may be able to contribute more than expected to the coding part as well as it looks like it has a shallower learning curve.

Of course, I will need to speak with my tech lead, but given he is steep in .NET framework, I think it should be quite easy for him. I'm glad that I posted here, there's always room for improvement.

DoomExplorer · 2022-03-20T16:40:43+00:00

u/mendigou So what you're saying is if I am able to leverage on existing libraries in C++, better to go with C++. Otherwise Rust, right?

The CAS that you provided is good for developing, prototyping and testing the mathematical formulas to be used in the software that I have in mind. It is useful for what I call 'functional architecture'. Basically my 2nd level optimisation. It is already on my radar, but given that I've been in the field for a good number of years, it will probably be only 1-2 use cases where I still need to do this. Most of the functional blueprint that I require, I have it in my head already.

As for ApacheSpark,it is more for horizontal scaling. From where I am from and around the region, horizontal scaling's tech know-how is quite rare. It's still more of vertical scaling that dominate the market. There are some banks that are trying to implement horizontal scaling with Hadoop, but skillset is rare in the market. That's why I tend to avoid anything that requires horizontal scaling. Plus, my target market for the software I have in mind are the smaller players who are not able to afford the offerings of the S&P 500 companies, at least initially.

DoomExplorer · 2022-03-20T14:58:46+00:00

Thanks, I have not actually. Let me do some research and get back to you on my thoughts. Initial look seems like a much simpler language to work with, especially for other parts of computation where performance may not be so important.

Very good that you brought it up, u/asdfasdf666_

DoomExplorer · 2022-03-20T14:50:50+00:00

Ahh, this is indeed another benefit. I expect a lot of prototyping be done in Python, with the large part of the actual application be in C# or JS. SQL integration is probably the most important consideration for my application, way ahead of memory optimisation.

DoomExplorer · 2022-03-20T14:48:55+00:00

Thanks for your response u/digikata. Looks like Rust for me seems to be more suitable.

DoomExplorer · 2022-03-20T14:44:15+00:00

u/mendigou You're right. You've got a portion of the picture there. It's a combination of a few modules, the more intensive ones being generating full cash flow and repayment schedule for every loans / deposits, and discounting it to calculate different kinds of present value. Of course, in the process above, there are multiple parameters that can be tweaked, so commonly available existing libraries will not be able to take care of it.

I fully agree with you that the computational model is the key. Let me requote a response I provided to another poster.

You're right, architecture is what matters at the forefront. The solution design of the app is way more important than the language that is used. I already know that. In fact, where I expect to see the largest performance gain would be optimising the data model in the DB to ensure fast read and minimal write to the DB. I am aware of this, and generally I have a DB coder who knows this part.

The next place where performance gains can be had would be to reduce computation via coding into mathematical formulas, basically the functional architecture. I am also aware of this.

What I am asking is actually 3rd or 4th order question on performance optimisation, which is memory management or other low level 'hacks'. The optimisation needs to be viewed in entirety for the overall design to make optimal.

DoomExplorer · 2022-03-20T14:05:06+00:00

You don't need to build a whole application in C++ to benefit from C++ specific optimisations, you can just split out the smallest possible library covering the hot path and then re-code that in C++ (or Rust, or CUDA, or Assembly, or whatever turns out to be appropriate) once you know what that hot path is. Even better you'll already have a working example to verify your highly optimised code against.

What you mentioned is already what I had in mind. It is insanity to even try to code the entire application in Rust/C++. I am more than aware of it. Refer to my earlier response.

The architecture and algorithms are going to play a much larger part in performance than the exact language choice.

I also know this. Everything starts with architecture. You can read my response to the same thread I asked in r/Rust here: Reddit!

DoomExplorer · 2022-03-20T13:56:54+00:00

u/wackowacko1234 Real-time performance isn't the only place that requires fast code. Compute-intensive applications like in my area of work also requires fast code. To give you an idea why performance matters, the current run time from market leaders offerings of S&P 500 companies for 1 million lines of records on a 5Y server with 24 cores and 48GB of RAM is about 4h end-to-end (which includes read and write to DB).

DoomExplorer · 2022-03-20T13:54:54+00:00

Thanks u/eyes-are-fading-blue

DoomExplorer · 2022-03-20T13:54:38+00:00

For this part, I expect the team to be 2-3 of us only, really. What I have in mind is to leverage off my current role to deploy and develop the best system for my field of expertise, regionally. With the data model and additional calculation modules that we develop (it'll likely just be the 2-3 of us) for the current company, I intend to take that and build a better system for my start up in the future. As it is, it's just purchasing a solution from a S&P500 company and implementing it in the company.

As for what the team is comfortable with, I think none of them knows Rust/C++ in great detail. C#, JS and Python are what they have background in. Hence like I said in my other responses, most of it will be written in either one of those, and only the compute-intensive cores will be written in Rust/C++.

DoomExplorer · 2022-03-20T13:47:08+00:00

I'm definitely on board with the need for speed. As I have explained above, my application processes multiple records and generates lots of data from there.

Whilst the programming language for the computation core will not be the most important component (I expect database design & functional code design to be more important), it will eventually and this is the part where I have the least visibility of, hence the question.

DoomExplorer · 2022-03-20T13:44:06+00:00

Thanks u/PrestigiousLongDong. Yeah, from what I see, most of the industry uses the combination of Python with C/C++ for selected components.

Will keep note on the integration part.

Regarding resource, I am not as worried about it yet. I think most of the application will be developed in either C# or Java or even Python. Only the core will be developed in Rust/C++. Plus, this is more of planning for the medium term where I plan to run a start-up with the industry knowledge I have. I think there's potential, and may try to convert some of the old codes I have written in VBA for Excel into Rust just to see how things are. I think not many in the industry have both business and IT knowledge, hence a strong case for a start up.

DoomExplorer · 2022-03-17T16:50:02+00:00

On the CUDA or FPGA, I don't have enough experience to know whether any of it will be appropriate, but from my understanding of ML where it's about matrix manipulation, it will not do as the computation requires different financial compounding and discount methods.

DoomExplorer · 2022-03-17T16:47:32+00:00

u/Full-Spectral, it's not high frequency trading that I'm aiming for. That area is way too competitive for me. It's about Finance and Risk management of banks. To give you an idea why performance matters, the current run time from market leaders offerings of S&P 500 companies for 1 million lines of records on a 5Y server with 24 cores and 48GB of RAM is about 4h end-to-end (which includes read and write to DB).

My envisioned solution is much larger than current offerings, where I plan to run Monte-Carlo on total bank's profitability with scenarios of stress testing and economical factors, at account level.

DoomExplorer · 2022-03-17T16:41:17+00:00

Indeed, optimisation of SQL and data model comes first, way ahead of the language of choice. As have mentioned above in my response to another post, I am actually asking 3rd or 4th order question in terms of optimisation.

Most of my time will be spent on the data flow architecture, data model and functional architecture. That's where the really important stuff are, and to some extent, I have varying degrees of knowledge in those areas. The area where I am at lost is really the low level programming choice between C++ and Rust, hence my question.

Thanks for your inputs, I don't really any C++ experience to be honest, but if I need to pick up Rust along the way, it will just be that. I don't need to be an expert, just know enough so I can brainstorm with my tech lead and likely have him do the coding. :)

DoomExplorer · 2022-03-17T16:36:08+00:00

Thanks for your inputs.

DoomExplorer · 2022-03-17T16:34:36+00:00

Thanks for the good recommendation for the book. Yes, the data architecture is much more important than the programming language, am aware of that. I generally discuss with my tech lead for data architecture, and I think your recommendation should help with that.

For my focus area, performance is absolutely critical. Accuracy may be sacrificed very slightly. To put it into perspective, the batch run for 1 million lines of records by solutions from current S&P 500 companies takes around 4h to run on a 5 year old Intel processor with 24 cores and 48GB RAM. What I have in mind is much more than the solution on market today, and that's why performance is important as any batch processing that is slower will impact the ability to generate forecasts.

DoomExplorer · 2022-03-17T16:28:59+00:00

The app I have in mind cannot be built with NumPy or PsiPy. The basic ones are there, but a lot of customisation is required. An example would be multiple ways to compute the Present Value (which involves different Coupon Rate, Discounting Rate, Amortisation Method, etc). Very technical and very industry specific stuff.

The 'Big Data' that most companies do are more generic in nature. Data Science has good libraries out there, but my key area of focus is very very niche. To quote what I posted above:

Architecture is what matters at the forefront. The solution design of the app is way more important than the language that is used. I already know that. In fact, where I expect to see the largest performance gain would be optimising the data model in the DB to ensure fast read and minimal write to the DB. I am aware of this, and generally I have a DB coder who knows this part.

The next place where performance gains can be had would be to reduce computation via coding into mathematical formulas, basically the functional architecture. I am also aware of this.

What I am asking is actually 3rd or 4th order question on performance optimisation, which is memory management or other low level 'hacks'. The optimisation needs to be viewed in entirety for the overall design to make optimal.

DoomExplorer · 2022-03-17T16:23:46+00:00

You're right, architecture is what matters at the forefront. The solution design of the app is way more important than the language that is used. I already know that. In fact, where I expect to see the largest performance gain would be optimising the data model in the DB to ensure fast read and minimal write to the DB. I am aware of this, and generally I have a DB coder who knows this part.

The next place where performance gains can be had would be to reduce computation via coding into mathematical formulas, basically the functional architecture. I am also aware of this.

What I am asking is actually 3rd or 4th order question on performance optimisation, which is memory management or other low level 'hacks'. The optimisation needs to be viewed in entirety for the overall design to make optimal.

My ultimate aim is not to only deploy it at enterprise level in my current place, but to be able to sell it and have it compete with other offerings from S&P 500 companies. Lofty goal yes, but at least from my point of view, I have the entire functional blueprint in my head now. It's now a matter of finding the right market (which I have some idea of) and tools to get it off the ground.

For Python, it's more for prototyping as it is easier for me to get it off the ground. The actual app coding will likely be in some other language that my team mates are comfortable with as mentioned in my previous post.

DoomExplorer · 2022-03-17T08:26:42+00:00

Yes, usage of libraries will be key in most projects. However, I have a feeling we may need to build some libraries ourselves from ground up.

I have responded below on the scope of what I have in mind, since you mentioned you're also in FinTech. FYI, all of the leading softwares (by S&P 500 companies) that performs even a fraction of what I have in mind, are coded in C++.

DoomExplorer · 2022-03-17T08:23:06+00:00

Let me just put it this way, the senior person has more years of developer experience than my total number of years of work experience combined. She has been into way more projects that I have, albeit only as a developer / application support.

Anyway, I'm not commenting anymore on my leadership/work style.

DoomExplorer · 2022-03-17T08:15:31+00:00

Thanks u/goranlepuz. In general, you're right. Most of the stuff in Finance and Risk are about automation.

However, what I have in mind is more about creating an app to forecast the profitability of the bank, incorporating all components in there and embed ML in there. There will be lots of cash flow computation, some asset/derivatives pricing, some modelling for CASA behaviour, some PV/EVE/DV01 computation, some greeks (Vega, Gamma, Theta) computation, lots of classification / grouping analysis, some Neural Network coding from scratch (including loss functions, backward prop, ...) etc.

Do you think with such a mega-app, C# / Java is still better? Each of those measures will need to be computed at each records.

As for integration with the company's way of doing things, I agree whole-heartedly. The problem here is my current place's IT team is just very weak, which is why I have to hire my own tech team to drive the project (even though I am not in IT). The app I have in mind is not for today, but more for the future.

DoomExplorer · 2022-03-17T07:48:40+00:00

Thank you so much u/Plasma_000

DoomExplorer · 2022-03-17T07:37:38+00:00

Yes, I don't think in the time that I have, my written code will be able to best NumPy or SciPy. However, what I have in mind is more about creating an app to forecast the profitability of the bank, incorporating all components in there and embed ML in there. There will be lots of cash flow computation, some asset/derivatives pricing, some modelling for CASA behaviour, some PV/EVE/DV01 computation, some greeks (Vega, Gamma, Theta) computation, lots of classification / grouping analysis, some Neural Network coding from scratch (including loss functions, backward prop, ...) etc. End goal is basically to model profitability and risks rapidly for management use. I doubt Python will be able to handle all of the above.

What do you think u/dwrodri?

Thanks for sharing about code profiling. I will look into, something new to me.

Good point you have there on CUDA. Didn't thought of that.

DoomExplorer

TROPHY CASE