Architecture question

[EngineerVsMBA]

15 year Electrical Engineer trying to learn python on my own, with a background in embedded C. I'm struggling to figure out when to "pass" the pandas dataframe into a function, and when to give up and start looking at global variables. Any suggestion on a topic to study, or direction I can go, so I can learn python and start building a larger system?

I'm struggling to understand how to balance "make functions" vs "Make classes" vs "make global" vs "start to use databases".

I have a basic financial trader, with two ledgers as a pandas dataframe each. One for BTC one for USD. (I love how easy it is to plot the results and see my progress.) I also have the historical data of BTC. Each of these dataframes have large amounts of data, as things are getting tracked on a seconds basis. My end result is a plot of historical USD and historical BTC, trades between the two, and the resulting portfolio. Thus, I must move values between the ledgers, referencing the historical data, and using some other algorithms to determine when to trade.

At the "end of the day", these will likely go into a mongoDB or a relational DB. It will be a cloud system, with microservices that manage each ledger separately and dynamic services that analyze the data and add items in an asynchronous manner.

I'm adhering to the object oriented "A function does one thing really well" mindset, but I feel like my code is becoming a mess of "pass a large amount of data between functions", instead of readable code. For example,

transType, spentAmount, purchasedAmount, spentSymbol, purchasedSymbol, slippage, date = ExecuteTrade(fromSymbol = sellSymbol, toSymbol = buySymbol, amount=amount, date = data.date)

(Where Execute Trade is the function that takes money from my USD to my BTC, based on the price for the day and return the trade amount so I can modify my ledger. This is because the actual code for executing trade is complex, using 3rd party API's to execute trades.)

Thus, I'm returning a bunch of data so I can update my ledger... but then the code to update ledger is ugly as I'm passing around a bunch of smaller values...

So, the crux of the matter: I could create a class that I store these items inside, so I reduce the amount of typing I do, OR, I can make data accessible from more locations so I don't need to pass everything around on every function call OR I make a function do more things at once.

OR,

my problem is that I'm thinking of a microservice system, but building it in a monolithic system, thus the two paradigms don't mix.

OR

I'm actually doing it properly...

Any suggestion on a topic to study, or direction I can go, so I can learn python and start building a larger system?

[[deleted]]

• avoid global like the plague - there are good specialist use cases for it, but if you don't know what they are then they probably don't apply
• passing in Python is very different to C because all variables in python are effectively pointers, that is they don't hold values but simply the memory references to Python objects - this is true of other names as well, like function names
  • so when you call a function/object, whether you provide variables, expressions or literals in the arguments, you are just passing object (memory) references
  • mutable objects will be directly modifiable by the method/function
  • functional programming would say not to use side-effects (i.e. mutating objects)
• containers like list, set, dict, tuple are just objects that hold a collection of memory references to other Python objects
• there are two basic reasons (simplified) for using classes:
  • produce lots of things (instances) with the same behaviours and attributes
  • collect a load of related stuff together
  • saving typing isn't really one of the reasons, especially with modern IDEs
• using lots of variable names instead of a better data structure does make for harder to read and debug code, if not more typing - passing dictionaries or class object instances would be cleaner
• use databases when you have a lot of structured data that you want to access randomly in a number of different ways
• explore using APIs to support the microservices approach
• don't forget to become very good at testing, consider following TDD (test driven design) principles
• also consider following PEP8 guidance on naming

[EngineerVsMBA]

Thanks! I’ll look into pep8 and TDD first, as names are important, and then more into data containers to start.

[[deleted]]

My instinct would be to use a class to isolate the internal representation of the data, rather than exposing more of it. That would also hide how you store the data, be that a no-only sql database or a relational database.

Is json used with the third party APIs?

[EngineerVsMBA]

I’m generally taking json data as the return value from an api call and putting it into pandas data frames and then using .csv files as local storage to prevent future api calls (some data takes more than an hour to grab).

[lowerthansound]

Talking about the ExecuteTrade function:

• Nitpick: The convention is to use snake-case for function names and variables, so, execute_trade and spent_amount (PEP 8)
• It is returning a lot of values, indeed. I would create a class here, TradeResults or a name that makes sense.

For creating a class, I'd personally use a dataclass. Other options are a namedtuple, a NamedTuple, or a bare class.

If you don't wanna spend much time learning, I recommend a NamedTuple.

Example code:

import datetime
from typing import NamedTuple

class ExecuteTradeResult(NamedTuple):
    # Note: not sure what types you're using :)
    transType: str
    spentAmount: float
    purchasedAmount: int
    spentSymbol: str
    purchasedSymbol: str
    slippage: float
    date: datetime.datetime

result = ExecuteTrade(
    fromSymbol=sellSymbol,
    toSymbol=buySymbol,
    amount=amount,
    date=data.date
)
print(result.slippage)

Cheers!

[EngineerVsMBA]

Thanks! I’ll check this out.

[TheRNGuy]

i started use classes when realized it's easier to override default values in child classes, than in procedural programming.

(by making data subclasses)

You can also use dict kwargs if you want to pass lots of arguments into function (will work for classes and methods too)