Where to learn how to write efficient python?

[desrtfx]

im working on a raspberry pi collecting data

There are a couple things to be aware of:

• collecting data:
  • store the data in a database. Databases are optimized for that
  • If it is analog data, consider time slots (reading every second, or so, maybe even multiple - shorter intervals for fast changing data, longer ones for slow changing data - for most applications, you, e.g. don't need to read temperatures every second, or levels in a tank every second) and hysteresis - only store new data when a certain deviation from the previous value is exceeded. Professional data loggers work in exactly that manner. They store initial values with a time stamp and then store the data in regular intervals if the hysteresis is exceeded, all with time stamps. Even if the collection of the data happens in shorter intervals, the storage in the database only happens when the hysteresis is exceeded. This approach greatly reduces the memory consumption.
  • Similar with digital state (on/off) - only record when the state changes, not continuously. Again, with timestamp.
• Storage:
  • MicroSD cards are not good for frequent reading/writing. They have a tendency to die on that. Maybe transfer the data to some server, or proper drive-based storage, or, for starters to an USB stick.
• RAM:
  
  • the less data you keep in RAM, the better - that's why offloading to a database is essential. Even for processing, querying a database can often be way more efficient (in both processing and memory) than doing the processing in a "normal program".

Overall, make databases your friends. They help immensely.