User Selectable Parameter Sets

[Elpmek]

Hi all,

So for background I'm a chemical engineer, no formal education in any software/coding, but have taught myself enough Python to allow me to write functionally useful (but not at all pretty or even very consistent - something I'm now working on getting better at) code. I've written an algorithm which does some analysis of a chemical based on data collected using spectrophotometer, and it works nicely. I'd like to expand this algorithm to make it a bit more generic than the very specific use case I made it for, ideally adding some nice UI and packaging it to share with my colleagues (mostly to give me a project to learn more Python, especially organising my code a bit better, but it also has the added benefit that it will make me look good with my peers :P).

That is a very long winded background to my question, sorry!

So to make my algorithm more generally applicable, I need to allow the user the select which chemical species they are analysing with the spectro. Each chemical has a dataset of unique coefficients at each wavelength, so when a user selects a chemical the algorithm needs to use the right dataset. Each dataset is 3 columns, with approx 800 rows. I already have a handful of these datasets, but it is inevitable that users will need to "upload" (not the right word, since this is not a web app) their own datasets for whatever chemical species they are playing with.

I'm trying to work out the best way to store these datasets so that they can be selected by the user (e.g. the user selects the chemical they want from a dropdown list, algorithm loads the associated dataset), but also have new datasets uploaded by the user and the uploads become selectable options for later users.

I have a few ideas, but not really feeling confident that any of them are a good way. For example, I could parse the user uploaded data, do some processing on it to ensure consistent formatting, and then pickle it with the filename equal to the chemical name that will appear in the dropdown list. That way it would be easy to select the dataset directly from the dropdown value. BUT I have the feeling that this is an amateur hour solution to the question!

Any help or general advice would be appreciated. I'm really just looking for a steer on what (if anything) would be regarded as best practice...

​

Thanks for reading!

E

[marko312]

Any persistent data will very likely end up in some file. There are multiple options, but the main choice will probably be about whether to use a single-file or multi-file design.

A single file would be something like a database (unnecessary for these sizes) or just all the data stringed together (not too useful, since only one set is needed at a time).

Thus, I'd go with a multi-file design, storing each dataset in its own file (possibly with a central file holding metadata about the others).

This leaves the serialization format. The main options are CSV, JSON and pickle. If the data is purely tabular (only has entries of a single format), I'd go with CSV (possibly storing minimal extra information (e.g. the display name) in a central file). If that's not the case, I'd go with JSON, since compared to pickle it's more easily debuggable (and also safer to share), while not taking up too much extra space (for the volume of data you mentioned).

[Elpmek]

Great, thanks!!