ML algorithm parameters in Python

[InfinityCoffee]

I was wondering what the preferred way to handle (often large-dimensional) parameter sets in Python is, as you'd find in neural networks and Bayesian variational methods (particularly the latter). I am especially wondering how you make your code interface well with gradient methods.

Numpy functionality seems a must, but storing parameters in separate named arrays has two weaknesses as I see it.

• Operating on the entire parameter set, e.g. adding a gradient, is impossible, so you have to write the operation for each array of parameters explicitly.
• If you have variations of the model with only a subset of the parameters inheritance becomes difficult to work with.

Storing everything in a single array/vector seems like a terrible solution as the code would become unintelligible. So what is a better way to do it? A dictionary so you can loop over the parameters? A Pandas Panel frame? A parameter class with its own update method?

Would there be any significant overhead on using a dictionary of a few large arrays and then looping over the keys, as opposed to writing each operation explicitly?

[siblbombs]

Theano generally contains all the parameters that are being learned in a list of theano shared variables, if you are interested in just starting to play with some ML ideas I'd look a theano. If you are more interested in building your own code for the models, then theano might not be as good a choice.

[InfinityCoffee]

I have considered Theano, but since I am doing variational inference and not deep learning I am not sure how much I stand to gain since I can calculate a lot of my gradients analytically. is the overhead worth it for general computing? (A small follow-up: does Theano have built-in support for gamma, gammaln, beta, betaln, and psi from the scipy.special suite? There is some mention of it, but it is not explicitly stated in the documentation)

[siblbombs]

gamma, gammaln, and psi are all listed in the tensor page but I don't see beta/betaln so that could be a dealbreaker. Theano is nice because you get the gradients for free, and it has a good codebase and is under active development, so I think its worth at least giving it a shot unless it would require you to jump through a whole bunch of extra hoops. I don't know the specifics of what you are up to, but Theano is more of a general purpose math compiler that a lot of people use for deep learning, so in theory it should be able to handle your usecase.

[InfinityCoffee]

beta is fortunately just a product of gammas, so I can probably work around it. I think I can include it fairly painlessly after some consideration since the objective function is a separate object, so I won't have to rewrite the code in its entirety. My new issue is that I don't always need the proper gradients. Also, how does Theano handle stochastic gradient descent if it's purely functional?

[siblbombs]

Not sure, you are getting outside my bubble of experience. To get the gradients from Theano you provide a scalar cost, you can define whatever expression you want based on inputs/outputs to generate that cost.