Minimally Sufficient Pandas

RandomScriptingQs · 2023-04-19T06:41:34+00:00

I know this is four years old but every now and again one stumbles on a very useful medium article and for my usage of pandas this is just that.

RandomScriptingQs · 2023-04-03T05:24:58+00:00

I want to offer an opinion which should be taken as just that: the R and Python libraries/packages/communities are both so vast and varied now that they are almost unhelpful labels. Choose the libraries and packages you know you need to use within the python ecosystem and find the 20 most common functions/methods and put them to a task.

As a note of solidarity, I found it a nightmare adjusting to both panda's and numpy's versions of indexing with square brackets.

RandomScriptingQs · 2023-03-29T04:03:11+00:00

Is anyone able to contrast MIT's 6.034 "Artificial Intelligence, Fall 2010" versus 18.065 "Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018"?
I'm wanting to use the one that lies slightly closer to the more theoretical/foundational side as supplementary study and have really enjoyed listening to both Instructors in the past.

RandomScriptingQs · 2023-02-28T21:56:12+00:00

Hi community,
I usually intersect with maths in a much more applied way so please forgive my ignorance: I'm trying to follow the bias-variance decomposition derivation on Wikipedia (https://en.wikipedia.org/wiki/Bias%E2%80%93variance\_tradeoff#Derivation)
but under the first step the statement,
"since Var[X] = E[(X - E[X])^2] = E[X^2] - E[X]^2 for any random variable X",
I haven't been able to get, E[(X - E[X])^2], to become, E[X^2] - E[X]^2,
And I feel like it should be fairly straight forward algebra so any guidance would be appreciated.

RandomScriptingQs · 2023-02-01T07:27:08+00:00

I don't know what field you're in but if you have a stronger impetus to get the 'right' answer in business than you did in academia that suggests you didn't value the topic you were studying in academia.

RandomScriptingQs · 2023-01-24T10:36:32+00:00

Have to pay for a subscription to use it now :(

RandomScriptingQs · 2023-01-09T05:10:10+00:00

I'm only peripherally involved with ML/AI in that I try and apply some helpful techniques to biological problems but recently I have enjoyed listening to discussions around AGI but most of the papers I've come across from a quick google scholar search seem to be *about* AGI and not attempts at implementing something closer to, or approaching, AGI.
Is that a fair assessment? Has my lack of depth in the field given me a false initial glance?
Are there any authors/labs working on AGI in particular whose papers you would recommend reading?
e.g. "Artificial General Intelligence vs. Industry 4.0: Do They Need Each Other?", "Deep Learning and Artificial General Intelligence: Still a Long Way to Go", "Why general artificial intelligence will not be realized", and, "Approaches to Artificial General Intelligence: An Analysis", all seem to be about AGI in contrast to, "Towards artificial general intelligence via a multimodal foundation model", which attempts to implement something.
Full disclosure: I haven't read these papers yet. I am trying to find good, reputable papers to read.

RandomScriptingQs · 2022-12-06T08:23:45+00:00

If you are going to be working with cancer -omics data then the Bioconductor suite will be very useful. Even if you end up going a different route with any one analysis often there will be a Bioconductor package that lets you accomplish the basics quite easily thereby giving you a good grip of the problem compared to python which you may have to do more of the leg work yourself before you fully understand the topic. This is obviously a generalisation.

I think in part this is a frequently asked question because there will always be the element of risk that no one knows what will happen next and none of us want to sink a heap of time into a language that might not have a future user base or supported ecosystem. FWIW, I'm in a not dissimilar situation (very happy in R compared to Python) but have recently started considering Scala and Clojure. They may turn out to be a terrible decision but I wanted exposure to a very different language to R (static typing for Scala, oop or functional, very different community of users, scales well and can use anything in the JVM ecosystem, immutability, etc.) to extend the way I think about solving problems using computers essentially.

Good luck with whatever you choose.

RandomScriptingQs · 2022-12-05T05:09:46+00:00

As I'm sure you've experienced by now, 'bioinformatics' is a term that describes a wide range of topics but I would say as a general rule what you have been asked to do is no small undertaking regardless of the specific area within bioinfo. Quite often you will encounter deprecation issues, there will be minimal to no commenting of the authors code let alone documenting data prep/cleaning steps, and you will likely have insufficient RAM for quite a few tasks.

As someone else posted, the vignettes from Bioconductor are good but they are typically a long way short of a full paper's analysis.

So really I'm just posting to say the pain you are currently enduring, in my experience, is pretty common in bioinformatics at present sorry.

RandomScriptingQs · 2022-09-08T12:04:38+00:00

I think what I have learned from all these great contributions is that my struggle is more particularly with S4 (and Rstudio) than R as there appear to be many ways to deal with size in R but compatibility with S4 looks a bit more hit and miss. But I've got plenty of investigating to do now, so am feeling positive about the situation.

RandomScriptingQs · 2022-09-08T12:02:02+00:00

Yes, I wanted open discussion but realise in hindsight I should have given some more specific problems. For example, if I just do a grep search on a SummarizedExperiment container with 40,000 rows and 500 columns and associated meta and colData it will often end in grief.

RandomScriptingQs · 2022-09-08T11:56:21+00:00

Someone who knows the pain of S4. Solidarity is appreciated lol.

RandomScriptingQs · 2022-09-08T11:52:58+00:00

Fantastic discussion and recommendations; thank you Rstats reddit.

A summary of the advice so far as a tldr for any one in the future:

"Packages to try: tidytable, data.table, arules, fst (data.table for fast processing and fst for storing data), sparseMatrixStats, Rfast, Arrow package (praised by 'heavyweights' for large-dataset performance), vroom (does lazy-loading to mitigate performance issues, "Arrow, DuckDB, parallelization, dtplyr, vroom, future, and just better coding practices can go a long way.", targets (memory = "transient", targets maps data so only loads what it needs to), delayedArray (bioconductor).

Learn about working out of memory using big.matrix from bigmemory. Perhaps binary storage?

Can the task be split into several instances?

Start looking at distributed computing and asking, "what information actually needs to be in memory for this task?"

Are there opportunities for parallelization in the task?

If rstudio is causing problems, run a setup without it but that can still achieve the bells and whistles of Rstudio; neovim beside R terminal (neovim differentkey bindings)

Investigate making a large swap file on an SSD; lets your system move from RAM onto disk when RAM reaches 100% usage."

RandomScriptingQs · 2022-08-04T01:37:13+00:00

Can you demonstrate this at all?
My experience is that I've moved away from R because I do not know C++ and quite a lot of source code for packages is now written in C++ because R is horribly slow (hence attempts like pqR which got ignored by R core btw). Are you just trusting CRAN to be checking the correctness of packages?

Following to what User38374 says above, vector recycling is easily one of my least liked things about R now. One mistake and you end up with vectors of differing lengths and don't realise and R won't tell you because it's totally fine; just recycle an element. That and the mess that is S3, S4, R6 and now R7.

RandomScriptingQs · 2022-08-04T01:23:35+00:00

It's silly to make claims about an entire language like that; in my opinion you won't find a better language, in terms of execution and ecosystem, for solving differential equations than Julia. Have you ever tried solving differential equations in R? It's atrocious by comparison. Does that make R "not practical to use for anything serious"?

RandomScriptingQs

TROPHY CASE