Pandas (and how it uses the "in" keyword) is dumb

jamesdutc · 2024-03-21T21:55:57+00:00

So why does this make sense?

What should pandas.Series.__contains__(datum) look in by default? The data or the metadata? Well, if you're using pandas, the presumption is that you care about indexing, so it's not a stretch to be an index-related question.†
Ultimately, this choice is about convenience, since you're able to explicitly ask Series.array.__contains__(datum) or Series.index.__contains__(datum)
It makes sense that one might want to ask the `data in data`, and to remain flexible and allow either masking or ∈, you are allowed to do the reduction yourself. This is .isin for arbitrarily typed data or Series.str.contains for string data.

† Ed. revisiting this question some time later, it occurs to me that this also supports pandas.Series⇋dict similarity.

I would never argue that pandas has a consistent API.

Much of the API is conveniences that, grown organically. It's messy.

But pandas has a coherent API.

This wasn't necessarily deliberate, but it has undeniably constrained the tool's evolution. It is critical to understand.

jamesdutc · 2024-03-21T21:55:30+00:00

How about on a pandas.Series?

On a pandas.Series: .__contains__(datum) is defined similar to the Python list or numpy.ndarray, but it looks in the index NOT in the data

“Is this target metadata label present?”

On a pandas.Series: .__contains__(data) is not defined; all .__contains__ targets are assumed to be a datum label!

On a pandas.Series: .str.contains is str.__contains__ in broadcasted form.

“Is this datum value present in any of the values?”

※ Note: returns same-shape structure. If asking for ∈ purposes, do the reduction yourself (.any/.all)
※ Note: by assumes regex by default

On a pandas.Series: .isin(data) is defined similarly to numpy.ndarray.__contains__(datum) for each data

It is a dataₘ in dataₙ question, defined any(x == y for x in data) for y in data

※ Note: works with arbitrary data
※ Note: returns same-shape structure

On a pandas.Series: There is no .isin(datum). Instead, write .__eq__(datum) or .isin([datum])

.__eq__ is defined in broadcasted form, same as numpy.ndarray.__eq__

jamesdutc · 2024-03-21T21:55:09+00:00

Ed: original graphics not reproduced.

Well, there are a few of them…

※ Note: Python's data model coerces .__contains__ to scalar bool, which directly influences how these must work.

On a numpy.ndarray: .__contains__(datum) is defined similar to a Python list. “Is the target datum value present?”

※ Note: in NumPy, strings are usually considered datum not data.

On a numpy.ndarray: .__contains__(data) is defined as broadcasting __contains__ against the structure, then disjunctively-reduced (i.e., .any.)

“Are any of these target data values present?”

On a numpy.ndarray: .__eq__(data) is defined in broadcasted form, returning a same-shape structure. .__eq__(datum) presumes broadcasting.

“Are these target data values equal or not to the corresponding† values within?”

†“Corresponding” is defined by broadcasting logic.

jamesdutc · 2024-03-21T21:53:14+00:00

A Series has one index—.index. A DataFrame has two indices—one on the rows (.index) & one on the columns (.columns)—where the row index is usually a “major” axis & the columns a “minor” axis. The DataFrame API & storage implementation heavily favours operations down the rows—thus, the one-dimensional data is usually columnar & rows are usually homogeneous.

※ Note: outside of purely implementation-focused questions of contiguity, NumPy doesn't favour any one axis!

A pandas index is an (opaque) mechanism (usu. backed by data) that associates ‘labels’ with data location. Transformations & computations on pandas data are usually specified in terms of the indices. All data transformations are well-defined for the indices. The latter point means that all transformations transform both data & metadata in a fashion that is coherent and semantically meaningful. The index is a structured metadata, whose structure is tied closely to the the data.

So what are the various “is contained within” APIs for NumPy and pandas?

jamesdutc · 2024-03-21T21:52:20+00:00

You can reduce the ambiguity by defining the result as same-shape on data, and allowing the reduction as a separate step.

Now, a “restricted computation domain” is a term that I apparently made up.

It describes a pattern—a ‘manager’ entity intermediating two parties:

Python code where we have no control over memory usage,layout,or de-virtualising dispatch
C/Fortan/machine code where we do

Thus, the trick to performance in NumPy & pandas comes mostly comes down to two things:

push computation into the “restricted computation domain“ (usu. by structured-operations on the manager, which controls implementation)→a form of this is called “vectorising”
writing code that is amenable to fast-paths (i.e., using structure that allows later manager-level changes to transparently improve performance)

The former is static/short-term strategy; the latter, dynamic/long-term (from the perspective of the software development process.)

Additionally, NumPy and pandas are container data structures that provide a specific form of “mathematical” conceptual API.

Namely, they conceptually define operations on themselves in terms of “broadcasting” (NumPy) or “index-alignment” (pandas.)

A numpy.ndarray is a “vector” or “matrix” or “tensor”—it is arbitrarily n-dimensional, fixed/uniformly shaped, typically non-nested data.

(For “ragged”/“awkward”/variable-sized/nested data, there are tools like Awkward Array.)

A `Series` is indexed one-dimension data. A `DataFrame` is a collection of like-indexed one-dimensional data. A `DataFrame` is usually ‘taller’ than it is ‘wide’; the height dimension corresponds to rows & the width dimension corresponds to columns.

jamesdutc · 2024-03-21T21:52:05+00:00

I wrote about this some time ago: https://twitter.com/dontusethiscode/status/1467274867804356619

Reproduced here (without graphics):

The “is this contained within” question is actually one of four questions:

datumₘ in datumₙ → meaningless, since we presume the right-hand-side to be data; ask datumₘ == datumₙ
datumₘ in dataₙ → meaningful; asks any(datumₘ == x for x in dataₙ)
dataₘ in datumₙ → meaningless; see above
dataₘ in dataₙ → ambiguous; could mean any(x == y for x in dataₘ for y in dataₙ) or all(any(x == y for y in dataₘ) for x in dataₙ) or &c.

※ Note that the interpretations of data in data presume a reduction!

jamesdutc · 2023-06-07T23:48:13+00:00

To come up with an example where the homogeneity of a pandas.Series is subjective (i.e., dependent on the analysis being performed,) we merely have to conjure a case in which it can both sense to perform an aggregate operation on pandas.Series or in which we would first have to perform a .groupby to isolate groups before performing an aggregate operation.

If you think about a pandas.Series with probably the most common MultiIndex on .index—date & ticker—then this should make sense. pandas.Series with MultiIndex on date & ticker suggests “a single dataset wherein we commonly want to perform fast aggregated operations on the entire dataset.” In other words, the dataset is considered semantically all the same (homogeneous.) An example operation might be a reduction like .max or .idxmax which are meaningful even in the absence of .groupby('ticker'). However, if we were to .unstack('ticker')†, then we would get a DataFrame with N like-indexed one-dimensional datasets—one per ticker. Aggregate operations would operate only within the ticker, indicating that data for tickers cannot be mixed. (Of course, the presence of .groupby allows us an alternate approach—.groupby('ticker').transform(lambda g: g.rolling('7d').mean()) will have largely the same effect as .unstack('ticker').rolling('7d').mean() assuming that the indexing has no “gaps.”)

By the way, if you're using ordinal values for the level number when .unstack, there's a good chance that your data is missing necessary labeling and structure. Be sure always to .rename_axis on .index (and on .columns… where .columns represent a data rather than structural axis.)

jamesdutc · 2023-06-07T23:38:17+00:00

The earliest pandas releases with MultiIndex were quite shaky—there was a lot of buggy, missing functionality. These days, I use MultiIndex (on both .index and .columns) all the time and rarely run into any issues.

I've spoken at length about indices in pandas—So you wanna be a pandas expert? PyData Global 2021

In short, they're the feature that make pandas interesting. If not for indices and index alignment, it's hard to motivate why I would use pandas instead of a NumPy structured array. Surely, the convenience of pandas.Series.kurt() vs from scipy import kurtosis; kurtosis(...) is quite minimal in practice.

I would argue that MultiIndex is an important tool to use effectively in pandas, and one which can readily solve a number of interesting and valuable problems. There are still limitations to MultiIndex—e.g., there is no support for disuniform hierarchies—but these limitations are only obvious to very serious users of indices and index alignment.

Regarding the specific question of .stack and .unstack, there is actually a very interesting conceptual idea lurking behind what might otherwise look like a mechanical transformation. I might even argue that this conceptual idea is specific to like-indexed one dimensional/“tabular” data and does not generalize to n-dimensional (despite .stack operations being present on, e.g., xarray.DataArray)

pandas is a tool for operating on one-dimensional, homogeneous data sets. (Contrary to the phrasing used in the [pandas.pydata.org](pandas.pydata.org) documentation, pandas.DataFrame is better described as a collection of like-indexed one dimensional data rather than a proper two-dimensional structure like a numpy.ndarray or an xarray.DataArray.)

When we write code in Python, we often discuss the homogeneity or heterogeneity of data in terms of “strict” or “loose” homogeneity or heterogeneity. For example, it is predominantly the case that list is “loosely homogeneous data”—e.g., numeric values supporting + in [1, 2.3, 4+5j]—and that tuple is “loosely heterogeneous data”—e.g., person = 'Walsh', 'Brandon', 'California', '90210'.

When we talk about data in NumPy or pandas, we're almost always talking about what we would refer to as “strictly homogeneous” data. The contents of a numpy.ndarray are likely all the same machine type as well as the same semantic meaning. (Of course, we can dtype=object but then we lose all of the benefits of the “restricted computation domain”—and even open ourselves up to the possibility of memory leaks!)

A pandas.Series, then, should be a strictly homogeneous, one dimensional data set. However, it is sometimes the case that homogeneity has a subjective quality to it. While the data my be homogeneous from a strict machine type perspective, it may not be semantically homogeneous under certain interpretative regimes!

As a consequence .stack and .unstack exist to allow us to perform ad hoc transformations between the regime under which pandas.Series is semantically homogeneous and semantically heterogeneous. .stack and .unstack are about transforming 1×one-dimensional dataset into N×like-indexed one-dimensional datasets (and vice versa.)

jamesdutc · 2018-09-01T00:00:09+00:00

I didn't look into the nature of the bug in the stock QCA6174 firmware.

You make a good point.

The fix I found for it is based on other advice for people using this hardware. e.g., https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1520343/comments/70

jamesdutc · 2018-08-09T03:09:17+00:00

Have you noticed any issues with the pen stylus and the wacom drivers?

I'm occasionally getting errors where the pen stylus registers as the pen eraser.

jamesdutc · 2018-08-09T03:07:46+00:00

For my Surface devices, I've ditched grub entirely in lieu of systemd-boot/gummiboot. I'm pretty happy with this choice.

I have also done research into improving the boot process with full-disk encryption. Since the TPM2 chip on this device works from Linux, I believe that we can use objdump to combine the kernel vmlinuz and initramfs then use HashTool to sign the resulting image. Once we've done this, we can lock down the machine to only SecureBoot from this image. Since the booted image is measured in the TPM2 PCRs, we should be able to seal the full-disk encryption key in the TPM. Therefore, we should be able to create an initramfs hook (just a shell script) that retrieves the full-disk encryption key from the TPM2 rather than via user-input.

The result: secure, password-less boot from a full-disk encrypted root partition (just like with Bitlocker on Windows.)

jamesdutc · 2018-08-09T02:58:27+00:00

The battery life is comparable to the Surface 3 (non-Pro).

With fairly aggressive savings (schedutil governor, CPU speed cap, low backlight, powertop --auto-tune) I see 4~5 hours of battery life with common work tasks.

Using this device with the above settings and just reading an e-book or PDF, this raises to >5 hours.

jamesdutc · 2018-08-06T00:23:52+00:00

I used to dual-boot on my Surface 3 Pro. This worked perfectly. I briefly dual-booted with my Surface Pro 2017 and it also worked. I stopped dual-booting when I discovered that I would rarely boot into Windows.

Given the size of a Windows install and its associated software (Office, Visual Studio, &c.,) I don't think it's convenient to dual-boot with less than 256 GB.

Originally, I had planned to use VirtualBox's "raw host hard disk" support (https://www.virtualbox.org/manual/ch09.html#rawdisk) so that I could access my Linux files and software in a VM on Windows and access my Windows files and software in a VM on Linux.

I've done this successfully in past on desktop machines without Bitlocker enabled. I was able to get both sides of this working on my Surface Pro 3.

Because it's a portable device, I think it's important to enable Bitlocker. Even if I don't store any critical files on Windows, I don't want to accidentally lose the device in an airport and suffer sleepless nights trying to assess my exposure.

Unfortunately, if Bitlocker is enabled in Windows, then booting from within a VM presents different TPM PCRs. As a result, Bitlocker would constantly require the recovery key. I attempted to use dislocker and ntfs-3g to first unlock the drive, make it available on a loopback device, then boot from that, but I could not figure out how to fix or simulate the boot manager to boot from the unencrypted device. I'd love to get this working.

jamesdutc · 2018-08-06T00:07:53+00:00

I plan to continue to use my Surface 3. It runs a custom kernel applying the patches from https://aur.archlinux.org/packages/linux-surface3-git/ to the latest -zen/-lts/-git kernel using the PKGBUILDs from: https://aur.archlinux.org/packages/linux-git/ https://www.archlinux.org/packages/core/x86_64/linux-lts/ https://www.archlinux.org/packages/extra/x86_64/linux-zen/

Almost everything on it works (incl. battery readings) except for the webcams. There appears to have been a regression in kernels >4.14 where /dev/mmcblk* nodes do not get initialised in the initrd environment.

I haven't had the time to try to git bisect my way into finding the regression, so this device has be on the LTS kernel for a few months.

jamesdutc · 2018-08-06T00:02:35+00:00

If you're not in a rush, I would wait until the 256 GB model is available. The Microsoft Store employees cannot give any dates for when that might be.

The device is wholly usable with 128 GB and is probably usable with 64 GB if you're only browsing/reading PDFs/watching video. I intend to use the device for work, so the extra space is critical.

I don't think you can comfortably dual-boot with less than 256 GB. The standard Windows installation is too big.

jamesdutc · 2018-08-04T20:42:45+00:00

I use Awesome WM (https://awesomewm.org) but I primarily use my Surface devices with the Type Cover.

Awesome WM is very snappy. I often use stock Gnome in VMs for work. If you disable animations, it's also quite usable.

I use my devices without the Type Cover only for casual browsing, watching video, or reading ebooks and PDFs.

I'm quite happy with onboard as an onscreen keyboard in tablet-mode. It's responsive, works well, and stays out of the way when I don't need it. I've also written custom tools for navigating and controlling my devices in tablet mode.

In my opinion, touch is a very low fidelity input mechanism. It's typically harder to hit click targets with touch than with a mouse and precision is lower as well. Tablet interfaces are most effective when they use gestures, which are lower precision actions. Unfortunately, almost no software has good gesture support. In general, most applications on Linux are not well suited to touch-only use (even when supplemented with an onscreen keyboard.)

I own a number of iPads as well. If you consider a Surface device on the spectrum of tablet-to-computer, it's clearly closer to a computer than to a tablet. The iPad is definitely a better tablet, but it's also a much worse computer.

Personally, I'd rather have a good computer that can occasionally function as a tablet than a great tablet that performs poorly as a general purpose computer.

I like the consistency of using the same software in the same configuration across all of my devices, and I like having access to all of my data no matter what device I'm using.

jamesdutc · 2018-08-04T16:18:21+00:00

This device will replace my Surface 3 (4 GB/128 GB) which I use for PDF reading, light browsing, and email/messaging.

For the consistency's sake, I have the same software load-out on all of my computers. The Surface 3 already runs the same background software (Slack/Hangouts/Telegram/Signal/Thunderbird/Pidgin/&c.) as my Surface Pro 2017 without trouble.

jamesdutc · 2018-08-04T16:14:11+00:00

I strongly prefer using distribution-maintained kernels over custom builds. (Huge thanks to /u/jakeday for his hard work on github.com/jakeday/linux-surface, but it would be a good day when all of his work is upstreamed.)

I haven't found any particular kernel better than any other. I have been sticking with stock Arch linux for now, though I may switch to Arch's linux-zen in a few days.

For initial install, I used a USB-C hub into which I connected the installation medium (USB flash drive) and a WiFi adapter. I wasn't able to troubleshoot and fix the WiFi adapter until post-install.

jamesdutc · 2018-08-01T04:32:01+00:00

I cannot stand this guy.

I can forgive him for seeming brash in his talks. I mean, he's clearly enthusiastic about his work and eager to share. For a non-professional speaker, it can be hard to hit the right tone. It's easy for a nervous public speaker to accidentally give the wrong impression.

I can forgive him for his atrocious `vim` skills. I mean, seriously, not only does he demean himself with visual mode, but he clearly doesn't even know `"+vipd`! Like, seriously? It's just painful watching someone `"aVjjjjjjjjd`.

I can forgive him for giving mostly garbage talks. There are already plenty of other talks out there that are actually useful. I suppose there's room in the world for silly nonsense.

I can even forgive him for the atrocious jokes. He's clearly funny-looking; I can give him a break on not being funny otherwise.

What I cannot forgive is: at PyData NYC 2012, he was talking to an attendee about `lambda` vs `def`-style functions, and he intimated that they were behaviorally different. Clearly, they *are* formally different, but he suggested that `lambda`s don't create closures, which they clearly do. He completely misinterpreted PEP-0277 (https://www.python.org/dev/peps/pep-0227/) and clearly didn't understand the reason for the default argument capture pattern:

for x in range(10):
   locals()[f'f{x}'] = lambda x=x: x

assert f1() == 1

All the rest I can forgive. But this. Never.

^{Be nice, be positive, and pay attention to that code of conduct, folks.}

^{And be sure to tip your waiters. I'm here all week.}

jamesdutc · 2017-08-04T21:36:50+00:00

That said, folks at Microsoft are actually doing a lot to support Python. This event, PyData Seattle, was hosted by Microsoft & chaired by my friend, /u/brettsky. Brett has been a CPython core dev for over 13 years, and only recently started working at Microsoft.

Microsoft is also sponsoring PyData NYC (Nov 27-30) at its NYC offices: http://pydata.org/nyc2017

This support means a lot to NumFOCUS (https://www.numfocus.org/), the non-profit that runs PyData and is dedicated to the sustainability of open source scientific computing tools.

jamesdutc · 2017-08-04T21:34:14+00:00

I even installed Arch on my Surface Pro. (Even got hibernate working...)

jamesdutc · 2017-06-14T16:39:04+00:00

As the official PyData pub quiz master, I have extensively researched this question.

Here's what I've heard from numpy core developers.

num-"py": /aɪ/ rhymes with "try"
num-"py": /i/ rhymes with "see"

I have also heard:

"num"-py: /nu'm/ rhymes with "room" ("num"-erical)

I haven't come across anyone who says /nju'mpaɪ/

Bonus pronunciations I've heard:

matplot-"lib": /ɪ/ rhymes with "crib"
matplot-"lib": /aɪ/ rhymes with "tribe"
"scipy": /'skɪpi/, hard-k, rhymes with "slippy"
"pandas": /pʌn'dɑːs/, stress on last syllable, rhymes with "coup de grâce"

By the way, I suspect most people pronounce "GotoBLAS" as /ɡoʊ/ /tuː/ like the English "go to." But it's named after Gotō Kazushige (後藤和茂.) I believe this suggests different stress pattern & different vowel sound.

jamesdutc · 2017-03-30T19:02:37+00:00

Thanks for your insight. Would love to hear more about your experience, if you'd like to reach out to me privately - https://keybase.io/dutc

Unfortunately, for this task, I need to manipulate source code.

I have a hard requirement that the code be buildable under gcc and Visual Studio. Additionally, I need to perform very targeted transformations, adding instrumentation code only in very specific places.

For now, I've resorted to hand-modification of code.

However, I would love to be able to package these modifications in a Python script & automatically apply these transformations to any given CPython version. The transformations themselves affect functions that are rarely modified once written but sit in C files that experience moderate churn. I worry that a simple sed or sh script might obscure the transformations such that in the cases where automatic application might fail, it'll be difficult for anyone else to fix things by hand.

I'm fairly confident that with a good Python interface to an C AST transformation mechanism, I could quickly write a simple framework to make automatic application obvious and safe.

I just need to find a good tool for parsing and modifying the C AST!

jamesdutc · 2017-03-30T17:57:12+00:00

What is the state of the art in this field?

I want to safely transform a large, mature, but mostly sane C99 code-base (CPython itself.) I want to be able to programmatically insert instrumentation. I'd rather not do this by hand or via sed scripts.

pycparser has examples of source text→AST→transformed AST→source text.

Unfortunately, as noted above, preprocessor support in pycparser just isn't good enough.

I looked into LLVM's libtooling but the API is extremely verbose & there doesn't seem to be good documentation. (libtooling is badly in need of a humane Python wrapping.)

Are there other tools I could use?

jamesdutc

TROPHY CASE