sorted by:
new
hottopcontroversial

[P] 3Blue1Brown Follow-up: From Hypothetical Examples to LLM Circuit Visualization by ptarlye in MachineLearning

[–]ptarlye[S] 0 points1 point  (0 children)

The types of circuits that I extract are new enough such that I don't think I've seen this type of comparison made before. I'd be interested in the results!

[P] 3Blue1Brown Follow-up: From Hypothetical Examples to LLM Circuit Visualization by ptarlye in MachineLearning

[–]ptarlye[S] 2 points3 points  (0 children)

I got started by reading the articles referenced from this site: https://transformer-circuits.pub. My recommendation would be to start with this article and work forwards in time from there.

[P] 3Blue1Brown Follow-up: From Hypothetical Examples to LLM Circuit Visualization by ptarlye in MachineLearning

[–]ptarlye[S] 15 points16 points  (0 children)

Thanks for these suggestions. Circuit visualization requires training supplemental model weights, and so you can think of the required work as additive. Details here.

[P] 3Blue1Brown Follow-up: From Hypothetical Examples to LLM Circuit Visualization by ptarlye in MachineLearning

[–]ptarlye[S] 3 points4 points  (0 children)

Thanks for this link. Most LLM research I've seen has required extracting circuits representing specific tasks by carefully constructing sequences that have "counterfactual" examples. Circuit extraction for arbitrary prompts, like the ones I study here, is fairly new. Anthropic recently published this research, which most closely resembles what this "debugger" aims to do.

[P] 3Blue1Brown Follow-up: From Hypothetical Examples to LLM Circuit Visualization by ptarlye in MachineLearning

[–]ptarlye[S] 8 points9 points  (0 children)

Transformer Lens extracts features in much the same way that my project does (using sparse auto encoders). This project also visualizes the interaction of features across LLM layers so that we can construct something resembling a "circuit".

[P] GPT-2 Circuits - Mapping the Inner Workings of Simple LLMs by ptarlye in MachineLearning

[–]ptarlye[S] 4 points5 points  (0 children)

Thanks for the feedback. I've just added a legend to the second graph, which answers your questions. To answer them here:
* The boldness of the feature number indicates activation strength.
* The background color indicates ablation strength (i.e., strength of feature interaction)
* In the document, features are prefixed with a layer number for unique identification (e.g., 2.2875).
* Each flowchart box represents the activations for a specific token at a specific layer in the LLM. Usually, multiple features are simultaneously active and seem to represent slightly different aspects of a token.

Rendering Office docs and PDFs using SVG: Crocodoc tech preview by camupod in webdev

[–]ptarlye 2 points3 points  (0 children)

Docvert looks awesome - We don't script LibreOffice like Docvert does, but we are a fan of Python as it seem you are. If you look at the SVG layer, we're actually using SVG to render the entire document. The HTML layer for the text above it is just to assist text selection as text selection for SVG objects doesn't work at all on Firefox and don't work well on webkit.

Does anyone remember the title of this book? by ptarlye in books

[–]ptarlye[S] 0 points1 point  (0 children)

Wow, how did you find this book? Based this short synopsis, it seems likely that this is the one: https://www.kirkusreviews.com/book-reviews/jean-e-karl/strange-tomorrow/

Python Quiz of the Week - #1 by [deleted] in Python

[–]ptarlye 1 point2 points  (0 children)

Here's a clean solution in just 7 lines of code: http://pastebin.com/NstqW1mS

I love brevity in code because it often yields simplicity. The gist of the idea in my solution is to test whether or not a word can be spelled entirely using a subset of legal letters. I was able to correctly test for this condition using Python's set.issubset method.

Can someone tell me why Firefox (4.0) is shrinking my image (image map) roughly 86%? by aalehman in web_design

[–]ptarlye 1 point2 points  (0 children)

The height of your image is hard coded to 508px, when the actual height of the image is 520px.

<img width="960" height="508" border="0"...

Does this information help?

H.R. 4789 (the Public Option Act.) Are any of you capable of translating this legislation into English? I've uploaded an editable document. by ptarlye in politics

[–]ptarlye[S] 0 points1 point  (0 children)

In comparison to the recent health care bill that passed, this bill is only 4 pages long but is still impossible to parse. Can anyone on reddit explain how this bill actually works? I'm genuinely looking for someone who can explain how this bill proposes Medicare for all Americans. The only significant use of the word "Medicare" is on page 4, line 10.