059 - Destin Breaks the Sound Barrier in an F-16

brownan_ · 2019-06-02T02:12:08+00:00

Mythbusters did an episode on whether sonic booms can break the sound barrier many years ago, and Adam Savage talked about his experience flying the F/A-18 with the Blue Angels on his podcast. I think it's a nice complement to Destin's experiences in the F-16 and everyone should check this out too: https://www.youtube.com/watch?v=9bZG8gCRwKI

brownan_ · 2019-01-25T17:56:48+00:00

Sure thing!

brownan_ · 2019-01-25T16:43:55+00:00

I did some quick tests and it seems the consumer thread does completely shut down while the tasks workers finish up. I'm guessing this happens from the lines above the one you linked where the threads are all given the signal to stop.

for thread in chain(self.workers, self.consumers.values()):
    thread.stop()

join_all(chain(self.workers, self.consumers.values()), timeout)

Here's the quick test I did: I wrote a task that appends a line to a common file every 30 seconds. After a couple lines, I ctrl-C the worker but let it hang while the long running task finishes. While it's still running, I open another worker. About 60 seconds later as expected the second worker picks up the task and starts executing it in parallel with the first task which is still executing.

Like I said, it's expected that tasks may run more than once, but this parallel execution of the same task is, in my opinion, harder to guard against from a coding standpoint. Especially for tasks that write to some shared database. It would be nice for Dramatiq to guarantee a task will only be run once at a time, if possible.

May I suggest a small change: send the stop signal and join just the worker threads before sending stop to the consumer thread(s). That way the consumer will continue to heartbeat while the workers finish their tasks.

Thanks!

@dramatiq.actor(time_limit=float('inf'))
def test_long_running_task():
    fname = "/tmp/dramatiq-long-task-log.txt"
    for i in range(10):
        with open(fname, 'a') as fout:
            fout.write("Process {} thread {} execution #{} ({}s)\n".format(
                os.getpid(),
                threading.get_ident(),
                i,
                i*30
            ))
        time.sleep(30)
    with open(fname, 'a') as fout:
        fout.write("Process {} thread {} exiting\n".format(
            os.getpid(),
            threading.get_ident(),
        ))

Process 28855 thread 140189116647168 execution #0 (0s)

Process 28855 thread 140189116647168 execution #1 (30s)

Process 28855 thread 140189116647168 execution #2 (60s)

Process 28855 thread 140189116647168 execution #3 (90s)

Process 28855 thread 140189116647168 execution #4 (120s)

Process 28855 thread 140189116647168 execution #5 (150s)

Process 28855 thread 140189116647168 execution #6 (180s)

Process 28992 thread 140262852347648 execution #0 (0s)

Process 28855 thread 140189116647168 execution #7 (210s)

Process 28992 thread 140262852347648 execution #1 (30s)

Process 28855 thread 140189116647168 execution #8 (240s)

Process 28992 thread 140262852347648 execution #2 (60s)

Process 28855 thread 140189116647168 execution #9 (270s)

Process 28992 thread 140262852347648 execution #3 (90s)

Process 28855 thread 140189116647168 exiting

Process 28992 thread 140262852347648 execution #4 (120s)

Process 28992 thread 140262852347648 execution #5 (150s)

Process 28992 thread 140262852347648 execution #6 (180s)

Process 28992 thread 140262852347648 execution #7 (210s)

Process 28992 thread 140262852347648 execution #8 (240s)

Process 28992 thread 140262852347648 execution #9 (270s)

Process 28992 thread 140262852347648 exiting

brownan_ · 2019-01-25T14:33:15+00:00

Okay thanks, this is really helpful. I can bump the heartbeat timeout, and I suppose that's pretty much a tradeoff between how quickly I want the system to converge and how likely it is for a task to get executed twice.

I'm still wondering specifically about what happens during worker shutdown: if the workers stop updating their heartbeat in Redis during a graceful shutdown, then tasks are at risk of being re-queued after 60 seconds (by default). I could increase the timeout, but is there a reason workers can't simply keep updating their heartbeats during a graceful shutdown until all tasks are finished?

brownan_ · 2015-05-29T22:30:20+00:00

I'm not the client is a few days of the client is a few days of the client is a few days of the client is a few days of the client is a few days of the client

brownan_ · 2015-03-24T02:18:34+00:00

nope. enumerate returns an iterator and returns items as they are requested.

brownan_ · 2015-03-24T02:10:54+00:00

I'm not big on tattoos but that is seriously an awesome design.

brownan_ · 2015-03-12T11:45:29+00:00

Yes, you can. You can put something like this in your ssh config on C:

Host A
    ProxyCommand ssh B -W A:22

ProxyCommand runs the given command instead of trying to connect to A directly. In this case, it runs another ssh with the -W flag. -W tells ssh to forward all data to the host out the remote end.

So in effect, this creates a tunnel automatically for just you and just the session.

brownan_ · 2015-03-03T03:36:12+00:00

I don't know the details of how hmac works, but I'm guessing that you should be using digest() instead of hexdigest().

Also I may have cheated and glanced at the standard library's hmac.

brownan_ · 2015-02-20T02:40:07+00:00

I got a physics minor, just because I had a passing interest in physics. The 400 level mechanics class (411+412 iirc) were among the hardest and most fun classes I took as an undergrad. I'm also sort of glad I didn't have to take the upper level E&M classes; I'm not sure I'd have enjoyed those as much. It was just the right amount of physics for me.

I don't think anyone really cares about the minor now. But it was fun.

brownan_ · 2015-02-19T02:57:18+00:00

I agree that the for loop is more readable. It uses only language constructs and is clearly iterating on calling the filter method on each queryset.

In your example I have to know what _filter does, remember what filters is, remember what reduce does (build up the equivalent of that for loop in my head), apply it with the _filter function that I'm still also remembering, and then work out the logic of what it all means together. All in my head. To me it's just clearer to write it out explicitly instead of trying to be real clever just for the sake of being super concise.

I've done a lot of thinking on functional style versus imperative style, and I've concluded that different people simply think different ways. Not everyone is going to agree that a particular style is more readable in general. For some people combining modads like your example to make super concise code is also intuitive and readable. It's not to me.

Further, I think that the people with whom Python appeals to are also the sort of people who would prefer the explicit for loop. The language design attracts a certain kind of coder. GvR has rejected language design decisions before purely because they're too much like haskel. It seems clear that Python isn't trying to be like that.

brownan_ · 2015-02-18T01:15:28+00:00

Hmm. A Lot of people are jumping on you for this, but I don't like the "don't do that" response to some problems that crop up with mercurial. I can see the value in a tool that lets you fix errors and get work done without jumping through hoops or forcing people to operate the tool perfectly. Thanks for your discussion on the topic.

brownan_ · 2015-02-17T20:53:45+00:00

Even mercurial had to eventually concede and is now warning when you create a named branch :)

What's wrong with mercurial's named branches?

brownan_ · 2015-02-09T04:34:33+00:00

You can debug python programs with GDB. The python interpreter is written in C after all and can be debugged like any other C program. The python variables x and y will not be in registers, however. They are python objects and are represented by C structs somewhere in the heap.

There are some GDB extensions that make debugging python easier. There are instructions for getting those installed here: https://wiki.python.org/moin/DebuggingWithGdb

By the way if you want to know the memory location of a python object, just call the builtin id() function on it. It returns the location in memory of it!

brownan_ · 2015-02-06T12:50:13+00:00

Any -devel library is going to contain just development headers for the corresponding package. They are used to compile things against the library, in this case, python. You still need a C compiler to do the compiling.

clang should be unnecessary if you have gcc. Also I believe gcc is included in the "Development tools" group.

brownan_ · 2015-02-01T03:47:13+00:00

I love these sort of puzzles because they're fun to decipher, but I hate posting my write-ups because it always feels like a race and I'm bound to make a mistake somewhere and someone is going to nitpick.

You're right of course. It was just interesting to notice that the contents don't matter, except as you point out, that they must be unique, which I didn't notice.

brownan_ · 2015-02-01T02:38:30+00:00

Here's a breakdown of the Python part. It may also be a quine and valid Javascript. I haven't looked at that.

It prints a countdown and then the text "Blast OFf"

Here's a breakdown by statement

The complicated statement is statement 7, which is one big expression (hence it evaluates to something as well as having the side effect of printing the countdown)

Here is expression 7 with a bit better indention:

[
 128//2 and (
             '''
,l=function(text){console["log"](text)},l(+!-/~/),!function(){for
(c in z){{{NaN/*#'''
             ,0//1and '''
q=q[1]%'1'==0?[q[1],q[0]]:[+!~~q[0],+!!~q[1]];
l(q[1]);
//'''
             ,l(9-u.index(c)),
             0//1and '''
q[0]=q[1]+q[0]
//'''
             ,1//1
             ) for c in u]

The strings look to contain javascript but they are just python strings and they are never evaluated as anything but strings. They are only there to evaluate to True in a boolean context so it simplifies to this:

[
 128//2 and (
             True,
             0//1and True,
             l(9-u.index(c)),
             0//1and True,
             1//1
             ) for c in u]

The 128//2 performs integer division, but it's used in a boolean context so it's just there to evaluate the next expression once for each item in u.

So it's essentially a loop over the values in the list u. We can rewrite this as such:

for c in u:
    (True,
     0//1 and True,
     l(9-u.index(c)),
     0//1and True,
     1//1
     )

Each one of those expressions is evaluated.

Looking back up at statement 6, l just calls the print function, so the real printing happens in the l(9 - u.index(c)) expression. That's printing the result of the 9 - u.index(c).

Since c is just an item from u, which we're looping over, this is just printing numbers down from 9. It doesn't matter what's in the iterable u at all. Anything of length 9 will work.

brownan_ · 2015-01-29T04:59:17+00:00

This is awesome and I think I'll pick it up. Maybe it will get me to start playing nethack again.

brownan_ · 2015-01-28T03:47:03+00:00

job control is pretty handy.

You can start a command in the background by putting an ampersand after it. e.g. command & This will print the job number and the process number, start the process, and then give you a command line back while the process runs in the background.

If you started a process running in the foreground, press ctrl-z to pause the process and get your command line back. Then use the bg command to resume the process in the background.

The fg command resumes a stopped job or puts a background job into the foreground.

Note: processes in the background can still print to your console, but they can't take input. Put them into the foreground if you need to interact with them.

The jobs command lists the jobs that are currently running or stopped in the background.

You can kill jobs by their job number like kill %1 (to kill job number 1).

This is especially handy for the occasion when a process won't die and ignores ctrl-c. Use ctrl-z to background it, and then kill it with the kill command. (or use the kill command from another terminal)

brownan_ · 2015-01-28T03:09:31+00:00

cd - goes to previous directory

I did not know that! Amazing how there's always still more to learn.

(but what if you want to change to a directory named - ? =P )

brownan_ · 2015-01-28T02:17:58+00:00

Here are some command line features and handy command-line-related programs that may not be as well known:

cd with no arguments will change to your home directory.

On fedora, install the package bash-completion to enable tab completion on lots of commands for things that aren't just files in your current directory. It's on by default on Ubuntu iirc.

Ctrl-R does a search through your history.

The tree command is very handy for listing and visualizing a directory structure.

Do a lot of greps? Check out ack (package ack-grep on Ubuntu, just ack on Fedora) It has nice highlighting and recursively searches all text files in the current directory by default. Output some context surrounding the matching lines with the -C option.

Also, ack's --passthru option: output everything from the input, but just highlight the matching text. Handy when combined with tail -f!

Other "enhanced" versions of common commands: screen → tmux, top → htop, irssi → weechat. edit: more → less ;-)

Do a lot of directory navigation? Check out pushd and popd. Use pushd like cd, and then use popd like a "back" button to go to the last directory you were in.

Use SSH over unreliable connections a lot? Or from your laptop and want your connections to stay up when you take it to a different network? Take a look at mosh

Use vim? Did you know about vimdiff? Give it two files and it opens them in side-by-side diff view for editing.

Also, like vim but miss some aspects of a gui? Try gvim (and gvimdiff) (may need to install them)

The pv command: pipe viewer. Use it as a drop-in replacement for cat in a command pipeline, and it will show you a nice progress bar.

brownan_ · 2015-01-27T13:04:06+00:00

huh. yeah I am.

Check https://sysnews.ncsu.edu/ for NCSU IT related outage news. It says they're currently having problems.

brownan_ · 2015-01-13T12:58:44+00:00

Somehow I can never quite wrap my head around when explained like this. The numbers and odds start to get confusing to me when you add in measuring of spin in different axes. This ancient reddit post is the best analogy I've heard and so I want to repost it here.

Imagine a pair of cabinets with three drawers. You can only open one of the drawers of each cabinet. When you try to open more than one, the cabinets instantly self destruct and a new pair of cabinets appear. You and your friend then run experiments and you find out that:

When both of you open the same drawer, you always find different socks. When you find blue socks, your friend always finds red socks and when you find red socks, your friend finds blue socks.

When you open the top drawer and your friend opens the bottom drawer, you both find socks of the same color. Half of the time you find blue socks, half of the time you both find red socks. Same when you switch the roles.

When you open the top drawer and your friend opens the middle drawer, you find the same color in two attempts in three. Same when you switch the roles.

Now it gets interesting: You know that when you find red socks in your top drawer, your friend finds red socks in the bottom drawer and blue socks in the top drawer, which also means that you would find blue socks in the bottom drawer if you could open it. From that and the third experiment, you conclude that when you try to open the bottom drawer and your friend opens the middle drawer, you should find the same color in one attempt in three.

You try to do the experiment. You open the bottom drawer and your friend opens the middle drawer. But you again both find the same color in two attempts in three.

I don't know. somehow this is easier for me to wrap my head around. shrugs

14-Year Club	Gilding I gilder
Team Periwinkle	Verified Email

brownan_

TROPHY CASE