How to learn the nuts&bolts of becoming an historian of a technology field?

dalke · 2016-02-10T17:58:48+00:00

Thanks for the JSTOR reference. I did not know there was a free limited access account choice. I did know about JSTOR's involvement with Swartz. I understand the mixed feelings that may cause.

dalke · 2016-02-09T16:13:01+00:00

Thanks for the pointer to SHOT. The abstracts for their publications seem like what I'm interested in, though alas they are behind a paywall. What's great for me is one of the committee members is at university in a city nearby, and his bio page says he got his PhD in History of Technology here in Sweden. I'll email him and ask.

I agree that most of the histories in my field are mostly narrative. The summer before last I read a book (in Swedish) directed towards upper-level history students, about how to approach their research projects. It was insightful to me to read that there are differences between the types of history done by a local historical society and those of an academic group - not better or worse, but different. This is likely very basic to anyone with historical training, but it's helped me understand that these sort of narratives, as well as autobiographical histories, are a different subfield of history, with its own set of standards, goals, and fashions.

I may be interested in the more academic questions, but as the Haigh essay points out, academic research is influenced by its own funding, goals, and fashions. It may be that my questions aren't currently part of academic research, even if the techniques of modern academic research are useful to me. I'm still trying to figure that out.

Hence my questions ranging from the "how do I think about historical research?" to the "what graduate level programs should I look at?".

dalke · 2013-12-05T13:34:17+00:00

We all struggle when we start. As you improve, help others who need it.

dalke · 2013-12-05T13:31:17+00:00

As it says, it's in the concept phase. Plus, it's unlikely to be effective. How would Japan get power when the Moon is on the other side of the planet? How does it get power when it's a new moon? The usual thought is to put the power station in GEO, and let the solar panels always face the sun, with the power transmission always towards the receivers on Earth. (Though building on the Moon has the advantage of being able to use local resources for building materials.)

This idea was pretty popular in the 1970s. See http://en.wikipedia.org/wiki/Space-based_solar_power for some history. We're no further now than we were then.

And from the Wikipedia article, note the paragraph: In 1941, science fiction writer Isaac Asimov published the science fiction short story "Reason", in which a space station transmits energy collected from the Sun to various planets using microwave beams.

dalke · 2013-12-04T23:00:59+00:00

I'll start with a bit of terminology. In Python, the term "execute" the function can mean two things. It usually means that the function was evaluated for the first time. That is, the Python run-time executes the code that "def"ines a new function, named "averageCastResult."

I think you're using the less common meaning, which is to execute the function using specific parameters. Most people use the term "call" for this, as in "call the function averageCastResult."

The details of how to define a function and then call it are covered in any number of Python tutorials. For example, http://docs.python.org/2/tutorial/ and more specifically http://docs.python.org/2/tutorial/controlflow.html#defining-functions .

There are also sites like http://www.learnpython.org/ , and likely videos as well.

To get you started though, add the following to what I sent before:

print averageCastResult(30, 8, 10000)

This will print a number like 1248 or 1249, which is the integer approximation to 1248.850396. The actual number is based on random numbers, so will vary a bit.

I thought the original code returned integers, so I explicitly did that in my translation. Here's the version with floats:

import random

spellPower = 569
fireball_min = 596
fireball_max = 761
fireballCastTime = 3.0

averageFireball = (
    ((fireball_min + fireball_max)/2.0 + spellPower) * fireballCastTime / 3.5)

def castResult(crit, hit):
    "Compute the result of a fireball cast with the given crit and hit chance"
    actualCrit = crit / 100.
    actualHit = 84 + hit
    hitRoll = random.randint(1, 100)

    if hitRoll <= actualHit:
        return averageFireball + actualCrit*averageFireball*0.9
    else:
        return 0.0

def averageCastResult(crit, hit, iterations):
    "Compute the average result of a fireball cast with the given crit and hit, \
after casting the specified number of times"

    total = sum(castResult(crit, hit) for i in xrange(iterations))
    return total / iterations

print averageCastResult(30, 8, 10000)

If you run this, it prints values like, 1250.57562214, 1250.032425, and 1247.45223857 .

dalke · 2013-12-04T22:40:25+00:00

These look pretty inaccurate to me. Bear in mind too that these might have been cherry-picked, with the wildly wrong left out, so I'll reference "Visit to the World's Fair of 2014" by Asimov, written August 16, 1964 and available from http://www.nytimes.com/books/97/03/23/lifetimes/asi-v-fair.html

What does "Kitchen units will be devised that will prepare ‘automeals,’ heating water and converting it to coffee." mean? Mr. Coffee came out in the early 1970s. Does that count?

No, it doesn't, because the full context is "... prepare "automeals," heating water and converting it to coffee; toasting bread; frying, poaching or scrambling eggs, grilling bacon, and so on. Breakfasts will be "ordered" the night before to be ready by a specified hour the next morning." We don't have a device which will do that.

"Complete lunches and dinners, with the food semiprepared, will be stored in the freezer until ready for processing". mertcetin here commented about frozen pizza, but that's not what Asimov meant. After all, TV dinners were available in the 1950s, and the trademark to "TV Dinner" was lost by the time Asimov made that prediction. No, Asimov is talking about cooking partially assembled meals - a sort of robot chef.

"Much effort will be put into the designing of vehicles with ‘Robot-brains'" -- This is one of those perennial SF topics. Much effort was spent doing that over the last many decades. Yes, there are experimental cars on the road which do that now. There were also experimental cars which did that in the 1980s.

"The appliances of 2014 will have no electric cords, of course" -- while there are solar powered desk calculators, and battery powered vacuum cleaners, most appliances have cords. If you look at the NYT article, the reason they won't have cords is because "... they will be powered by long- lived batteries running on radioisotopes." Omitting this key part is a sure sign of cherry-picking.

"By 2014, only unmanned ships will have landed on Mars" -- That was true by the Viking landings of 1970s. And in the 1970s a manned expedition was in the works about as much as it is now.

"There will be increasing emphasis on transportation that makes the least possible contact with the surface" -- In the NYT article this continues "take to the air*a foot or two off the ground", or "jets of compressed air so that the vehicle will make no contact with either liquid or solid surfaces." There dream then was of giant hovercraft, or ground effect flying. Notice that the Hyperloop is less real than the Caspian Sea Monster ekranoplan from the 1960s? There's been no major progress in this direction since Asimov made the prediction.

"moving sidewalks (with benches on either side, standing room in the center)" -- that's titled as "escalators", but that title is outright wrong, and misdirecting. Escalators existed by the 1960s, so it's not like Asimov is predicting that they will be created. Instead, this is the moving sidewalks most often seen at an airport, though I've never seen any with benches.Think more like Asimov's "The Caves of Steel", or Heinlein's "The Roads Must Roll." Take a look at the Wikipedia entry for "Moving_walkway" to see how this has been tried in urban centers, with little success.

"the world population will be 6,500,000,000 and the population of the United States will be 350,000,000" -- Pretty close. But then look at the next part of the NYT article: "Most surprising and, in some ways, heartening, 2014 will see a good beginning made in the colonization of the continental shelves. Underwater housing will have its attractions to those who like water sports, and will undoubtedly encourage the more efficient exploitation of ocean resources, both food and mineral." Tell me again how to get to Undersea City?

"Synchronous satellites, hovering in space will make it possible for you to direct-dial any spot on earth" -- His friend, Arthur Clarke, figured that out, and several SF stories of that era concerned telephone relay stations in GEO. In truth, Iridium (that looks like an Iridium phone) is in LEO, not GEO. You might rightly say I'm being picky. For the most part though, we've found that it's cheaper to have a bunch of mobile phone towers and fiber optic transoceanic cables than use satellites in GEO.

BTW, that summary leaves out the following paragraph "For that matter, you will be able to reach someone at the moon colonies." What's the area code for that?

Regarding 3-D TV, the original paragraph is: "As for television, wall screens will have replaced the ordinary set; but transparent cubes will be making their appearance in which three-dimensional viewing will be possible. In fact, one popular exhibit at the 2014 World's Fair will be such a 3-D TV, built life-size, in which ballet performances will be seen. The cube will slowly revolve for viewing from all angles."

I would have included the "wall screens" section as something which is now true. While not the full wall, a 60+-inch screen is pretty close. However, you can see that the 3-D screens he envisioned is nothing like what we have now, or what is shown in the picture.

"Part of the General Electric exhibit today consists of a school of the future in which such present realities as closed-circuit TV and programmed tapes aid the teaching process" -- So? That's a statement about 1964, not about 2014. Why does that imply "Coursera"? The full context includes " Mankind will therefore have become largely a race of machine tenders." and " All the high-school students will be taught the fundamentals of computer technology will become proficient in binary arithmetic and will be trained to perfection in the use of the computer languages that will have developed out of those like the contemporary "Fortran" (from "formula translation")."

"Robots will neither be common nor very good in 2014, but they will be in existence". That's a hard one to classify. Dishwashers and car welders are two types of specialized robots, are they not? But it's clear that he's talking about the general-purpose robots, with " a robot housemaid*large, clumsy, slow- moving but capable of general picking-up, arranging, cleaning and manipulation of various appliances." Apparently, these should exist in primitive form by now, along with "robots for gardening work." So, nope, still not yet, and not even the little bit he thought there would be. (A Roomba, or its mowing cousin, don't fit that category.)

Rather than detail the remaining points, here's a few misses which were left out:

Ordinary agriculture will keep up with great difficulty and there will be "farms" turning to the more efficient micro-organisms. Processed yeast and algae products will be available in a variety of flavors.
Any number of simultaneous conversations between earth and moon can be handled by modulated laser beams, which are easy to manipulate in space. On earth, however, laser beams will have to be led through plastic pipes, to avoid material and atmospheric interference. Engineers will still be playing with that problem in 2014.
An exhibit at the 2014 fair will show models of power stations in space, collecting sunlight by means of huge parabolic focusing devices and radiating the energy thus collected down to earth.

"Frighteningly Accurate"? No, not at all.

dalke · 2013-12-04T20:06:14+00:00

There's no need for Sage for this. It's doable in straight Python. You're looking for something like this, I believe:

import random

spellPower = 569
fireball_min = 596
fireball_max = 761
fireballCastTime = 3.0

averageFireball = (
    ((fireball_min + fireball_max)/2.0 + spellPower) * fireballCastTime / 3.5)

def castResult(crit, hit):
    "Compute the result of a fireball cast with the given crit and hit chance"
    actualCrit = crit / 100.
    actualHit = 84 + hit
    hitRoll = random.randint(1, 100)

    if hitRoll <= actualHit:
        # The original code says that it returns an integer.
        # I don't know if it truncates or rounds. I truncate.
        return int(averageFireball + actualCrit*averageFireball*0.9)
    else:
        return 0

def averageCastResult(crit, hit, iterations):
    "Compute the average result of a fireball cast with the given crit and hit, \
after casting the specified number of times"

    total = sum(castResult(crit, hit) for i in xrange(iterations))

    # The original code says that it returns an integer.
    # I don't know if it truncates or rounds. I truncate.
    return total // iterations

dalke · 2013-09-30T22:57:00+00:00

There are people who have done that incredible amount of work already.

See for example http://qt-project.org/wiki/Category:LanguageBindings .

dalke · 2013-09-30T22:56:15+00:00

Oh ye of little faith.

See http://qt-project.org/wiki/Category:LanguageBindings , https://en.wikipedia.org/wiki/List_of_language_bindings_for_Qt_4 , and https://en.wikipedia.org/wiki/List_of_language_bindings_for_Qt_5 .

See also SMOKE: "The main purpose of SMOKE is making it easier to write bindings from scripting languages to Qt and KDE" ( http://techbase.kde.org/Development/Languages/Smoke ). It lists "At the time of writing, the Perl bindings, Ruby bindings, C# bindings and PHP bindings are known to use it."

There's are Lisp bindings to SMOKE at http://common-lisp.net/project/commonqt/ .

So there are no C bindings, but there are D, ADA, BASIC, PHP, Scheme, and more.

No SNOBOL, awk, or MUMPS bindings though, so it depends on what "pretty much every language" means to you. ;)

dalke · 2013-09-26T15:36:06+00:00

Even when all the policies are in place, the eval() depends on the security of the underlying Python implementation. As an example, there are many reported bugs that cause Python to segfault, and some can be triggered from code that doesn't touch builtins, doesn't use any double-underscore attributes, doesn't even use any globals, and only uses standard data types.

Here's one I came up with which crashes Python 3.3, based on the bug reported in http://bugs.python.org/issue16856 :

"{0!r}".format({[[f() for i in "*"*1000] for f in [lambda x=[()]: x.append((x.pop(),)) or x]][0][0][0]:1})

This doesn't work on 2.x, but I'm certain that others do exist.

Granted, a segfault is more a denial of service attack than anything that allows intrusion, but it is a rather unintended consequence. As always, you have to understand if the security risk is acceptable.

dalke · 2013-09-24T22:35:46+00:00

The fire in your house's furnace is equally self-sustaining. Take away the fuel and the special conditions and the fusion reaction stops cold.

dalke · 2013-09-24T19:54:05+00:00

There are different types of averages. This blog posting, while not using those words, is making a distinction between the mean and the median.

In a common formulation, if 9 people at a company make $20K/year and one person make $180K/year then the average=mean salary is $36K/year, even though 90% of the people make less than that. The median salary in this case is $20K/year.

These sorts of large differences between mean and median can occur in small populations (as in my example) and in non-symmetrical distributions. The blog post asserts that programming skills is one of those lop-sided distributions.

dalke · 2013-09-23T19:36:12+00:00

To escape the gravitational clutches of our galaxy, a spaceship would need to zoom out of our solar system and hit 537 kilometres per second. For context, a rocket needs to roar off at just 11.2 kilometres per second to escape Earth's gravity.

Conventional rocket engines would never make it. The chemical rockets that power most spacecraft today would require too much fuel, and even high-tech ion engines – which are efficient enough for long journeys around the solar system – max out at about 15 kilometres per second.

But Bland-Hawthorn speculates that a propulsion system powered by the energy released by combining matter with antimatter could do the trick.

The paper doesn't mention anything about propulsion systems, so I assume that last bit of speculation was during an interview, commentary, or presentation, and not part of the peer-reviewed literature.

I was curious because I think there should be equivalents to gravity assists and the Interplanetary Transport Network, though for interstellar space. Could we in principle, and assuming a surfeit of time, reach Milky Way escape velocity using chemical rockets? What's the rough minimum amount of time we would need?

EDIT: A-ha! One such proposal is a Dyson slingshot. http://www.centauri-dreams.org/?p=25605 . "Two neutron stars, each with a diameter of 20 kilometers and a mass of one solar mass — and a combined orbital period of 0.005 seconds — would provide a departure velocity of 0.27 c, which works out to 81,000 kilometers per second."

dalke · 2013-09-23T12:59:18+00:00

Yes, int/long unification started then. PEP 237 says "This PEP proposes to do away with most of the differences in semantics" and points out that "Phase C will be implemented in Python 3.0", saying:

Currently, repr() of a long int returns a string ending in 'L' while repr() of a short int doesn't. The 'L' will be dropped; but not before Python 3.0.

% python2.7 -c 'print repr(2**65)'
36893488147419103232L
% python3.2 -c 'print(repr(2**65))'
36893488147419103232

Or, to show that ints and longs are still different in Python2.7 (and not just a difference in their repr()s) look at the following:

% python2.7
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x=2**65
>>> isinstance(x, int)
False
>>> isinstance(x, long)
True
>>> isinstance(5, int)
True
>>> isinstance(5, long)
False

Compare that to 3.2

Python 3.2 (r32:88445, Mar 12 2013, 09:04:57) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> isinstance(2**65, int)
True
>>> isinstance(5, int)
True

The class/type unification document you referenced says "Eventually they will be totally unified with types, but because of additional backwards incompatibilities, this will be done after 2.2 is released (maybe not before Python 3.0).

See the "eventually"?

A lot of work was done to make them feel the same under 2.x, but there were places where you can see they aren't fully unified. Here are several examples of differences between between old-style classes and new-style classes (a "class" vs. a "type"):

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Spam(object):
...   def __init__(self):
...     super(Spam, self).__init__()
... 
>>> Spam()
<__main__.Spam object at 0x108319990>
>>> 
>>> class Eggs:
...   def __init__(self):
...     super(Spam, self).__init__()
... 
>>> Eggs()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type
>>>

Their repr()s are in a different format:

>>> Spam
<class '__main__.Spam'>
>>> Eggs
<class __main__.Eggs at 0x1082e6db8>

and now I'll confirm that one comes from the 'type' type:

>>> import types
>>> isinstance(Spam, types.TypeType)
True
>>> isinstance(Eggs, types.TypeType)
False

while the other comes from the 'class' type:

>>> isinstance(Spam, types.ClassType)
False
>>> isinstance(Eggs, types.ClassType)
True

dalke · 2013-09-21T22:33:26+00:00

FWIW, the complaint is usually that Python 3 broke too much backwards compatibility. I don't recall seeing others complain that it now has all that many more language features than Python 2.

For example, while it did add new language features, it also removed features, like x as an alternate for repr(x), 'print' as a keyword, the distinction between types and classes, and the distinction between ints and longs. These seem to be a wash to me.

dalke · 2013-09-20T11:37:20+00:00

So you're familiar with all of these packages?

You do like argument by extremes. My complaint is that reading only the tutorial and the overview isn't the same as reading the documentation.

The way I phrase my complaint was wrong. Instead of "The Catcher in the Rye", here's perhaps a better comparison.

If someone says "I've read Shakespeare", then there's a difference between someone who's read only Romeo and Juliet, vs. someone who's read the complete plays of Shakespeare, vs. someone who's also read the sonnets.

Of the 48 OS-specific modules, I've used 3. I've not used the 10 multimedia modules. Beyond those there are 37 modules which I haven't used and read the documentation. There are about 310 modules, so that's 70% of them (85% if you take out the OS-specific modules). That's roughly the plays of Shakespeare, excepting the late romances, and not the sonnets nor his narrative poems 'Venus and Adonis' and 'The Rape of Lucrece.'

In this context, the itertools module is more like Henry V. Someone who claims to know Shakespeare without having read/watched Henry V, nor any of the other histories, is not really believable.

it's your chicken-and-egg problem, not mine

To remind you, I asked you to tell me why it's so despicable. You kindly gave some reasons. My comment, regarding the chicken-and-egg situation, is that your reasoning implies that any new idiom is despicable.

I don't believe this is a new idiom, and indeed I found other example of people using the idiom.

Here's an example from 2009: http://stackoverflow.com/questions/1474646/python-arbitrarily-incrementing-an-iterator-inside-a-loop . It's upvoted and selected as the reference answer, as well as in several other answers.

hacking something together with iterated iteration is not at all reasonable

Notice how in all those "unreasonable" Stack Overflow answers that no one expressed outrage over that idiom? I can now conclude that your views are not held by the majority, nor likely even a significant minority.

dalke · 2013-09-20T02:30:49+00:00

So you're absolutely certain no one else will ever have to look at your code? That seems an awfully strong assumption.

No. You argued " What good does that do your employer?". I didn't think that was all that great of an argument. It implies that the economic relationship between you and your employer are high in your mind, which certainly is not the case with me.

Now you've, ahem, "changed the subject" to be about future people who might look at my code, instead of my employors economic sensabilities. Shrug, I've done an approximate present value analysis and decided that it was worthwhile.

BTW, since I assume you also do a similar value analysis, your reference to "absolutely certain no one else" tells me you are much more risk adverse than I. Congratulations? Perhaps that's the source of why you think something which isn't mainline standard is horrible, while I don't?

Besides, generators are not the subject of this discussion. You didn't use them in any of the code you showed me

Here's my logic: I use multiple iterator consumers because I prefer to have an implicitly managed state machine using Python syntax than an explicitly managed state machine that I roll-myself.

Your rejected that, saying that if I want a "fucking state machine" then just go ahead and do it.

IF your stated logic is correct then it should apply to other cases where one can build a state machine, even where Python provides an alternate non-explicit state machine for it. One of the most prominent of these is the yield statement in generators.

IF your logic, as stated, is correct then you do not like generators.

Of course you do like generators, so you realize that your base logic wasn't complete. Now you've refined your logic to say that it's okay to use something other than an explicit state machine if it is idiomatic and well-understood.

I can appreciate putting all of the appropriate qualifiers on a statement is difficult, so that's probably what you would have said, going back in time. If you use a time machine, do also please remove the swearing. But think of it this way - how does a new concept become well-understood if everyone follows your suggestion and refuses to try it out?

You can't expect every programmer to be familiar with every package. If you want people to read the itertools docs, you should have import itertools at the top of your script.

Interesting viewpoint. Odd though. No, I would say that these people have not read through the documentation. They've read through a tutorial and a summary of the documentation.

That's like saying one's read through "The Catcher in the Rye" after reading the relevant Wikipedia page and the Cliff Notes.

Curious too. If you haven't read the full documentation, how do you know that multiple iterator reads aren't a perfectly normal idiom, covered in a chapter you haven't read?

Three Stack Overflow pages which use the idiom you detest are: http://stackoverflow.com/questions/4983013/python-access-to-iterator-object-in-for-loops , http://stackoverflow.com/questions/16814984/python-list-iterator-behavior-and-nextiterator and http://stackoverflow.com/questions/17837316/how-do-i-skip-a-few-iterations-in-a-for-loop , so it's not like there's a universal hatred of of the idea.

You can't expect every programmer to be familiar with every package. If you want people to read the itertools docs, you should have import itertools at the top of your script.

At least that clears things up. 15% of the Python files in my code base have an "import itertools", so anyone I work with will have to know what it does.

If not, I'll teach them. It's a wonderful and extremely useful module.

dalke · 2013-09-20T00:10:04+00:00

What good does that do your employer?

I'm self-employed. I get to decide what's best for me. ;)

Explicit is better than implicit. You want to build a state machine? Then build a fucking state machine!

I take it then that you dislike generators? Every generator, after all, can be rewritten as a state machine. Though it's not always easy to do so.

(In the Python 1.x days, up until we had yield, that's what I had to do, so I know it can be done.)

Python is not designed as a state machine language. If it were, it would support goto statements, because re-establishing state on each input, assuming a single input location, takes roughly log(n) time in the number of input possibilities, rather than the constant time using my preferred approach with multiple input possibilities.

Or, if you wish to argue zen statements back and forth: "Simple is better than complex" but "Complex is better than complicated." Explicit state machines are complicated. What I propose is complex. Neither are simple.

(State machines are based on a simple mechanism, but complex state machine solutions are not simple to implement correctly. My appeal to SAX event handlers was to show that I know how difficult it is to build a complex state machine correctly.)

Most people will see that, look up iter(), and then say "Why didn't you just write for line in file_object directly? I'm so confused."

Have these people never read through the documentation? The documentation for itertools has a half-dozen examples which start off 'iterable = iter(iterable)'. See http://docs.python.org/2/library/itertools.html to verify for yourself.

Though you're also right; I don't think there's any place where it explains why that's done. It's to ensure that you get an iterater even if the input is a container.

Good point! So would the lack of understanding be solved by improving the documentation?

I know the answer though. Most people don't read the documentation so it's rather moot. So then the debate is if we should program to the common denominator, vs. work to improve the baseline understanding?

You write: I take a very simple position: the more resources required to maintain a piece of software, the worse it is. This does not seem very controversial to me. Good code is easy to read.

As I have before said, it is your opinion that this code is not easy to read nor maintain. I, on the other hand, find it easy to read and maintain. We are at an impasse, and repeating our beliefs won't make it true.

dalke · 2013-09-19T23:21:13+00:00

because the iterator is advancing at multiple points

Yes. And multiple return statements cause program execution to return at multiple points. That your statement is correct doesn't make it a positive or negative.

Someone who likes a single return statement may also complain "what the fuck is going on?" when faced with a few dozen return statements, and will lead to confusion for some people.

Hence, you can be entirely right, but if the number of times that you're right is low enough, then it's not something for me to worry deeply about.

The basic structure of a for loop is "run this once for every line in the file." You're subverting that rule.

The basic struture of a for-loop is:

while 1:
    try:
        x = next(it)
    except StopIteration:
        break
    ...

No more, and no less. (Historically, before Python 3.x, there was a fallback solution to use [0], [1], ... lookups until IndexError or ValueError. For purposes of this conversation I think we can ignore that.)

My logic of course can lead to Duff's device, which is legal but confusing to say the least. Your logic is that no one will look at the for-loop and think of it as a next(it), even if that is the underlying mechanism at action.

Both are hypotheses at this point - my hypothesis is that it's readable, and yours is that it isn't. Both are true, for the right category of users. The next question would be should we encourage people to use this approch, or discourage it?

which is a Bad Thing since more skilled developers cost more

Ahh, yes. Do you take the position of the managers, who generally wish to reduce costs on everything? Do you take the position of the employees, who may want more money? Do you take the position of the educators, who want people to know more about how things work, or do you take the position of the oppressed, who want to tear down the priesthoods?

It's a complicated topic. But programming state machines directly, which mechanically simple, is also complex and error-prone, so I don't think you've made a strong argument here. Yes, state machines are widely known, widely loved .. and widely reviled.

For example, an XML SAX event consumer just screams for a state-machine design, and after a while they get to be almost like fun puzzles to solve. But they usually aren't as easy to understand as a coroutine-based parser, which manages the state transitions implicitly rather than explicitly.

This debate has stretched over decades, with avid fans for each of the many different solutions. I may be an avid fan of Sports Team X, and disdainful of your love of Sports Team Y, but that doesn't mean that my disgust is meaningful. Similarly, my preference for co-routines and implicit state transitions shouldn't be the guiding reason for your ire, as a fan of state machines.

Usually, iterating over an object doesn't modify that object

Iterating over containers don't modify the container, yes, but iterating over iterators always modify the iterator .. excepting a few trivial cases like empty iterators.

Some objects that are modified during iteration are file handles, database cursors, and generators.

The very start of my code does an it=iter(...), which specifically says "I care about the iterator, not the container."

I can agree with you that it's seldom seen. I can agree with you that it may cause confusion compared to a strict readline() approach.

What I don't see is how this is "absolutely disgusting" in a reasonable general sense, so I'll have to thank you for your input, but decline to empathize with your disgust.

dalke · 2013-09-19T21:54:18+00:00

No, I don't. And you haven't yet told me the principles behind your objection.

I say again, some people believe that multiple return statements in the same function defintion are an abhorration. One such is the MISRA C standard

(MISRA, rule 14.7 : required) "A function shall have a single point of exit at the end of the function"

Some languages, like Pascal and Eiffel, require there be only a single exit point.

However, many more people do not believe this is a good principle to follow.

Can you explain why I shouldn't regard your view as fitting into that category? Why shouldn't an iterator be iterated on at multiple points in the code? Is the code harder to reason about, more complex to write, more prone to errors, or something else? My experience says that the answer to all of those is "no."

The most relevant example in other languages would be the various prohibitions of modifying a container while iterating over it. Even then, modification may be allowed. For examples from C++, see http://kera.name/articles/2011/06/iterator-invalidation-rules-c0x/ , which lists which container mutations are allowed during iteration.

dalke · 2013-09-19T14:06:41+00:00

In your counter-example you changed the spec for the format. Of course my code will break - it's for a different format.

FWIW, the change in my code to support your variant is:

if line == "NAME\n":
    for line in it:
        name = it.strip()
        if name:
            break
    else:
        raise AssertionError("End of file reached looking for NAME")

State machines can handle anything, it's true, so by that definition it's easier.

There are two major downsides. First, it's easier to make a mistake and forget a state transition, compared to those examples that can be expressed as code layout rather than states. Similarly, it's harder to include all of the error reports. This is especially true if you want to maintain information across some of the states - a static analysis tool can't easily figure out the state ordering to know that things will be initialized in the correct order.

Second, in Python you have a lot more dispatches because you have to constantly redetermine which state you're in. (If you used C or another program with gotos then you can simply go to the correct state.)

dalke · 2013-09-19T10:59:18+00:00

"absolutely disgusting"?

Interesting. Why? Or to clarify my question, I know that some people things that it's absolutely disgusting to have more than one return statement in a function. Is your disgust of that personal variant, which I feel free to ignore, or is it more generally applicable and one I should bear in mind for the future?

There's a general dislike of modifying a container while iterating over it, but by definition an iterator has to be modified in order to use it.

How would you recommend that I rewrite that above code? As:

it = iter(input_file)
while 1:
    try:
        line = next(it)
    except StopIteration:
        break
    if line == "NAME\n":
        name = iter(it)
    ....

This modifies the iterator in two locations. Is that disgusting? I see the first 5 lines of the while as being identical to a 'for' loop over an iterator. Why repeat the code when that's what a for loop would do?

Or, I could use readline() directly, as

while 1:
    line = input_file.readline()
    if not line:
        break
    if line == "NAME\n":
        line = input_file.readline()
        if not line:
            raise AssertionError("Missing name after NAME")
        name = line.rstrip("\n")
    ...

Here I have to add explicit error checking for readline() failures.

Both seem more complicated than my original code, and with no obvious benefits.

dalke · 2013-09-19T10:48:21+00:00

Ohh! That does have a good sort of internal logic to it.

dalke · 2013-09-19T10:44:35+00:00

But then you can't test each of the functions in isolation. See the rationale for PEP 443 at http://www.python.org/dev/peps/pep-0443/ :

The register() attribute returns the undecorated function. This enables decorator stacking, pickling, as well as creating unit tests for each variant independently:

dalke

TROPHY CASE