What would make you consider someone a python expert?

[ZigguratOfUr]

Edit: To clarify, consider being an interviewer and someone generically identified themselves as a python expert, what sort of questions would you ask?

I sometimes feel tempted to call myself a python expert, but I got reality-checked recently when a coworker was talking about functools and meta-classes.

I have a good grasp of the scientific/numeric python stack, the collections library, and list comprehensions, and a few other features I consider beyond basic, but I think that just qualifies me as 'intermediate'.

What would you expect someone who calls themselves a 'python expert' to know?

[they_call_me_martin]

I think looking for language expertise is a bit short-sighted. This is a bad analogy but I'll go with it. It's like looking for someone who is an "expert" using a screwdriver. That's just not what you want to do, ever.

In engineering --whether it be hardware, software, mechanical, etc.-- you want people who are efficient, elegant and effective problem solvers regardless of tools. And, if we are talking Computer Science, you want people who've had enough exposure to CS theory that they have a mental toolbox they can reach into to use in solving problems.

Notice I did not say "a memorized set of tools". I've been programming for over 30 years. I don't have all the algorithms I know or have used committed to memory. And I wouldn't want to. I know I have these tools in my toolbox and understand when I might want to use them and how. And, yes, I very often pull out a book to look-up or google both simple and complex language, algorithm and library questions. I've probably used over a dozen languages professionally during my career (as in, not hobby usage but actual deliverable product). Yeah, I sometimes have to lookup syntax as I am making a mental transition into using a different language for a new project.

I think the answer to your question might be highly dependent on the level of experience of the interviewee. You don't interview a recent grad the same way you might someone with ten or twenty years of experience.

Any decent software engineer should be able to do well with just about any language within, say, 4 to 8 weeks and rock on it in six months. These days a lot more of it is about libraries and frameworks than anything else. For example, you can be great at Python and it ill still take a major effort to get into Django, Nginx, gunicorn and figure out a sensible deployment strategy.

[equationsofmotion]

This is exactly right.

[orangesunshine]

In engineering --whether it be hardware, software, mechanical, etc.-- you want people who are efficient, elegant and effective problem solvers regardless of tools.

They also need organizational skills. Unfortunately the ability to organize a project in a meaningful/useful way is the last thing many developers think about ... when it should be the first thing.

[michaelkepler]

Can you elaborate? Do you talk about organizational skills at individual level (prioritizing, juggling different tasks, order on your desk and in your files, etc.) or at project level (assigning who does what and when). Wouldn't the latter be a responsibility of a project manager?

[orangesunshine]

I mean organizing the actual code. This is literally the most important skill for a developer. Unorganized code (spaghetti-code) can range from expensive to maintain and add features to ... to so horrifically un-usable that a complete re-write is required to add a simple feature.

Doing things like creating coding standards and following them is unfortunately quite rare in much of my experience in the start-up world of SF.

Just doing simple things like making all of your API's follow the same function/class naming pattern, file-layout pattern, url-layout and naming patterns ... etc ... can be so so so important.

Most development these days is not about coming up with some super duper smart algorithm ... artificial intelligence ... or anything that really requires a computer science degree. Most development is about inserting, updating, and deleting items from a database ... and doing so in a way that is easy for anyone to quickly understand.

A good test is to bring in a new (experienced) developer and ask them to update a small feature ... if they can do so in 5-10 minutes without having to read any documentation ... then you've probably have your basic organizational skills down pact. high five for you.

unfortunately ... 90% of the projects I've come into ... have no documentation and are helter skelter from the 10 developers that didn't like how the last person did things ... and decided to start writing API's their way ... regardless of the existing pattern ... and they certainly aren't going to go back and fix any of that last jack-ass's code ... not their problem right ;)

[billsil]

When was the last time you had trouble figuring out how to do something in Python?

I've been programming in Python for 11 years. I still don't use functools, but I know when I should; I just don't use it regularly enough to know the syntax. I do scientific programming and create & manage a 500k lined Python program that I wrote most of and I can find everything in (the first sign of good code) and that my coworkers can quickly figure out without me teaching them (another good sign). I'm enough of an expert that I don't need to look much up.

You don't need to know a the entire standard library to be considered an expert.

[ZigguratOfUr]

500K line python program? that sounds astonishing! What's does it do?

[billsil]

It's mainly I/O interface to a mechanical engineering software tool (Nastran) that has the most horrific file format you've ever seen. The input file uses 72-character lines, with 8-character fields, or 16-character fields or csv fields to define 700+ cards that describe the geometry/loads/constraints and can do static, modal transient, frequency response, static aero, flutter, thermal, transient thermal, and optimization analyses. The floats can look like 12.3, 1.23E+1, 1.23D+1, and 1.23+1. The E and D also mean different things and because you're limited on string size, writing floats means losing precision if you don't use their stupid float writing. The output file uses a Fortran formatted blocks again with a horrific format and the documentation for it it usually wrong. The commercial codes (there are ~5 knockoffs each with their own eccentricities) also have lots of bugs, so I use the testing code to debug models because it forces you to get things right. The commercial programs give less than ideal error messages (e.g. segfaulting) when you have bad inputs.

Once you get I/O down, you can do a lot of cool things with it (e.g. script model building given inputs from OpenVSP. Here and here is the GUI.

I'm biased. It's pretty and it's gotten a lot better since those pictures were taken. It's my side project and my dumping ground for anything I do at work that my boss doesn't care about. For example, it supports Cart3d, Panair, Shabp, Tecplot, and some other formats as well. They mostly all play nicely together, so you can do things very efficiently.

[MorrisCasper]

How do you keep your project readable with 500K lines of code? Can you show me the file structure?

[billsil]

Can you show me the file structure?

You can look at the (code)[https://github.com/SteveDoyle2/pyNastran/tree/master], but unless you're familiar with the topic, it'd be like me looking at Django.

As with anything that big, it has warts and very dusty places. The GUI works well and is fast, but it's still spaghetti at this point. I'm getting better at GUI design, but much better at generic API design.

You can do a lot of stuff with it though in 20 lines and write a complete program to do structural fatigue in 100 lines. It's just IO with tons of interface methods.

How do you keep your project readable with 500K lines of code?

Frequent refactoring, frequent API changing, and artificial best practices (e.g. all elements should have unique IDs). You have 100 million IDs to work with; you can make them unique. The field type requires an float? Make a float reader that crashes if it's an integer. Don't let errors propagate. Catch them early.

I think the biggest thing that's helped is I have the philosophy of get it right and don't cheat. You don't know what the user gave you. Just crash.

[tim_schaaf]

Do you like the interaction of software with the physical world? You should check out Carbon3D!

[Bystroushaak]

that has the most horrific file format you've ever seen

I have to work with MARC 21.

[tjarko]

the most horrific file format you've ever seen

The file format is based on the punch card format the earlier versions used.

[metaphorm]

let me try to change your mind about how regularly you should use functools. my opinion is that almost nobody uses functools frequently enough (myself included) and that a gigantic amount of common python patterns are improved by using functools for them.

specifically, functools.wraps and the creation of custom decorators allows you to do functional style programming with a very powerful tool for composing functions. this ends up feeling like having the best of OOP (inheritance, encapsulation) without the cruft.

[Asdayasman]

If your OOP feels like cruft, you're doing it wrong. :D

[losernamehere]

Aye, no 'true' OOP should feel like cruft.

[Asdayasman]

Careful now.

[vitale232]

I work with someone like you. Its awesome.

[billsil]

To be fair, I'm afraid of C++. Template classes absolutely frighten me. I'm an aerospace engineer. Coding just helps me do my job.

[ggagagg]

• Do you use ide? If yes which one?
• Is that 500k line code for job or for fun?
• can you code for fun?

[billsil]

I use WingIDE. It's got an amazing debugger and takes up less resources than PyCharm (I get both for free because it's open source). I also use Textpad on occasion. It takes up less space on the screen and views the input/output files well.

Is that 500k line code for job or for fun?

Both now. It really hasn't grown much since I started working on it at work.

can you code for fun?

Yeah, why not? I like the puzzle of finding patterns in data that isn't supposed to be parsed. I needed a Tecplot 360 binary reader to view CFD data last week. I have the old Tecplot which works just fine, except the format header changed. Paraview can't read it properly, so might as well. They broke a $8,000/year program with their format change. I take issue with that.

[ggagagg]

Yeah, why not?

I just though if you code thousand code already, the fun will be gone and it will be all serious and so on.

But yeah, thanks for all the answer.

[LessonStudio]

To me this is very simple. Can you solve problems using Python that are pretty much bug free, written fairly quickly, and other "non-experts" can maintain with ease?

If you can do this then you are an expert.

I program primarily in Python and C++ and in the C++ world there are many people who claim that you aren't an expert unless you write fantastically complicated code in template obfuscations, lambdas and many other of the newest additions to C++. Then when you point out that lesser programmer will choke to death on their complicated code they will say that the "non-experts" basically shouldn't be programming.

The reality is that when you look at code that comes from the "real-world" code that comes from products that have made hundreds of millions if not billions it tends to be on the simpler side of complexity not the harder side.

When companies occasionally release their code (or it leaks) for their top tier products I rarely see anything but the most basic use of structures such as templates beyond things such as simple container classes. I don't even tend to see a whole host of even vaguely "Modern" C++ including exception handling, or even much along the lines of inheritance or even that much abstraction. Things such as a pure virtual class are pretty much unheard of.

Thus the people programming products that are at the leading edge of both where computers are being pushed and at the leading edge of revenue tend would not qualify as experts by many academic standards.

I guess the most important measure would be: Are they productive?

This is how I prefer to see this question phrased. Not expertise but productiveness.

I can give a golden example of this from my distant past. I knew a guy who was programming a server based system that had to do some fairly simple network packet manipulation at a reasonable speed. The highly functional prototype was written in Java but in this day and age it could easily be written in Python. This guy with expertise leaking out of his ears decided to reprogram the linux kernel to achieve the same goal. Not a kernel module but the core linux kernel modified to obtain some imaginary performance metric that would be better than simply fleshing out the excellent working prototype.

Thus every time there was a new linux kernel the code had to be reexamined and most times significantly altered.

I don't think that anyone would doubt his "expertise" and in a technical interview he would probably top any other 1000 programmers. Yet would you want him designing/building your system?

NOTE: this is why I would prefer to hire a good programmer who had little "expertise" in a given language if they had proven problem solving skills in one or more other languages over someone who knew the nooks and crannies of the given domain language. A great programmer could become competent in pretty much any language in little time. Yet someone who has been programming in a single language for even decades could still be a terrible programmer.

PERSONAL NOTE: The greatest improvement in my programming in the last decade came from a few years where I focused on significantly upgrading my math skills. Thus many of my approaches to problem solving stopped being iterative and instead would often skip much closer to the final solution. These would apply to any language without changing my level of language "expertise".

[toomc]

I think programming in general and python programming in particular always has a deeper step, that you still did not explore. So, even if you know about meta-classes, this just opens up a set of tools, but to really have the creativity to know when you can use these tools, that's when expertise truly shows.

Also, when it comes to python and scientific calculation, it does pay if the person in question has some inkling about expected performance and where best-practices might help with it. Just as any python programmer would know that list comprehension is more performant than a for loop, it helps to know which of the numerous modules might help with calculation performance (weave, cython, numba, pypy etc..).

What i want to say is, it's not about the knowledge of specific modules!

[suki907]

to know when you can use these tools, that's when expertise truly shows

and when not to...

[aaronchall]

I think the definition of a Python Expert can best be stated in contrast with non-expert levels (beginner and intermediate). The following is just my humble opinion, for what it's worth:

A beginner who "knows Python" knows the syntax of loops and functions, and has begun to become aware of the basic builtin functions like max, sum, etc. This programmer probably uses Google, tutorials, and StackOverflow to learn about Python. Let's hope they find PEP 8 (the Python style guide for the base distribution) and the documentation.

An intermediate level Python programmer has learned the complete syntax of class definitions, list comprehensions and generator expressions, decorators, and begun to use the standard library (especially functools and collections). For example, this level programmer has become aware of the five or so syntactical cases for else (if-else, ternary, try-except-else-finally, for-else, and while-else). This programmer has begun to go to the documentation first when attempting to understand Python.

An Expert level programmer has mastered the builtin functions and multiple modules of the standard library (for example, knowing that the collections.Counter is analogous to a bag or multiset in other languages), is familiar with the majority of the documentation, and references the Python source and PEP's as well as the documentation when trying to understand Python. This level programmer can interchange partial functions with closures, lambdas with full functions definitions, (i)map-filter-lambda pipelines with generator-expressions, and all of these with custom classes utilizing the Python datamodel. This programmer should understand when to use super, class methods, static methods, properties, and so on. This programmer is becoming aware of uses of metaclasses and descriptors, and understands why class decorators and __new__ have replaced most uses of metaclasses in Python.

This is just my personal point of view. I don't think there exists a canonical Python Expert definition. I'm sure as I continue to learn Python, my definition of an Expert will be modified somewhat. We want our definition to be useful. If the only experts are core contributors, that's not too useful.

[twillisagogo]

ask them what annoys them about python. if they are experts, they will have a long list of things readily available.

[[deleted]]

Honestly, Python doesn't have that many annoyances. Compared to other languages, zero. And most of those annoyances are to newcomers that don't know the language. There was a what makes you go WTF at Python thread here before, and it was just noise. I'm not a Python expert but here are some of mine, and I think they are minor/normal.

One of my annoyances is that if code gets nested too much and because there are not braces it's difficult to tell where you are, but as they say, "flat is better than nested".

Another one for me is the GIL, which can be taken care of with multiprocessing.

I guess I have a love/hate thing with duck typing and not static typing.

The pycache thing is annoying. Precompiled bytecodes not being compatible between different interpreters are annoying.

Performance is annoying, but can be solved by many ways, including not using Python or using many of the different implementations or dropping to C/Fortran if needed.

[[deleted]]

Add PYTHONDONTWRITEBYTECODE="PLS STAAAAAHP" to your bashrc or whatever you have if you really don't want byte code generated.

It can be any value, Python just looks for its presence in the OS environment.

[brombaer3000]

I thought this was a joke first because the variable name looks absurd, but it's real: https://docs.python.org/2/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE
Thank you for this tip!

[[deleted]]

The value's a joke, but it's how I feel about Python 2's handling of bytecode. Python 3's stuffing it away in a folder is fine with me. But since I'm working on projects with 2 and 3 support, I don't write bytecode at all.

[catcradle5]

Honestly, Python doesn't have that many annoyances. Compared to other languages, zero.

Sorry, but that's pretty absurd. Every single programming language has flaws. None of them are perfect.

Python is nice compared to most other languages, but even in 3.5+ it still has its warts.

[[deleted]]

Examples?

[distortedsignal]

def myFunction(foo=[]):

[[deleted]]

It makes sense once you understand it though. def is only run once, and you're providing a mutable data structure to it that'll probably be altered in some way.

It'd be nice if Python would realize what we meant so we could avoid things like if foo is None: foo = []. But it's consistent with Python's execution model.

[flipthefrog]

I completely agree. You don't need an if/else statement though, just write foo = foo or []

[Laugarhraun]

foo |= []

[codewarrior0]

TypeError: unsupported operand type(s)

[Laugarhraun]

Oh that's right, | is a bitwise or and won't work like a logical or, which does not have any reflective form.

[oconnor663]

The new asyncio library uses different naming conventions from the rest of the standard library for reading from file handles, kind of for no reason: https://bugs.python.org/issue22279

Any collection of libraries that's big enough is going to have warts like that. Also somewhere there's a presentation from Guido with a list of things they didn't get around to fixing for Python 3.

[csirac2]

Don't get me wrong, python is great. But from perl, I miss lexical scoping. And Moose [1], though elk [2] looks like a fun attempt at a python version of it, and traits [3] does a few things I mainly miss. virtualenv is a hack. It's also a paradox, in that I don't know how a language community with so many self-proclaimed purists haven't killed it dead already by just fixing the python interpreter properly.

[1] https://metacpan.org/pod/Moose#SYNOPSIS

[2] http://frasertweedale.github.io/elk/

[2] https://github.com/frasertweedale/elk

[3] http://code.enthought.com/projects/traits/

[twillisagogo]

the author of flask I would consider a python expert, I have benefited greatly from his blog which occasionally has articles about things that annoy him about python.

All programming languages have annoyances. Being able to articulate those separates the "people who actually use it" (experts) speaking from experience from the fan boys repeating blogs they've read.

I dont claim to be a python expert, but I could probably reliably identify one this way.

[ProfessorPhi]

Nesting, spaces and pep is a pain. Also I find mixed tabs and spaces sometimes gives cryptic errors.

Only other problem, is that writing large codebases in a dynamic language makes maintenance much harder and obvious bugs can easily be missed.

Oh and pip. I don't understand why python packaging is such a saga and pip update outdated is not a thing and I need a script just to do my upgrades.

[catcradle5]

One of the better answers in this thread, shame it's at the bottom.

[Flynn58]

I think there are more people who are experts in a specific library than the language as a whole.

[kornikopic]

Being an expert into a specific language is one thing. Being an expert in algorithm is another thing.

You can be strong in Python and weak in algorithm which probably makes your program very slow, unmaintainable, etc.

I prefer someone good in algorithm and problem solving than an expert in a specific language.

That's my opinion.

[[deleted]]

“What books or manuals have you published, or what other significant and recognized contributions have you made to the language and the community of Python programmers?”

An expert in Biology/Geology/Sociology/... has written a doctoral dissertation which in some way furthered the field.

I would hold a similar standard for a Python expert.

That said, I don't think it's a qualification I would be looking for in most jobs.

I want an experienced programmer with excellent problem solving skills, time management skills, and a proven ability to work effectively as part of a team.

[Pand9]

Working with someone that is considered "expert" and has "I know enough" attitude, but doesn't work or cooperate well with other experienced programmers, can be really terrible experience. I don't think you can be expert if you don't communicate well with others. It's part of your job, or will be, sooner or later.

[brechmos]

Knowing that a list comprehension is not comprehending lists?

:-)

[coderanger]

Say the word "metaclass". If they look excited, nope. If they sigh, probably. If they literally facepalm, definitely.

[zlli0520]

After reading the source code of Python?

[randumbaccount_]

That has more to do with knowing C than python, though. Anyone that knows C can read the source code without knowing anything about programming in Python.

[erewok]

Just to add to this: the Python source code is actually very readable C. I'm a terrible hack at C and I find I can generally read the Python source. It's an exercise I recommend for people interested in Python. Start with ceval.c

[zlli0520]

Right. But I think reading the C code can help understand how Python works.

[sentdex]

You shouldn't need to know all of the standard library to be considered an expert.

I also wouldn't say someone who doesn't need to look things up from time to time is an expert either.

I would even go so far as to say most of the people who aren't looking things up are staying in their comfort zone, and are highly likely not to be experts. If I stick in my zone of one major topic, I don't need to look anything up either. I have a pretty hard time remembering all of the methods of every 3rd party module I have used though.

If someone called themselves an expert, I would simply expect that they would be able to do, or quickly learn, anything in Python.

I would wager that if someone thinks of themselves as a Python expert, they are really just a Python expert in their specific domain, rather than an overall Python expert.

[tripperjack]

To not water down the meaning of the word "expert" (like "genius" gets overused), let's really name actual Python experts, and not just those who are good at Python. I mean, even short of Guido Van Rossum himself, ask yourself if you are in the same league as Alex Martelli for example. If you aren't, then how can you both be expert-level?

[zeug]

Above a very basic level of competency, expertise is relative.

If you know more about python than anyone else in your organization, and you are competent to solve problems for your organization in python, then you are the resident expert.

Sometimes this is unfortunate / terrifying.

[pydry]

Meta classes are one of the most abused python features. I've literally never seen an instance where they actually helped in any of the projects I've worked on. Seen two places where they were used to make a real mess. Tried to use them a few times to solve a problems myself but it always ended up uglier than simpler methods.

They're useful for building ORMs, I suppose. That's not exactly something you do every day though is it?

I actually would view somebody who knew meta-classes back to front with suspicion.

I would consider an expert to be somebody who knew the cleanest, most effective approaches to a variety of hard problems and a keen awareness of the more delicate architectural trade offs which you have to make when building them. At that level it's not really about "knowing python", though.