Python Performance: Why 'if not list' is 2x Faster Than Using len()
Python Performance: Why 'if not list' is 2x Faster Than Using len()
blog.codingconfessions.com
Python Performance: Why 'if not list' is 2x Faster Than Using len()
53 comments
-
I don't know about others ... but I'm not using Python for execution speed.
Typically the biggest problem in a program is not 100 million calls of
len(x) == 0. If it was, the interpreter could just translate that expression during parsing.-
I mean, nobody uses Python for execution speed precisely because it is so slow.
-
Even so,
not xis a pretty nice-to-read pattern, and it's nice that it's faster than the less nicelen(x) == 0. I generally do not care to distinguish whether a list isNoneor empty, I just want to know if there's something there. If I care, then I'll typically separate those checks anyway:if x is None: ...,if len(x) == 0: ....-
I prefer the clarity of
len(x) == 0personally; it's a pretty low syntactic penalty to clarify intent.Obviously, if your variable names are good and you're consistent about your usage
not listcan be fine too, but it is a very JavaScript-y thing to me to be that vague and/or rely on "truthiness."-
The notion of truthiness is defined by the language.
Here are most of the built-in objects considered false:
- constants defined to be false: None and False
- zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)
- empty sequences and collections: '', (), [], {}, set(), range(0)
It's not something that happens to work, it's literally defined that way.
if not xis the common way to tell if you have data or not, and in most cases, the difference betweenNoneand empty data ([],{}, etc) isn't important.len(x) == 0will raise an exception if you give itNone, and that's usually not what you want. So I guess the verbose way to do that isif x is None or len(x) == 0:, but that's exactly equivalent toif not x, with the small caveat that it doesn't check if the value has__len__implemented. If you're getting wonky types thrown around (i.e. getting a class instance when expecting a list), you have bigger problems.I use type hinting on pretty much all "public" methods and functions, and many of my "private" methods and functions as well. As such, sending the wrong types is incredibly unlikely, so
not xis more than sufficient and clearly indicates intent (do I have data?).-
I did not say it's not semantically well defined.
https://en.wikipedia.org/wiki/Brainfuck#Hello_World! -- this is semantically well defined, but it's still vague. Vagueness is a property of how well the syntax is conveying intent.
-
It's only vague if coming from a language where it's invalid or vague semantically. For example:
- Javascript -
[]is truthy for whatever reason - C -
int x[] = {};evaluates to true because it's a pointer; C only evaluates to false if something is 0 - Rust - invalid because you cannot convert a vec -> bool directly, and there's no concept of null (same w/ Go, but Go has nil, but requires explicit checks)
- Lua - empty tables, zero, and empty strings are truthy; basically, it's truthy unless it's
nilorfalse
The only surprising one here is Javascript. I argue Lua and Python make sense for the same reason, Lua just decided to evaluate truthiness based on whether the variable is set, whereas Python decided to evaluate it based on the contents of the variable. I prefer the Python approach here, but I prefer Lua as a language generally (love the simplicity).
- Javascript -
-
-
-
-
The interpreter can't make the replacement until it's about to execute the line as
__bool__and__len__are both (Python's equivalent of) virtual functions, so it's got to know the runtime type to know if the substitution is valid. Once it's that late, it'll often be faster to execute the requested function than work out if it could execute something faster instead. Even with type hints, it's possible that a subclass with overridden methods could be passed in, so it's not safe to do anything until the real runtime type is known.Once there's a JIT involved, there's an opportunity to detect the common types passed to a function and call specialised implementations, but I don't think Python's JIT is clever enough for this. LuaJIT definitely does this kind of optimisation, though.
-
Hm... I'll admit I wasn't awkward of the
.__len__function.However, does this mean it's possible to write alen(x) == 0that's diverges fromnot x?If not, then the substitution is still valid (andnotpresumably also considers the same fundamental. If so, that's kind of silly.EDIT: I missed the part of your comment about
.__bool__... so yeah in theory you could have something where these two operations are not equivalent.Arguably, you could just say that's pathological and invalid. Then, still have an optimized path to prefer
not .__bool__()if.__len__() == 0is the comparison you'd be making. Even with the extra interpreter check during evaluation, that would quite possibly be faster if the overhead is truly high.EDIT 2: you'd probably need a little bit more overhead than a straight substitution anyways because you need to maintain the semantic of "if this thing is
Noneit's not valid if the syntax was originallylen(x)."
-
-
This. I rarely boot up Python for the tasks I need to do, and if they are, they are one of the following:
- Assistant code for other coding language
- Throwaway script
- Prototype before using a faster language
-
Assuming an equivalent package is produced, what's the maintenance cost (factoring in coder availability) difference between the Python vs faster language implementations?
^^ therein lies the rub
Reminds of the expression,
premature optimization is the root of all evilif not swimming in funding, might be a darwinic move to choose the faster language and have to code everything yourself from scratch
-
53 comments