Python is annoying

2019-Feb-07, Thursday 09:38
mindstalk: (Default)
Our code works with binary data (hashes/digest) and hexstring representations of such data, a lot. It was written in Python 2, when everything was a string, but some strings were "beef" and some were "'\xbe\xef'"

Then we converted to Python 3, which introduced the 'bytes' type for binary data, and Unicode strings everywhere, which led to some type problems I had figured out, but a recent debugging session revealed I had to think about it some more. Basically we can now have a hexstring "beef", the bytes object b'\xbe\xef' described by that hexstring... and the bytes b"beef" which is the UTF-8 encoding of the string.

In particular, the function binascii.hexlify (aka binascii.b2a_hex) which we used a lot, changed what it returned.

Python 2:
>>> binascii.a2b_hex("beef")
>>> binascii.hexlify(_)

Python 3:
>>> binascii.a2b_hex("beef")
>>> binascii.hexlify(_)

>>> binascii.a2b_hex("beef")
>>> _.hex()

I found it easy to assume that if one of our functions was returning b"beef" and the other "beef" that they were on the same page, when really, not.

Bunch of examples in the cut.

Grah Python )
mindstalk: (Default)
Once again, into the Pyth circle of hell. This time trying to convert our code from Python 2 to 3. 2to3 does much fo the grunt work of print() and 'from . import', though it didn't always get the latter right. But instead of string and unicode, we now have string and bytes value types, and a strong barrier between them. And of course no static compiler to find up front when types might be mixed up. And yes, we're weak on unit tests, especially tests that exercise all possible code paths. Things seem to mostly work now, but will they under all conditions? Who knows?
mindstalk: (Default)
Speaking of aggravating values of fun... I foresaw a day of boring editing in my future? Well, I managed to make that half a day. I'd looked for programmatic solutions but they all seemed half-assed in bad ways, especially in breaking the logging information (filename, function name) we want. So I went through by hand... and a good mastery of vim commands and maps. It's funny, I haven't had to make a new vim map in forever, but the skills are there.

Samples: (Edit: all the [] should be angle brackets; I forgot how Dreamwidth swallows them.)

map CP :s/,/+/g #turns commma to plus, down a line
map KE cwkinnlog.error([Esc]A)[ESC]CP
#changes word under cursor to 'kinnlog.error(', appends ')' to line, invokes CP
map KX cwkinnlog.exception([Esc]A)[ESC]CP
map fp /print^M # fast search for 'print', since CP changes the current search to ','
map St istr(^[ #inserts 'str(' and goes back to command more.
map stt a)^[ #inserts ')' and goes back to command more
map kld cf(kinnlog.debug(^[ #changes everything from cursor to a '('

I got to show off more simply last week, too; I was setting up the product for Co-worker, and had to edit the IP address in a bunch of files.
$ vi *.json
/[IP 1]
n #search next
. #ditto
:wn #write, move to next file -- which preserves all the state, so I can keep going 'n' and '.'

She was impressed by my speed. (If you're wondering why not regexes, there were only 1 or 2 addresses per file, this seemed as fast or faster as recalling a ':%s/...')


All that said and done... Python makes it easy to write code. Python has an awesomely nice standard library, including a logger module with many handlers like RotatingFileHandler. Python makes it easy to rip out the guts of a stub logging module and replace it with something sophisticated.

Python 2 has a print statement which can take an indefinite list of things, and print them all, without any type information.

Python 2 does not provide any function that does something similar, so converting from those print statements to calls to actual logging functions was terribly painful and/or un-typesafe. I mean, a lot of the time I can't tell easily what the type of the value of a variable is where it's being used. Granted you can wrap almost anything in 'str()', and maybe I should have been more thorough about doing just that. Python doesn't have a compiler, and as long as it's basically syntactically correct, will be happy until it tries to actually execute the code. I really hate this.

Hoisting Shadows

2017-Feb-05, Sunday 11:35
mindstalk: (Default)
A bit after writing the previous post on shadowing variables in JavaScript, I came across this page on hoisting. JavaScript re-writes your code so that declarations but not initializations to the top of current scope, meaning the script or function[1]. So

var a=5;

turns into

var a;

Put that way, it's clear why the problem happens, if not why the language was designed this way.

Does the same thing happen in Python? Searching did not get clear results. I saw pages contrasting JavaScript with Python, which doesn't even *have* declarations. OTOH, the same behavior occurs. So... I dunno.

[1] JavaScipt does not have block scoping the way C and Perl[2] do; scopes are delimited by curly braces, but random curly braces do nothing to isolate variables.

{ a=5; }

will work just fine. :(

[2] As I was verifying this in Perl, I ran into behavior I'd forgotten. I'd thrown in "use strict;" but wasn't getting the errors I expected. Eventually I recalled that $a and $b have special meaning in Perl (I think for comparison functions), and I guess are pre-declared, and I was using $a a la the above code, so strict didn't complain about trying to access $a before assigning to it. Sigh.
mindstalk: (juggleface)
I spent longer than I expected figuring out the problem described in Robbie's Shadows of Python post. I kept revising my e-mail to him: "You're wrong because of X. No, I mean Y. Um, Z?" Eventually I got it, I think, and distilled it to a simpler piece of contrasting code:


def f():

def g():

>>> f()
>>> g()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 8, in g
UnboundLocalError: local variable 'x' referenced before assignment

Simply because we will (or even might, it works if the assignment is in an if clause) assign to x in the future, it becomes a local variable for the scope of the whole function, blocking access to the outer scope definition. This compile-time annoyance from the same language that has no equivalent to Perl's 'use strict;' to stop you from using misspelled variables. Thanks, Python.

Checking against the other big scripting language, Perl:


sub f {
    print $x, "\n";
sub g {
    print $x, "\n";


[Zefiris:0] perl

Happy access to a global variable. If you use 'my $x=2;' in g(), then the change to the global-level variable is avoided, of course.

Python would let you use 'global x' in g(), or 'nonlocal x' for intermediate scopes, but that gives you full access to x. There's no "I'm going to create a new local variable from this point on, while having used an outer variable earlier in the function." And this doesn't work:

def h():

You can do that in Perl, with or without 'my', getting expected behavior in each case.

This is part of the reason I'm willing to work in C++ again; yes, it's annoying, but I've seen no language that doesn't have mindbendingly stupid shit in it somewhere.

Edit: someday I'll remember I want the pre tag, not the code tag, for preserving indentation.

April 2019

141516 17181920

Expand Cut Tags

No cut tags

Style Credit

Most Popular Tags


RSS Atom