Rich and Strange Aeons

python JSON datetime at last

Thu, 26 Aug 2021 16:47:56 GMT

content warning: geeky computer stuff

At work I have a common flow of Python code querying MySQL and dumping results into JSON. This works great except our tables tend to have columns for create and modify dates. And Python's json library won't convert datetime types to string, so dumping the result of "SELECT * FROM" errors. And I've had bits of ad hoc code to remote datetime values, then a little function for it, but it was still a pain. Especially today, when for stupid VPC reasons I'm writing a simple generic SQL service.

In annoyance I tried searching again and found, buried in the middle of a stackoverflow with answers about subclassing and shit:

json.dumps(thing, default=str)

Bam, done. The function passed to 'default' gets invoked on anything that errors at first. And in Python 'str' works on pretty much anything.

If I cared about the exact format I could write a function that inspected type and converted dates (or just assume datetime is the only cause of errors so just invoke a method when triggered), but str is fine for now.

Soooo simple. Thanks, python, and stackoverflow person who gave the simplest answer.

comments

my first ever Python bug

Sun, 14 Jun 2020 04:01:28 GMT

Years and years ago, when Perl was still the dominant scripting language, with Python and Ruby nipping at its heels, and CPAN was where you would look for cool libraries, I wrote my first Python program. I don't recall what it did, whether it was meant to do something useful or just do something like "10 9 1 blastoff" as a test, but I do know that it looped based on a command line parameter. And that it didn't work, running nigh-forever when it had a parameter of '20' and should stop after 20 times. It took me like 20 minutes to figure out what was wrong.

I can replicate the basic problem:

import sys
count = 0
while count < sys.argv[1]:
    print(count)
    count += 1
else:
    print("Blastoff!")

And the behavior still replicates -- if you run it in Python 2. In Python 3, you'll get

TypeError: '<' not supported between instances of 'int' and 'str'

Because sys.argv is a list of strings. Thing is, Perl is a very accommodating language; the equivalent code in Perl will just work, as I well knew. Or rather, Perl has separate comparison operators for numeric and string values, so this code, using '<', would quietly try to convert sys.argv[1] into a number for comparison with 'count'. (If you wanted lexicographic comparison, you would use 'count lt sys.argv[1]', except of course it would be more like '$count lt $argv[1]' or something; anyway, there the value of $count would be converted to a string for comparison.)

I remember being very put out. Though I did go write a 'real' script in Python, and showed it to my boss, who was able to read and understand it with no Python knowledge. Go go executable pseudocode.

These days, on the rare occasions I poke at Perl, I'm more likely to be annoyed by the lack of a REPL, or easy ways to print nested data structures, or the quirky way function arguments are handled. Perl still does some things better ('use strict;') but I don't miss it.

But dang, that bug.

comments

wtf Python

Mon, 13 Apr 2020 03:24:25 GMT

I was writing some code, in a file called 'code.py'. I tried adding doctests and running them; this would hang, or complain about circular imports. It seems that 'import doctest' will run a file in the directory called 'code.py' -- e.g. make a 'code.py' file like

while True:
    pass

start python, import doctest, and watch it hang.

This seems poor, and undocumented. Probably I should file a bug later.

I guess I have a new candidate for interview questions about problem solving.

comments

Python is annoying

Thu, 07 Feb 2019 14:53:14 GMT

Our code works with binary data (hashes/digest) and hexstring representations of such data, a lot. It was written in Python 2, when everything was a string, but some strings were "beef" and some were "'\xbe\xef'"

Then we converted to Python 3, which introduced the 'bytes' type for binary data, and Unicode strings everywhere, which led to some type problems I had figured out, but a recent debugging session revealed I had to think about it some more. Basically we can now have a hexstring "beef", the bytes object b'\xbe\xef' described by that hexstring... and the bytes b"beef" which is the UTF-8 encoding of the string.

In particular, the function binascii.hexlify (aka binascii.b2a_hex) which we used a lot, changed what it returned.

Python 2:
>>> binascii.a2b_hex("beef")
'\xbe\xef'
>>> binascii.hexlify(_)
'beef'

Python 3:
>>> binascii.a2b_hex("beef")
b'\xbe\xef'
>>> binascii.hexlify(_)
b'beef'

vs.
>>> binascii.a2b_hex("beef")
b'\xbe\xef'
>>> _.hex()
'beef'

I found it easy to assume that if one of our functions was returning b"beef" and the other "beef" that they were on the same page, when really, not.

Bunch of examples in the cut.

( Grah Python )

comments

refactoring $#&# Python

Fri, 23 Mar 2018 02:04:09 GMT

Once again, into the Pyth circle of hell. This time trying to convert our code from Python 2 to 3. 2to3 does much fo the grunt work of print() and 'from . import', though it didn't always get the latter right. But instead of string and unicode, we now have string and bytes value types, and a strong barrier between them. And of course no static compiler to find up front when types might be mixed up. And yes, we're weak on unit tests, especially tests that exercise all possible code paths. Things seem to mostly work now, but will they under all conditions? Who knows?

comments

vigorous vim; frigging Python

Wed, 12 Apr 2017 01:41:31 GMT

Speaking of aggravating values of fun... I foresaw a day of boring editing in my future? Well, I managed to make that half a day. I'd looked for programmatic solutions but they all seemed half-assed in bad ways, especially in breaking the logging information (filename, function name) we want. So I went through by hand... and a good mastery of vim commands and maps. It's funny, I haven't had to make a new vim map in forever, but the skills are there.

Samples: (Edit: all the [] should be angle brackets; I forgot how Dreamwidth swallows them.)

map CP :s/,/+/g #turns commma to plus, down a line
map KE cwkinnlog.error([Esc]A)[ESC]CP
#changes word under cursor to 'kinnlog.error(', appends ')' to line, invokes CP
map KX cwkinnlog.exception([Esc]A)[ESC]CP
map fp /print^M # fast search for 'print', since CP changes the current search to ','
map St istr(^[ #inserts 'str(' and goes back to command more.
map stt a)^[ #inserts ')' and goes back to command more
map kld cf(kinnlog.debug(^[ #changes everything from cursor to a '('

I got to show off more simply last week, too; I was setting up the product for Co-worker, and had to edit the IP address in a bunch of files.
$ vi *.json
/[IP 1]
C[IP2]",
n #search next
. #ditto
:wn #write, move to next file -- which preserves all the state, so I can keep going 'n' and '.'

She was impressed by my speed. (If you're wondering why not regexes, there were only 1 or 2 addresses per file, this seemed as fast or faster as recalling a ':%s/...')

*******

All that said and done... Python makes it easy to write code. Python has an awesomely nice standard library, including a logger module with many handlers like RotatingFileHandler. Python makes it easy to rip out the guts of a stub logging module and replace it with something sophisticated.

Python 2 has a print statement which can take an indefinite list of things, and print them all, without any type information.

Python 2 does not provide any function that does something similar, so converting from those print statements to calls to actual logging functions was terribly painful and/or un-typesafe. I mean, a lot of the time I can't tell easily what the type of the value of a variable is where it's being used. Granted you can wrap almost anything in 'str()', and maybe I should have been more thorough about doing just that. Python doesn't have a compiler, and as long as it's basically syntactically correct, will be happy until it tries to actually execute the code. I really hate this.

comments

Hoisting Shadows

Sun, 05 Feb 2017 16:49:15 GMT

A bit after writing the previous post on shadowing variables in JavaScript, I came across this page on hoisting. JavaScript re-writes your code so that declarations but not initializations to the top of current scope, meaning the script or function[1]. So

console.log(a);
var a=5;

turns into

var a;
console.log(a);
a=5;

Put that way, it's clear why the problem happens, if not why the language was designed this way.

Does the same thing happen in Python? Searching did not get clear results. I saw pages contrasting JavaScript with Python, which doesn't even *have* declarations. OTOH, the same behavior occurs. So... I dunno.

[1] JavaScipt does not have block scoping the way C and Perl[2] do; scopes are delimited by curly braces, but random curly braces do nothing to isolate variables.

{ a=5; }
console.log(a);

will work just fine. :(

[2] As I was verifying this in Perl, I ran into behavior I'd forgotten. I'd thrown in "use strict;" but wasn't getting the errors I expected. Eventually I recalled that $a and $b have special meaning in Perl (I think for comparison functions), and I guess are pre-declared, and I was using $a a la the above code, so strict didn't complain about trying to access $a before assigning to it. Sigh.

comments

Simplified Shadows of Python

Mon, 08 Aug 2016 21:01:55 GMT

I spent longer than I expected figuring out the problem described in Robbie's Shadows of Python post. I kept revising my e-mail to him: "You're wrong because of X. No, I mean Y. Um, Z?" Eventually I got it, I think, and distilled it to a simpler piece of contrasting code:

x=1

def f():
    print(x)

def g():
    print(x)
    x=2

>>> f()
1
>>> g()
Traceback (most recent call last):
  File "", line 1, in 
  File "r.py", line 8, in g
    print(x)
UnboundLocalError: local variable 'x' referenced before assignment

Simply because we will (or even might, it works if the assignment is in an if clause) assign to x in the future, it becomes a local variable for the scope of the whole function, blocking access to the outer scope definition. This compile-time annoyance from the same language that has no equivalent to Perl's 'use strict;' to stop you from using misspelled variables. Thanks, Python.

Checking against the other big scripting language, Perl:

$x=1;

sub f {
    print $x, "\n";
}
sub g {
    print $x, "\n";
    $x=2;
}

f();
g();
f();

[Zefiris:0] perl r.pl
1
1
2

Happy access to a global variable. If you use 'my $x=2;' in g(), then the change to the global-level variable is avoided, of course.

Python would let you use 'global x' in g(), or 'nonlocal x' for intermediate scopes, but that gives you full access to x. There's no "I'm going to create a new local variable from this point on, while having used an outer variable earlier in the function." And this doesn't work:

x=1
def h():
    x=x
    print(x)
h()

You can do that in Perl, with or without 'my', getting expected behavior in each case.

This is part of the reason I'm willing to work in C++ again; yes, it's annoying, but I've seen no language that doesn't have mindbendingly stupid shit in it somewhere.

Edit: someday I'll remember I want the pre tag, not the code tag, for preserving indentation.

comments