Rich and Strange Aeons

My phone was slow, so I rebooted it. And logged in again to my shell server. And got an error message about now finding 'zsh-syntax-highlighting/' I hadn't seen before. And my shell didn't have highlighting. I connected to the server's screen session and confirmed that yes, old shells have highlighting. Something's definitely wrong!

'ls', and I find 'zsh-syntnax/' [sic]. WTF? Why is that? Did something go horribly wrong with my last OS upgrade? Have I been hacked and pranked?

Fortunately I keep a really deep shell history, and found

'mv zsh-syntax-highlighting zsh-syntnax' from 9 days ago.

That answers how, but not who or why; I don't remember doing that!

But the history provides context, several other 'mv' commands renaming long filenames to shorter ones. Aha, I do remember doing that! I wanted 'ls' output to fit into more columns. So I did that to this directory without considering that some part of my shell configuration was actually using it. Whoops! And somewhat excusable, once my configuration is set up I don't have to twiddle with it much, I have aspects that are 25+ years old yet contributing positively to my experience.

One quick rename and edit of .zshrc to use the new name, and all is well again.

I learned a neat trick this week, which I think I can explain to non-technical readers.

The problem: say you want to have data on the cloud, to be shared or synced across multiple machines, like laptop and phone, new laptop and work phone, etc. Data like managed passwords or bookmarks or hell, emails. Data you would like to keep secret. What to do?

Approach 1: log into a cloud service provider with your passphrase and upload your data. This is terrible, they can see your data, and even if they're mostly trustworthy, a bad employee or a hacker could make off with your bank passwords.

Approach 2: log into the provider with your passphrase, and upload your data encrypted with the passphrase. This is barely any better; with standard login mechanisms, the provider sees your passphrase, even if they ideally don't store it in a visible form[1], and could trivially use it to decrypt your data and make off with your bank passwords.

Approach 3: encrypt your data with a separate passphrase, and upload that encrypted data when you log in. This is solidly secure (assuming you chose strong passphrases). Many geeks like me probably do a manual equivalent, encrypting files with gpg and copying them remotely.[2] Only problem is, you need to manage two passphrases.

Approach 4: This is what I learned, and is the approach of Firefox Sync. Good presentation, and more technically gory presentation. But I'll give my own description.

The key idea is that you don't log in with your passphrase. Instead, a credential is made *from* the passphrase, via a one-way hash. (One-way meaning the passphrase cannot be recovered from the hash.) That is what is sent to the server and used for log in. Which leaves your passphrase free to encrypt the data you upload, which is now safe because the provider never sees your passphrase. You don't have to trust them[2]; even if they broadcast your files to the world, ideally your data is safely encrypted. The provider only stores [(login credential), (passphrase-encrypted data)]. But any client you log in from can download the data and decrypt it with the passphrase you entered locally.

Put another way, the secret of your passphrase can be used to generate multiple secrets, for login and encryption, that don't generate each other. So you get the security of Approach 3 with the convenience of Approach 2.

On the flip side, if you forget and have to reset your passphrase, your encrypted data has to be thrown away; no one can recover it. That's not a big deal for Sync, especially as any device that has a copy of your data can then upload it again.

There's a bunch of complexity to the actual Firefox Sync process, but that's the fundamental insight.

***

One bit of complexity is straightforward. Approach 4 as I described it means that if you change your passphrase, your data has to be re-encrypted, which could get annoying. So instead have your client, at sign-up, generate a strong random data encryption key. Use that to encrypt your data, and encrypt the key with your passphrase. Now the provider stores [(login credential), (passphrase-encrypted key), (key-encrypted data)]. Your data is still secure since the provider never sees your passphrase or the actual key. But if you change your passphrase, only the small (login credential) and (passphrase-encrypted key) have to change, not the arbitrarily large (key-encrypted data).

***

Other bits of complexity are less interesting, or even baffling. Firefox actually encrypts your data key not with the passphrase, but with another secret derived from the passphrase. There are long steps that make brute force attacks less feasible. The key is protected with XOR rather than some fancier encryption mechanism. And weirdest of all, instead of your browser generating a data key and sending the encrypted key to the server, what happens is that the server makes up some random number ( wrap(wrap(kB)) in the Firefox writeups ) from which the client derives the key. The math works out but it's a reversal of expected flow. My best guess is that they feel they can make better random numbers than the client, which might be true if they have good hardware randomness generators one their servers.

(Though I'm not sure if it really matters; seems like they could assign '0000...' to everyone, and the data keys would still differ based on people's passphrases.)

****

[1] Passwords are supposed to be stored salted (add some non-secret extra stuff to defeat various attacks) and digested/hashed. So someone who steals the password file can't see the actual passwords. Login means you present your password, it's salted and digested and compared to what's in the file, then your password is thrown away. Whether any particular Internet site follows that protocol is another matter, which is one reason you're told not to re-use passwords across sites.

[2] In theory open-source clients can be examined, so that they can be fully trusted. How many of us do that, though? Not me... My gut feels that a single-purpose program like gpg is easier to secure, to make sure it's never doing anything like opening a network connection while decrypted my data, compared to some client or web browser that does everything, so it would be harder to make sure it was only opening the right network connections and not secretly sending your data somewhere.

When I was in college, I participated in vi vs. emacs editor wars. I don't recall how serious they were. My good reasons were that I was pulled into vi first, and emacs was slower to start up. At some point I'd definitely matured to finding them less interesting, and granting that emacs was probably at least as powerful as vi, or even the later vim, but I had little reason to switch.

Recently a Windows-using friend was snarking at vim as well as at git, and a bunch of good-natured ribbing ensued, but I wandered off into serious musings:

* To be honest, I'm not in a good position to say vim is 'better' than other editors, because I don't know other editors well!

* That said, vim has a lot of word-processor-like and other features which I'm fairly sure casual text editors don't. To whit:

o Macros: the ability to define or even record sequences of commands or text. I've had macros that invoked other macros.

o History: not just an unbounded undo history of text changes, but a memory for the last command and the ability to trivially repeat it, as well as a history of :commands such as replaces. I think vim also has a notion of history as a *tree*, though I've never grokked that. But the "do the last edit I did"? I use that one all the time.

o Formatting: separate commands for reformatting ordinary text and code.

o Movement: sophisticated movement commands, useful not just for moving your cursor, but for applying edit commands to (so, "delete 3 sentences", or "replace 5 paragraphs"), or (in vim) for selecting text. You could use a mouse for similar selections, but often the keyboard commands would be faster and take less arm movement.

o Marks: you can make bookmarks within files to easily jump between points, and these marks can persist across multiple files.

o IDE-like features: syntax highlighting, smart indentation, bouncing between brackets, more. A dedicated IDE is probably even better -- but if you use multiple languages, having one decent interface across them can beat having to use multiple interfaces, especially if the IDE is less sophisticated as an editor.

* Registers: if you want, you can suck text into up to 27 different clipboards.

o Completion that defaults to using the current file as a dictionary, but that can take arbitrary other dictionaries.

o And for those who can't stand funky movement keys in command mode, vim has long recognized arrow keys in editing mode, and there's a graphical vim that probably recognizes mice.

Some practical cases:

* Having to edit an IP address in lots of files:
vi *.json #edit multiple files
/192 #search for old IP
cW{new IP} #change word to new one
n. #search for next instance, and repeat last action
n. #ditto
:wn #write this file, go to the next file
n. #continue the search and replace
n.
:wn
n. #...etc, for like a dozen files

* I've had to edit the etc/hosts file on my Windows machine between two different states. Notepad++ makes a single search and replace fairly easy, and remembers the last one done, but not the last two, and the time between changes was long enough that I wasn't keeping it open to exploit undo/redo. vim could easily remember two different replace commands to invoke.

(An alternative would have been to keep two different files around, but Windows GUI doesn't make copy-and-rename that easy either, compared to a command line.)

So how does all this compare to a good Windows text editor like Notepad++? Looking at it, it does have lots of features I haven't used. There's a Macro menu, via Recording, so that's promising -- though I wonder what its notion of commands can be. The Search menu has a couple of bracket options (find matching, select to matching), though using the % key in vi would definitely be faster; also has Bookmark. Some options I doubt vim has, like 'sort lines'. (Checks: I was wrong, vim does have it.)

Overall it's a decent set of features, and for the casual or new user, the menus will be more discoverable than reading through vim documentation. But I doubt, not just from gut feeling but from comparing myself to my boss and his IDE use, that it could ever be as fast as vim mastery. Between vim and 'screen' (for switching between windows, and for cut and paste), my hands typically never have to leave the alphanumeric keyboard, not even as far as the laptop touchpad or function keys, during coding. And personally, the lack of command history would be very nearly crippling.

I was going to give Notepad++ an advantage on spellchecking, but I looked, and hey, vim has an option for that too:
:setlocal spell spelllang=en_us

And though I've never used it, there is a GUI version of vim, with at least some menus. I just found a screenshot where its Edit menu has Undo, Redo, and Repeat, and a university guide says

"Next to every menu item is a keyboard shortcut to execute that function. gVim allows a new user to easily find common functions through the menu bar, but also gives the user the keystrokes that access the function faster, and are compatible with vim and usually vi." So even the discoverability advantage of Notepad++ fades away: between it and gvim, we have two fairly powerful editors, except that the latter allows everything to be done with the keyboard and has a potent command history.

Huh. When I started this post, I was going to avoid "my editor is better", and stick to "my editor does these cool things." And then I was checking Notepad++ features and being impressed by what it had (I hadn't known it had macros). But now I seem to have argued myself into "vim, or at least gvim, quite plausibly *is* better: it can do everything[1] yours can, *and* these things unique to it (and maybe emacs.)"

[1] Not everything a great IDE can do, but I'm fine with that; I don't need my editor to compile code for me, there's a command line for that.

Edit: All that said, I just installed gvim, and its menus do seem less complete than Notepad++'s. In particular, while Repeat is there, I don't see any menu options for creating macros, plus many of the other things N++ makes available, like "sort lines". (On the IDE front, it *does* have "make" and stuff like "find next error".) I know that you can create new menus for gvim, and maybe someone has created more comprehensive ones, but a newbie won't know that. And while there's online help, it's vimmy in navigation. So I have to give newbie friendliness back to Notepad++.

Edit 2: After some further thought and reading, I'll posit that for fast and flexible general text editing, nothing beats vim mastery, not even emacs. For general programmability beyond being an editor, nothing beats emacs. For integration with some specific language, hopefully an IDE gives features that even IDE-like features of vim don't... but it probably is inferior as a text editor, which makes it a matter of tradeoffs: what do you want to optimize for? And how much do you want to be tied to a language-specific tool?

At work today, Boss suggested I look at sqlite a bit, since our client code uses it. What I thought might be a brief glance turned into hours of reading, as it became rather fascinating. For those who don't know, it's an embedded SQL database, with not much code, unlike the client/server databases of Oracle or anything else you've probably heard of. As their docs put it, they're not competing with such databases, they're competing with fopen() and other filesystem access.

They call their testing "aviation grade", possibly without hyperbole: 100% branch coverage, 100% coverage of something stronger than branches, 700x more testing code than actual library code and a lot of that generates tests parametrically... it sounds pretty nuts. They worship Valgrind but find compiler warnings somewhat useless; getting warnings to zero added more bugs than it solved. https://www.sqlite.org/testing.html

They claim "billions and billions of deployments", which sounded like humorous hyperbole until they added being on every iPhone or Android phone, every Mac or Windows 10 machine, every major browser install... There are over 2 billon smartphones, so just from the phone OS and the phone browser, you've got 4 billion installs...

They also make a pitched case for consider a sqlite database any time you'd be considering some complex file format. With almost no code to write, you'd get consistency robustness, complex queries, machine and language independence, and at least some ability to do partial writes[1], compared to throwing a bunch of files into a zipfile.

https://www.sqlite.org/appfileformat.html
https://www.sqlite.org/affcase1.html

They also had a nicely educational description of their rollback and write-ahead models. https://www.sqlite.org/atomiccommit.html
https://www.sqlite.org/wal.html

[1] I do wonder about this. One odd thing about sqlite is a looseness about types, and AIUI cramming numeric values into the smallest range that will hold them. So I'd think that if you UPDATED a value 100 to a value 1000000000000, you'd have to shuffle the trailing part of the file, compared to a format that e.g. reserved 8 bytes for a numeric type. But maybe they do buffer numeric or string storage. And not having to write the whole file, or not having to read the whole file (e.g. to decompress it) seem like at least partial wins.

Firefox 52 dropped support for ALSA systems. Arch Linux users were insulated from this; something like the code was still there but not enabled by default, but it was in the Arch package. As of FF 54 though, poof, it's gone for good. I'm not sure exactly why Pulseaudio is avoided, but I'm still avoiding it... so need another browser.

There are lots, actually! Currently trying Seamonkey on the work VM, mostly because it's the one alternative which is both based on the same engine as FF (so familiar, and plugins should work) while having a supported package on Arch. There are a couple more, including Pale Moon, but they need the AUR, and I'm lazy.

I think I mentioned not long ago that I found I'd been aliasing fi=finger which breaks if loops in my shell, and marveled that it took so long to find that. It makes more sense to me now.

1) Yeah, I didn't script much.
2) When I did do an ad hoc script at the prompt, it was a for loop.
3) Scripts you get are mostly bash scripts.
4) Even an explicitly written zsh script wouldn't have a problem: my aliases are loaded by .zshrc, which is loaded by interactive shells, i.e. not script shells[1].
5) Only when I tried pasting an if loop into a *function*, also loaded by .zshrc after my aliases, did a problem occur. Possibly it had occurred before and I simply gave up on some unnecessary function that mysteriously didn't work.

[1] This also sheds light on past failures to ssh in somewhere and invoke a function directly: not an interactive shell, so no functions loaded. When I try 'ssh ... "zsh -i script_invoking_function"', it works. So if I want remote function invocation, I'll need to use -i or to load functions outside of .zshrc.

My boss apparently figured out the problem with the VMware clone: fakes3 (for faking a local Amazon S3 service) apparently behaves badly if given the local hostname (or just localhost?) rather than an IP address, and I'd probably edited the clones files to use a name because why wouldn't you.

And I tackled my VBox again, and got shared directories working! I'd found a different set of instructions, which worked for manual mount, and then even automount from fstab. Going back, I reproduced the error I'd gotten from the Arch instructions: I'd been using '/vmshare /vmshare' (Window and guest locations, a la VMware FUSE command syntax) when it actually wanted 'vmwshare /vmshare' (short name of the shared directory in VBox, guest location). "Protocol error" is a pretty terrible error message, but I can see now what I was doing wrong.

I also found VBox's Seamless Mode, which I don't quite see the deep point of yet, though it does reclaim screen space from the Windows title bar while still leaving the start/monitor bar at the bottom, but it's allegedly similar to VMware Unity Mode, except Unity says it doesn't support Linux Guests.

So VirtualBox seems strictly better than VMware in features, since it does everything one would want, while VMware doesn't do Unity or (more important to me) touchpad scrolling. OTOH we probably have the VMware clone at least behaving the way we expected it to. Though I'm not sure this was actually tested on it, I think it's a prognosis based on fixing something on the Ubuntu machine.

Also VBox is open source and doesn't charge you money for running more than one VM at once. OTOH we already paid VMware the money.

I haven't tried comparing performance.

LXDE was happy starting from startx, but it doesn't have a way to configure move-on-focus. Searching about "you can't" and "apply this 100 line XML file somewhere". So I moved on to xfce -- the full thing, not just xfwm. It's not happy starting from startx/.xinitrc, or I'm doing something wrong, but it provides its own startxfce4. It resizes, doesn't crash, and had a single step for turning on the One True Mouse behavior. Was also able to configure my usual X keys (I have some function keys mapped to window raise/lower/minimize). Plus it's supposed to be fairly light weight. So that's where I'm at for now. Haven't explored it much since; if it lets me move my windows around, I'm good. (I'm used twm after all, which pretty much is nothing but that.)

On the downside, touchpad scroll still doesn't work. This may be a VMware problem. I was working on a VirtualBox image, but it didn't go smoothly. First I tried export/import, which didn't seem to work -- frozen boot screen. But when I went to Close it, the proper display flashed up, and I've been able to find that it does boot and have my account. But it's not *usable*. Possibly X would fix that. I also tried a pure install, but after applying some tweak to make console resize, it didn't want to boot at all. I haven't had time to go back and try pure vanilla. And the VMware image I'm working with is getting more and more developed, it may be hard to switch.

I also found a program to make *Windows* use move-on-focus. I should probably tell you what it was, but I don't remember, and the info seems to be only on the office machine. But it's SO useful, at least for my workflow which uses overlapping windows a lot. (Often a full-screened browser or VM (or browser in VM) and some other window I'm taking notes in.)

OTOH I really wish I could make *Windows* raise and lower windows with a key.

New co-worker coming Monday, she'll need a VM too. Rather than re-installing, I simply cloned mine. Easy! And purged my limited personal info on it, a bit more work. And got the system working on it... that was a lot more work, we've got too much hardwiring of local IP address. Which will interfere with putting code in source control too, so we've got a double incentive to fix that.

My phone (Android 4, CM 11) swipe input is weird when it comes to profanity. So, there are three levels: its first guess for your word, two alternates to the side, and then a list you can bring up. 'suck', 'sucking', 'shit', and 'dick' will never appear in the first two levels, even if I go slowly and letter by letter -- 'shi' turns into 'shot', with 'shirt' and 'shoot' as alternates. Peter thinks it's a probability weight thing; I figure it's hardwired, with a short list of words being simply barred from your being able to input them too quickly (i.e. accidentally.)

But 'fuck', 'fucking', and 'fucker' I can enter quite easily. 'cunt' too. And 'pussy', with a bit of care (it's hard to swipe a double letter.)

Peter thinks it's also probabilities weighted by how you've used the phone, but I use 'suck' and 'sucking' in texts far more than any of the others... I've probably never tried to swipe 'cunt' before.

My thought for a while was that maybe fuck* weren't in the dictionary at all, until I added them, so wouldn't be barred, but I checked my personal word list and nope, they're not there.

So I dunno. Maybe it's more of an "accidentally unprofessional" filter, like words on the edge of acceptability are barred so you don't tell your boss how much something sucks, but the designers figured if you want to go full vulgar you knew what you were doing. Not how I'd do things.... and no, I don't see a profanity filter I might have turned off.

Wait, I'm wrong! I just checked again, and yes, Android Keyboard has a "block offensive words" option which is off, so I probably did that at some point... and I'm *still* not getting 'suck' or 'shit' as choices above the third level.

***

Totally unrelatedly, I finally have Japanese input working on my laptop! 日本語よ! I'd tried UIM a while back, per the Arch Linux default recommendation, but it didn't work. Then I tried SCIM a few days ago, and it seemed to not work either, but later I found myself typing in Japanese suddenly. Ctrl-Space turns it on, Ctrl-Shift cycles through modes (Anthy [Japanese], Unicode, English/European [which doesn't seem to do anything]. I should look into configuring that, because I need Ctrl-Shift-C and -V to copy and past from/to my Terminator terminal, so there's an annoying conflict there. Still, woo!

Is this more than a toy, given my weak Japanese skills? Slightly: online dictionaries tend to work best with Japanese input, not romaji, so now I can actually use them. And I've started studying it again, so that helps.

most: the pager sequence now goes more, less, most. most's big thing seems to be displaying multiple windows. It's also good on scrolling sideways, but I just found less is too, so I'm not sure there's a difference there. OTOH, I just learned less can scroll sideways.

rlwrap: applies a readline wrapper to interactive programs that don't use readline directly, like ocaml or 'perl -de 1'. If you use rlwrap -m, then ^^ summons an editor on your input.

man 7: there's a whole lots of odd information there.

.inputrc lines for better history:
"\e[A" history-search-backward
"\e[B" history-search-forward
searches (up-arrow) with what you typed so far as a prefix. There's also ^R, which you type first, followed by what you're looking for, and use ^R again to search other matches.

I recall that years ago, zsh searched on prefix. Then for a long time it had gone to only searching on the first word. I finally got the behavior I wanted back, though it's more involved.
.zshrc:
(* edit: these lines don't actually work for me. I thought I did but I must have tested in a shell with the other ones already loaded, not in a fresh shell.
autoload -Uz up-line-or-beginning-search down-line-or-beginning-search
zle -N up-line-or-beginning-search
zle -N down-line-or-beginning-search
[[ -n "${key[Up]}" ]] && bindkey "${key[Up]}" up-line-or-beginning-search
[[ -n "${key[Down]}" ]] && bindkey "${key[Down]}" down-line-or-beginning-search
*)

or
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
zle -N history-beginning-search-forward-end history-search-end
bindkey "^[[A" history-beginning-search-backward-end
bindkey "^[[B" history-beginning-search-forward-end

I don't know the difference between the two, if anyway. The -end stuff in the second case is to make it move the cursor to the end; otherwise it just leaves it where you left it, which I hate.

zsh tricks:
ls > file1 > file2, or ls > file1 | file 2. Duplicates the output. More compact than messing with tee.

I just spent an embarrassing number of seconds trying to see if "ls | cat | cat" would "duplicate output", before I remembered what piping *does*.

You can set zsh options so tab completion lets you scroll around the choices. My mind is blown. I'm not sure what the minimal set needed is.
setopt auto_menu auto_list
seems like a good starting point. But I'd just re-started the configuration wizard and turned almost everything on and stuff started being cooler.

I played with the shell fish ("friendly interactive shell") again. I'll probably never leave zsh at this point, but fish does lots of neat things out of the box, vs. having to turn them on in zsh via research or going through the startup wizard. I think zsh's completions are more powerful, but it's a close race.

[Edit: hmm, I just found that for ocaml, zsh doesn't provide anything, but fish does. I'd guess fish is doing its parsing of man pages thing, rather than knowing about ocaml from installation.]

kill completions:
fish:

phoenix@mindstalk ~/zoot> kill 
1            (systemd)  1430           (bioset)  1985  (systemd-journal)
2           (kthreadd)  1431           (bioset)  1990          (kauditd)
3        (ksoftirqd/0)  1432           (bioset)  2155    (systemd-udevd)
5       (kworker/0:0H)  1466           (bioset)  3048            (crond)
…and 46 more rows

zsh:

[mindstalk:0] kill 3360
 3360 pts/3    00:00:04 zsh                                                    
18729 pts/3    00:00:00 zsh                                                    
18730 pts/3    00:00:00 ps 

bash (with bash-completion package):

[phoenix@mindstalk zoot]$ kill 
Display all 151 possibilities? (y or n)
1      1422   1474   1499   15636  18739  2      3048   3337   491    779
10     1423   1477   1500   16300  18740  2155   3049   3338   5      780
10445  1424   1480   1501   16972  18741  23548  3050   3353   637    782

(and lots more PIDs).

As you can see, fish and zsh try to give you useful information, or at least a name. zsh seems limited to the tty process, which bash lists all the processes, regardless of whether you can kill them. fish also lists every process. No doubt zsh completion could be configured to do so as well. (I'd actually want just listing all of my processes, not like I can kill root ones.)

Likewise, bash's idea of command option completion is to just list them; the other two shells give descriptions. (What I really learned tonight was that bash does such advanced completion at all.)

sudo !!
A very old simple trick for when you try to do something but it needs sudo. If I'd known about !! I'd forgotten until today, though I knew about !num to get at a specific history command.