This is another case of a udev rule that took over an hour to figure out, and now seems very clear and obvious. Which probably means it won't next time I need it.

I have a bunch of USB to serial adapters, all using pl2303, all so cheap that there's no product ID or any way to tell them apart. This is cheaper than buying actual console server hardware nowadays. So how to know which ttyUSB* device is hooked up to what? The solution is to use the USB bus ID, which corresponds to the usb port the device is plugged into. My /etc/udev/rules.d/z25_persistent_usb_serial.rules:

SUBSYSTEMS=="usb", ID=="3-2", KERNEL=="ttyUSB*", NAME="ttyUSB_front_upper"
SUBSYSTEMS=="usb", ID=="2-2", KERNEL=="ttyUSB*", NAME="ttyUSB_front_lower"

The ID numbers can be found in dmesg.