Could ASLR be used to prevent the Spectre attack?

The way Spectre mitigations are shaping up, it's going to require modification of every program that deals with sensitive data, inserting serialization instructions in the right places. Or programs can be compiled with all branch prediction disabled, with more of a speed hit.

Either way, that's going to be piecemeal and error-prone. We'll be stuck with a new class of vulnerabilities for a long time. Perhaps good news for the security industry, but it's going to become as tediously bad as buffer overflows for the rest of us.

Also, so far the mitigations being developed for Spectre only cover branching, but the Spectre paper also suggests the attack can be used in the absence of branches to eg determine the contents of registers, as long as the attacker knows the address of suitable instructions to leverage.

So I wonder if a broader mitigation can be developed, if only to provide some defense in depth from Spectre.

The paper on Spectre has this to say about ASLR:

The algorithm that tracks and matches jump histories appears to use only the low bits of the virtual address (which are further reduced by simple hash function). As a result, an adversary does not need to be able to even execute code at any of the memory addresses containing the victim’s branch instruction. ASLR can also be compensated, since upper bits are ignored and bits 15..0 do not appear to be randomized with ASLR in Win32 or Win64.

I think ASLR on Linux also doesn't currently randomize those bits of the address; cat /proc/self/maps shows the lower 3 bytes always 0. But, could it be made to randomize more of the address, and so guard against Spectre?

(Then we'd only have to get all binaries built PIE, which is easily audited for and distributions are well on their way toward already. Indeed, most executables on my laptop are already built PIE, with the notable exception of /bin/bash and all haskell programs.)

I don't have an answer to this question, but am very curious about thoughts from anyone who knows about ASLR and low level CPU details. I have not been able to find any discussion of it aside from the above part of the Spectre paper.