[Cryptography] How to find hidden/undocumented instructions

[grarpamp]

On Wed, Aug 2, 2017 at 9:17 AM, Henry Baker <hbaker1 at pipeline.com> wrote:
> https://raw.githubusercontent.com/xoreaxeaxeax/sandsifter/master/references/domas_breaking_the_x86_isa_wp.pdf
>
> Breaking the x86 ISA
> Christopher Domas  xoreaxeaxeax at gmail.com  July 27, 2017
>
> A processor is not a trusted black box for running code; on the
> contrary, modern x86 chips are packed full of secret instructions and
> hardware bugs.  In this paper, we demonstrate how page fault analysis
> and some creative processor fuzzing can be used to exhaustively search
> the x86 instruction set and uncover the secrets buried in a chipset.
> The approach has revealed critical x86 hardware glitches, previously
> unknown machine instructions, ubiquitous software bugs, and flaws in
> enterprise hypervisors.

"Lastly, a so-called `halt and catch fire' instruction was
discovered on an as-yet unnamed x86 processor. This
instruction, executed in ring 3 from an unprivileged process,
appears to lock the processor entirely. To rule out kernel bugs,
the instruction was tested against three Linux kernels and two
Windows kernels, yielding the same results. Kernel debugging
with serial I/O and interrupt hooks appeared to corroborate the
results. At the time of this paper's publishing, the vendor has
not been provided sufficient time to respond to the issue."

This is nice work. These sort of fuzzers and searchers need a
distributed network version to cover more space deeper and faster.