Thread by @KeyJ_trbl

Gather round, folks. I'm going to write a @Foone-style thread about a rather interesting debugging adventure I had during the last days.

So there was this new release of the #ao486 PC clone core for the #MiSTerFPGA emulation system the other day. It was a highly anticipated major release with a 4x increase in CPU performance.
I wrote a thing or two about it here:

https://twitter.com/KeyJ_trbl/status/1294383405627576320

I also did some benchmarks (

https://twitter.com/KeyJ_trbl/status/1294748909189955585

) using @philscomputerlb's nice suite. Some of the benchmark programs failed to run, but I initially didn't investigate any further. (I mean, c'mon folks, Doom is playable now, what else could you ask for?)

One of the benchmarks that didn't run was TOPBENCH, which turned out to be written by fellow demoscener @MobyGamer. When he specifically asked for results, I felt obliged to see what's up with that.

So what made TOPBENCH fail? Actually nothing: it just exited immediately with a detailed message explaining that it's not a good idea to run this program while EMM386 is enabled, and that I should try again without EMM386 or use a command-line option to override the warning.

So I booted again without EMM386, ran TOPBENCH, and ...

Runtime error 200 at 26A0:01C4

Oh, that's bad. Maybe there's some command line option I can set? Run TOPBENCH -h, and

Runtime error 200 at 26A0:01C4

Oh fsck. It crashes even before parsing the command line!

Now that "runtime error 200" thing is nothing new. TOPBENCH is written in Turbo Pascal, and the overwhelming majority of programs written in that environment had exactly that issue with faster CPUs back in the day, due to a delay loop calibration routine that ran too fast.

It was still no plausible explanation, because
- ao486 is not fast enough to trigger the issue
- @MobyGamer knows about that issue and wouldn't ship a program where this is still present
- the various tools that patch Turbo Pascal executables to fix this didn't find anything

So it had to be something else.

But wait ... didn't I get some meaningful output from the program when running under EMM386?
Booted again (this time with EMM386), and sure enough, the program *just works*. I could actually do some benchmarks with it:

https://twitter.com/KeyJ_trbl/status/1294785965094055937

To recap: I got a program that
- works perfectly fine when EMM386 and its EMS emulation are active
- crashes with a mysterious "runtime error 200" (which translates to "division by zero", by the way) during early initialization when EMM386 is *not* active

(Cue a few hours of frustrating work collecting together the source code of TOPBENCH, all of its dependencies, and a working IDE and compiler, and putting this all into a single directory because that's easier to transfer into ao486 because of reasons.)

With my own build, I got the exact same behavior as with the official release executable - good!

I could also confirm that it's really during early initialization, as the TP IDE wouldn't let me step a single line of code before hitting the crash - not good, but expected.

In the meantime, @MobyGamer sent me an executable that had some startup logging enabled, but the crash site was even before the earliest logs he inserted. No cake.

But I got an idea this way: [>>]

If one of the two dozen "units" (modules in Turbo Pascal) crash during initialization, I can create a simple test application that does nothing but include some of these units, and narrow it down this way.

That worked perfectly, and I quickly found a unit that crashed.

I narrowed it down further to a function that queried the current video mode by calling the appropriate video BIOS function:

function GetMode: byte;
var regs: registers;
begin
with regs do begin
ax := $0F00;
intr($10, regs);
GetMode := al;
end;
end;

This is not a call that should fail under any circumstances! But well ... unsure how to proceed, I replaced it by its equivalent in inline assembly:

function GetMode: byte;
var res: byte;
begin
asm
mov ax, 0F00h
int 10h
mov res, al
end;
GetMode := res;
end;

In theory, there shouldn't be any functional difference between these two versions.

In practice, however, there was: The original code that uses the Intr() library function crashes, while the inline assembly version works just fine!

Note that this was just a pyrrhic victory: My test program worked with the alternate GetMode implementation, but TOPBENCH as a whole still didn't. There were obviously more calls that needed modifications.

But before fixing those, I got this nagging feeling: WHY does it crash?

Different register contents maybe? (After all, the remainder of the "registers" structure was uninitialized!) No, that wasn't it.

Meanwhile, @MobyGamer sent me the assembly source of Turbo Pascal's standard library's Intr function, I looked at it ... and then it hit me.

You know, the "int" instruction in x86 is only available with an immediate argument, no registers or memory. The Intr function, however, needs to call *any* of the 256 possible interrupt service routines, selected by a function call argument.

So Intr contains an "int 0" instruction, but the interrupt number is replaced by the actually requested number, at runtime, earlier in the function: "int 0" (CD 00 in hex) becomes "int xx" (CD xx), and that's what is ultimately executed ... except it isn't!

A normal, proper x86 CPU would execute the "int xx" instruction, because hey, that's what's in memory, and it shouldn't matter *when* it was written there, right? Unfortunately, it's a little bit more complicated, because of caches and pipelines.

The "int 0" instruction might already be in the instruction cache, or some CPU-internal pipeline, or even already decoded, at the point when it's overwritten. CPUs go to some lengths to roll things back in such an event to ensure that the modified instruction is executed.

In #ao486, though, there seems to be a bug that causes the CPU to ignore that the instruction has been changed. It still executes "int 0", so it calls the interrupt service routine for interrupt 0, which is ... division by zero.

Mystery solved, bug filed! github.com/MiSTer-devel/a…

I still don't know exactly what this has to do with EMM386, but I can guess: If EMM386 (and, specifically, the EMS emulation) is active, the CPU is usually in Protected Mode and paging is enabled. This might cause the core to behave differently for "int" instructions.

If I do these self-modifying code shenanigans with "normal" instructions, like "mov", they are also perfectly reproducible with EMM386 present. It only goes away when there are lots (more than 40) additional instruction bytes between the modifying and the modified code.

Another interesting observation: The crash in Intr happens only in software compiled with Turbo Pascal 7.0, not 6.0 (didn't check earlier versions). I thought that maybe the code in 6.0's library just had the required extra between the "int" and the code modifying it, but no ...

Intr in TP6 just works in a completely different way. It doesn't use an "int" instruction at all! It grabs the ISR routine's address from the interrupt vector table and just jumps there.

Did I say "it jumps there"? Just kidding! No, of course it sets up the stack so that it can use a *return* instruction to jump there. That's obviously way cooler.

(I bet there's a good reason to do it this way, but I don't want to investigate. If you know it, please respond!)