Grunt Work

Today seemed more like work than anything has so far.  All because of Turbo Pascal overlays.

I noticed before that some parts of the LORD code weren't automatically diassembled.  Sometimes I went ahead and manually did the area, and just assumed maybe these were functions that weren't used anymore and had no CALLs to them for the disassembler to look there.  I still had a nagging suspicion about it though.

Well, today I decided to go through and find all of these and convert them into data and code, just to see what was there.  I did this to what I could (with some problems here and there) until I eventually got up into an area of some kind of labeled structures, sometimes followed by JMPs to somewhere in the overlay.  Sometimes these jumps seemed to go to places which didn't necessarily make sense compared to what the upper level function doing the calling would have intended to do.  And some of the time functions were calling into one of these areas but it was crap code there instead of jumps.  I knew I had suddenly found a serious problem with disassembling the overall program if main parts of the game were trying to call things which were simply broken.

Well, the first part of these mystery structures was a call to interrupt 3F.  This is the overlay manager interrupt.  But that still didn't help me any.  I looked and looked trying to find information on how Turbo Pascal overlays work, but was coming up very empty.  Luckily, through a link on some forum, I found a single website which explained aspects of it.  And I'm going to go ahead and link it here, just for anyone else who might stumble onto this post trying to find the same information that I was.

The article covers Turbo Pascal 5 and 6 overlays, and points out that TP7 had an implementation to take advantage of protected mode on 286 hardware for storing overlays in higher memory.  But I guess LORD was compiled to not use that (I don't remember what kind of options you had for overlays in the TP configuration), because I found the information in the article to still hold true.

In a nutshell, Turbo Pascal puts stub segments into your main EXE, one for each unit in the overlay.  You have a 32-byte structure of information about the unit, such as where it's located in the overlay and all that.  The article documents the structure's format.  Then following the 32-byte structure are a number of 5-byte records, one for each function exported from the unit.  The first two bytes are an interrupt 3F call again.  The second two bytes are the offset in the unit to the function you're trying to call.  The fifth byte is zero.  When you make a call in your main EXE to a function in the overlay, the call comes to one of these 5-byte records in a stub.  So through Turbo Pascal magic, the interrupt call dynamically changes the code in that 5-byte location to be a far jmp to wherever it loaded this unit from the overlay into memory.  All future calls to this overlay function then go immediately to the code you want through this far jump.  That is, until you call something else in another unit, and it needs to free memory of a previously loaded unit for the new one.  So it sets that old record back to the original 5-bytes (so that it'll call the overlay manager interrupt again if you try to use it later), and goes on to repeat this whole process to load the new unit for whatever function you called.

Once I understood this, I also had to understand that IDA (the diassembler I'm using) had already tried to translate some of the bytes in these 5-byte records in each stub.  Sometimes it did it correctly and created proper jumps into the overlay.  Other times it wrote the far jump, but it actually jumped to the wrong function in a unit.  And other times it just made a royal mess out of the byes to the point that it was junk code.  I didn't even know these 5-byte records actually existed in the EXE until I looked at it in a hex editor, outside of IDA.  It wasn't until then that I actually knew the website I linked was still relevant to TP7.  In the hex editor, I could clearly see that each 5-byte record started with an interrupt 3F, so I knew it was what I was looking for.

So now came the task of manually fixing IDA's disassembly.  This meant doing the following: go to a stub segment in IDA (e.g. stub002).  Now match up bytes in the 32-byte header structure to bytes in the external  hex editor, so that I know I'm looking at the same stub in both.  Now back to IDA, bring up the segments list and look at the corresponding overlay segment which matches the value on the stub (e.g. stub002 and ovr002).  Make a note of the segment's base address.  Now go back to your hex editor, and look up the first 5-byte record below the 32-byte structure you located.  It'll start with "CD 3F" which is an interrupt call (not to be confused with the same interrupt call at the start of the 32-byte struct above).  You can ignore those two bytes.  The second two are what you want: they're the function offset.  Make a note of them.  Now go back to IDA, and into the stub segment again.  Click on the first byte of the first 5-byte record below the header structure.  Now go to your Hex View window in IDA (not your hex editor!), and you should be synchronized to the corresponding byte (assuming you have synchronization on, which you should).  Right-click on the byte and pick Edit.  Now you can type in new hex values.  Start by typing "EA" as the first byte, which is a far jump.  Next, type in your function's offset, which you got from your hex editor.  Following those two, now type in the low and high bytes of the corresponding overlay segment's base address which you already noted (remember to swap bytes, since little-endian is low-byte first).  Right-click now and Commit your changes, and go back to the normal IDA code view.  You should be able to convert that 5-byte record into code (if it's not already), and it should turn into a legal far jump to somewhere in the overlay.  If you then convert that into a function, its name will stay synced to whatever the overlay function gets named (just with a j_ prefix, and making it easier to keep up with when code in the main EXE calls the overlay function through this stub).  You now have to repeat this for the rest of the 5-byte records in the stub (the number of which is held in the 32-byte record above, check the site linked to know where).  And then repeat it for the rest of the stubs, too.  If I were you, I'd double-check ones which are already legal far jumps, because they might very well be jumping to the wrong thing!

Luckily I only had to do about a dozen 5-byte records in all for LORD.  But it was still rather tedious.  And if I hadn't found that website with the stub structures, I would have been up the creek without a paddle!

So hopefully this helps anyone else who might have had overlay problems in IDA.  Or maybe just someone who wants to better understand TP overlays in general.  If that website ever goes down, or if anyone has any questions about something I said, feel free to ask.  Trust me, I know what it's like to not be able to find any information on this stuff!

Reference Referral

I don't have anything outstanding to mention from my exploring and labeling of assembly today.  I mostly have started going to the deepest nested levels of the unlabeled functions I have left.  The best way to identify a function you don't understand is usually to identify the other functions that it calls.  This led to not only labeling many functions today, but also many variables.  So I made a lot of progress, even if not much of it was actual game code and was more just support functions.  But it's always satisfying to figure one out, regardless of its purpose.

Today it was mostly functions and variables which had to do with OS detection and serial port interaction, with a tad bit of FOSSIL driver stuff.  This meant lots of referring to interrupt and I/O port lists to figure out what was going on.  And that's why I decided to make this post.

There's something out there which I've known about for many many years now, and it's something that I think a lot of lower-level and/or old-school programmers all have a copy of.  I'm talking about Ralf Brown's Interrupt List.  Or simply the RBIL if you prefer.  It's a very detailed listing of near everything which ever used an interrupt, I/O port, or memory address in a PC, up until about the year 2000.

When I first got a copy, it was late 90s, though I had no idea that the thing had been around since 1981.  This was around that time when I'd started breaking out of my Turbo Pascal cage, having thoroughly read through the built-in help and learning about I/O ports, interrupt hooking, and memory access in the process.  So finding this collection of information in the RBIL was like a pot of gold.  I liked making games at the time, so messing with the video card's registers and memory was my biggest draw.  I could do things better and faster than Pascal normally allowed.  Simple curiosity eventually led me to also hooking interrupts, directly accessing the keyboard and serial ports, etc.  Basically, anything in my PC I could mess with, I tried it.  And I was doing all of this before I even knew C!

None of that would have been possible without the RBIL.  It would have been like when I had an Atari 800 years before that, and had seen the POKE and PEEK commands used in various Basic programs, but had no idea what the heck they did or what kind of possibilities they allowed because I had no kind of reference.  So it makes me wonder just how many other developers that the RBIL inspired to do more.  And I wonder how many game developers and software companies relied on this list to make their products.  Maybe it was indirectly responsible for games like Wolfenstein 3D and Doom, for all we know!

Ralf Brown may have not contributed all of the information himself, but he maintained it.  And nearly twenty years is an incredible amount of dedication to ensuring that there was one central list of information that software and hardware developers could rely on.  Even though twelve years have passed since it was last updated, and a lot has changed inside of PCs in the meantime, a lot of the information in the RBIL is still just as relevant as it ever was.  And it will continue to be relevant until the PC architecture is dead, when/if that ever happens.

So kudos to Ralf, and to everyone who contributed to making the RBIL possible.  All of us geeks much appreciate it!

Parms and Pads

Just some minor things found in the land of LORD assembly today that might be of interest.

The first is with command line parameters.  There's /E to supposedly put the door driver into EMS, /L for local mode, /D for "drew" which it requires to start normally, /N for the old node system, /P to set the drop file location, and /R to force RIP graphics.  Then there's the "NOEMS" parameter, to disable using EMS memory for the overlay, which is actually not documented in the v3.55 LORD.DOC.  It's "introduced" as a new feature in LORD v4.  Maybe he just forgot to mention it sooner?  Or maybe it wasn't ready.  Who knows.  Without having dug into it too deeply, all I can tell you that it at least acknowledges that you used the switch when you start the game with it.

The other thing related to command line switches is that there's supposed to be a /B to lock the baud rate.  Except there is no /B.  There's a /C though, which appears to parse a numeric parameter.  I haven't deduced what it actually does yet, but if I had to guess, I would bet it was a simple typo and people probably never could lock the rate using /B as the docs specified!  Forget this paragraph, I later learned there is a /B and a /C.  The latter locks the com port.

Now on to something else, in a function to pad a string with spaces on the end.  You give it the string to pad, and the value for how long the string should be.  But it accepts a second numerical value, which apparently has the sole purpose of making the function immediately exit if it's a negative number.  If it's 0, though, then it carries on as normal and pads the string.  But the interesting thing is that if you send a positive value for that argument, it will apparently just go into a perpetual loop.

I looked at call references to the function, and they almost always passes zero for that argument.  But I did notice at least one where it could potentially send a -1.  It's after a CWD operation (extends a word into a dword).  DX is pushed for the numerical argument for the function, which as a result of CWD means it's always either going to be 0 or -1 depending on the sign of AX.  And I think AX was set based on the length of the string about to be passed.  All I can gather from that instance is that if the string was going to be longer than 127 characters, it would immediately exit without doing any padding.  I'd have to look into it more to really know for sure.  Either way, I don't understand the purpose of it even being there.

Anyhow, the thing that stands out about it is simply that the possibility for a perpetual loop exists, even though I doubt that ever actually occurs in the game.  And considering the implementation, I can't say with any certainty that Seth actually did it himself this time, or whether it was just a compilation thing.

That's it for today!

Double Dumbass On You!

An interesting little find in LORD.  I found a function in the overlay which has the sole purpose of returning a string of one of the locations you can search for the girl to rescue, based on an index value you pass as an argument.  All of the normal ones are there: Castle Coldrake, Fortress Liddux, Gannon Keep, Penyon Manor, and Dema's Lair.  When you first enter the function, however, it automatically sets up "Dumbass World" to be returned.  If a value other than 1-5 is passed, you get that string instead.

Discovering Old Bugs

In my efforts to parse through the disassembled Legend of the Red Dragon code, I found a quirk which bugged me enough to really look into.

The problem was/is, when creating a new character, if you put spaces after your name, it tells you your name isn't long enough. Why is that? I thought I had identified what the functions which interacted with the inputted string were doing. One stripped leading spaces, one stripped trailing spaces, and the other stripped out bad color codes and fowl language. It made no sense to me.

So I took a more active approach. I tried cracking the EXE, individually removing calls to the above mentioned functions which would have modified the name you typed. I wanted to see which would make the "try a longer name" stop happening. Turned out, the function call to strip trailing spaces was in fact misbehaving. But again, why? I looked through it many times and it seemed okay. Was it some slight difference in a flag being set on a modern CPU, maybe? I've heard of an instance between 8086 and 286, but nothing really beyond that. Was the Turbo Pascal function to delete a character from a string somehow at fault, in need of some kind of modern patch? Both seemed unlikely, but I was at a loss otherwise. And I wanted/needed to know.

The only way to know what was really happening was to see it in action. So I used a version of DOSBox with the built-in debugger, tracked down the function in the game to strip trailing spaces, and stepped through it line by line. That's when I realized that it was nothing as complicated as I might have thought. Seth Able had just programmed it wrong the whole time!

The function is looping through every character, starting at the end of the string, checking if it's a space, deleting the last character of the string if so, then checking if the string is empty before starting the loop over. Except the bug is that he wasn't updating the variable which held the string position to check for the next space character. Instead of actually stepping backwards through the string, it was checking the exact same memory location each time through the loop. So if the last character of the initial string was a space, it was going to see a space every time through the loop, regardless of what the last character of the updated string actually was. So it just kept shortening the string length until it was 0 chars long. He should have either been decrementing the position variable at the end of the loop, or getting the string length at the beginning of the loop.

The function is used in a few other places in the game, too, so they would have the same problem.

So, that's why when you type a name with spaces on the end, it thinks your name is too short. Because technically, it is!



p.s. This exists in both LORD 3.55 and 4.00a. I don't know how many other versions, but those are the last ones from Seth himself. I checked 4.07 from Michael Preslar, and the bug has since been corrected.

Code Guts

I thought I'd set up a blog here to document interesting things I find in the world of disassembly and programming.  We'll see where it goes!

My current project at the moment is the classic BBS door game Legend of the Red Dragon, developed by Seth Able Robinson over at RTSoft.  It's since been sold to Gameport, where it's mostly stagnant now.  But I thought it would be fun to go through it and see what there is to see.

By the way, the version I'm disassembling is v3.55.  This was kind of by accident.  When I started, I assumed it was 4.00a, the very last one from Seth Able.  I'd copied it out of an old BBS directory I had without checking.  But it's not a big deal, for a few reasons.  First, the only versions after 3.55 were 4.00 and 4.00a.  Second, v4 didn't really introduce that much based on the change log.  In terms of gameplay, it was mostly just a new blackjack game.  I think I remember it, and sure, it was neat.  But the fact is, I played on 3.55 way more back then, so for nostalgia's sake it's fine that this ended up being the version I'm working with.  I may still do 4.00a later, who knows.

Let's get started!