Obstinate Overlays

I came to find out I wasn't quite as thoroughly done labeling all the functions in Legend of the Red Dragon assembly as I thought.  As I was going through the code in more detail, I found a call to a function in what looked like another broken stub.  I thought no problem, it must just link to one of the functions which I've already labeled, since I'd gone through all of the overlay segments already.  I just needed to trace down which function it was and create the JMP manually like before.

Nope!

After a bit of poking around, it turned out that the segment in which this Turbo Pascal stub was wanting to jump into wasn't loaded into IDA at all.  This meant somehow I had to manually load this segment in.  Luckily you can do that, with the Additional Binary File in the Load File menu.  First I had to find the unmangled Turbo Pascal stub in the EXE, since the stubs IDA can't straighten out aren't always the same as the original data in the EXE anymore.  Once that was found, I could calculate where the segment was in the LORD.OVR overlay file that I needed.  I then used this offset and length to load the section into IDA that I needed.

After loading this, I still had to create the segment again so that I could set the base address, which you apparently can't do when creating the segment in the Additional Binary dialog.  But if you make a new segment for the same memory area then it just replaces the other one, basically.  Then I had to set the assumed data segment.  But even after doing this, the segments in the Program Segmentation tab don't display DS properly.  So I dunno how to fix that, although it doesn't seem to matter.  I think that's just the default DS.

Then came the next challenge: the relocations.  I quickly noticed that there were calls all red and not linking to anywhere else.  I knew immediately that it was because there were no relocations applied, but I just wasn't sure how to go about fixing it.  My first step was to go back to the Turbo Pascal stub from the EXE and determine the size of the relocation section in the overlay (which is always directly below each segment).  I extracted that, and started trying to match up the first relocation to the first CALL in the assembly via IDA's hex editor.  Matched up perfectly.  Then I realized that I actually only needed to change the segment of each call manually to add 0x1000 to them, since that's IDA's overall base address or whatever.  But when you realize just how many calls you're going to have to manually set, you see what a huge pain in the ass that's about to become.

So I did some looking into the IDC script language which IDA uses.  Two functions are useful here:  Word(), and PatchWord().  I tested it out by manually putting together a script command to patch the first segment address to add 0x1000 to it, and it worked fine.  So, I wrote a little C app to convert the relocation table from the overlay into an IDC script, and pasted the lines in several at a time.  Worked like a charm!

Then I found ANOTHER broken stub and realized I was going to have to do this all over again.  So I opted to make my life a little easier.  I wrote another C app to scan the LORD.EXE and locate all of the overlay stubs, then using that information, extract each segment and relocation table into separate binary files, while printing out code and relocation sizes and all that as it goes.  I was then able to easily import the code chunk I needed, parse the relocation binary with my other little C app to make the relocation script, patch it, and done.  That segment went in much quicker.

I used my stub finder to determine that every stub was now in fact added into IDA.  So I proceeded to label those few functions which got added (related to DarkCloak Tavern as it turns out) and can now say I have nearly all of the game's assembly processed/labeled/etc.  Again.

This also indirectly resulted in setting me on a path to determine what a couple more of the settings in PLAYEXT.DAT are specifically, so it wasn't all bad!


p.s. The alternative to writing the relocation-to-IDC-script utility would have been one to just patch the segment directly before inserting it.  But since I'd already manually inserted the first one straight out of the overlay, I opted to just fix what was already there than try to do it again.  This was before I realized I was in fact going to have to do it a second time anyway for the other segment that was missing!

Color Quandary

I haven't posted here lately, but for the record, I've labeled and identified basically 99% of the game at this point, so there's not much left to do in terms of disassembly.  So I've been working on something related to it.  But I did come across some stuff earlier which might warrant a mention.  It's in the function to display LORD color codes locally.

First of all, there is no foreground black, aka color 0.  You can use LORD code `^, which sets the foreground color to 16, which will technically be black, but this is not correct.  It will be black, sure, but it will also enable blinking text.  So if you have a background other than black, you'll see the characters on it, and they will in fact be blinking.

In the Turbo Pascal TextColor function, anything over color 15 will set the blinking bit.  15 or less (aka the standard 16 DOS colors) resets the bit to 0.  I'm not exactly sure why Borland chose to do it this way, since the blinking bit is actually at the end of the background color nibble of the text attribute byte in video memory.  The bits are broken up as: bBBBFFFF, where b is blink, BBB is background, and FFFF is foreground.  The TextBackground function only seems to accept 0..7 according to the help, though, so I'm doubting that blink can be enabled from it.  It would make more sense from a technical standpoint to let you set the flashing bit that way, by setting the background color higher than 7 as you would directly into memory.  Maybe they thought it would be confusing to people, since the foreground characters are what blink when it's enabled, so they figured all foreground-related stuff should be together?  Something I wonder about though is whether Turbo Pascal can actually accept higher than 8 colors for the background, because EGA and up lets you disable blinking in the hardware in order to use the last bit to enable a full 16 colors for the background like the foreground can do.  If TextBackground limits the value, you'd have to use a roundabout method of setting the extended background color when in that mode.

Anyway, the second LORD thing is regarding color 12.  In this local printing function, for some reason the `@ LORD code translates to color 9, which `9 also does.  Maybe since this local print function isn't used for a whole lot of stuff other than during startup, he just never noticed.

Next is something the background colors.  For that you use `rX, where X is the background color.  This time we do have a 0 for black.  But we also have an 8.  As mentioned above, though, this might not be possible and simply display black like with 0.  Then again, maybe it displays black and enables blinking?  No way to find out unless I wrote a test program in Pascal itself.

Related to the above is an oddity.  Apparently Seth tried to implement setting the background color twice, probably as an oversight.  The first time uses a bunch of comparison checks for each ASCII number value.  The second one though works differently; it actually makes a short string of the value, and uses the Turbo Pascal "Val" function to convert it into a number, and then sets the background color.  This would let you set 0..9, even.  

But here's the thing: that second iteration will never get called normally.  Because of the way the comparison checks work, after the first test for `r, the string position variable is incremented in order to get the background digit.  By the time it reaches the other comparison check for `r, the current character won't be `r anymore, it'll be the position of the digit, so the comparison never matches.  BUT, it can happen, I suppose, if you use an invalid LORD code.  For example, `rr4 would trigger it.  This can't be tested in the game itself, though, because all of the editable content (LORDTXT.DAT etc) go through a different color code parsing function.

You can look at the source code to Barak's House to find its local color-setting function which has this exactly same "bug."  

On a side note, looking through Barak's House is actually fun, because I can see the original (and sometimes much more simple) Pascal code of some of the same functions I spent the time disassembling and figuring out in LORD.

I guess that's it.  A lot of rambling about some simple color stuff!

Multi-day Dump

I haven't updated this recently, because individual little things I came across during my disassembly didn't really warrant a post to themselves.  But I can take the time to touch on a few things for a single update!

I was starting to run dry on sources to find information on various variables and functions.  But then I decided to take a different approach: disassemble LORDCFG.EXE as well.  This turned out to be a great idea, because it let me figure out the format of LORD.DAT, as well as figure out how the node files work.  I even figured out the registration code algorithm.  So I took this information with me back to LORD.EXE and quickly started filling in lots of stuff, and figuring out the bits of LORD.DAT which are impossible to know from LORDCFG.  I don't remember ever seeing anyone post what LORD.DAT's structure was before, now that I think about it, so maybe I'll do that here:

regcode1 : long
regcode2 : long
day_last_run : word ; The day of the month in which the game was last run
regcode3 : long
gamewinner : string[146] ; Name of the last winner of the game
multinode_support : byte ; Enable multi-node support
regcode4 : long
allow_bank_transfers : byte ; Allow bank transfers
max_bank_transfer : long ; Maximum amount of money transferable via the bank
human_fights : long ; Number of player vs player fights a day
regcode5 : long
grace_deletion_days : long ; Number of days until players are deleted
ANSI_score_filename : string[60] ; Path + filename to generated ANSI score file
ASCII_score_filename : string[60] ; Path + filename to generated ASCII score file
unknown_string : string[60] ; Nothing uses this that I can find
clean_mode: byte ; Enable clean mode
forest_fights : long ; Forest fights a day
registered_thanks : string ; Who to thank for registering the game
bank_transfers_per_day : byte ; Number of bank transfers allowed a day

I took the time just now to convert that from assembly stuff like "db" and "dw" to a Turbo Pascal-ish syntax, to match the PLAYER.DAT structure Seth gives in LORDSTRC.PAS.  Keep in mind that off the top of my head I don't know which of those are actually boolean/char/byte, signed/unsigned, etc.  If in question, assume it's signed.  If you're only interested in size though (like for loading it in C), long is 4 bytes, word is 2 bytes, byte is 1 byte, and strings are always 1 byte longer than specified (to hold the string length).  If no size is specified, the string is 1 size byte + 255 characters.  The entire structure is 628 bytes.

Let's see, what else?  I've been figuring out formats to all the in-game files like OUT.x, INFO.x, FIGHTx.DAT, etc.  I had to make a list of just all of the files the game uses, to keep track!  I've also been deducing/documenting the conditions which are required for certain random game events to occur.  And I started making a list of all of the special LORD color codes, since they're not all documented.

That reminds me.  Something strange I noticed is that LORDTXT.DAT has some ANSI art in it which is damaged.  I think Seth used a utility written in Pascal (probably something he wrote himself) to create the file, but unfortunately Pascal's string length limit of 255 characters resulted in some of the really long lines of ANSI getting chopped.  So the ANSI graphic of the dragon's lair, and I think the New World one, maybe another but I can't remember, have some damage; ANSI sequences just abruptly end on 2-3 lines.  That would imply it was broken like that back in the old days on the BBSes too, but I can't remember either way.

Thanks in part to Barak's House, I found the Turbo Pascal source files of some add-on libraries Seth used, which has made labeling some of the support sections and internal variables a little easier.  A lot of it I don't really need to have labeled for disassembling the game itself.  But doing so helps me find the stuff Seth himself wrote from the 3rd-party helper stuff he linked in.  But to be honest, I think I've identified 99% of the functions Seth made.  It's mostly some variables left which are giving me a headache to figure out.

I thought there was something else I wanted to mention too, but I can't think of it.  Maybe next time!

Grunt Work

Today seemed more like work than anything has so far.  All because of Turbo Pascal overlays.

I noticed before that some parts of the LORD code weren't automatically diassembled.  Sometimes I went ahead and manually did the area, and just assumed maybe these were functions that weren't used anymore and had no CALLs to them for the disassembler to look there.  I still had a nagging suspicion about it though.

Well, today I decided to go through and find all of these and convert them into data and code, just to see what was there.  I did this to what I could (with some problems here and there) until I eventually got up into an area of some kind of labeled structures, sometimes followed by JMPs to somewhere in the overlay.  Sometimes these jumps seemed to go to places which didn't necessarily make sense compared to what the upper level function doing the calling would have intended to do.  And some of the time functions were calling into one of these areas but it was crap code there instead of jumps.  I knew I had suddenly found a serious problem with disassembling the overall program if main parts of the game were trying to call things which were simply broken.

Well, the first part of these mystery structures was a call to interrupt 3F.  This is the overlay manager interrupt.  But that still didn't help me any.  I looked and looked trying to find information on how Turbo Pascal overlays work, but was coming up very empty.  Luckily, through a link on some forum, I found a single website which explained aspects of it.  And I'm going to go ahead and link it here, just for anyone else who might stumble onto this post trying to find the same information that I was.

The article covers Turbo Pascal 5 and 6 overlays, and points out that TP7 had an implementation to take advantage of protected mode on 286 hardware for storing overlays in higher memory.  But I guess LORD was compiled to not use that (I don't remember what kind of options you had for overlays in the TP configuration), because I found the information in the article to still hold true.

In a nutshell, Turbo Pascal puts stub segments into your main EXE, one for each unit in the overlay.  You have a 32-byte structure of information about the unit, such as where it's located in the overlay and all that.  The article documents the structure's format.  Then following the 32-byte structure are a number of 5-byte records, one for each function exported from the unit.  The first two bytes are an interrupt 3F call again.  The second two bytes are the offset in the unit to the function you're trying to call.  The fifth byte is zero.  When you make a call in your main EXE to a function in the overlay, the call comes to one of these 5-byte records in a stub.  So through Turbo Pascal magic, the interrupt call dynamically changes the code in that 5-byte location to be a far jmp to wherever it loaded this unit from the overlay into memory.  All future calls to this overlay function then go immediately to the code you want through this far jump.  That is, until you call something else in another unit, and it needs to free memory of a previously loaded unit for the new one.  So it sets that old record back to the original 5-bytes (so that it'll call the overlay manager interrupt again if you try to use it later), and goes on to repeat this whole process to load the new unit for whatever function you called.

Once I understood this, I also had to understand that IDA (the diassembler I'm using) had already tried to translate some of the bytes in these 5-byte records in each stub.  Sometimes it did it correctly and created proper jumps into the overlay.  Other times it wrote the far jump, but it actually jumped to the wrong function in a unit.  And other times it just made a royal mess out of the byes to the point that it was junk code.  I didn't even know these 5-byte records actually existed in the EXE until I looked at it in a hex editor, outside of IDA.  It wasn't until then that I actually knew the website I linked was still relevant to TP7.  In the hex editor, I could clearly see that each 5-byte record started with an interrupt 3F, so I knew it was what I was looking for.

So now came the task of manually fixing IDA's disassembly.  This meant doing the following: go to a stub segment in IDA (e.g. stub002).  Now match up bytes in the 32-byte header structure to bytes in the external  hex editor, so that I know I'm looking at the same stub in both.  Now back to IDA, bring up the segments list and look at the corresponding overlay segment which matches the value on the stub (e.g. stub002 and ovr002).  Make a note of the segment's base address.  Now go back to your hex editor, and look up the first 5-byte record below the 32-byte structure you located.  It'll start with "CD 3F" which is an interrupt call (not to be confused with the same interrupt call at the start of the 32-byte struct above).  You can ignore those two bytes.  The second two are what you want: they're the function offset.  Make a note of them.  Now go back to IDA, and into the stub segment again.  Click on the first byte of the first 5-byte record below the header structure.  Now go to your Hex View window in IDA (not your hex editor!), and you should be synchronized to the corresponding byte (assuming you have synchronization on, which you should).  Right-click on the byte and pick Edit.  Now you can type in new hex values.  Start by typing "EA" as the first byte, which is a far jump.  Next, type in your function's offset, which you got from your hex editor.  Following those two, now type in the low and high bytes of the corresponding overlay segment's base address which you already noted (remember to swap bytes, since little-endian is low-byte first).  Right-click now and Commit your changes, and go back to the normal IDA code view.  You should be able to convert that 5-byte record into code (if it's not already), and it should turn into a legal far jump to somewhere in the overlay.  If you then convert that into a function, its name will stay synced to whatever the overlay function gets named (just with a j_ prefix, and making it easier to keep up with when code in the main EXE calls the overlay function through this stub).  You now have to repeat this for the rest of the 5-byte records in the stub (the number of which is held in the 32-byte record above, check the site linked to know where).  And then repeat it for the rest of the stubs, too.  If I were you, I'd double-check ones which are already legal far jumps, because they might very well be jumping to the wrong thing!

Luckily I only had to do about a dozen 5-byte records in all for LORD.  But it was still rather tedious.  And if I hadn't found that website with the stub structures, I would have been up the creek without a paddle!

So hopefully this helps anyone else who might have had overlay problems in IDA.  Or maybe just someone who wants to better understand TP overlays in general.  If that website ever goes down, or if anyone has any questions about something I said, feel free to ask.  Trust me, I know what it's like to not be able to find any information on this stuff!

Reference Referral

I don't have anything outstanding to mention from my exploring and labeling of assembly today.  I mostly have started going to the deepest nested levels of the unlabeled functions I have left.  The best way to identify a function you don't understand is usually to identify the other functions that it calls.  This led to not only labeling many functions today, but also many variables.  So I made a lot of progress, even if not much of it was actual game code and was more just support functions.  But it's always satisfying to figure one out, regardless of its purpose.

Today it was mostly functions and variables which had to do with OS detection and serial port interaction, with a tad bit of FOSSIL driver stuff.  This meant lots of referring to interrupt and I/O port lists to figure out what was going on.  And that's why I decided to make this post.

There's something out there which I've known about for many many years now, and it's something that I think a lot of lower-level and/or old-school programmers all have a copy of.  I'm talking about Ralf Brown's Interrupt List.  Or simply the RBIL if you prefer.  It's a very detailed listing of near everything which ever used an interrupt, I/O port, or memory address in a PC, up until about the year 2000.

When I first got a copy, it was late 90s, though I had no idea that the thing had been around since 1981.  This was around that time when I'd started breaking out of my Turbo Pascal cage, having thoroughly read through the built-in help and learning about I/O ports, interrupt hooking, and memory access in the process.  So finding this collection of information in the RBIL was like a pot of gold.  I liked making games at the time, so messing with the video card's registers and memory was my biggest draw.  I could do things better and faster than Pascal normally allowed.  Simple curiosity eventually led me to also hooking interrupts, directly accessing the keyboard and serial ports, etc.  Basically, anything in my PC I could mess with, I tried it.  And I was doing all of this before I even knew C!

None of that would have been possible without the RBIL.  It would have been like when I had an Atari 800 years before that, and had seen the POKE and PEEK commands used in various Basic programs, but had no idea what the heck they did or what kind of possibilities they allowed because I had no kind of reference.  So it makes me wonder just how many other developers that the RBIL inspired to do more.  And I wonder how many game developers and software companies relied on this list to make their products.  Maybe it was indirectly responsible for games like Wolfenstein 3D and Doom, for all we know!

Ralf Brown may have not contributed all of the information himself, but he maintained it.  And nearly twenty years is an incredible amount of dedication to ensuring that there was one central list of information that software and hardware developers could rely on.  Even though twelve years have passed since it was last updated, and a lot has changed inside of PCs in the meantime, a lot of the information in the RBIL is still just as relevant as it ever was.  And it will continue to be relevant until the PC architecture is dead, when/if that ever happens.

So kudos to Ralf, and to everyone who contributed to making the RBIL possible.  All of us geeks much appreciate it!

Parms and Pads

Just some minor things found in the land of LORD assembly today that might be of interest.

The first is with command line parameters.  There's /E to supposedly put the door driver into EMS, /L for local mode, /D for "drew" which it requires to start normally, /N for the old node system, /P to set the drop file location, and /R to force RIP graphics.  Then there's the "NOEMS" parameter, to disable using EMS memory for the overlay, which is actually not documented in the v3.55 LORD.DOC.  It's "introduced" as a new feature in LORD v4.  Maybe he just forgot to mention it sooner?  Or maybe it wasn't ready.  Who knows.  Without having dug into it too deeply, all I can tell you that it at least acknowledges that you used the switch when you start the game with it.

The other thing related to command line switches is that there's supposed to be a /B to lock the baud rate.  Except there is no /B.  There's a /C though, which appears to parse a numeric parameter.  I haven't deduced what it actually does yet, but if I had to guess, I would bet it was a simple typo and people probably never could lock the rate using /B as the docs specified!  Forget this paragraph, I later learned there is a /B and a /C.  The latter locks the com port.

Now on to something else, in a function to pad a string with spaces on the end.  You give it the string to pad, and the value for how long the string should be.  But it accepts a second numerical value, which apparently has the sole purpose of making the function immediately exit if it's a negative number.  If it's 0, though, then it carries on as normal and pads the string.  But the interesting thing is that if you send a positive value for that argument, it will apparently just go into a perpetual loop.

I looked at call references to the function, and they almost always passes zero for that argument.  But I did notice at least one where it could potentially send a -1.  It's after a CWD operation (extends a word into a dword).  DX is pushed for the numerical argument for the function, which as a result of CWD means it's always either going to be 0 or -1 depending on the sign of AX.  And I think AX was set based on the length of the string about to be passed.  All I can gather from that instance is that if the string was going to be longer than 127 characters, it would immediately exit without doing any padding.  I'd have to look into it more to really know for sure.  Either way, I don't understand the purpose of it even being there.

Anyhow, the thing that stands out about it is simply that the possibility for a perpetual loop exists, even though I doubt that ever actually occurs in the game.  And considering the implementation, I can't say with any certainty that Seth actually did it himself this time, or whether it was just a compilation thing.

That's it for today!

Double Dumbass On You!

An interesting little find in LORD.  I found a function in the overlay which has the sole purpose of returning a string of one of the locations you can search for the girl to rescue, based on an index value you pass as an argument.  All of the normal ones are there: Castle Coldrake, Fortress Liddux, Gannon Keep, Penyon Manor, and Dema's Lair.  When you first enter the function, however, it automatically sets up "Dumbass World" to be returned.  If a value other than 1-5 is passed, you get that string instead.