Oblivion.exe file and directory thrashing

Overview

This information was found on Bethesda forums and copied here to keep the information from being purged like so many other helpful topics that have been lost to "auto archiving"

Note from Wrye

A recent PM exchange with bg2408 pointed out some problems with Oblivion.exe directory thrashing, too which I can add a memory of an old post plus some anecdotal recent evidence.

Inactive ESP/ESM Reading?

First it was observed by Dark0ne a couple of years ago that Oblivion seems to be doing something with esps and esms that are in the data directory, but inactive. His experience was that simply having too many esps/esms in Data caused some weird performance/FPS glitches, and that moving these mods to a subdirectory of Data resolved the problem. (He used Bash's File: Hid feature, which moves files to Data\Bash\Hidden.) I think that I have seen some other mentions of this problem.

Related to that I recently had a problem where Oblvion crashed apparently due to a large semi-clone of Oblivion.esm that I had made while testing some bash features. The clone was not active, and so should not have been read, but its mere presence in the directory caused CTDs. (The clone was also likely not perfect, so perhaps that played a part.) It may have been named "Oblivion something or other.esm" and perhaps Oblivion has special handling for files that start with "Oblivion". (BSA files are definitely treated special.)

So, if this is true, it is annoying because it means that you simply cannot have too many mods in you data directory -- even if they are inactive.

Directory Thrashing

But worse than that is what bg2408 mentioned in his PM...

Some time ago I had problems with random crashes. I have investigated them and found that Oblivion sometimes starts
to randomly searches all its directories, and if this takes too long, it goes haywire. Ever seen the recommendation
to not have several hundreds unused .esp files in your data folder? Likely a naive observation based on this behavior.


Now, as seen with procmon, Oblivion does not only search \Data\, but ALL folders in its install and mygames-directory.
After I have seen this, and removed all junk (which were several ten thousand files...) the crashes were gone. Now OBMM
stores omods in a subfolder of Oblivion's install folder, therefore omods are affected by this search, too.

I have not had a chance to investigate this myself, but it sounds even crazier than scanning inactive esps. Why the heck would you scan randome chunks of the install directory AND the savegames directory? But I wouldn't be totally surprised either -- we have seen the engine do some pretty strange stuff.

So, it would be useful if this were investigated a bit more. Main questions are:

1. How often does it happen?
2. What triggers it? (searching for missing files?)
3. When does it become a problem?

Possibilities

Then there is the question of what to do about it.

1. Add something to OBSE to stop this behavior. This would definitely be good, but the OBSE team have not really addressed these types of problems. (And I have asked about this one and gotten a "Maybe later.") Also, of course, OBSE fix would only help OBSE users.
2. Not put files a lot of extra files in those directories. E.g. I personally have been storing all of my downloads under Oblivion\Downloads. Maybe the advice for folks should be "Do not do that -- put them in the directory above Oblivion."
3. Modify tools to not store files in these directories. As bg2408 points out, OBMM stores files under Oblvion directory. Similarly Bash stores backup and hidden mods files under Oblivion\Data\Bash. And the new Bain feature stores Installers under Oblivion\Installers. Those tools might be modified to use different directories.

For Bash, I could move establish a new directory at the same level as Oblivion directory, and then move installers and mod backups to that directory. But it will be a mild hassle for me to program and for users to use -- so I would rather not do it unless necessary.

Next Steps

At this point, the main thing that I would like to see is a bit more understanding/verification. If it is confirmed, and there is no easy way around it, then I will go ahead and rig bash to use an alternate directory. But I would prefer not to do that unless necessary.

References

• Bethesda Forums Topic