Short: Prevent Memory Fragmentation! Author: Eric Sauvageau (merlin@thule.icenet.no) Uploader: dream step polymtl ca Type: util/boot Architecture: m68k-amigaos LowFrag v1.3 (20-Dec-96) - By Eric Sauvageau. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0. History ~~~~~~~~~~ 1.0 - First version - limited release. 1.2 - Wouldn't work if no threshold value was specified from the command line. - Fixed major enforcer hits :( 1.3 - The AllocMem() patch wasn't properly patched (Wrong LVO). Patching the right one slowed down the system too much, so I simply removed it. - Now promotes allocs larger than the treshold instead of the smaller (more common) ones. It's a bit faster. 1. Introduction ~~~~~~~~~~~~~~~ "Out of Memory" the program told you. What the heck?! You have 2 megs of memory, 1 of which is free, and this 300 Kb program tells you "Out of Memory"? Then you open a Shell, type "Avail" and press Return. And Avail tells you this: Type Available In-Use Maximum Largest chip 1004888 1091240 2096128 207000 fast 0 0 0 0 total 1004888 1091240 2096128 207000 This is called "memory fragmentation". You have a "playground" of 2 Megabytes of RAM, but little programs are playing all over it, leaving only small areas free between each of them rather than one large free area. Yes, programs are like kids, they love to scatter everywhere. Unfortunately, it's pratically impossible to "unfrag" memory on the Amiga. Best thing you can do is to try an "Avail Flush" in a Shell, hoping that it will remove a library from memory, allowing two small areas of 200 Kb each to join together into a 400 Kb large area, the library separating them being freed from memory. Ok, so we can't unfrag memory. How about giving a little more organization to these "kids", telling them to play together rather than scattering all over the playground? Well, it ain't that simple. Kids never listens. You know this, you've been there too. Fortunately, the memory "playground" has a little more order than a playground invaded by kids. Before getting access to the "playground", you must ask Mr. Exec if you can go play on it, and where to play. But you see, Mr. Exec is very busy watching Mrs. Paula sunbathing on the other side of the park, so he's doing a quick'n'dirty job: he starts looking from the closest area, spots the first area large enough for the one asking for space, and directs it there. And Mr. Exec goes back to the sight of Mrs. Paula, sunbathing in a tiny bikini. 2. But I have an Amiga, no a kid playground! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is how Exec works. When a program needs a memory area for its code or data, it asks Exec to assign it a free area, and goes there. This is what the DOS does when you load a program: it asks Exec for areas where it can put the program. And then, the programs asks Exec for areas where it can put the various data, libraries, etc... But problem is, when the program leaves, some of the data can stay in memory: 1) Files created in the Ram Disk. 2) Libraries/devices loaded. 3) Files in the clipboard, usualy located in the Ram Disk. 4) Areas "forgotten" by the program. 5) Small "patches" left in memory. And there is surely a few other cases. Or, another situation: Two programs are loaded one after the other. The first one leaves, but not the second one. Result: an area of the size of the first program is left, separated from the remaining free memory by the second program. We can't really fix this problem. But we can help reducing the number of situations where it can occur. This is what LowFrag can do. 3. Soooo... What does LowFrag do? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ LowFrag will eliminate one of the cases where memory gets fragmented: the case where a program is loaded (100-500 Kb) in memory, and exists leaving some data behind it in memory (clipboard, libraries, etc...). Or when another program puts small allocations just after the large program, so when the large program quites, it leaves a rather large block separated from the main area. How does LowFrag works? Usually, the data left in memory are small blocks (small when compared to the programs, that is). So, we could reduce fragmentation by separating the large allocations (usually applications) from the smaller allocations (usually data, patches, etc...). Exec's memory allocation has a flag which will do just that: MEMF_REVERSE -> Allocate memory starting from the top of memory. When this flag is used on a memory request, Exec will start looking from the TOP of memory for a free area, rather than from the bottom (as usual). So, we can separate allocations by putting the "MEMF_REVERSE" flag to the smaller allocations, so they stay out of the way of the larger allocations. This is what LowFrag does. When an allocation is larger than a threshold value, the allocation gets "stamped" with a "MEMF_REVERSE" flag, telling Exec to find it a suitable area by starting from the top of memory. So when the application leaves, any smaller allocations left in memory won't clutter the area usually needed by large. Here's a concrete case that I tested myself where LowFrag reduce fragmentation: 1) Launch a terminal (Terminus) - Around 400 Kb 2) Cut some text into the clipboard - A few bytes 3) Exit the terminal. Without LowFrag: The memory is splitted by the clipboard content. But with LowFrag, the clipboard content is at the top of the memory. So rather than "cutting" a 400 Kb block from the main area, it will probably cut around 10-50 Kb (small allocations done by the terminal: serial.device, various libraries, etc...). We can call this an improvement I guess :) 4. Sounds like fun... I want to use it! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Simple: copy LowFrag in your C: directory, and somewhere in your startup-sequence (usually just after SetPatch), add this single line: Run <>NIL: c:LowFrag value value is the threshold limit: allocations smaller than this value will get "promoted" with MEMF_REVERSE. If you don't put any value, then the default threshold of 32768 (32 Kb) will be used. I found this value to work very well on my own system, so try the patch first with this value, adjusting it afterward to suit your system. LowFrag can be aborted at any time by sending it a CTRL-C code, altough this isn't recommended: if another program patches AllocMem() or AllocVec() afterward, it's most likely that the system will crash. 5. That's it ~~~~~~~~~~~~ So enjoy. The whole thing is released as public domain. If you have any ideas on how to improve the thing, the source code is there in the archive (written in Amiga E, with some inline assembly), so just go ahead. A few possible ideas: - Promote allocations which has the flag MEMF_PUBLIC. - Have the user define a list of tasks which should always get promoted (altough this could probably slow down the system. Memory allocation is one of the cornerstones of the OS, so any changes there will reflect on the whole system's performances. 6. Hints & tips for programmers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can help reducing the memory fragmentation problem. a) When possible, use the memory pool system available since Exec V39 (it's been in amiga.lib since V33 I think?). b) For large executables, split it into separate hunks, so they could be loaded even on a fragmented system. c) Make sure you don't leave anything behind when you quit your program, unless when needed. Testing for memory leaks should be part of every debugging routine (Enforcer won't intercept by itself ALL kind of bugs, guys.) d) When doing an allocation meant to STAY in memory, why don't use the MEMF_REVERSE flag? e) For large applications, it is often a good idea to put part of the code into a shared library, and keep the library open only when it is needed. This way, it can be flushed from memory when the system needs more memory. 7. The Author ~~~~~~~~~~~~~ I can be reached via Fidonet: Eric Sauvageau>1:242/907.0 Or via Internet: merlin@thule.icenet.no dream@step.polymtl.ca (this is a friend's account) http://thule.icenet.no/~merlin This whole little thing is released on the Public Domain. Do whatever you want with it. As for I, I decline any responsability if you encounter problems while using this patch. It's been in use on my system and a friend's for many weeks before I released it, without any problem. I also wrote a few other goodies for the Amiga: TDPrefs - Preferences editor for the trackdisk.device. MFormat - Replacement for CBM's "Format" command. DevsMan - Manager for the Devs: and Storage directories. NewIcons3 - The ultimate icon enhancer. XPKatana - GUI/ARexx interface for the XPK packing system. See ya! Merlin ------