MegaCopy
========
A drop-in replacement for the AmigaDOS `Copy` command that does the same
thing, accepts the same arguments, and finishes sooner.
*** READ THIS FIRST: THIS IS AN EARLY BETA ***
===============================================
MegaCopy is new code. It has been reviewed hard and tested, but it has
NOT been through years of daily use the way C:Copy has, and it has had
very little time on real hardware.
It is a tool that WRITES FILES. A bug in it can destroy data.
YOU USE THIS AT YOUR OWN RISK. THERE IS NO WARRANTY OF ANY KIND.
The author is not responsible for lost files, corrupted volumes, broken
installations or wasted time. If your data matters to you, you must have
a backup BEFORE you point this at anything. That is your job, not the
program's. Nobody is going to be able to give your files back to you.
If you are not willing to accept that, do not use it. That is a
completely reasonable decision and no one will think less of you for it.
- Do not point it at anything you do not have a backup of.
- Do not put it in a boot script yet.
- Test it on something you can afford to lose first.
What has been verified so far: a 5758-file / 414-directory / 229 MB tree
copied byte-for-byte identical (verified from outside the Amiga), a
9-case behaviour suite against the real AmigaOS 3.1 `copy.c` source, and
an 8-case wildcard suite. All of that was under emulation. Real hardware
coverage is thin.
Bug reports are very welcome.
THE SAFE WAY TO TRY IT: DON'T INSTALL IT AT ALL
===============================================
You do NOT have to replace C:Copy to use MegaCopy. That is the fast
route, not the only one, and it is not the one to start with.
Just drop the `MegaCopy` binary into your `C:` drawer and call it by its
own name:
MegaCopy DH0:Work DH1:Backup ALL CLONE
That is it. Nothing is replaced, nothing is renamed, your `C:Copy` is
never touched, and there is nothing to undo. If MegaCopy turns out to
have a bug, it can only affect the commands where you typed `MegaCopy`
yourself. Your Startup-Sequence, your installers and every script on the
machine keep using the Copy they have always used.
It takes exactly the same arguments as Copy, so anything you know how to
type for one works for the other.
**Start here.** Use it by hand for a while, on trees you have backed up.
Only consider installing it as C:Copy once you trust it -- and read the
INSTALLING section carefully first, because that step is the one with
consequences.
WHY IT IS FASTER
================
Not magic, and not a bigger buffer. The reason is boring and countable:
**fewer DOS packets per file.**
Every operation on an AmigaDOS file is a message to a filesystem handler
process: a round trip with a task switch at each end. On a tree of small
files -- a WHDLoad collection is the perfect example -- that traffic, not
the data, is what takes the time.
Per file, with CLONE, the floor is four packets:
FINDOUTPUT -> open the destination
WRITE -> write the data
END -> close it
SetFileDate -> stamp the date
The stock Copy pays those four plus two more: `SetProtection` and
`SetComment`, on every single file, unconditionally.
MegaCopy skips both when it can prove they cannot change anything:
- a file created with MODE_NEWFILE is born with protection 0, so if
the source protection (masked of the archive bit) is also 0 -- which
is the case for the vast majority of files -- then SetProtection
would write the value that is already there;
- a newly created file has no comment, so if the source has none
either, SetComment is a packet that changes nothing.
Both skips only apply when the destination is genuinely new. If the file
already existed, the old bits and the old comment are still there and
must be overwritten, so both calls are made.
**6 packets down to 4.** If packets were the only cost, that predicts
33% faster. Measured: 19-23% depending on the tree. The difference is
the data itself, which neither program can avoid.
That the prediction and the measurement agree is the point. It is not a
number that came out of a benchmark and got rationalised afterwards.
Directory scanning uses `ExAll` with ED_COMMENT, which returns dozens of
entries per packet complete with type, size, protection, date and
comment. In a 500-file directory that is roughly 10 packets instead of
about 1000.
Measured, under WinUAE, on the same tree, alternating runs:
tree of 5758 files / 414 dirs / 229 MB
C:Copy 41-43 s
MegaCopy 33-34 s 21-23% faster, reproduced 5 times
tree of 807 files / 77 dirs / 51 MB (MegaBench, 1/50s clock)
C:Copy 6.02 s (spread 5.98 - 6.08)
MegaCopy 4.88 s (spread 4.86 - 4.90) 19% faster
Also worth knowing: with NOREP, re-syncing a tree where nothing changed
took 9 s against 43 s for a full copy. The V40 Copy has no NOREP at all,
so there is nothing to compare against there -- it is simply something
the stock command cannot do.
WHERE IT COULD BE FASTER, BUT IS NOT PROVEN
===========================================
Everything above is measured. Everything in this section is theory, and
is written down precisely so nobody mistakes it for a result.
**Overlapped I/O.** MegaCopy keeps two DOS packets in flight: while the
destination writes block N, the source is already reading N+1. Between
two different devices this should approach 2x on the data. It has NOT
been demonstrated. Measurement under emulation could not confirm it,
because the host's disk cache removes exactly the latency the overlap is
supposed to hide. Worse, on an A1200 with IDE the transfer is PIO --
reading and writing consume the *same* CPU -- so there is no physical
overlap of data there at all; the honest expectation is 1.1-1.3x. Real
DMA (SCSI on Zorro II/III) is where this should show. Untested.
**Lockstep re-sync.** When re-syncing, the source directory and the
destination directory are served by two *different* handler processes,
so their two `ExAll` scans can be in flight simultaneously instead of
one after the other. Implemented; predicted around 2x on re-sync; not
yet measured on hardware where it could matter.
**MaxTransfer and Mask.** A filesystem will only DMA straight out of
your buffer if the buffer respects the device's `de_Mask` and
`de_MaxTransfer` from its DosEnvec. If it does not, the FFS quietly
bounce-copies through its own buffer and you pay for every byte twice.
MegaCopy does not read those values yet. This cannot be tested under a
directory-mount emulator, which has no FileSysStartupMsg at all -- it
needs real hardware with a real device.
**Where the real floor is.** On the destination side, an FFS writing a
small file does 3-4 block writes plus 2-3 rewrites of the same header,
because close re-stamps the date and SetProtection and SetFileDate are
separate packets with no ACTION that merges them. On a period hard disk
that is 50-70 ms per file. At that point the limit is the destination
filesystem, not the copier. Getting past it means changing the approach
(raw writes to a verified-empty volume) or changing the filesystem
(PFS3 handles metadata atomically and in batches). Neither is in here.
One measurement did come out clean and is worth stating: on the test
rig, a full copy took **exactly as long as the write-only path**. Adding
the real read of the source cost no measurable time. So on that machine
there was nothing left to win on the reading side, and everything that
remained was the destination.
Two things that sound like they should help and were measured as doing
**nothing**: raising task priority, and asking the filesystem for more
cache buffers with ACTION_MORE_CACHE. Both are still available as
switches (TURBO, CACHE) because on real hardware they may behave
differently, but do not expect anything from them.
COMPATIBILITY
=============
The argument template is the V46/V54 Copy template, character for
character, plus three extensions at the end:
FROM/M,TO/A,ALL/S,Q=QUIET/S,BUF=BUFFER/K/N,CLONE/S,DATES/S,
NOPRO=NOPROTECTION/S,COM=COMMENT/S,NOREQ=NOREQUESTERS/S,
NOREP=NOREPLACE/S,INTER=INTERACTIVE=ASKREP=ASKREPLACE/S,
FORCE/S,NEWER/S,ARC=ARCHIVE/S,COPYLINKS/S,SPEED/S,STATS/S,
PROGRESS/S,TURBO/S,CACHE/S
Behaviour was checked against the actual AmigaOS 3.1 `copy.c` source
rather than against the manual, because the two disagree in places.
Things that follow the source and might surprise you:
- Several directories as FROM get their *contents* merged into TO. It
does not create one subdirectory per source. That is what the stock
does (it steps into non-wildcard directories).
- A directory matched by a *wildcard* is different: that one is
recreated as TO/name, and is not descended into unless ALL is given.
- FROM is optional. With no FROM it copies the current directory.
- A wildcard in TO is rejected.
- Return codes match the stock, including the distinction between the
first FROM missing (fails, 20, and stops) and a later one missing
(warns, 5, and carries on).
- On an I/O error, the partial destination file is deleted, exactly as
the stock does. It does not leave a plausible-looking truncated file
behind.
- Ctrl-C returns 20 and re-signals the shell, so your script stops.
This matters: a script of the form "copy, then delete the original"
must not proceed after an interrupted copy.
THE EXTRA ARGUMENTS
===================
Everything the V40 Copy understands, MegaCopy understands, and it means
the same thing. Nothing was removed and nothing changed meaning. Anything
you already type at `Copy` works unchanged -- that is the whole design
constraint, because the moment it is installed as C:Copy, every script on
the system is depending on it.
On top of that there are extra switches. Eight of them are not really
"ours": they are from the LATER AmigaOS Copy (V46/V54), so MegaCopy is a
superset of the 3.1 command rather than an invention. Three are ours.
From the newer Copy:
NOREP (NOREPLACE)
Do not touch files that already exist in the destination. This is
what turns a copy into a re-sync, and it is where MegaCopy is
furthest ahead: it reads the destination directory ONCE with ExAll
and answers every lookup from memory, instead of a Lock+Examine
pair per file. The V40 command does not have NOREP at all.
NEWER
Copy only when the source is newer than the file already in the
destination. The usual companion to NOREP for backups.
INTER (INTERACTIVE, ASKREP, ASKREPLACE)
Ask before replacing each existing file.
FORCE
Overwrite even when the destination is delete- or write-protected.
It clears the protection bits, copies, and then puts the source's
bits on -- which are the ones that belong there anyway. Directories
it had to unprotect get their original bits restored on the way
out, including if you interrupt with Ctrl-C.
ARC (ARCHIVE)
Incremental backup. Copies only files whose archive bit is clear
(i.e. changed since the last archive run) and then sets that bit on
the SOURCE, so the next run skips them.
COPYLINKS
Follow links instead of skipping them.
SPEED
Accepted and ignored. It exists so scripts that pass it do not
break; MegaCopy always goes as fast as it can.
STATS
Print the final summary (files, directories, KB, elapsed, KB/s)
even when QUIET is on. Useful for logging a scripted copy.
Ours:
PROGRESS
Force the progress display even under QUIET. For files above 128K
it draws a bar; below that it refreshes a counter on one line every
16 files, so a tree of thousands of small files does not sit there
looking frozen. It only draws if the output really is an
interactive console, so a redirect to a file or a pipe never
receives control characters.
TURBO
Raise the task priority by 1 while copying, restored on exit.
HONEST WARNING: measured as doing nothing on the test rig. Kept
because real hardware may differ.
CACHE
Ask both volumes for more filesystem cache buffers
(ACTION_MORE_CACHE), returned on exit. Same honest warning: it
measured as a no-op. It also asks for 256 buffers per volume, which
on a 512K machine is a lot of memory to hand over.
Everything is restored when the program exits, including on Ctrl-C:
priority, cache buffers, the shell's current directory, and the
requester suppression from NOREQ.
Requires Kickstart 2.04 (V37). It uses ReadArgs, AllocDosObject, SendPkt,
CreateMsgPort and ExAll, which are 2.0-era calls; `ExAllEnd` is 3.0 and
is gated at runtime. Built for plain 68000, so it runs on an A500 with a
2.04 ROM as well as on an 060.
Known deliberate difference: `BUF=0` in the stock means "allocate a
buffer exactly the size of the file", the classic trick to get the FFS
to lay the file out contiguously. MegaCopy uses a fixed two-buffer
pipeline and gets contiguity from SetFileSize instead, which serves the
same purpose. The switch is accepted and means "as large as possible".
INSTALLING (AND UNDOING IT)
===========================
Read the "SAFE WAY TO TRY IT" section above first. You do not need this
step at all, and it is the only step that has consequences.
Installing only buys you one thing: it speeds up the *rest of the
system*, because scripts, installers and the Workbench start using
MegaCopy without knowing it. That is also exactly why it is the risky
option -- a bug now reaches everything, including your Startup-Sequence.
Concretely, what starts going faster without you doing anything:
- **Directory Opus**, when it is configured to copy by shelling out to
C:Copy rather than with its own internal routine. A lot of DOpus 4
button and menu setups are exactly that, and file management is
precisely the load MegaCopy is built for: many small files with
directory traffic dominating.
- **Installer scripts** and the whole `Install` / `InstallerLG` family,
which copy hundreds of small files one at a time -- the worst case
for the stock command and the best case for this one.
- **Your own boot and backup scripts**, and anything in
S:User-Startup that moves files around.
- **WHDLoad collection housekeeping**: unpacking, moving and
re-syncing game directories.
If a program does its own copying internally instead of calling C:Copy
-- and many do -- installing changes nothing for it. Nothing breaks; you
simply do not get the speed-up there.
**IT IS FULLY REVERSIBLE.** Option 2 in the installer puts your original
Copy back, byte for byte, with its protection bits, its date and its
comment intact. And even if the installer itself will not run, one shell
command undoes it:
Rename C:Copy_original C:Copy
Nothing is hidden, encrypted or scattered around the system. There is
exactly one file moved and one file added, and you can undo both by
hand with commands you already know. Still: the responsibility for
deciding to do this on your machine is yours.
Double-click `InstallMegaCopy` (or run it from a shell). It shows you
what is currently installed, whether a backup exists, and offers:
1 = Install MegaCopy as C:Copy
2 = Restore the original Copy
3 = Quit
**It is reversible.** Option 1 saves your real `C:Copy` to
`C:Copy_original` first; option 2 puts it back. The backup is treated as
untouchable: if one already exists it is never overwritten, so
installing twice cannot bury the genuine Copy under a copy of MegaCopy.
Safety rules it follows, because getting this wrong leaves a machine
that cannot copy files to fix itself:
- Nothing is written over `C:Copy` until the replacement has been
written to a temporary file, closed with the close *checked*, and
verified by size. A close that fails to flush is exactly the case
"everything wrote fine and the file is wrong", and it is checked.
- The swap is rename-rename-delete, never delete-then-rename. There is
no instant when `C:Copy` does not exist, and if anything fails the
original is put straight back.
- Protection bits, date and comment are preserved, so a restored Copy
is the same file you started with -- including the `p` bit, without
which `Resident C:Copy ADD` would start failing.
- A pre-existing backup is validated as a real executable before being
trusted, so a truncated or empty `Copy_original` cannot lead you
into a dead end.
If your startup scripts use `Resident`, it warns you. MegaCopy is NOT
reentrant and must not be made resident.
**Write this down before installing.** If you ever boot the same disk
under Kickstart 1.3, neither MegaCopy nor the installer will run, both
need 2.04, and `C:Copy` will be dead. The way out is one command:
Rename C:Copy_original C:Copy
The installer prints this on screen before it asks you anything.
THE BENCHMARKING TOOLS
======================
Included because the speed claims above should be something you can
check yourself, not something you have to believe.
**MegaBench** is a program, not a script, and that is deliberate: a
shell script can only time with `Date`, which has one-second resolution.
On a six-second run that is +-17% of error -- the measurement would be
pure noise. MegaBench uses `DateStamp`, so 1/50 s.
MegaBench <from> <to> [PASSES <n>] [KEEP]
What it does to make the comparison honest:
- Sends ACTION_FLUSH to both volumes between runs. Without that, the
second program to run finds half the first one's metadata still in
the filesystem's buffers and wins for the wrong reason.
- Alternates the order: odd passes start with the stock, even passes
start with MegaCopy, so leftover warm cache is shared out instead of
always going to the same one.
- Discards the first pass entirely, as warm-up.
- Deletes and recreates the destination before every single run.
- Uses `C:Copy_original` if it exists. This is not a detail: once you
install MegaCopy as C:Copy, a benchmark that calls plain `Copy` is
comparing MegaCopy against itself and will report a dead heat
forever. It tells you which one it used.
- Checks the return code of every run. A copy that failed also
"finishes quickly", and that number is worthless.
What it cannot do, stated plainly: under an emulator the host's own disk
cache is in charge and no program inside the Amiga can flush it. Numbers
from an emulator are good for comparing A against B, not as absolutes.
The way to beat the host cache is to use a tree larger than the host's
RAM.
Note that `PASSES 3` means eight full copies of the tree (three measured
passes plus a warm-up, times two programs). For a large tree use
`PASSES 1`.
Also included, as AmigaDOS scripts:
**Bench** - MegaCopy against the stock Copy, alternating, with
everything logged to a file so you do not need to
photograph the screen.
**Patrones** - eight wildcard cases (#?.txt, #? with and without ALL,
wildcards in two path components, no match, TO that
does not exist), each with a listing to check.
**Verificar** - nine behaviour cases against the stock: FROM omitted,
several directories merging, wildcard TO, NIL: as a
destination, QUIET really being quiet, and the return
codes. Each case prints OK or MAL.
**Probar** - a quick smoke test on a small tree with edge cases
(zero-byte file, empty directory, a file larger than
the buffer, four levels deep).
The scripts write their output to a log next to themselves, so they can
be run on a machine whose screen you cannot easily read.
CREDITS
=======
Pancho Manera / PAL-N Studios.
Written with the Macumba Editor, the same environment used for
PocketLauncher and the rest of the PAL-N Studios tools.
The AmigaOS 3.1 `copy.c` source was used as the reference for
behavioural compatibility, and the dos.library autodocs and the
`pattern.asm` source for everything about ExAll, MatchFirst and the
packet protocol. Where the manual and the source disagreed, the source
won.
Built with FPC for m68k, targeting plain 68000 with the small-puddle
OS heap so it runs on chip-RAM-only machines.
|
aminet net>