Ported by LouiSe

more info and other AMIGA ports at:
http://louise.amiga.hu

-----------------------------------------


HTDUMP

Copyright 2000 by Ren Hoek (renarak.cs.hro.nl)



Purpose
-------
A program to use in a script to easily dump http content
in a file.

The program will accept links in the standard URL form
and will support tweaking options as specifying the referrer
address and supporting resuming of files.

Importance is put on being sure files get transferred correctly
without errors and being able to tweak headers to overcome
anti-leeching procedures on websites.

Offcourse it's a valuable tool too for web developers like me
who work a lot of CGI's..

Ok most of the info is moved into htdump itself..
Try a 'htdump --help | less' and read. Or try the rather 
complex example right below. On the website there are
more examples available. Don't hesitate to email me
if you have any questions.

(Note.. I need to make this pretty.. and more mainstream
perhaps.. dunno.. let's see if anybody reads this first
*grin*)



Example
-------
Netscape has released version 4.72 and this time everybody
is allowed to download the 128-bit strong encryption
version, except the usual 'evil' people. To this
end Netscape has devised a method to protect the download
of their program. Since I'm Dutch, I can download this
program and don't have to worry about probable domain name
checks to see if I'm from an 'evil' country.

There are 2 reasons why I wanted to see how this
particulair situation worked, because 
I can't use a download manager like Getright
and Netscape froze up on me after it downloaded 89% of
the new install version. *grr* And also because
of all the check and weird checks, it's a good
test of htdump

So let's see if we can make it download..
First, let's see the page where we can download it

(line below is 1 line, I cut is in bits so you can
read it on 80 char monitors)

https://wwwus.netscape.com/usdl-bin/pdms_dnstest.cgi?
PRODUCT=communicator4.72-win32-en-complete-128&
COMPONENTS=CLIENT&TEMPLATES=NSCP

If we click 'Agree' a download is automatically
started. 

View the source and see:

<FORM METHOD="POST" ACTION="pdms_promise2.cgi">
<INPUT TYPE="SUBMIT" NAME="Accept" VALUE="I Accept">
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<INPUT TYPE="BUTTON" NAME="Accept" 
  VALUE="I Do Not Accept" onClick="window.history.back();">
<INPUT TYPE="hidden" NAME="PRODUCT"
  VALUE="communicator4.72-win32-en-complete-128">
<INPUT TYPE="hidden" NAME="COMPONENTS" VALUE="CLIENT">
</FORM>

If we check out the info for this page you will see

Form 1:
  Action URL: https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi
  Encoding: application/x-www-form-urlencoded (default)
  Method: Post

Ok now we have a target and we know what we have to post.
Only the hidden things get send to the cgi if I'm not mistaken..
SUBMIT's and BUTTON's don't get send.. So make the following
command (cut up again, it's 1 line)

htdump -d -P 
 "PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT"
 https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi 
 -u "Mozilla/4.05 [en] (Win95; U ;Nav)"


I had to fidget around to get it to work though.. it turned out
that Netscape also checks on User Agent.. Leave it out and
you get an internal server error back. :) So I just inserted
something nice :)


And we get this result:

-------------------------------------
Username: []
Password: []
B64_pass: []
Host    : [wwwus.netscape.com]
Port    : [443]
Path    : [usdl-bin/pdms_promise2.cgi]
Debug   : [1]
-------------------------------------
POST /usdl-bin/pdms_promise2.cgi HTTP/1.0
User-Agent: Mozilla/4.05 [en] (Win95; U ;Nav)
Content-type: application/x-www-form-urlencoded
Content-length: 64

PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT
-----------------------------------
SSL connection using RC4-MD5
header: [HTTP/1.1 302 Moved Temporarily]
header: [Server: Netscape-Enterprise/3.6]
header: [Date: Sun, 12 Mar 2000 00:35:14 GMT]
header: [Content-type: text/html]
header: [Set-cookie: NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_Rm
 UVRvydzyKhqQ== ; expires=Sunday, 12-Mar-00 04:35:15 GMT ; 
 domain=.netscape.com; secure]
header: [Refresh: 0; URL=http://home.netscape.com/download/thanks.html]
header: [Location: https:pdms_pick2.cgi/cc32d472.exe?product=
 communicator4.72-win32-en-complete-128&ext=.exe]
header: []
-------------------------------------
Content-length: 0
Errorcode: 302
-------------------------------------


Ok as you can see we get a load of data.. what is interesting
is the Cookie and the redirection we get. So a new target
(Again this is 1 line, just cut up to make it readable)
 
htdump
 "https://wwwus.netscape.com/usdl-bin/pdms_pick2.cgi/cc32d472.exe?
 product=communicator4.72-win32-en-complete-128&ext=.exe"
 -c "NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_RmUVRvydzyKhqQ==" 
 -f cc32i472.exe &

This will make it download in the background. Come back a few
hours later and it will be there complete. I tried to use
Ranges (-R) but it still spit out the same. It's a dumb script
that doesn't understand resume unfortunatly.. But even if
my primary goal failed I learned of the 'Location' header
which I never seen before :) You learn something every day.

I think you should be able to use htdump efficiently now.
Use the debug option to see what is going on, and experiment.

Have fun

        Ren Hoek (renarak.cs.hro.nl)

 PERMSSN    UID  GID    PACKED    SIZE  RATIO     CRC       STAMP          NAME
---------- ----------- ------- ------- ------ ---------- ------------ -------------
[generic]                 2617    5581  46.9% -lh5- c2d1 Oct 18  2001 htdump-0.9x.readme
[generic]                32189   32269  99.8% -lh5- ef64 Oct 18  2001 htdump-0.9x/arguments.c
[generic]                 1099    2448  44.9% -lh5- 3fa0 Oct 18  2001 htdump-0.9x/arguments.o
[generic]                  333    1281  26.0% -lh5- 1513 Oct 18  2001 htdump-0.9x/delnchar.c
[generic]                  193     248  77.8% -lh5- c000 Oct 18  2001 htdump-0.9x/delnchar.o
[generic]                 1209    2478  48.8% -lh5- 32b1 Oct 18  2001 htdump-0.9x/Documentation/CHANGELOG
[generic]                 1206    3154  38.2% -lh5- 33a1 Oct 18  2001 htdump-0.9x/Documentation/htlink.c
[generic]                 1185    2570  46.1% -lh5- 1070 Oct 18  2001 htdump-0.9x/Documentation/INSTALL
[generic]                 1985    4980  39.9% -lh5- 543a Oct 18  2001 htdump-0.9x/Documentation/LICENSE
[generic]                 2488    5271  47.2% -lh5- eabd Oct 18  2001 htdump-0.9x/Documentation/README
[generic]                19424   57601  33.7% -lh5- 02c5 Oct 18  2001 htdump-0.9x/Documentation/rfc1630.txt
[generic]                43116  137582  31.3% -lh5- f13d Oct 18  2001 htdump-0.9x/Documentation/rfc1945.txt
[generic]               129562  432138  30.0% -lh5- c449 Oct 18  2001 htdump-0.9x/Documentation/rfc2616.txt
[generic]                24684   79545  31.0% -lh5- a2f3 Oct 18  2001 htdump-0.9x/Documentation/rfc2617.txt
[generic]                 1894    4147  45.7% -lh5- 33e2 Oct 18  2001 htdump-0.9x/Documentation/TECHNICAL
[generic]                  768    1430  53.7% -lh5- 66b9 Oct 18  2001 htdump-0.9x/Documentation/TODO
[generic]                  755    2246  33.6% -lh5- 3caf Oct 18  2001 htdump-0.9x/enc_password.c
[generic]                  561     764  73.4% -lh5- 2b96 Oct 18  2001 htdump-0.9x/enc_password.o
[generic]                 1381    4380  31.5% -lh5- ef75 Oct 18  2001 htdump-0.9x/file.c
[generic]                  905    1774  51.0% -lh5- 098d Oct 18  2001 htdump-0.9x/file.o
[generic]                 7099   21813  32.5% -lh5- 6097 Oct 18  2001 htdump-0.9x/getopt.c
[generic]                 1815    4376  41.5% -lh5- ef6a Oct 18  2001 htdump-0.9x/getopt.h
[generic]                 1698    3757  45.2% -lh5- d351 Oct 18  2001 htdump-0.9x/getopt.o
[generic]                 1440    5303  27.2% -lh5- b78b Oct 18  2001 htdump-0.9x/global.h
[generic]                 9572   19188  49.9% -lh5- 047c Oct 18  2001 htdump-0.9x/htdump
[generic]                 1568    6878  22.8% -lh5- 8ec4 Oct 18  2001 htdump-0.9x/init.c
[generic]                 1343    3396  39.5% -lh5- bb40 Oct 18  2001 htdump-0.9x/init.o
[generic]                 1413    5834  24.2% -lh5- f382 Oct 18  2001 htdump-0.9x/main.c
[generic]                  780    1581  49.3% -lh5- eba5 Oct 18  2001 htdump-0.9x/main.o
[generic]                  544    1118  48.7% -lh5- befe Oct 18  2001 htdump-0.9x/Makefile
[generic]                  537    1862  28.8% -lh5- ffb4 Oct 18  2001 htdump-0.9x/mem2hex.c
[generic]                  457     921  49.6% -lh5- 97f7 Oct 18  2001 htdump-0.9x/mem2hex.o
[generic]                  925    3590  25.8% -lh5- 73a4 Oct 18  2001 htdump-0.9x/read_headers.c
[generic]                  485     924  52.5% -lh5- be75 Oct 18  2001 htdump-0.9x/read_headers.o
[generic]                  190     291  65.3% -lh5- 857b Oct 18  2001 htdump-0.9x/sgets.c
[generic]                  196     253  77.5% -lh5- d80b Oct 18  2001 htdump-0.9x/sgets.o
[generic]                  236     952  24.8% -lh5- ed65 Oct 18  2001 htdump-0.9x/socket-ssl.c
[generic]                 2125    7985  26.6% -lh5- 0326 Oct 18  2001 htdump-0.9x/socket.c
[generic]                  973    2046  47.6% -lh5- 7bc0 Oct 18  2001 htdump-0.9x/socket.o
[generic]                 2129    7193  29.6% -lh5- 7a26 Oct 18  2001 htdump-0.9x/tailor.h
[generic]                  314    1104  28.4% -lh5- fba3 Oct 18  2001 htdump-0.9x/title.c
[generic]                 1150    3493  32.9% -lh5- 3b46 Oct 18  2001 htdump-0.9x/usage.c
[generic]                 1067    2138  49.9% -lh5- 9cdd Oct 18  2001 htdump-0.9x/usage.o
---------- ----------- ------- ------- ------ ---------- ------------ -------------
 Total        43 files  305610  887883  34.4%            Oct 20  2001