Loading a web page with Lispworks

Hi, I am a Lisp newbie. I am trying to load a web page into a string
with Lispworks, in order to parse the html code.
I am trying to use open-tcp-stream, but I am  about the value
I have to use for the parameter "service".
Basically I just want to do:

(with-open-stream (web-page (comm:open-tcp-stream
"www.thewebpage.com" ?))
(loop for line = (read-line web-page nil nil)
while line
do (write-line line))))

First, I don't know what the value of "?" should be. Second, I have
the hunch this will not work anyway. Can anyone tell me what am I
missing?
Finally, is there a place where I can look for some examples? I have
looked at the Lispworks Reference Manual, which is very complete, but
I get overwhelmed. Is there something simpler?

Thanks a lot for any help.

PS: I am still undecided about whether to use Lispworks or CLISP, so
the same question applied to CLISP would also help me a lot.

On Apr 1, 10:33 pm, littlelis...@hotmail.com wrote:
> Hi, I am a Lisp newbie. I am trying to load a web page into a string
> with Lispworks, in order to parse the html code.
>
> (with-open-stream (web-page (comm:open-tcp-stream
> "www.thewebpage.com" ?))
>   (loop for line = (read-line web-page nil nil)
>         while line
>         do (write-line line))))
>
> PS: I am still undecided about whether to use Lispworks or CLISP, so
> the same question applied to CLISP would also help me a lot.

In CLISP (using the construct you used):

(with-open-stream (stream (ext:open-http "http://
www.thewebpage.com/"))
(loop for line = (read-line stream nil nil nil)
while line
do (write-line line)))

...should work.

--
Phil
http://phil.nullable.eu/

On 1 Apr., 23:33, littlelis...@hotmail.com wrote:
> Hi, I am a Lisp newbie. I am trying to load a web page into a string
> with Lispworks, in order to parse the html code.
> I am trying to use open-tcp-stream, but I am  about the value
> I have to use for the parameter "service".
> Basically I just want to do:

... use Drakma - a http-client for Common Lisp: http://weitz.de/drakma/

ciao,
Jochen

On Tue, 01 Apr 2008 14:51:08 -0700, philip.armitage wrote:

> On Apr 1, 10:33 pm, littlelis...@hotmail.com wrote: 
>
> In CLISP (using the construct you used):
>
> (with-open-stream (stream (ext:open-http "http:// www.thewebpage.com/"))
>   (loop for line = (read-line stream nil nil nil)
>     while line
>     do (write-line line)))
>
> ...should work.

almost .. you need to send the initial request line before you start
reading, like:

(princ "GET / HTTP/1.1" stream)

..followed by two "newlines":

(defvar *crlf* (format stream "~C~C" #\Return #\Linefeed))
(princ *crlf* stream)
(princ *crlf* stream)

..but yeah, use Drakma.. :)

--
Lars Rune Nøstdal
http://nostdal.org/

On Wed, 02 Apr 2008 01:27:09 +0000, Lars Rune Nøstdal wrote:
>
> (defvar *crlf* (format stream "~C~C" #\Return #\Linefeed)) (princ *crlf*
> stream)

look at what my news client did, again .. pan was better before 2.x ...

i'm going to turn "wrap text" off for good this time and see if it helps

even if i do some mistakes when wrapping text "manually" maybe the damage
will be less .. *sigh*

--
Lars Rune Nøstdal
http://nostdal.org/

On Apr 2, 2:27 am, Lars Rune N=F8stdal <larsnost...@gmail.com> wrote:
> On Tue, 01 Apr 2008 14:51:08 -0700, philip.armitage wrote: 
> 
>
> almost .. you need to send the initial request line before you start
> reading, like:
>
> (princ "GET / HTTP/1.1" stream)
>
> ..followed by two "newlines":
>
> (defvar *crlf* (format stream "~C~C" #\Return #\Linefeed))
> (princ *crlf* stream)
> (princ *crlf* stream)

No, it works fine without doing all that (I tested before I posted the
code!). Probably you would need to do that on a raw TCP stream.

--
Phil
http://phil.nullable.eu/

philip.armitage@gmail.com wrote:
>
> No, it works fine without doing all that (I tested before I posted the
> code!). Probably you would need to do that on a raw TCP stream.

you're right, i did not see the "ext:open-http"-part

--
Lars Rune Nøstdal
http://nostdal.org/

On Apr 2, 10:06 am, philip.armit...@gmail.com wrote:
> On Apr 2, 8:56 am, littlelis...@hotmail.com wrote:
>
>
> 
> 
> 
he 
> 
> 
> 
> 
> 
>
> Can you post the exact code you used in CLISP? That error message
> often indicates that a duff URL was supplied (note that in my original
> post, the URL got wrapped where it shouldn't have...).
> --
> Philhttp://phil.nullable.eu/

Thanks for the quick reply. I used:

(with-open-stream (stream (ext:open-http
"http://www.google.com/"))
(loop for line =3D (read-line stream nil nil nil)
while line
do (write-line line)))

It says:

;; connecting to "http://www.google.com"...

and then the above error.

On Apr 2, 9:15 am, littlelis...@hotmail.com wrote:

> Thanks for the quick reply. I used:
>
> (with-open-stream (stream (ext:open-http
> "http://www.google.com/"))
>   (loop for line = (read-line stream nil nil nil)
>     while line
>     do (write-line line)))
>
> It says:
>
> ;; connecting to "http://www.google.com"...
>
> and then the above error.

Hmmmm...I'm honestly not sure then. That works fine for me with clisp
2.41 on Linux and 2.42 on Windows. Probably one to take to the clisp
mailing list I guess.

--
Phil
http://phil.nullable.eu/

<littlelisper@hotmail.com> wrote:
+---------------
| Hi, I am a Lisp newbie. I am trying to load a web page into a string
| with Lispworks, in order to parse the html code.
| I am trying to use open-tcp-stream, but I am  about the value
| I have to use for the parameter "service".
| Basically I just want to do:
|
| (with-open-stream (web-page (comm:open-tcp-stream
| "www.thewebpage.com" ?))
|   (loop for line = (read-line web-page nil nil)
|         while line
|         do (write-line line))))
|
| First, I don't know what the value of "?" should be. Second, I have
| the hunch this will not work anyway. Can anyone tell me what am I
| missing?
+---------------

You're missing quite a bit. See RFC 1945 "Hypertext Transfer
Protocol -- HTTP/1.0" <http://www.ietf.org/rfc/rfc1945.txt>
and RFC 2616 "Hypertext Transfer Protocol -- HTTP/1.1"
<http://www.ietf.org/rfc/rfc2616.txt> for details.

Note that for simple client apps you can usually get away with
implementing only the HTTP/1.0 protocol, *except*... even when
doing HTTP/1.0 you should *always* send the HTTP/1.1 "Host:" request
header anyway, since so many web sites these days use name-based
virtual hosting. [Fortunately, all of the popular web servers will,
as required by HTTP/1.1, correctly interpret the "Host:" request
header even if the request uses HTTP/1.0 protocol otherwise.]


-Rob

p.s. A simple standards-conforming HTTP/1.0 [+"Host:"] "GET"-only
client *can* be written in only ~100 lines of CL. [Handling "POST"
is only slightly more complicated.]

-----
Rob Warnock			<rpw3@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607