| Alexandre Ferrieux 2008-03-30, 5:16 am |
| On Mar 30, 5:00=A0am, Fredderic <my-name-h...@excite.com> wrote:
> On Sat, 29 Mar 2008 05:03:50 -0700 (PDT),
>
> Alexandre Ferrieux <alexandre.ferri...@gmail.com> wrote:
>
> I've only ever used [fcopy] once, and now I remember why. =A0Doesn't that
> kind of usage strike anyone as being a seriously bodgy way to get
> progress feedback, speed limiting, or, well, to do just about anything
> other than pure automagical data movement...?
>
> I tried using [ftell] on the inbound channel, from a periodic timer
> that runs until the [fcopy -command] gets invoked, and it just reports
> -1 all the time. =A0That would at least make progress reporting possible.
>
> I suppose, if we could tell how much data is waiting at the inbound
> channel, we can cap it at some arbitrary value and use that as the -size
> argument. =A0But I don't know how to do that, and you'd still need your
> own [fileevent] listener to run [fcopy].
>
> Likewise for transfer rate limiting, there's no way to tell [fcopy] to
> transfer one buffer full and finish so we can update our progress,
> insert a pause, or whatever else. =A0Perhaps a -once option would be
> useful, but even then we still have to implement our own [fileevent]
> handler.
>
> And as soon as you start to dip into [fileevent] handling, you get the
> problems I mentioned in my previous post. =A0If the handler you're
> listening to is on a channel that's significantly faster than the other
> one, it'll spin like a mad man, so you need a means to listen to
> whichever channel is slower. =A0And just how big do you make that
> transfer size? =A0If your input stream is coming in sporadically, it'll
> sit there waiting because the amount of data that came through in the
> last burst was a single byte shorter than the size you specified. =A0In
> the meantime you could have posted the current progress, and/or done
> your speed limiting calculations and associated adjustments.
>
> I think personally, that being able to copy a single block (or as much
> of one as is available), would make [fcopy] actually useful for
> non-trivial functions. =A0It's probably still not entirely optimal with
> all the setup I suspect it's doing, but it's probably the best bet for
> making it truly useful anytime soon.
>
>
> Alternatively, moving data from one channel to another is a fairly
> common activity from what I've seen, both with and without processing
> inbetwee. =A0So I wonder if would be worth providing a means to create a
> [fileevent] that only triggers when BOTH the input and output channels
> are ready. (One-shot [fileevent]s would probably help there, we could
> then post something lean and mean on the wiki to do the job, without
> having to fiddle with continually setting and unsetting the [fileevent]
> handlers.)
>
> My choice, would be to split [fcopy] in half. =A0An engine that invokes a
> callback when there's both data waiting and somewhere to put it (output
> buffers contain less than a block of data, for example), and a
> mechanism for transferring up to a certain amount of data from one
> channel to another with minimal memory copying:
>
> =A0 =A0 =A0 =A0 fcopy infile outfile -onready command
> =A0 =A0 =A0 =A0 fcopy infile outfile -single ?-size size?
>
> The basic existing [fcopy] command could then be "expanded" as:
>
> =A0 =A0 =A0 =A0 fcopy $in $out -onready [list fcopy $in $out -single]
>
> The -onready option would be compatible with -command, though probably
> not -size (although having -size pre-load the input buffer when used in
> conjunction with -onready might offer some benefit, or at least
> make [chan pending] more useful). =A0In essence, the default -onready
> command would simply invoke one execution of [fcopy]'s existing data
> transfer code, which is exactly what the -single option does. =A0I
> haven't looked at the source to [fcopy], but I wouldn't imagine it to
> be a particularly complex change.
>
> Most importantly, rate limiting, progress reporting, and whatever else,
> would grow quite readily out of that basic split in functionality.
> Although for rate limiting you probably will want some means to "pause"
> and/or cancel the handler (return value?), or a -continue option (or
> something similar) that would normally be invoked internally by -single
> (perhaps even just "-single -size 0" would be acceptible, albeit ugly).
>
> Comments, anyone?
>
> Fredderic
Hmmm... I think you've missed one tiny bit about [fcopy -size]'s
semantics :-)
It does exactly what you call "-once mode".
Thus, today's fcopy offers in a single "primitive" (in the sense that
it cannot be emulated at full efficiency in pure Tcl, though it can be
aproached):
- both synchronous and asynchronous transfers
- progress notification with configurable granularity
- bandwidth throttling with a simple [after] in the callback
- all of this with the guarantee of never accumulating data in
memory if the input is faster than the output (a mistake that is easy
to make when doing it by hand)
The above doesn't strike me as "seriously bodgy". I see it rather as a
well-designed compromise.
(Notice that this doesn't weaken the importance of fixing its current
implementation bugs, which we are actively analyzing right now. The
point is that the API is good, regardless of the quality of the
underlying code at any given time).
Now if there's one practical situation that it doesn't handle, or does
awkwardly, please describe it.
This may result in another attempt to push the envelope, and we both
enjoy that :-)
-Alex
|