[project @ Arch-1:robey@lag.net--2003-public%secsh--dev--1.0--patch-156]

rewrite channel pipes to work on windows
the pipe system i was using for simulating an os-level FD (for select) was
retarded.  i realized this week that i could just use a single byte in the
pipe to signal "data is ready" and not try to feed all incoming data thru
the pipe -- and then i don't have to try to make the pipe non-blocking (which
should make it work on windows).  a lot of duplicate code got removed and now
it's all going thru the same code-path on read.

there's still a slight penalty on incoming feeds and calling 'recv' when a
pipe has been opened (by calling 'fileno'), but it's tiny.

removed a bunch of documentation and comments about things not working on
windows, since i think they probably do now.

4 files changed, 34 insertions, 255 deletions

diff --git a/MANIFEST.in b/MANIFEST.in
index fab5a4dc..b7e551e9 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,3 +1,3 @@
-include ChangeLog LICENSE test.py demo.py demo_simple.py demo_server.py demo_windows.py forward.py
+include ChangeLog LICENSE test.py demo.py demo_simple.py demo_server.py forward.py
 recursive-include docs *
 recursive-include tests *.py *.key
diff --git a/README b/README
index 526ffe43..8ff90bbc 100644
--- a/README
+++ b/README
@@ -47,17 +47,10 @@ also think it will work on Windows, though i've never tested it there.  if
 you run into Windows problems, send me a patch: portability is important
 to me.
 
-the Channel object supports a "fileno()" call so that it can be passed
-into select or poll, for polling on posix.  once you call "fileno()" on a
-Channel, it changes behavior in some fundamental ways, and these ways
-require posix.  so don't call "fileno()" on a Channel on Windows.  this is
-detailed in the documentation for the "fileno" method.
-
 python 2.2 may work, thanks to some patches from Roger Binns.  things to
 watch out for:
 * sockets in 2.2 don't support timeouts, so the 'select' module is
-  imported to do polling.  this may not work on windows.  (works fine on
-  osx.)
+  imported to do polling.  
 * logging is mostly stubbed out.  it works just enough to let paramiko
   create log files for debugging, if you want them.  to get real logging,
   you can backport python 2.3's logging package.  Roger has done that
@@ -102,32 +95,23 @@ connection is not secure!)
 
 the following example scripts get progressively more detailed:
 
-demo_windows.py
-    executes 'ls' on any remote server, loading the host key from your
-    openssh key file.  (this script works on windows because it avoids
-    using terminal i/o or the 'select' module.)  it also creates a logfile
-    'demo_windows.log'.
-
 demo_simple.py
     calls invoke_shell() and emulates a terminal/tty through which you can
     execute commands interactively on a remote server.  think of it as a
-    poor man's ssh command-line client.  (works only on posix [unix or
-    macosx].)
+    poor man's ssh command-line client.
 
 demo.py
     same as demo_simple.py, but allows you to authenticiate using a
-    private key, and uses the long form of some of the API calls.  (posix
-    only.)
+    private key, and uses the long form of some of the API calls.
 
 forward.py
     command-line script to set up port-forwarding across an ssh transport.
-    (requires python 2.3 and posix.)
+    (requires python 2.3.)
 
 demo_server.py
     an ssh server that listens on port 2200 and accepts a login for
     'robey' (password 'foo'), and pretends to be a BBS.  meant to be a
-    very simple demo of writing an ssh server.  (should work on all
-    platforms.)
+    very simple demo of writing an ssh server.
 
 
 ***  USE
@@ -235,3 +219,4 @@ v0.9 FEAROW
 
 * ctr forms of ciphers are missing (blowfish-ctr, aes128-ctr, aes256-ctr)
 * server mode needs better documentation
+* why are big files so slow to transfer?  profiling needed...
diff --git a/demo_windows.py b/demo_windows.py
deleted file mode 100755
index f3f9f902..00000000
--- a/demo_windows.py
+++ /dev/null
@@ -1,129 +0,0 @@
-#!/usr/bin/python
-
-# Copyright (C) 2003-2005 Robey Pointer <robey@lag.net>
-#
-# This file is part of paramiko.
-#
-# Paramiko is free software; you can redistribute it and/or modify it under the
-# terms of the GNU Lesser General Public License as published by the Free
-# Software Foundation; either version 2.1 of the License, or (at your option)
-# any later version.
-#
-# Paramiko is distrubuted in the hope that it will be useful, but WITHOUT ANY
-# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
-# A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
-# details.
-#
-# You should have received a copy of the GNU Lesser General Public License
-# along with Paramiko; if not, write to the Free Software Foundation, Inc.,
-# 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA.
-
-
-#   This demo is like demo_simple.py, but it doesn't try to use select()
-#   to poll the ssh channel for reading, so it can be used on Windows.
-#   It logs into a shell, executes "ls", prints out the results, and
-#   exits.
-
-
-import sys, os, base64, getpass, socket, traceback
-import paramiko
-
-if os.environ.has_key('HOME'):
-    # unix
-    HOME = os.environ['HOME']
-else:
-    # windows
-    HOME = os.environ['HOMEDRIVE'] + os.environ['HOMEPATH']
-
-
-#####   utility functions
-
-def load_host_keys():
-    filename = HOME + '/.ssh/known_hosts'
-    keys = {}
-    try:
-        f = open(filename, 'r')
-    except Exception, e:
-        print '*** Unable to open host keys file (%s)' % filename
-        return
-    for line in f:
-        keylist = line.split(' ')
-        if len(keylist) != 3:
-            continue
-        hostlist, keytype, key = keylist
-        hosts = hostlist.split(',')
-        for host in hosts:
-            if not keys.has_key(host):
-                keys[host] = {}
-            if keytype == 'ssh-rsa':
-                keys[host][keytype] = paramiko.RSAKey(data=base64.decodestring(key))
-            elif keytype == 'ssh-dss':
-                keys[host][keytype] = paramiko.DSSKey(data=base64.decodestring(key))
-    f.close()
-    return keys
-
-
-# setup logging
-paramiko.util.log_to_file('demo_windows.log')
-
-# get hostname
-username = ''
-if len(sys.argv) > 1:
-    hostname = sys.argv[1]
-    if hostname.find('@') >= 0:
-        username, hostname = hostname.split('@')
-else:
-    hostname = raw_input('Hostname: ')
-if len(hostname) == 0:
-    print '*** Hostname required.'
-    sys.exit(1)
-port = 22
-if hostname.find(':') >= 0:
-    hostname, portstr = hostname.split(':')
-    port = int(portstr)
-
-
-# get username
-if username == '':
-    default_username = getpass.getuser()
-    username = raw_input('Username [%s]: ' % default_username)
-    if len(username) == 0:
-        username = default_username
-password = getpass.getpass('Password for %s@%s: ' % (username, hostname))
-
-
-# get host key, if we know one
-hostkeytype = None
-hostkey = None
-hkeys = load_host_keys()
-if hkeys.has_key(hostname):
-    hostkeytype = hkeys[hostname].keys()[0]
-    hostkey = hkeys[hostname][hostkeytype]
-    print 'Using host key of type %s' % hostkeytype
-
-
-# now, connect and use paramiko Transport to negotiate SSH2 across the connection
-try:
-    t = paramiko.Transport((hostname, port))
-    t.connect(username=username, password=password, hostkey=hostkey)
-    chan = t.open_session()
-    print '*** Here we go!'
-    print
-
-    print '>>> ls'
-    chan.exec_command('ls')
-    f = chan.makefile('r+')
-    for line in f:
-        print line.strip('\n')
-
-    chan.close()
-    t.close()
-
-except Exception, e:
-    print '*** Caught exception: ' + str(e.__class__) + ': ' + str(e)
-    traceback.print_exc()
-    try:
-        t.close()
-    except:
-        pass
-    sys.exit(1)
diff --git a/paramiko/channel.py b/paramiko/channel.py
index 7862471f..81e42ec4 100644
--- a/paramiko/channel.py
+++ b/paramiko/channel.py
@@ -31,12 +31,6 @@ from ssh_exception import SSHException
 from file import BufferedFile
 
 
-# this is ugly, and won't work on windows
-def _set_nonblocking(fd):
-    import fcntl
-    fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
-
-
 class Channel (object):
     """
     A secure tunnel across an SSH L{Transport}.  A Channel is meant to behave
@@ -63,7 +57,7 @@ class Channel (object):
         subclass of L{Channel}.
 
         @param chanid: the ID of this channel, as passed by an existing
-        L{Transport}.
+            L{Transport}.
         @type chanid: int
         """
         self.chanid = chanid
@@ -84,6 +78,7 @@ class Channel (object):
         self.name = str(chanid)
         self.logger = util.get_logger('paramiko.chan.' + str(chanid))
         self.pipe_rfd = self.pipe_wfd = None
+	self.pipe_set = False
         self.event = threading.Event()
         self.combine_stderr = False
         self.exit_status = -1
@@ -504,9 +499,6 @@ class Channel (object):
         out = ''
         self.lock.acquire()
         try:
-            if self.pipe_rfd != None:
-                # use the pipe
-                return self._read_pipe(nbytes)
             if len(self.in_buffer) == 0:
                 if self.closed or self.eof_received:
                     return out
@@ -526,6 +518,9 @@ class Channel (object):
             if len(self.in_buffer) <= nbytes:
                 out = self.in_buffer
                 self.in_buffer = ''
+		if self.pipe_rfd != None:
+		    # clear the pipe, since no more data is buffered
+		    self._clear_pipe()
             else:
                 out = self.in_buffer[:nbytes]
                 self.in_buffer = self.in_buffer[nbytes:]
@@ -754,24 +749,21 @@ class Channel (object):
         
     def fileno(self):
         """
-        Returns an OS-level file descriptor which can be used for polling and
-        reading (but I{not} for writing).  This is primaily to allow python's
+        Returns an OS-level file descriptor which can be used for polling, but
+        but I{not} for reading or writing).  This is primaily to allow python's
         C{select} module to work.
 
         The first time C{fileno} is called on a channel, a pipe is created to
         simulate real OS-level file descriptor (FD) behavior.  Because of this,
-        two actual FDs are created -- this may be inefficient if you plan to
-        use many channels.
+        two OS-level FDs are created, which will use up FDs faster than normal.
+	You won't notice this effect unless you open hundreds or thousands of
+	channels simultaneously, but it's still notable.
 
-        @return: a small integer file descriptor
+        @return: an OS-level file descriptor
         @rtype: int
         
-        @warning: This method causes several aspects of the channel to change
-        behavior.  It is always more efficient to avoid using this method.
-
-        @bug: This does not work on Windows.  The problem is that pipes are
-        used to simulate an open FD, but I haven't figured out how to make
-        pipes enter non-blocking mode on Windows yet.
+        @warning: This method causes channel reads to be slightly less
+	    efficient.
         """
         self.lock.acquire()
         try:
@@ -779,12 +771,8 @@ class Channel (object):
                 return self.pipe_rfd
             # create the pipe and feed in any existing data
             self.pipe_rfd, self.pipe_wfd = os.pipe()
-            _set_nonblocking(self.pipe_wfd)
-            _set_nonblocking(self.pipe_rfd)
             if len(self.in_buffer) > 0:
-                x = self.in_buffer
-                self.in_buffer = ''
-                self._feed_pipe(x)
+		self._set_pipe()
             return self.pipe_rfd
         finally:
             self.lock.release()
@@ -876,10 +864,9 @@ class Channel (object):
             if self.ultra_debug:
                 self._log(DEBUG, 'fed %d bytes' % len(s))
             if self.pipe_wfd != None:
-                self._feed_pipe(s)
-            else:
-                self.in_buffer += s
-                self.in_buffer_cv.notifyAll()
+                self._set_pipe()
+            self.in_buffer += s
+	    self.in_buffer_cv.notifyAll()
         finally:
             self.lock.release()
 
@@ -1025,83 +1012,19 @@ class Channel (object):
         self._log(DEBUG, 'EOF sent')
         return
 
-    def _feed_pipe(self, data):
+    def _set_pipe(self):
         "you are already holding the lock"
-        if len(self.in_buffer) > 0:
-            self.in_buffer += data
-            return
-        try:
-            n = os.write(self.pipe_wfd, data)
-            if n < len(data):
-                # at least on linux, this will never happen, as the writes are
-                # considered atomic... but just in case.
-                self.in_buffer = data[n:]
-            self._check_add_window(n)
-            self.in_buffer_cv.notifyAll()
-            return
-        except OSError, e:
-            pass
-        if len(data) > 1:
-            # try writing just one byte then
-            x = data[0]
-            data = data[1:]
-            try:
-                os.write(self.pipe_wfd, x)
-                self.in_buffer = data
-                self._check_add_window(1)
-                self.in_buffer_cv.notifyAll()
-                return
-            except OSError, e:
-                data = x + data
-        # pipe is very full
-        self.in_buffer = data
-        self.in_buffer_cv.notifyAll()
+	if self.pipe_set:
+	    return
+	self.pipe_set = True
+	os.write(self.pipe_wfd, '*')
 
-    def _read_pipe(self, nbytes):
+    def _clear_pipe(self):
         "you are already holding the lock"
-        try:
-            x = os.read(self.pipe_rfd, nbytes)
-            if len(x) > 0:
-                self._push_pipe(len(x))
-                return x
-        except OSError, e:
-            pass
-        # nothing in the pipe
-        if self.closed or self.eof_received:
-            return ''
-        # should we block?
-        if self.timeout == 0.0:
-            raise socket.timeout()
-        # loop here in case we get woken up but a different thread has grabbed everything in the buffer
-        timeout = self.timeout
-        while not self.closed and not self.eof_received:
-            then = time.time()
-            self.in_buffer_cv.wait(timeout)
-            if timeout != None:
-                timeout -= time.time() - then
-                if timeout <= 0.0:
-                    raise socket.timeout()
-            try:
-                x = os.read(self.pipe_rfd, nbytes)
-                if len(x) > 0:
-                    self._push_pipe(len(x))
-                    return x
-            except OSError, e:
-                pass
-        pass
-
-    def _push_pipe(self, nbytes):
-        # successfully read N bytes from the pipe, now re-feed the pipe if necessary
-        # (assumption: the pipe can hold as many bytes as were read out)
-        if len(self.in_buffer) == 0:
-            return
-        if len(self.in_buffer) <= nbytes:
-            os.write(self.pipe_wfd, self.in_buffer)
-            self.in_buffer = ''
-            return
-        x = self.in_buffer[:nbytes]
-        self.in_buffer = self.in_buffer[nbytes:]
-        os.write(self.pipe_wfd, x)
+	if not self.pipe_set:
+	    return
+	os.read(self.pipe_rfd, 1)
+	self.pipe_set = False
 
     def _unlink(self):
         if self.closed or not self.active: