Prior art

Some similar projects you may be interested in looking at, if only for "how not to do things" ideas:

• Gitso (more recent fork) - reverse VNC connector: lacks strong authentication and the web of trust, delegates firewall issues to the developer, gives full access to the developer, cross-platform
• Corkscrew - SSH over HTTP tunnel, allows you to run whatever on top of even the most restrictive firewalls
• Tor - useful to bypass fascist firewalls, NAT and all sort of restrictive technologies, even if both ends are behind NAT, keeps both ends geolocation more private
• Iodine - IP over DNS tunnel, when even Tor fails
• Mosh - the mobile shell, interesting concepts of local terminal emulation and predictive capabilities
• Monkeysphere - OpenPGP web of trust applied to SSH and HTTP servers and clients
• watch-my-terminal - Allows a remote users to watch your terminal session via any modern browser (script+node on the broadcaster side, javascript on the watchers' side). Terminal emulation is quite good; it correctly renders vim, less and aptitude.
• teleconsole - can share terminal sessions through SSH or the browser through a SSH proxy

It would seem to me preferable to reuse existing components (SSH, especially) instead of reinventing the wheel too much here. So much trouble with terminal emulation and so on, as I am sure you know...