Tox.im

[edhelas]

Let me clarify some things here,

The XMPP protocol is not here to transport binary data, we have a bunch 
of other protocols for that. XMPP is only here to transport the 
signalisations to create bitstreams connections between the different 
end-points.

I've implemented Jingle over WebRTC threw XMPP in my project, WebRTC 
re-use a lot of existing protocol (RTP, SRTP, RTPC, SDP), Jingle is 
just here to do a little bit of signalisation ("someone is calling 
you"…) and to convert the standard SDP (Session Description Protocol, 
which is used in SIP too) to a standard XML Jingle package.

Creating a tool that convert SDP to Jingle is quite easy ( check  our 
PHP implementation 
http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/SDPtoJingle.php 
and 
http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/JingletoSDP.php).

When you've established a RTP connection (RTP take place on top of UDP) 
you can send 4K 60FPS 3D video, it doesn't matter.

Secondly, saying that the XML overhead is a big issue is for me a 
non-sense. A couple of XMPP servers already support XML compression 
(with a minimum of 30% to more than 50% of bandwidth gain), see XEP 
like XEP-0138: Stream Compression 
(http://xmpp.org/extensions/xep-0138.html) and XEP-0229: Stream 
Compression with LZW (http://xmpp.org/extensions/xep-0229.html). 
Furthermore the whole web is based on XHTML (on top of HTTP which 
create way more overhead than XMPP on top of TCP).

The big issue that we have now on XMPP is definitly not the XML size 
(every 3G connections and cheap smartphones can now handle this amount 
of data). The issue is more on improving how the XMPP stanza are sent 
threw the network and how to limit useless stanza to be send to the 
client (and between the servers). For the ejabberd server with Pubsub, 
we have sometime more than 50% of the stanza (XMPP packets) that are 
totally useless (the client already have the information, it's a old 
information…).

As far as I know, base64 heavy data are only used to send and receive 
the users avatars (and for some authentications exchanges in SASL2, I 
think mostly to prevent encoding issues).

Finally the Websocket are not here to improve the XMPP connections on 
desktops and mobiles clients, they are already using a pure TCP 
connection for that. The Websocket extension is mostly here to replace 
the BOSH system (which mimic a full-duplex XMPP communication on top of 
HTTP) which, yes, have a big overhead and serious performances issues.

The aim of XMPP over Websockets is just to bring XMPP to the modern 
browsers.

For the XMPP clients, there's already a couple of them, with strenghts 
and weaknesses. From my point of view Gajim has a quite good 
implementation of XMPP but the UI is really not "user friendly". Pidgin 
is quite great but the XMPP support is not as good as expected. Empathy 
start to be really interesting and the integration of the client in the 
OS is quite great.

You can also take a look at the HipChat proprietary client which offer 
a nice user experience on top of XMPP.

For me, Google has dropped the XMPP protocol for a couple of reasons :
- Yes, the XMPP Standard Fundation can be slow to react and they are 
not working as fast as Google would like.
- They are currently replacing XMPP with Hangout which is a 
proprietary, centralized and a totally obscur system. They are not 
replacing XMPP because its lack of features, they are building 
something totally different. And because the XMPP network is 
decentralized and totally open, you can't control and monitor it (or 
tape/spy it) easily.
- Some clients already used the Gmail XMPP connexion for other purposes 
than the standard IM stuffs.
- With Hangout, Google can now control the network, the servers but 
also the clients. To create Hangout-compatible clients we have to 
reverse-engineer their protocol (like we does for MSN, Skype or QQ).

I'm fighting for more than 5 years now to built a full social network 
on top of XMPP, and yes I still think that this protocol is totally 
underrated. With XMPP you can do way more than just IM.

If I compare your little 'requirement list' with XMPP :
- XMPP is already built as a client-server protocol on top of TCP (with 
a couple of peer-to-peer extensions)
- As I said the overhead here is not the XML protocol but the way you 
send data threw the network, and we are working with the XSF to 
optimize theses stuffs
- Efficient encoding, XMPP is in full UTF-8 and handle perfectly any 
encoding (my client is used on many differents countries and I never 
had any bad feedbacks for this thing).
- Stream encryption is supported C-S and S-S, the XSF is trying to push 
full encryptions between the servers 
(https://xmpp.net/result.php?domain=jabber.org&type=client for 
example). For the record, Goggle has never put any encryption between 
the Gmail XMPP server and the rest of the network, all the data are 
broadcasted in plain text. The OTR protocol is already used by a a lot 
of XMPP clients (on mobile too http://www.xabber.org/).
- XMPP support "buge data" streaming (SOCKS5 bitestream 
http://xmpp.org/extensions/xep-0065.html) or Jingle can be used for 
this purpose.
- The core of XMPP is built with a nice presence and ressources system. 
There's also extensions like XEP-0085: Chat State Notifications 
(http://xmpp.org/extensions/xep-0085.html) or XEP-0224: Attention 
(http://xmpp.org/extensions/xep-0224.html) to add nice features to a 
chat conversation. There is also extensions to handle archiving, 
message synchronisation between the clients and so on.
- The core of XMPP is built on one-to-one and one-to-many system. You 
have several way to do this : chatrooms for simple messaging, you can 
do CC/mailing-list/transfert of messages to one or several contacts, 
and there is PubSub which add a nice stream management feature (which 
we used with Atom to publish and/or retrieve posts) on the PubSub nodes.
- Just have a look of the ~300 extensions already shipped with XMPP 
http://xmpp.org/xmpp-protocols/xmpp-extensions/

I'll just finish with this : http://xkcd.com/927/

Regards,

Jaussoin Timothée aka edhelas

On jeu., juil. 10, 2014 at 12:27 , Dāvis Mosāns <davispuh at gmail.com> 
wrote:
> 2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <l at odewijk.nl>:
>> 2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh at gmail.com>:
>> 
>>> Also what about NUL bytes? I bet most parsers are implemented in 
>>> C/C++ using typical char * null-terminated string, how'll pass this 
>>> JSON to someone? because well NUL...
>> 
>> At the parser level you would find a "b" character that's not 
>> between brackets, signalling a binary header is coming. A binary 
>> header is actually just the number of bytes that follow in binary 
>> format. The following bytes are then a binary file, to be assigned 
>> to a string as if it were a variable. We have a binarybuffer in 
>> javascript, that sort of thing. It would contain the NULL byte if 
>> you like it to.
>> 
>> [...]
>> 
>> I think it's much more serious that you have to serve the BJSON 
>> completely as a binary file. It's not like you can dump it onto a 
>> webpage anymore. You can't slip it into your normal HTTP text 
>> transfer bodies either, has to work with attachments. Attachments 
>> should be secure when facing malformed transfers* etc. Inconvenient, 
>> but not the end.
>> 
> 
> A lot of protocols are text based, often implemented using C strings 
> and that's what I mean, you can't embed a JSON with binary data 
> containing NUL there (because NUL will terminate that string), so you 
> handle it like typical binary file and then what's the point of JSON 
> to use in first place, because I don't see how it can be any better 
> than any other proper binary data. Such binary JSON gives only 
> overhead but no advantages.
> 
> 
> 2014-07-09 7:25 GMT+03:00 Bill Stewart <billstewart at pobox.com>:
>> I haven't used it in years, but I was always quite fond of XDR
>> https://en.wikipedia.org/wiki/External_Data_Representation
>> Sun's External Data Representation coding from the 80s, RFC-1014.
>> Defines a bunch of variable types, and gives you tools for packing 
>> and unpacking them.
>> 
> 
> It's actually pretty good, but there are reasons why Protobuf was 
> created and used instead. The main benefit of Protobuf is that it's 
> easily extendable and can have optional fields. If you add or remove 
> optional fields to server all old clients will still work like 
> nothing have changed. But with XDR you can't do that unless you add 
> another layer on top of it, but that's more work comparing to just 
> taking Protobuf and using it. Also currently Protobuf is much more 
> popular and have more libraries available for dozens of langauges.
> 
> 
> 2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" 
> <cyberkiller8 at gmail.com>:
>> 
>> It's really nice that so many of you got into the spirit and start
>> thinking about how to change xmpp to make it something new, but what 
>> are
>> you achieving here? It will end up being a new protocol, incompatible
>> with existing xmpp, it will take a few years to finish the spec, then
>> another 10+ years until any meaningful applications start using it 
>> (if
>> at all)... So yeah, except being "coder porn" it does nothing to help
>> the problem here and now.
>> 
> 
> The goal would be to create smaller overhead and thus be more 
> performance effective. Also it doesn't have to be incompatible. It 
> could be incorporated in XMPP so that new applications could use it 
> but other's just use same legacy XMPP and everything keeps working 
> fine and people wouldn't know what's happening under the hood, nor 
> they would care. And I think it would be trivial to convince people 
> to use and enable this "Binary" XMPP mode (if it's implemented in 
> their client) which makes their chat client app to use 100x times 
> less bandwidth and 50x times less CPU time (spent in parsing), thus 
> your phone's battery would last longer. And yes XML overhead is that 
> big.
> 
>> 
>> A technically pretty proto won't help, today’s world has a huge 
>> problem
>> with taking anything new. Better to stay with existing stuff, make it
>> maybe less efficient because of it, but it will be here fast, when 
>> it's
>> needed. Plus being less efficient is a no issue today, with fast
>> machines (you can use compression on the fly, yes really :-P ), 
>> loads of
>> storage, broadband connections (even the 3G data caps are getting 
>> larger
>> and larger each year), etc. People are sending gigabytes of binary 
>> files
>> in base64 each day in email messages, so why even care? ;-)
>> 
> 
> I guess you don't know that nothing is ever fast or good enough. 
> People will always want things faster. What about real-time video 
> call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some 
> really good binary protocol is used so that it's not your software 
> that creates a bottleneck, but if it does then your software is bad 
> and why would I use it over other that can do it, the one that was 
> designed for it, for example see Elemental Demonstrates 4K HEVC Video 
> at 60 fps in London
> 
> Anyway, I must admit that I haven't studied XMPP enough to know how 
> good or bad it is, but always should try to minimize any overhead, 
> basically you want to process as little as possible.
> 
> Here straight from wiki, weaknesses:
> Does not support Quality of Service (QoS)
> XMPP does not have the ability to set the timing flow of messages, 
> preventing XMPP from becoming practical for many embedded distributed 
> realtime, Machine-to-Machine, or IoT applications.
> High overhead for embedded applications
> As a text based protocol, XMPP has a relatively high computing and 
> network overhead.
> In-band binary data transfer is inefficient
> Binary data must be first base64 encoded before it can be transmitted 
> in-band. Therefore any significant amount of binary data (e.g., file 
> transfers) is best transmitted out-of-band, using in-band messages to 
> coordinate. The best example of this is the Jingle XMPP Extension 
> Protocol, XEP-0166. This issue are being adressed by the experimental 
> XEP-0322: Efficient XML Interchange (EXI) Format.
> 
> that sounds really really bad. But it's not all lost, Jingle actually 
> seems good as it have option to switch to Real-time Transport 
> Protocol (RTP) and then it's just pure binary stream with minimal 
> overhead. And looks like they are aware of these issues as EXI is 
> being developed, but still while it's a big step forward, it will 
> never beat pure binary protocol.
> 
> Also from wiki, this is good idea:
> 
> A perhaps more efficient transport for real-time messaging is 
> WebSocket, a web technology providing for bi-directional, full-duplex 
> communications channels over a single TCP connection. Experimental 
> implementations of XMPP over WebSocket exist, and a (now-expired) 
> Internet-Draft documenting this approach was published at the IETF 
> but not yet standardized.
> 
> 
>> In my opinion the bottom line is - a small addition to existing xmpp 
>> has
>> a far larger chance of being widely adopted (by applications and by 
>> the
>> users) than a completely new protocol. And despite how awesome coder 
>> one
>> might be - you won't be able to write all those implementations 
>> yourself
>> or convince the masses to switch (again!).
>> 
> 
> Maybe yes, maybe no. I think if you've written specification in very 
> clear and understandable way and if you've reference implementation 
> library which everyone could just link against and if your protocol 
> does it better than current existing solutions then I don't see why 
> it wouldn't get adapted. Besides you don't need it implemented 
> everywhere, you need it so that it's in application you use and you 
> could contribute there yourself.
> 
> 
> 2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <l at odewijk.nl>:
>> [...]
>> 
>> There is no masses using XMPP. Masses of coders, maybe, and they 
>> will use the best tool for the job.
>> 
>> All the extensions have succeeded in making any XMPP app lacking in 
>> usability. I sure haven't found any nice XMPP clients, nice enough 
>> to compare with native clients.
>> 
>  That's true indeed, currently there aren't any decent XMPP client 
> (atleast I'm not aware of any). I mean from user's usability point 
> (UX/UI). There are good either proprietary clients (eg. Skype) or 
> good open source clients (eq. Quassel) that doesn't support XMPP :D
> 
>> In fact I'm willing to bet everyone in the western world uses FB, 
>> Google chat and MSN (slackers and slowpokes). They all have limited 
>> XMPP implementations, they native clients do more. And there's no 
>> good app for interacting with XMPP.
>> 
> About which Western wold you're talking about? I don't know, but I 
> would assume that in Europe, Skype would be one of the most popular 
> clients. Atleast here MSN never was a thing and everyone have always 
> been using Skype and almost everyone still does. FB isn't really used 
> that much (here we've better alternative). And about Google Talk only 
> some people are aware that it even exists. I know that in Russia it's 
> ICQ and in China it's QQ that's dominating there. Anyway I think they 
> don't implement XMPP because it's too much work for too small 
> benefit. Also then how they would compete with others if they would 
> be limited with XMPP if it doesn't do something that they want to do. 
> So own protocol might be safer bet for a lot of companies. By the way 
> Google Talk dropped XMPP support.
> 
> 
> I actually really hoped that Tox would succeed so that I could drop 
> Skype and convince others to do same :D But now I'm not so sure if 
> there will be some replacement in nearest future. Actually I think 
> one of reason why Tox created new protocol and didn't used XMPP is 
> that generally XMPP consists of client-server architecture and P2P 
> support is only with extension which isn't widely known nor 
> implemented much. Also XMPP seems way too complicated than it should 
> be.
> 
>> Pidgin really isn't good, it's just the only one out there. And it 
>> is still in the MSN era. I've switched to Office 2013 from 
>> Libre/OpenOffice and it really is in a different league all 
>> together. And it sucks that it is. But what can we do?
>> 
> I personally don't like Pidgin too. And about Office and LibreOffice 
> they both have their own advantages and disadvantages but I wouldn't 
> say that any of them would be significantly better or worse than 
> other. Also it's nothing to do with open souirce, it's just we've 
> more users that want everything to be perfect without any effort for 
> free than go and help and contribute to projects.
> 
> 
> Ok, so anyway I've come to conclusion that I'm not sure anymore 
> whether it's better to try to fix XMPP or just create new much 
> simpler and better protocol. But in any case we really should 
> summarize all our points about what's bad in XMPP and send them to 
> XMPP group and see if they're willing to fix them and accept our 
> proposals.
> 
> If we want to create a new protocol, then I've few ideas about it. 
> Firstly it's really a shame that a lot of things are getting 
> reinvented over and over again. So the main goal would be don't 
> reinvent stuff but reuse as much as possible from existing things. It 
> would help a lot as there's libraries for already everything and 
> would just have to combine them. Next, think about high-level 
> differences between instant message, group message, offline chat 
> message, email, SMS, MMS, mailing list. What are differences? 
> Actually if you think about it, there are no differences at all, it's 
> the same message being delivered in various ways. But why single 
> protocol couldn't handle it all? In fact it could. We're already 
> using browser for all of that. It's just that there's no 
> specification to unify it all in single protocol. Imagine if we 
> wouldn't need a separate application for IRC, for email nor for other 
> chats. It all could be handled by one general protocol. And why stop 
> here, actually calls, sound and video are also exactly same. There's 
> no really difference between text or sound/video, it's just a data 
> and it depends only how you interpret it. Also security, there's only 
> one option, just encrypt it all before sending. It's just so simple.
> 
> This next generation protocol would be a superset of typical media 
> protocols and thus it wouldn't matter anymore what other people use. 
> You would just use some middle-ware protocol layer that would 
> translate that foreign protocol to this next-gen protocol and you 
> simply use your favorite client. From user's point of view it doesn't 
> matter at all which protocol is used under the hood.
> 
> So the main features and design goals of this protocol should be:
> Support for peer-to-peer and client-server architectures over both 
> TCP and UDP
> Lightweight, minimal overhead, generally be just pass-thru
> Efficient encoding, basically binary streams
> Encryption by default with option for OTR
> Data stream itself can contain anything: text, images, sound, video, 
> animations, screen sharing (and even remote desktop control would be 
> possible)
> Dozens of events: joined, left, started typing and so on
> One to one and one to many streams
> maybe more things
> 
> 
> So how that all would be possible? It's extremely simple. In 
> client-server architecture, open connection to server, establish TLS 
> (for example) now send this next-gen protocol header message (to 
> specify what kind of data you're sending) and then send your data, it 
> could be for example a raw Theora stream (directly from your camera) 
> or just a text message. Then server just relays it to all parties.
> I don't think it could be any simpler and nothing will beat this in 
> terms of performance. You've just some basic dependencies and what 
> would be the reason to complicate this? It's trivial to implement as 
> there's already libraries for those. In case of offline messages 
> server would just store them and in fact it could do same for sound 
> and videos too. You know Vine, Snapchat? what about YouTube, 
> Soundcloud, basically it could support them all. IMO this is millions 
> worth idea :D
> 
> 
> It's worth looking at WebSockets, WebRTC and probably other standards.
> 
> 
> 
> I really don't understand why currently it all have been made so 
> complicated because it isn't.
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/html
Size: 25187 bytes
Desc: not available
URL: <https://lists.cpunks.org/pipermail/cypherpunks/attachments/20140710/d22fc672/attachment-0001.txt>