Let me clarify some things here, The XMPP protocol is not here to transport binary data, we have a bunch of other protocols for that. XMPP is only here to transport the signalisations to create bitstreams connections between the different end-points. I've implemented Jingle over WebRTC threw XMPP in my project, WebRTC re-use a lot of existing protocol (RTP, SRTP, RTPC, SDP), Jingle is just here to do a little bit of signalisation ("someone is calling you"…) and to convert the standard SDP (Session Description Protocol, which is used in SIP too) to a standard XML Jingle package. Creating a tool that convert SDP to Jingle is quite easy ( check our PHP implementation http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/SDPtoJingle.ph... and http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/JingletoSDP.ph...). When you've established a RTP connection (RTP take place on top of UDP) you can send 4K 60FPS 3D video, it doesn't matter. Secondly, saying that the XML overhead is a big issue is for me a non-sense. A couple of XMPP servers already support XML compression (with a minimum of 30% to more than 50% of bandwidth gain), see XEP like XEP-0138: Stream Compression (http://xmpp.org/extensions/xep-0138.html) and XEP-0229: Stream Compression with LZW (http://xmpp.org/extensions/xep-0229.html). Furthermore the whole web is based on XHTML (on top of HTTP which create way more overhead than XMPP on top of TCP). The big issue that we have now on XMPP is definitly not the XML size (every 3G connections and cheap smartphones can now handle this amount of data). The issue is more on improving how the XMPP stanza are sent threw the network and how to limit useless stanza to be send to the client (and between the servers). For the ejabberd server with Pubsub, we have sometime more than 50% of the stanza (XMPP packets) that are totally useless (the client already have the information, it's a old information…). As far as I know, base64 heavy data are only used to send and receive the users avatars (and for some authentications exchanges in SASL2, I think mostly to prevent encoding issues). Finally the Websocket are not here to improve the XMPP connections on desktops and mobiles clients, they are already using a pure TCP connection for that. The Websocket extension is mostly here to replace the BOSH system (which mimic a full-duplex XMPP communication on top of HTTP) which, yes, have a big overhead and serious performances issues. The aim of XMPP over Websockets is just to bring XMPP to the modern browsers. For the XMPP clients, there's already a couple of them, with strenghts and weaknesses. From my point of view Gajim has a quite good implementation of XMPP but the UI is really not "user friendly". Pidgin is quite great but the XMPP support is not as good as expected. Empathy start to be really interesting and the integration of the client in the OS is quite great. You can also take a look at the HipChat proprietary client which offer a nice user experience on top of XMPP. For me, Google has dropped the XMPP protocol for a couple of reasons : - Yes, the XMPP Standard Fundation can be slow to react and they are not working as fast as Google would like. - They are currently replacing XMPP with Hangout which is a proprietary, centralized and a totally obscur system. They are not replacing XMPP because its lack of features, they are building something totally different. And because the XMPP network is decentralized and totally open, you can't control and monitor it (or tape/spy it) easily. - Some clients already used the Gmail XMPP connexion for other purposes than the standard IM stuffs. - With Hangout, Google can now control the network, the servers but also the clients. To create Hangout-compatible clients we have to reverse-engineer their protocol (like we does for MSN, Skype or QQ). I'm fighting for more than 5 years now to built a full social network on top of XMPP, and yes I still think that this protocol is totally underrated. With XMPP you can do way more than just IM. If I compare your little 'requirement list' with XMPP : - XMPP is already built as a client-server protocol on top of TCP (with a couple of peer-to-peer extensions) - As I said the overhead here is not the XML protocol but the way you send data threw the network, and we are working with the XSF to optimize theses stuffs - Efficient encoding, XMPP is in full UTF-8 and handle perfectly any encoding (my client is used on many differents countries and I never had any bad feedbacks for this thing). - Stream encryption is supported C-S and S-S, the XSF is trying to push full encryptions between the servers (https://xmpp.net/result.php?domain=jabber.org&type=client for example). For the record, Goggle has never put any encryption between the Gmail XMPP server and the rest of the network, all the data are broadcasted in plain text. The OTR protocol is already used by a a lot of XMPP clients (on mobile too http://www.xabber.org/). - XMPP support "buge data" streaming (SOCKS5 bitestream http://xmpp.org/extensions/xep-0065.html) or Jingle can be used for this purpose. - The core of XMPP is built with a nice presence and ressources system. There's also extensions like XEP-0085: Chat State Notifications (http://xmpp.org/extensions/xep-0085.html) or XEP-0224: Attention (http://xmpp.org/extensions/xep-0224.html) to add nice features to a chat conversation. There is also extensions to handle archiving, message synchronisation between the clients and so on. - The core of XMPP is built on one-to-one and one-to-many system. You have several way to do this : chatrooms for simple messaging, you can do CC/mailing-list/transfert of messages to one or several contacts, and there is PubSub which add a nice stream management feature (which we used with Atom to publish and/or retrieve posts) on the PubSub nodes. - Just have a look of the ~300 extensions already shipped with XMPP http://xmpp.org/xmpp-protocols/xmpp-extensions/ I'll just finish with this : http://xkcd.com/927/ Regards, Jaussoin Timothée aka edhelas On jeu., juil. 10, 2014 at 12:27 , Dāvis Mosāns <davispuh@gmail.com> wrote:
2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:
2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh@gmail.com>:
Also what about NUL bytes? I bet most parsers are implemented in C/C++ using typical char * null-terminated string, how'll pass this JSON to someone? because well NUL...
At the parser level you would find a "b" character that's not between brackets, signalling a binary header is coming. A binary header is actually just the number of bytes that follow in binary format. The following bytes are then a binary file, to be assigned to a string as if it were a variable. We have a binarybuffer in javascript, that sort of thing. It would contain the NULL byte if you like it to.
[...]
I think it's much more serious that you have to serve the BJSON completely as a binary file. It's not like you can dump it onto a webpage anymore. You can't slip it into your normal HTTP text transfer bodies either, has to work with attachments. Attachments should be secure when facing malformed transfers* etc. Inconvenient, but not the end.
A lot of protocols are text based, often implemented using C strings and that's what I mean, you can't embed a JSON with binary data containing NUL there (because NUL will terminate that string), so you handle it like typical binary file and then what's the point of JSON to use in first place, because I don't see how it can be any better than any other proper binary data. Such binary JSON gives only overhead but no advantages.
2014-07-09 7:25 GMT+03:00 Bill Stewart <billstewart@pobox.com>:
I haven't used it in years, but I was always quite fond of XDR https://en.wikipedia.org/wiki/External_Data_Representation Sun's External Data Representation coding from the 80s, RFC-1014. Defines a bunch of variable types, and gives you tools for packing and unpacking them.
It's actually pretty good, but there are reasons why Protobuf was created and used instead. The main benefit of Protobuf is that it's easily extendable and can have optional fields. If you add or remove optional fields to server all old clients will still work like nothing have changed. But with XDR you can't do that unless you add another layer on top of it, but that's more work comparing to just taking Protobuf and using it. Also currently Protobuf is much more popular and have more libraries available for dozens of langauges.
2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" <cyberkiller8@gmail.com>:
It's really nice that so many of you got into the spirit and start thinking about how to change xmpp to make it something new, but what are you achieving here? It will end up being a new protocol, incompatible with existing xmpp, it will take a few years to finish the spec, then another 10+ years until any meaningful applications start using it (if at all)... So yeah, except being "coder porn" it does nothing to help the problem here and now.
The goal would be to create smaller overhead and thus be more performance effective. Also it doesn't have to be incompatible. It could be incorporated in XMPP so that new applications could use it but other's just use same legacy XMPP and everything keeps working fine and people wouldn't know what's happening under the hood, nor they would care. And I think it would be trivial to convince people to use and enable this "Binary" XMPP mode (if it's implemented in their client) which makes their chat client app to use 100x times less bandwidth and 50x times less CPU time (spent in parsing), thus your phone's battery would last longer. And yes XML overhead is that big.
A technically pretty proto won't help, today’s world has a huge problem with taking anything new. Better to stay with existing stuff, make it maybe less efficient because of it, but it will be here fast, when it's needed. Plus being less efficient is a no issue today, with fast machines (you can use compression on the fly, yes really :-P ), loads of storage, broadband connections (even the 3G data caps are getting larger and larger each year), etc. People are sending gigabytes of binary files in base64 each day in email messages, so why even care? ;-)
I guess you don't know that nothing is ever fast or good enough. People will always want things faster. What about real-time video call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some really good binary protocol is used so that it's not your software that creates a bottleneck, but if it does then your software is bad and why would I use it over other that can do it, the one that was designed for it, for example see Elemental Demonstrates 4K HEVC Video at 60 fps in London
Anyway, I must admit that I haven't studied XMPP enough to know how good or bad it is, but always should try to minimize any overhead, basically you want to process as little as possible.
Here straight from wiki, weaknesses: Does not support Quality of Service (QoS) XMPP does not have the ability to set the timing flow of messages, preventing XMPP from becoming practical for many embedded distributed realtime, Machine-to-Machine, or IoT applications. High overhead for embedded applications As a text based protocol, XMPP has a relatively high computing and network overhead. In-band binary data transfer is inefficient Binary data must be first base64 encoded before it can be transmitted in-band. Therefore any significant amount of binary data (e.g., file transfers) is best transmitted out-of-band, using in-band messages to coordinate. The best example of this is the Jingle XMPP Extension Protocol, XEP-0166. This issue are being adressed by the experimental XEP-0322: Efficient XML Interchange (EXI) Format.
that sounds really really bad. But it's not all lost, Jingle actually seems good as it have option to switch to Real-time Transport Protocol (RTP) and then it's just pure binary stream with minimal overhead. And looks like they are aware of these issues as EXI is being developed, but still while it's a big step forward, it will never beat pure binary protocol.
Also from wiki, this is good idea:
A perhaps more efficient transport for real-time messaging is WebSocket, a web technology providing for bi-directional, full-duplex communications channels over a single TCP connection. Experimental implementations of XMPP over WebSocket exist, and a (now-expired) Internet-Draft documenting this approach was published at the IETF but not yet standardized.
In my opinion the bottom line is - a small addition to existing xmpp has a far larger chance of being widely adopted (by applications and by the users) than a completely new protocol. And despite how awesome coder one might be - you won't be able to write all those implementations yourself or convince the masses to switch (again!).
Maybe yes, maybe no. I think if you've written specification in very clear and understandable way and if you've reference implementation library which everyone could just link against and if your protocol does it better than current existing solutions then I don't see why it wouldn't get adapted. Besides you don't need it implemented everywhere, you need it so that it's in application you use and you could contribute there yourself.
2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:
[...]
There is no masses using XMPP. Masses of coders, maybe, and they will use the best tool for the job.
All the extensions have succeeded in making any XMPP app lacking in usability. I sure haven't found any nice XMPP clients, nice enough to compare with native clients.
That's true indeed, currently there aren't any decent XMPP client (atleast I'm not aware of any). I mean from user's usability point (UX/UI). There are good either proprietary clients (eg. Skype) or good open source clients (eq. Quassel) that doesn't support XMPP :D
In fact I'm willing to bet everyone in the western world uses FB, Google chat and MSN (slackers and slowpokes). They all have limited XMPP implementations, they native clients do more. And there's no good app for interacting with XMPP.
About which Western wold you're talking about? I don't know, but I would assume that in Europe, Skype would be one of the most popular clients. Atleast here MSN never was a thing and everyone have always been using Skype and almost everyone still does. FB isn't really used that much (here we've better alternative). And about Google Talk only some people are aware that it even exists. I know that in Russia it's ICQ and in China it's QQ that's dominating there. Anyway I think they don't implement XMPP because it's too much work for too small benefit. Also then how they would compete with others if they would be limited with XMPP if it doesn't do something that they want to do. So own protocol might be safer bet for a lot of companies. By the way Google Talk dropped XMPP support.
I actually really hoped that Tox would succeed so that I could drop Skype and convince others to do same :D But now I'm not so sure if there will be some replacement in nearest future. Actually I think one of reason why Tox created new protocol and didn't used XMPP is that generally XMPP consists of client-server architecture and P2P support is only with extension which isn't widely known nor implemented much. Also XMPP seems way too complicated than it should be.
Pidgin really isn't good, it's just the only one out there. And it is still in the MSN era. I've switched to Office 2013 from Libre/OpenOffice and it really is in a different league all together. And it sucks that it is. But what can we do?
I personally don't like Pidgin too. And about Office and LibreOffice they both have their own advantages and disadvantages but I wouldn't say that any of them would be significantly better or worse than other. Also it's nothing to do with open souirce, it's just we've more users that want everything to be perfect without any effort for free than go and help and contribute to projects.
Ok, so anyway I've come to conclusion that I'm not sure anymore whether it's better to try to fix XMPP or just create new much simpler and better protocol. But in any case we really should summarize all our points about what's bad in XMPP and send them to XMPP group and see if they're willing to fix them and accept our proposals.
If we want to create a new protocol, then I've few ideas about it. Firstly it's really a shame that a lot of things are getting reinvented over and over again. So the main goal would be don't reinvent stuff but reuse as much as possible from existing things. It would help a lot as there's libraries for already everything and would just have to combine them. Next, think about high-level differences between instant message, group message, offline chat message, email, SMS, MMS, mailing list. What are differences? Actually if you think about it, there are no differences at all, it's the same message being delivered in various ways. But why single protocol couldn't handle it all? In fact it could. We're already using browser for all of that. It's just that there's no specification to unify it all in single protocol. Imagine if we wouldn't need a separate application for IRC, for email nor for other chats. It all could be handled by one general protocol. And why stop here, actually calls, sound and video are also exactly same. There's no really difference between text or sound/video, it's just a data and it depends only how you interpret it. Also security, there's only one option, just encrypt it all before sending. It's just so simple.
This next generation protocol would be a superset of typical media protocols and thus it wouldn't matter anymore what other people use. You would just use some middle-ware protocol layer that would translate that foreign protocol to this next-gen protocol and you simply use your favorite client. From user's point of view it doesn't matter at all which protocol is used under the hood.
So the main features and design goals of this protocol should be: Support for peer-to-peer and client-server architectures over both TCP and UDP Lightweight, minimal overhead, generally be just pass-thru Efficient encoding, basically binary streams Encryption by default with option for OTR Data stream itself can contain anything: text, images, sound, video, animations, screen sharing (and even remote desktop control would be possible) Dozens of events: joined, left, started typing and so on One to one and one to many streams maybe more things
So how that all would be possible? It's extremely simple. In client-server architecture, open connection to server, establish TLS (for example) now send this next-gen protocol header message (to specify what kind of data you're sending) and then send your data, it could be for example a raw Theora stream (directly from your camera) or just a text message. Then server just relays it to all parties. I don't think it could be any simpler and nothing will beat this in terms of performance. You've just some basic dependencies and what would be the reason to complicate this? It's trivial to implement as there's already libraries for those. In case of offline messages server would just store them and in fact it could do same for sound and videos too. You know Vine, Snapchat? what about YouTube, Soundcloud, basically it could support them all. IMO this is millions worth idea :D
It's worth looking at WebSockets, WebRTC and probably other standards.
I really don't understand why currently it all have been made so complicated because it isn't.