2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <[1]l@odewijk.nl>: 2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <[2]davispuh@gmail.com>: Also what about NUL bytes? I bet most parsers are implemented in C/C++ using typical char * null-terminated string, how'll pass this JSON to someone? because well NUL... At the parser level you would find a "b" character that's not between brackets, signalling a binary header is coming. A binary header is actually just the number of bytes that follow in binary format. The following bytes are then a binary file, to be assigned to a string as if it were a variable. We have a binarybuffer in javascript, that sort of thing. It would contain the NULL byte if you like it to. [...] I think it's much more serious that you have to serve the BJSON completely as a binary file. It's not like you can dump it onto a webpage anymore. You can't slip it into your normal HTTP text transfer bodies either, has to work with attachments. Attachments should be secure when facing malformed transfers* etc. Inconvenient, but not the end. A lot of protocols are text based, often implemented using C strings and that's what I mean, you can't embed a JSON with binary data containing NUL there (because NUL will terminate that string), so you handle it like typical binary file and then what's the point of JSON to use in first place, because I don't see how it can be any better than any other proper binary data. Such binary JSON gives only overhead but no advantages. 2014-07-09 7:25 GMT+03:00 Bill Stewart <[3]billstewart@pobox.com>: I haven't used it in years, but I was always quite fond of XDR [4]https://en.wikipedia.org/wiki/External_Data_Representation Sun's External Data Representation coding from the 80s, RFC-1014. Defines a bunch of variable types, and gives you tools for packing and unpacking them. It's actually pretty good, but there are reasons why Protobuf was created and used instead. The main benefit of Protobuf is that it's easily extendable and can have optional fields. If you add or remove optional fields to server all old clients will still work like nothing have changed. But with XDR you can't do that unless you add another layer on top of it, but that's more work comparing to just taking Protobuf and using it. Also currently Protobuf is much more popular and have more libraries available for dozens of langauges. 2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" <[5]cyberkiller8@gmail.com>: It's really nice that so many of you got into the spirit and start thinking about how to change xmpp to make it something new, but what are you achieving here? It will end up being a new protocol, incompatible with existing xmpp, it will take a few years to finish the spec, then another 10+ years until any meaningful applications start using it (if at all)... So yeah, except being "coder porn" it does nothing to help the problem here and now. The goal would be to create smaller overhead and thus be more performance effective. Also it doesn't have to be incompatible. It could be incorporated in XMPP so that new applications could use it but other's just use same legacy XMPP and everything keeps working fine and people wouldn't know what's happening under the hood, nor they would care. And I think it would be trivial to convince people to use and enable this "Binary" XMPP mode (if it's implemented in their client) which makes their chat client app to use 100x times less bandwidth and 50x times less CPU time (spent in parsing), thus your phone's battery would last longer. And yes XML overhead is that big. A technically pretty proto won't help, today’s world has a huge problem with taking anything new. Better to stay with existing stuff, make it maybe less efficient because of it, but it will be here fast, when it's needed. Plus being less efficient is a no issue today, with fast machines (you can use compression on the fly, yes really :-P ), loads of storage, broadband connections (even the 3G data caps are getting larger and larger each year), etc. People are sending gigabytes of binary files in base64 each day in email messages, so why even care? ;-) I guess you don't know that nothing is ever fast or good enough. People will always want things faster. What about real-time video call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some really good binary protocol is used so that it's not your software that creates a bottleneck, but if it does then your software is bad and why would I use it over other that can do it, the one that was designed for it, for example see [6]Elemental Demonstrates 4K HEVC Video at 60 fps in London Anyway, I must admit that I haven't studied XMPP enough to know how good or bad it is, but always should try to minimize any overhead, basically you want to process as little as possible. Here straight from wiki, weaknesses: * Does not support Quality of Service (QoS) * XMPP does not have the ability to set the timing flow of messages, preventing XMPP from becoming practical for many embedded distributed realtime, Machine-to-Machine, or IoT applications. High overhead for embedded applications * As a text based protocol, XMPP has a relatively high computing and network overhead. In-band binary data transfer is inefficient * Binary data must be first base64 encoded before it can be transmitted in-band. Therefore any significant amount of binary data (e.g., file transfers) is best transmitted out-of-band, using in-band messages to coordinate. The best example of this is the [7]Jingle XMPP Extension Protocol, [8]XEP-0166. This issue are being adressed by the experimental [9]XEP-0322: Efficient XML Interchange (EXI) Format. that sounds really really bad. But it's not all lost, [10]Jingle actually seems good as it have option to switch to [11]Real-time Transport Protocol (RTP) and then it's just pure binary stream with minimal overhead. And looks like they are aware of these issues as EXI is being developed, but still while it's a big step forward, it will never beat pure binary protocol. Also from wiki, this is good idea: A perhaps more efficient transport for real-time messaging is [12]WebSocket, a web technology providing for bi-directional, full-duplex communications channels over a single TCP connection. Experimental implementations of XMPP over WebSocket exist, and a (now-expired) Internet-Draft documenting this approach was published at the IETF but not yet standardized. In my opinion the bottom line is - a small addition to existing xmpp has a far larger chance of being widely adopted (by applications and by the users) than a completely new protocol. And despite how awesome coder one might be - you won't be able to write all those implementations yourself or convince the masses to switch (again!). Maybe yes, maybe no. I think if you've written specification in very clear and understandable way and if you've reference implementation library which everyone could just link against and if your protocol does it better than current existing solutions then I don't see why it wouldn't get adapted. Besides you don't need it implemented everywhere, you need it so that it's in application you use and you could contribute there yourself. 2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <[13]l@odewijk.nl>: [...] There is no masses using XMPP. Masses of coders, maybe, and they will use the best tool for the job. All the extensions have succeeded in making any XMPP app lacking in usability. I sure haven't found any nice XMPP clients, nice enough to compare with native clients. That's true indeed, currently there aren't any decent XMPP client (atleast I'm not aware of any). I mean from user's usability point (UX/UI). There are good either proprietary clients (eg. Skype) or good open source clients (eq. Quassel) that doesn't support XMPP :D In fact I'm willing to bet everyone in the western world uses FB, Google chat and MSN (slackers and slowpokes). They all have limited XMPP implementations, they native clients do more. And there's no good app for interacting with XMPP. About which Western wold you're talking about? I don't know, but I would assume that in Europe, Skype would be one of the most popular clients. Atleast here MSN never was a thing and everyone have always been using Skype and almost everyone still does. FB isn't really used that much (here we've better alternative). And about Google Talk only some people are aware that it even exists. I know that in Russia it's ICQ and in China it's QQ that's dominating there. Anyway I think they don't implement XMPP because it's too much work for too small benefit. Also then how they would compete with others if they would be limited with XMPP if it doesn't do something that they want to do. So own protocol might be safer bet for a lot of companies. By the way [14]Google Talk dropped XMPP support. I actually really hoped that Tox would succeed so that I could drop Skype and convince others to do same :D But now I'm not so sure if there will be some replacement in nearest future. Actually I think one of reason why Tox created new protocol and didn't used XMPP is that generally XMPP consists of client-server architecture and P2P support is only with extension which isn't widely known nor implemented much. Also XMPP seems way too complicated than it should be. Pidgin really isn't good, it's just the only one out there. And it is still in the MSN era. I've switched to Office 2013 from Libre/OpenOffice and it really is in a different league all together. And it sucks that it is. But what can we do? I personally don't like Pidgin too. And about Office and LibreOffice they both have their own advantages and disadvantages but I wouldn't say that any of them would be significantly better or worse than other. Also it's nothing to do with open souirce, it's just we've more users that want everything to be perfect without any effort for free than go and help and contribute to projects. Ok, so anyway I've come to conclusion that I'm not sure anymore whether it's better to try to fix XMPP or just create new much simpler and better protocol. But in any case we really should summarize all our points about what's bad in XMPP and send them to XMPP group and see if they're willing to fix them and accept our proposals. If we want to create a new protocol, then I've few ideas about it. Firstly it's really a shame that a lot of things are getting reinvented over and over again. So the main goal would be don't reinvent stuff but reuse as much as possible from existing things. It would help a lot as there's libraries for already everything and would just have to combine them. Next, think about high-level differences between instant message, group message, offline chat message, email, SMS, MMS, mailing list. What are differences? Actually if you think about it, there are no differences at all, it's the same message being delivered in various ways. But why single protocol couldn't handle it all? In fact it could. We're already using browser for all of that. It's just that there's no specification to unify it all in single protocol. Imagine if we wouldn't need a separate application for IRC, for email nor for other chats. It all could be handled by one general protocol. And why stop here, actually calls, sound and video are also exactly same. There's no really difference between text or sound/video, it's just a data and it depends only how you interpret it. Also security, there's only one option, just encrypt it all before sending. It's just so simple. This next generation protocol would be a superset of typical media protocols and thus it wouldn't matter anymore what other people use. You would just use some middle-ware protocol layer that would translate that foreign protocol to this next-gen protocol and you simply use your favorite client. From user's point of view it doesn't matter at all which protocol is used under the hood. So the main features and design goals of this protocol should be: * Support for peer-to-peer and client-server architectures over both TCP and UDP * Lightweight, minimal overhead, generally be just pass-thru * Efficient encoding, basically binary streams * Encryption by default with option for OTR * Data stream itself can contain anything: text, images, sound, video, animations, screen sharing (and even remote desktop control would be possible) * Dozens of events: joined, left, started typing and so on * One to one and one to many streams * maybe more things So how that all would be possible? It's extremely simple. In client-server architecture, open connection to server, establish TLS (for example) now send this next-gen protocol header message (to specify what kind of data you're sending) and then send your data, it could be for example a raw [15]Theora stream (directly from your camera) or just a text message. Then server just relays it to all parties. I don't think it could be any simpler and nothing will beat this in terms of performance. You've just some basic dependencies and what would be the reason to complicate this? It's trivial to implement as there's already libraries for those. In case of offline messages server would just store them and in fact it could do same for sound and videos too. You know Vine, Snapchat? what about YouTube, Soundcloud, basically it could support them all. IMO this is millions worth idea :D It's worth looking at [16]WebSockets, [17]WebRTC and probably other standards. I really don't understand why currently it all have been made so complicated because it isn't. References 1. mailto:l@odewijk.nl 2. mailto:davispuh@gmail.com 3. mailto:billstewart@pobox.com 4. https://en.wikipedia.org/wiki/External_Data_Representation 5. mailto:cyberkiller8@gmail.com 6. http://www.streamingmediaglobal.com/Articles/Editorial/Featured-Articles/Elemental-Demonstrates-4K-HEVC-Video-at-60-fps-in-London-93707.aspx 7. http://en.wikipedia.org/wiki/Jingle_(protocol) 8. http://xmpp.org/extensions/xep-0166.html 9. http://xmpp.org/extensions/xep-0322.html 10. http://xmpp.org/about-xmpp/technology-overview/jingle/ 11. http://en.wikipedia.org/wiki/Real-time_Transport_Protocol 12. http://en.wikipedia.org/wiki/WebSocket 13. mailto:l@odewijk.nl 14. https://www.eff.org/deeplinks/2013/05/google-abandons-open-standards-instant-messaging 15. http://www.theora.org/ 16. https://www.websocket.org/ 17. http://www.webrtc.org/