I don't agree, I think XMPP could be good solution, while yes attack surface is quite large but it will be in any case, because even if you create the very minimalist chat protocol possible (let's say basically use asymmetric cryptography for messages which are plaintext without any features) you still can have bugs in cryptography library, network stack, OS/kernel. This part will be same no matter what messaging protocol you use. So by changing plaintext to other payload such as XMPP we introduce another layer but this layer could be parsed in a sandbox / virtual machine thus even if you receive malicious message it couldn't exploit other parts of your system and it would work exactly like that simple plaintext protocol. Now but what if there's a bug in cryptography library, well you have already lost even with your basic plaintext protocol... 2014-07-07 11:41 GMT+03:00 stef <s@ctrlc.hu>:

On Mon, Jul 07, 2014 at 09:11:24AM +0200, edhelas wrote:

...

I really think that we need to focus on an existent standard and improve it, and for me XMPP seem to be the perfect protocol for all theses things : - Standard IM + chatroom - Video/Audio conferencing (with Jingle, we are using it with WebRTC on Movim) - Pubsub (for newsfeeds, blogging) - Geolocation - Vcard4 support - SASL2 authentication - OTR support - Full encryption between the servers (https://xmpp.net/list.php) - and so on…

i dunno, but xml based protocol (attack surface), geolocation (privacy), video/audio conferencing (traffic analysis), etc are all attributes i do not want in a secure communication protocol and a protocol that supports these is considered bloated. also the huge amounts of known/guessable plaintext in xmpp are quite worrisome. i agree NIH is bad, but xmpp is as bad for a post-snowden adversary model.

-- otr fp: https://www.ctrlc.hu/~stef/otr.txt

security is always a trade-off with convenience/usability and IMO that layer on top of plaintext protocol would be minimal comparing to already your OS surface. And if you go in that direction then why not go further? develop a basic custom minimalistic OS (in a way that compiled code could be verified in case of compiler backdoor) for just single purpose for secure messaging. It could be booted from CD-ROM or read-only flash, would self-verify itself and PC hardware for known anomalies, present you with a hash of environment so you've memorized it and if it ever changes you know someone have touched something on your PC, maybe BIOS, maybe other firmware maybe your boot medium etc. Then you would plugin your security token with encrypted GPG key and you could securely message. But actually no, you wouldn't use just general purpose computer, you would have developed a custom computer from ground-up with every single chip and transistor to be verifiable and it would serve only this single purpose of secure messaging. But now what if your friend doesn't do the same? it's all bets off and you've lost because it will be easier to "attach" to other end than you. Anyway I see a reason for both of these use cases, encrypted feature full messaging and just extremely secure basic plaintext messaging. But if you go with latter then I wouldn't stop in middle that is I wouldn't use same general OS but something trimmed down. I think currently Tails is pretty good and it comes with Pidgin OTR and you can use it over IRC network which is basically a simple plaintext protocol so your case is already covered I think. So for this first case of feature full messaging, XMPP seems to be a good choice. 2014-07-07 17:55 GMT+03:00 rysiek <rysiek@hackerspace.pl>:

Dnia poniedziałek, 7 lipca 2014 16:06:47 Dāvis Mosāns pisze:

...

I don't agree, I think XMPP could be good solution, while yes attack surface is quite large but it will be in any case, because even if you create the very minimalist chat protocol possible (let's say basically use asymmetric cryptography for messages which are plaintext without any features) you still can have bugs in cryptography library, network stack, OS/kernel. This part will be same no matter what messaging protocol you use.

Exactly. And that's an argument for NOT minimizing the attack surface beyond these problems... how exactly?

I mean, your argument is basically: "don't wash your hands, as there might be salmonella in the eggs anyway". Dafuq?

-- Pozdr rysiek

On 7/8/14, "Łukasz \"Cyber Killer\" Korpalski" <cyberkiller8@gmail.com> wrote:

W dniu 07.07.2014 16:55, rysiek pisze:

...

Dnia poniedziałek, 7 lipca 2014 16:06:47 Dāvis Mosāns pisze:

...

I don't agree, I think XMPP could be good solution, while yes attack surface is quite large but it will be in any case, because even if you create the very minimalist chat protocol possible (let's say basically use asymmetric cryptography for messages which are plaintext without any features) you still can have bugs in cryptography library, network stack, OS/kernel. This part will be same no matter what messaging protocol you use.

Exactly. And that's an argument for NOT minimizing the attack surface beyond these problems... how exactly?

I mean, your argument is basically: "don't wash your hands, as there might be salmonella in the eggs anyway". Dafuq?

I'm going to defend XMPP too, but on the grounds that it's an already established and widely used protocol, the overhead is minimal looking from a modern point of view (even when not using the potentially privacy-risky elements) and it was designed to be extendable. These are imo good arguments to use xmpp instead of creating something new (again :-P ).

As has been said over the decades: start correct, 'good' easy to maintain code, secure of course, and optimize later, eg 1-1 mapping from XMPP (XML I assume?) to say msgpack: MessagePack: http://msgpack.org/ - a fast, binary replacement for JSON Such optimizations ought be behind a library anyway! (From user app point of view.) As someone else said, think of the stack, separate the concerns: IP, user addressing, persistence of ids, persistence of addresses, crypting, dht, distributed storage, libs, user apps. For impatient programmers wanting instant gratification, work on one layer in the stack.

2014-07-08 18:05 GMT+02:00 Dāvis Mosāns <davispuh@gmail.com>:

XMPP design,

I used XMPP with facebook chat. It didn't support even a quarter of the cookiejar of features. This was with Pidgin, afaik the only serious rich crossplatform manyprotocol chat program. So forgive me for being a little underwhelmed on the protocol itself. It is also by design the most common denominator, with extensions infrequently supported (read: less useful). The most common denominator is of course chat and user accounts. But chat is not the atomic networked message that we're talking about. VOIP over base64 also seems kind of like banging your head into the wall. It feels pretty good though, because it means state of the art can be improved. It gives some reason for the current state of things. We have a go-to solution, why improve?

I think it's unreal/unpractical and not worth going for and there's just no benefits for it (just use proper binary serialization), JSON was meant as human-readable serialization format and introducing binary format there, then what was the point of using JSON in first place? Why not just some proper binary serialization? (eg. Protobuf). It just really seems that people throw a lot of stuff on JSON, XML and others even if it was never intended to be used for those purposes. There are different tools each for it's own specific purpose and people should not abuse them. Current JSON parsers treat " as special token to separate strings, so if you want to include " in JSON you've to escape, so it will be "\"" and now with any binary encoding you're complicating this because either you've to escape " or track whether you're inside binary data or not and it will crash for non-binary aware parsers. Also what about NUL bytes? I bet most parsers are implemented in C/C++ using typical char * null-terminated string, how'll pass this JSON to someone? because well NUL... 2014-07-08 22:22 GMT+03:00 Cathal (Phone) <cathalgarvey@cathalgarvey.me>:

Biggest FAIL in json is lack of binary. I think a quick extension to bencoding is in order: "u<len>" prefix for utf8, "b<len>" for binary. Other types are pretty generally useful as-is. Replace "d", "l" and "e" with curly and square braces for readability.

Bencoding's structure and basic idea is nice as it's terse and understandable, but also easy to make security guarantees about: length prefix, and on parse errors just dump the input and error out. Minimal overhead for raw binary, which is what you want for crypto, file transfers, and VoiP streams. Trivial to write in any language so rapidly portable, and can be coded recursively with relative ease without sacrificing understandability or security (much).

Thoughts? Bencoding 2.0?

On 8 July 2014 17:05:22 GMT+01:00, "Dāvis Mosāns" <davispuh@gmail.com> wrote:

...

yeah I agree that using XML was bad idea in XMPP design, there's no good reason to use it, but XMPP is already thought out unlike any new protocol. But actually I think could use same XMPP protocol and just map on different encoding. What is XML? basically it's just a language for data mapping (an encoding) and it would be perfectly possible to use same XMPP protocol concepts and map them on different data structure. And this is the thing I think should be pursued for. Which encoding to use is debatable. I would say using Google Protocol Buffers <https://developers.google.com/protocol-buffers/> are perfect for network protocols. I haven't investigated how good is MessagePack, but it could be usable too. Only about JSON and similar I don't like that they're "type-less", they have just some basic data types like String, Number etc and you loose information that way, say you've uint32 and you store and transmit that with JSON and on other end it will be probably int64 because that CPU is 64bit, of course you could find shortest fitting type, but that's not practical because you don't know limits of this field, maybe next message it will be bigger. In Protobuf there's types for everything int32, unit64 and so on.

2014-07-08 11:25 GMT+03:00 Zenaan Harkness <zen@freedbms.net>:

...

On 7/8/14, "Łukasz \"Cyber Killer\" Korpalski" <cyberkiller8@gmail.com> wrote:

...

W dniu 07.07.2014 16:55, rysiek pisze:

...

Dnia poniedziałek, 7 lipca 2014 16:06:47 Dāvis Mosāns pisze:

...

I don't agree, I think XMPP could be good solution, while yes attack surface is quite large but it will be in any case, because even if you create the very minimalist chat protocol possible (let's say basically use asymmetric cryptography for messages which are plaintext without any features) you still can have bugs in cryptography library, network stack, OS/kernel. This part will be same no matter what messaging protocol you use.

Exactly. And that's an argument for NOT minimizing the attack surface beyond these problems... how exactly?

I mean, your argument is basically: "don't wash your hands, as there might be salmonella in the eggs anyway". Dafuq?

I'm going to defend XMPP too, but on the grounds that it's an already established and widely used protocol, the overhead is minimal looking from a modern point of view (even when not using the potentially privacy-risky elements) and it was designed to be extendable. These are imo good arguments to use xmpp instead of creating something new (again :-P ).

As has been said over the decades: start correct, 'good' easy to maintain code, secure of course, and optimize later, eg 1-1 mapping from XMPP (XML I assume?) to say msgpack: MessagePack: http://msgpack.org/ - a fast, binary replacement for JSON

Such optimizations ought be behind a library anyway! (From user app point of view.)

As someone else said, think of the stack, separate the concerns: IP, user addressing, persistence of ids, persistence of addresses, crypting, dht, distributed storage, libs, user apps.

For impatient programmers wanting instant gratification, work on one layer in the stack.

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

W dniu 08.07.2014 22:31, Lodewijk andré de la porte pisze:

2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh@gmail.com <mailto:davispuh@gmail.com>>:

Also what about NUL bytes? I bet most parsers are implemented in

(---snip the technical discussion about a new protocol---)

secure enough against those attacks?

Stop right there... It's really nice that so many of you got into the spirit and start thinking about how to change xmpp to make it something new, but what are you achieving here? It will end up being a new protocol, incompatible with existing xmpp, it will take a few years to finish the spec, then another 10+ years until any meaningful applications start using it (if at all)... So yeah, except being "coder porn" it does nothing to help the problem here and now. I agree that xmpp is not perfect, it has some problems of its own, but it is an already established and widely used and standard protocol, with lots of implementations. From a practical point of view the best course of action to get something fast is to use it, and put whatever new stuff there is needed inside xmpp, keeping it compatible with the existing spec. A technically pretty proto won't help, today’s world has a huge problem with taking anything new. Better to stay with existing stuff, make it maybe less efficient because of it, but it will be here fast, when it's needed. Plus being less efficient is a no issue today, with fast machines (you can use compression on the fly, yes really :-P ), loads of storage, broadband connections (even the 3G data caps are getting larger and larger each year), etc. People are sending gigabytes of binary files in base64 each day in email messages, so why even care? ;-) In my opinion the bottom line is - a small addition to existing xmpp has a far larger chance of being widely adopted (by applications and by the users) than a completely new protocol. And despite how awesome coder one might be - you won't be able to write all those implementations yourself or convince the masses to switch (again!). -- Łukasz "Cyber Killer" Korpalski mail: cyberkiller8@gmail.com xmpp: cyber_killer@jabster.pl site: http://website.cybkil.cu.cc gpgkey: 0x72511999 @ hkp://keys.gnupg.net //When replying to my e-mail, kindly please //write your message below the quoted text.

2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:

2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh@gmail.com>:

Also what about NUL bytes? I bet most parsers are implemented in C/C++

...

using typical char * null-terminated string, how'll pass this JSON to someone? because well NUL...

At the parser level you would find a "b" character that's not between brackets, signalling a binary header is coming. A binary header is actually just the number of bytes that follow in binary format. The following bytes are then a binary file, to be assigned to a string as if it were a variable. We have a binarybuffer in javascript, that sort of thing. It would contain the NULL byte if you like it to.

[...]

I think it's much more serious that you have to serve the BJSON completely as a binary file. It's not like you can dump it onto a webpage anymore. You can't slip it into your normal HTTP text transfer bodies either, has to work with attachments. Attachments should be secure when facing malformed transfers* etc. Inconvenient, but not the end.

A lot of protocols are text based, often implemented using C strings and that's what I mean, you can't embed a JSON with binary data containing NUL there (because NUL will terminate that string), so you handle it like typical binary file and then what's the point of JSON to use in first place, because I don't see how it can be any better than any other proper binary data. Such binary JSON gives only overhead but no advantages. 2014-07-09 7:25 GMT+03:00 Bill Stewart <billstewart@pobox.com>:

I haven't used it in years, but I was always quite fond of XDR https://en.wikipedia.org/wiki/External_Data_Representation Sun's External Data Representation coding from the 80s, RFC-1014. Defines a bunch of variable types, and gives you tools for packing and unpacking them.

It's actually pretty good, but there are reasons why Protobuf was created and used instead. The main benefit of Protobuf is that it's easily extendable and can have optional fields. If you add or remove optional fields to server all old clients will still work like nothing have changed. But with XDR you can't do that unless you add another layer on top of it, but that's more work comparing to just taking Protobuf and using it. Also currently Protobuf is much more popular and have more libraries available for dozens of langauges. 2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" < cyberkiller8@gmail.com>:

It's really nice that so many of you got into the spirit and start thinking about how to change xmpp to make it something new, but what are you achieving here? It will end up being a new protocol, incompatible with existing xmpp, it will take a few years to finish the spec, then another 10+ years until any meaningful applications start using it (if at all)... So yeah, except being "coder porn" it does nothing to help the problem here and now.

The goal would be to create smaller overhead and thus be more performance effective. Also it doesn't have to be incompatible. It could be incorporated in XMPP so that new applications could use it but other's just use same legacy XMPP and everything keeps working fine and people wouldn't know what's happening under the hood, nor they would care. And I think it would be trivial to convince people to use and enable this "Binary" XMPP mode (if it's implemented in their client) which makes their chat client app to use 100x times less bandwidth and 50x times less CPU time (spent in parsing), thus your phone's battery would last longer. And yes XML overhead is that big.

A technically pretty proto won't help, today’s world has a huge problem with taking anything new. Better to stay with existing stuff, make it maybe less efficient because of it, but it will be here fast, when it's needed. Plus being less efficient is a no issue today, with fast machines (you can use compression on the fly, yes really :-P ), loads of storage, broadband connections (even the 3G data caps are getting larger and larger each year), etc. People are sending gigabytes of binary files in base64 each day in email messages, so why even care? ;-)

I guess you don't know that nothing is ever fast or good enough. People will always want things faster. What about real-time video call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some really good binary protocol is used so that it's not your software that creates a bottleneck, but if it does then your software is bad and why would I use it over other that can do it, the one that was designed for it, for example see Elemental Demonstrates 4K HEVC Video at 60 fps in London <http://www.streamingmediaglobal.com/Articles/Editorial/Featured-Articles/Elemental-Demonstrates-4K-HEVC-Video-at-60-fps-in-London-93707.aspx> Anyway, I must admit that I haven't studied XMPP enough to know how good or bad it is, but always should try to minimize any overhead, basically you want to process as little as possible. Here straight from wiki, weaknesses: - Does not support Quality of Service (QoS) - XMPP does not have the ability to set the timing flow of messages, preventing XMPP from becoming practical for many embedded distributed realtime, Machine-to-Machine, or IoT applications. - High overhead for embedded applications - As a text based protocol, XMPP has a relatively high computing and network overhead. - In-band binary data transfer is inefficient - Binary data must be first base64 encoded before it can be transmitted in-band. Therefore any significant amount of binary data (e.g., file transfers) is best transmitted out-of-band, using in-band messages to coordinate. The best example of this is the Jingle <http://en.wikipedia.org/wiki/Jingle_%28protocol%29> XMPP Extension Protocol, XEP-0166 <http://xmpp.org/extensions/xep-0166.html>. This issue are being adressed by the experimental XEP-0322: Efficient XML Interchange (EXI) Format <http://xmpp.org/extensions/xep-0322.html>. that sounds really really bad. But it's not all lost, Jingle <http://xmpp.org/about-xmpp/technology-overview/jingle/> actually seems good as it have option to switch to Real-time Transport Protocol (RTP) <http://en.wikipedia.org/wiki/Real-time_Transport_Protocol> and then it's just pure binary stream with minimal overhead. And looks like they are aware of these issues as EXI is being developed, but still while it's a big step forward, it will never beat pure binary protocol. Also from wiki, this is good idea: A perhaps more efficient transport for real-time messaging is WebSocket <http://en.wikipedia.org/wiki/WebSocket>, a web technology providing for bi-directional, full-duplex communications channels over a single TCP connection. Experimental implementations of XMPP over WebSocket exist, and a (now-expired) Internet-Draft documenting this approach was published at the IETF but not yet standardized. In my opinion the bottom line is - a small addition to existing xmpp has

a far larger chance of being widely adopted (by applications and by the users) than a completely new protocol. And despite how awesome coder one might be - you won't be able to write all those implementations yourself or convince the masses to switch (again!).

Maybe yes, maybe no. I think if you've written specification in very clear and understandable way and if you've reference implementation library which everyone could just link against and if your protocol does it better than current existing solutions then I don't see why it wouldn't get adapted. Besides you don't need it implemented everywhere, you need it so that it's in application you use and you could contribute there yourself. 2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:

[...]

There is no masses using XMPP. Masses of coders, maybe, and they will use the best tool for the job.

All the extensions have succeeded in making any XMPP app lacking in usability. I sure haven't found any nice XMPP clients, nice enough to compare with native clients.

That's true indeed, currently there aren't any decent XMPP client (atleast I'm not aware of any). I mean from user's usability point (UX/UI). There are good either proprietary clients (eg. Skype) or good open source clients (eq. Quassel) that doesn't support XMPP :D In fact I'm willing to bet everyone in the western world uses FB, Google

chat and MSN (slackers and slowpokes). They all have limited XMPP implementations, they native clients do more. And there's no good app for interacting with XMPP.

About which Western wold you're talking about? I don't know, but I would assume that in Europe, Skype would be one of the most popular clients. Atleast here MSN never was a thing and everyone have always been using Skype and almost everyone still does. FB isn't really used that much (here we've better alternative). And about Google Talk only some people are aware that it even exists. I know that in Russia it's ICQ and in China it's QQ that's dominating there. Anyway I think they don't implement XMPP because it's too much work for too small benefit. Also then how they would compete with others if they would be limited with XMPP if it doesn't do something that they want to do. So own protocol might be safer bet for a lot of companies. By the way Google Talk dropped XMPP support <https://www.eff.org/deeplinks/2013/05/google-abandons-open-standards-instant-messaging> . I actually really hoped that Tox would succeed so that I could drop Skype and convince others to do same :D But now I'm not so sure if there will be some replacement in nearest future. Actually I think one of reason why Tox created new protocol and didn't used XMPP is that generally XMPP consists of client-server architecture and P2P support is only with extension which isn't widely known nor implemented much. Also XMPP seems way too complicated than it should be. Pidgin really isn't good, it's just the only one out there. And it is still

in the MSN era. I've switched to Office 2013 from Libre/OpenOffice and it really is in a different league all together. And it sucks that it is. But what can we do?

I personally don't like Pidgin too. And about Office and LibreOffice they both have their own advantages and disadvantages but I wouldn't say that any of them would be significantly better or worse than other. Also it's nothing to do with open souirce, it's just we've more users that want everything to be perfect without any effort for free than go and help and contribute to projects. Ok, so anyway I've come to conclusion that I'm not sure anymore whether it's better to try to fix XMPP or just create new much simpler and better protocol. But in any case we really should summarize all our points about what's bad in XMPP and send them to XMPP group and see if they're willing to fix them and accept our proposals. If we want to create a new protocol, then I've few ideas about it. Firstly it's really a shame that a lot of things are getting reinvented over and over again. So the main goal would be don't reinvent stuff but reuse as much as possible from existing things. It would help a lot as there's libraries for already everything and would just have to combine them. Next, think about high-level differences between instant message, group message, offline chat message, email, SMS, MMS, mailing list. What are differences? Actually if you think about it, there are no differences at all, it's the same message being delivered in various ways. But why single protocol couldn't handle it all? In fact it could. We're already using browser for all of that. It's just that there's no specification to unify it all in single protocol. Imagine if we wouldn't need a separate application for IRC, for email nor for other chats. It all could be handled by one general protocol. And why stop here, actually calls, sound and video are also exactly same. There's no really difference between text or sound/video, it's just a data and it depends only how you interpret it. Also security, there's only one option, just encrypt it all before sending. It's just so simple. This next generation protocol would be a superset of typical media protocols and thus it wouldn't matter anymore what other people use. You would just use some middle-ware protocol layer that would translate that foreign protocol to this next-gen protocol and you simply use your favorite client. From user's point of view it doesn't matter at all which protocol is used under the hood. So the main features and design goals of this protocol should be: - Support for peer-to-peer and client-server architectures over both TCP and UDP - Lightweight, minimal overhead, generally be just pass-thru - Efficient encoding, basically binary streams - Encryption by default with option for OTR - Data stream itself can contain anything: text, images, sound, video, animations, screen sharing (and even remote desktop control would be possible) - Dozens of events: joined, left, started typing and so on - One to one and one to many streams - maybe more things So how that all would be possible? It's extremely simple. In client-server architecture, open connection to server, establish TLS (for example) now send this next-gen protocol header message (to specify what kind of data you're sending) and then send your data, it could be for example a raw Theora <http://www.theora.org/> stream (directly from your camera) or just a text message. Then server just relays it to all parties. I don't think it could be any simpler and nothing will beat this in terms of performance. You've just some basic dependencies and what would be the reason to complicate this? It's trivial to implement as there's already libraries for those. In case of offline messages server would just store them and in fact it could do same for sound and videos too. You know Vine, Snapchat? what about YouTube, Soundcloud, basically it could support them all. IMO this is millions worth idea :D It's worth looking at WebSockets <https://www.websocket.org/>, WebRTC <http://www.webrtc.org> and probably other standards. I really don't understand why currently it all have been made so complicated because it isn't.

A lot of protocols are text based, often implemented using C strings and that's what I mean, you can't embed a JSON with binary data containing NUL there (because NUL will terminate that string), so you handle it like typical binary file and then what's the point of JSON to use in first place, because I don't see how it can be any better than any other proper binary data. Such binary JSON gives only overhead but no advantages.

You can also shove NUL bytes into any other transfer encoding to make headaches for the recipient, it's not an encoding-specific problem. It's up to the author of a decoder to account for their language's issues. In C, you'd have to treat the data as binary, not a string, to avoid NUL termination, and only decode (safely) that which is explicitly described as a UTF8 string by leading with "u"; if that contains premature NULs, it's an error condition. On 09/07/14 23:27, Dāvis Mosāns wrote:

2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:

...

2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh@gmail.com>:

Also what about NUL bytes? I bet most parsers are implemented in C/C++

...

using typical char * null-terminated string, how'll pass this JSON to someone? because well NUL...

At the parser level you would find a "b" character that's not between brackets, signalling a binary header is coming. A binary header is actually just the number of bytes that follow in binary format. The following bytes are then a binary file, to be assigned to a string as if it were a variable. We have a binarybuffer in javascript, that sort of thing. It would contain the NULL byte if you like it to.

[...]

I think it's much more serious that you have to serve the BJSON completely as a binary file. It's not like you can dump it onto a webpage anymore. You can't slip it into your normal HTTP text transfer bodies either, has to work with attachments. Attachments should be secure when facing malformed transfers* etc. Inconvenient, but not the end.

A lot of protocols are text based, often implemented using C strings and that's what I mean, you can't embed a JSON with binary data containing NUL there (because NUL will terminate that string), so you handle it like typical binary file and then what's the point of JSON to use in first place, because I don't see how it can be any better than any other proper binary data. Such binary JSON gives only overhead but no advantages.

2014-07-09 7:25 GMT+03:00 Bill Stewart <billstewart@pobox.com>:

...

I haven't used it in years, but I was always quite fond of XDR https://en.wikipedia.org/wiki/External_Data_Representation Sun's External Data Representation coding from the 80s, RFC-1014. Defines a bunch of variable types, and gives you tools for packing and unpacking them.

It's actually pretty good, but there are reasons why Protobuf was created and used instead. The main benefit of Protobuf is that it's easily extendable and can have optional fields. If you add or remove optional fields to server all old clients will still work like nothing have changed. But with XDR you can't do that unless you add another layer on top of it, but that's more work comparing to just taking Protobuf and using it. Also currently Protobuf is much more popular and have more libraries available for dozens of langauges.

2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" < cyberkiller8@gmail.com>:

...

It's really nice that so many of you got into the spirit and start thinking about how to change xmpp to make it something new, but what are you achieving here? It will end up being a new protocol, incompatible with existing xmpp, it will take a few years to finish the spec, then another 10+ years until any meaningful applications start using it (if at all)... So yeah, except being "coder porn" it does nothing to help the problem here and now.

The goal would be to create smaller overhead and thus be more performance effective. Also it doesn't have to be incompatible. It could be incorporated in XMPP so that new applications could use it but other's just use same legacy XMPP and everything keeps working fine and people wouldn't know what's happening under the hood, nor they would care. And I think it would be trivial to convince people to use and enable this "Binary" XMPP mode (if it's implemented in their client) which makes their chat client app to use 100x times less bandwidth and 50x times less CPU time (spent in parsing), thus your phone's battery would last longer. And yes XML overhead is that big.

...

A technically pretty proto won't help, today’s world has a huge problem with taking anything new. Better to stay with existing stuff, make it maybe less efficient because of it, but it will be here fast, when it's needed. Plus being less efficient is a no issue today, with fast machines (you can use compression on the fly, yes really :-P ), loads of storage, broadband connections (even the 3G data caps are getting larger and larger each year), etc. People are sending gigabytes of binary files in base64 each day in email messages, so why even care? ;-)

I guess you don't know that nothing is ever fast or good enough. People will always want things faster. What about real-time video call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some really good binary protocol is used so that it's not your software that creates a bottleneck, but if it does then your software is bad and why would I use it over other that can do it, the one that was designed for it, for example see Elemental Demonstrates 4K HEVC Video at 60 fps in London <http://www.streamingmediaglobal.com/Articles/Editorial/Featured-Articles/Elemental-Demonstrates-4K-HEVC-Video-at-60-fps-in-London-93707.aspx>

Anyway, I must admit that I haven't studied XMPP enough to know how good or bad it is, but always should try to minimize any overhead, basically you want to process as little as possible.

Here straight from wiki, weaknesses:

- Does not support Quality of Service (QoS) - XMPP does not have the ability to set the timing flow of messages, preventing XMPP from becoming practical for many embedded distributed realtime, Machine-to-Machine, or IoT applications. - High overhead for embedded applications - As a text based protocol, XMPP has a relatively high computing and network overhead. - In-band binary data transfer is inefficient - Binary data must be first base64 encoded before it can be transmitted in-band. Therefore any significant amount of binary data (e.g., file transfers) is best transmitted out-of-band, using in-band messages to coordinate. The best example of this is the Jingle <http://en.wikipedia.org/wiki/Jingle_%28protocol%29> XMPP Extension Protocol, XEP-0166 <http://xmpp.org/extensions/xep-0166.html>. This issue are being adressed by the experimental XEP-0322: Efficient XML Interchange (EXI) Format <http://xmpp.org/extensions/xep-0322.html>.

that sounds really really bad. But it's not all lost, Jingle <http://xmpp.org/about-xmpp/technology-overview/jingle/> actually seems good as it have option to switch to Real-time Transport Protocol (RTP) <http://en.wikipedia.org/wiki/Real-time_Transport_Protocol> and then it's just pure binary stream with minimal overhead. And looks like they are aware of these issues as EXI is being developed, but still while it's a big step forward, it will never beat pure binary protocol.

Also from wiki, this is good idea:

A perhaps more efficient transport for real-time messaging is WebSocket <http://en.wikipedia.org/wiki/WebSocket>, a web technology providing for bi-directional, full-duplex communications channels over a single TCP connection. Experimental implementations of XMPP over WebSocket exist, and a (now-expired) Internet-Draft documenting this approach was published at the IETF but not yet standardized.

In my opinion the bottom line is - a small addition to existing xmpp has

...

a far larger chance of being widely adopted (by applications and by the users) than a completely new protocol. And despite how awesome coder one might be - you won't be able to write all those implementations yourself or convince the masses to switch (again!).

Maybe yes, maybe no. I think if you've written specification in very clear and understandable way and if you've reference implementation library which everyone could just link against and if your protocol does it better than current existing solutions then I don't see why it wouldn't get adapted. Besides you don't need it implemented everywhere, you need it so that it's in application you use and you could contribute there yourself.

2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <l@odewijk.nl>:

...

[...]

There is no masses using XMPP. Masses of coders, maybe, and they will use the best tool for the job.

All the extensions have succeeded in making any XMPP app lacking in usability. I sure haven't found any nice XMPP clients, nice enough to compare with native clients.

That's true indeed, currently there aren't any decent XMPP client (atleast I'm not aware of any). I mean from user's usability point (UX/UI). There are good either proprietary clients (eg. Skype) or good open source clients (eq. Quassel) that doesn't support XMPP :D

In fact I'm willing to bet everyone in the western world uses FB, Google

...

chat and MSN (slackers and slowpokes). They all have limited XMPP implementations, they native clients do more. And there's no good app for interacting with XMPP.

About which Western wold you're talking about? I don't know, but I would assume that in Europe, Skype would be one of the most popular clients. Atleast here MSN never was a thing and everyone have always been using Skype and almost everyone still does. FB isn't really used that much (here we've better alternative). And about Google Talk only some people are aware that it even exists. I know that in Russia it's ICQ and in China it's QQ that's dominating there. Anyway I think they don't implement XMPP because it's too much work for too small benefit. Also then how they would compete with others if they would be limited with XMPP if it doesn't do something that they want to do. So own protocol might be safer bet for a lot of companies. By the way Google Talk dropped XMPP support <https://www.eff.org/deeplinks/2013/05/google-abandons-open-standards-instant-messaging> .

I actually really hoped that Tox would succeed so that I could drop Skype and convince others to do same :D But now I'm not so sure if there will be some replacement in nearest future. Actually I think one of reason why Tox created new protocol and didn't used XMPP is that generally XMPP consists of client-server architecture and P2P support is only with extension which isn't widely known nor implemented much. Also XMPP seems way too complicated than it should be.

Pidgin really isn't good, it's just the only one out there. And it is still

...

in the MSN era. I've switched to Office 2013 from Libre/OpenOffice and it really is in a different league all together. And it sucks that it is. But what can we do?

I personally don't like Pidgin too. And about Office and LibreOffice they both have their own advantages and disadvantages but I wouldn't say that any of them would be significantly better or worse than other. Also it's nothing to do with open souirce, it's just we've more users that want everything to be perfect without any effort for free than go and help and contribute to projects.

Ok, so anyway I've come to conclusion that I'm not sure anymore whether it's better to try to fix XMPP or just create new much simpler and better protocol. But in any case we really should summarize all our points about what's bad in XMPP and send them to XMPP group and see if they're willing to fix them and accept our proposals.

If we want to create a new protocol, then I've few ideas about it. Firstly it's really a shame that a lot of things are getting reinvented over and over again. So the main goal would be don't reinvent stuff but reuse as much as possible from existing things. It would help a lot as there's libraries for already everything and would just have to combine them. Next, think about high-level differences between instant message, group message, offline chat message, email, SMS, MMS, mailing list. What are differences? Actually if you think about it, there are no differences at all, it's the same message being delivered in various ways. But why single protocol couldn't handle it all? In fact it could. We're already using browser for all of that. It's just that there's no specification to unify it all in single protocol. Imagine if we wouldn't need a separate application for IRC, for email nor for other chats. It all could be handled by one general protocol. And why stop here, actually calls, sound and video are also exactly same. There's no really difference between text or sound/video, it's just a data and it depends only how you interpret it. Also security, there's only one option, just encrypt it all before sending. It's just so simple.

This next generation protocol would be a superset of typical media protocols and thus it wouldn't matter anymore what other people use. You would just use some middle-ware protocol layer that would translate that foreign protocol to this next-gen protocol and you simply use your favorite client. From user's point of view it doesn't matter at all which protocol is used under the hood.

So the main features and design goals of this protocol should be:

- Support for peer-to-peer and client-server architectures over both TCP and UDP - Lightweight, minimal overhead, generally be just pass-thru - Efficient encoding, basically binary streams - Encryption by default with option for OTR - Data stream itself can contain anything: text, images, sound, video, animations, screen sharing (and even remote desktop control would be possible) - Dozens of events: joined, left, started typing and so on - One to one and one to many streams - maybe more things

So how that all would be possible? It's extremely simple. In client-server architecture, open connection to server, establish TLS (for example) now send this next-gen protocol header message (to specify what kind of data you're sending) and then send your data, it could be for example a raw Theora <http://www.theora.org/> stream (directly from your camera) or just a text message. Then server just relays it to all parties.

I don't think it could be any simpler and nothing will beat this in terms of performance. You've just some basic dependencies and what would be the reason to complicate this? It's trivial to implement as there's already libraries for those. In case of offline messages server would just store them and in fact it could do same for sound and videos too. You know Vine, Snapchat? what about YouTube, Soundcloud, basically it could support them all. IMO this is millions worth idea :D

It's worth looking at WebSockets <https://www.websocket.org/>, WebRTC <http://www.webrtc.org> and probably other standards.

I really don't understand why currently it all have been made so complicated because it isn't.

-- T: @onetruecathal, @IndieBBDNA P: +353876363185 W: http://indiebiotech.com

Hi everyone,

I'm working on the Movim project since 2008, our aim is to create a full social network on top of the XMPP protocol. As I see again, the guys of the Tox project are trying to reinvent the wheel… again. Now, to do IM, we have Skype, BBM, Line, WhatsApp, MSN, QQ, AIM, ICQ, IRC, XMPP, Facebook Messenger…

Same for the social networks as Davis said (PumpIO, TentIO…)

I really think that we need to focus on an existent standard and improve it, and for me XMPP seem to be the perfect protocol for all theses things : - Standard IM + chatroom - Video/Audio conferencing (with Jingle, we are using it with WebRTC on Movim) - Pubsub (for newsfeeds, blogging) - Geolocation - Vcard4 support - SASL2 authentication - OTR support - Full encryption between the servers (https://xmpp.net/list.php) - and so on…

XMPP can do a lot more than just IM, it's a full social-communication protocol it just need to be implemented, tested and debugged :)

Tim

On lun., juil. 7, 2014 at 6:00 , Dāvis Mosāns <davispuh@gmail.com> wrote:

2014-07-06 23:28 GMT+03:00 rysiek <rysiek@hackerspace.pl>:

...

Dnia niedziela, 6 lipca 2014 22:25:59 piszesz:

...

hmm, I wonder are there any such open protocol specification created? I know about XMPP, but nothing more...

Well, there's the Diaspora protocol: https://wiki.diasporafoundation.org/Federation_protocol_overview

And... StatusNet/OStatus, PumpIO, TentIO, ActivityStreams, BuddyCloud (XMPP- based, I guess), and quite a few others I don't really remember. Some of them are related, all are incompatible. And all the devs are showing strong symptoms of the NIH syndrome.

Which is absurd.

-- Pozdr rysiek

that indeed is stupid and so no one have solved it yet... for social network or basically any IM/chat/etc to be usable it must have majority of people (eg. your friends) users there, otherwise without people they are totally useless so currently we're stuck with no-so-great applications/protocols only because everyone already are on them like Facebook and Skype. On that mailing list there were discussion about a polyglot protocol/application which could support all networks so users wouldn't be forced to migrate which I think is essential because a lot of people won't bother. There was mention to Sockethub <http://sockethub.org/>which seems quite cool, only for a bit different use case I would say. Another thing I would like to mention is BitlBee <http://bitlbee.org> it is a gateway between various IM/chat networks and IRC so you can chat with friends on Facebook using your favorite IRC client, or post a tweet on your Twitter and use various other protocols. It even supports OTR.

okay so I've quickly reviewed Movim, idea is really good and it seems to be nice, but I haven't yet tried to run it, will do that someday. It looks

2014-07-07 10:11 GMT+03:00 edhelas <edhelas@movim.eu>: like you haven't really marketed it good enough because this is first time I hear about it despite it being an somewhat old project. For example Tox is pretty new but it's already quite popular and I keep hearing about it every few months. I would suggest to post more on various social sites, forums and just let people know it exists (eg. post to Reddit) Another thing I would suggest is add a video to website of example usage so people could see how it is actually used, explain various features and such as users might not immediately discover some features. Now I'll tell a few things, but that's only my personal opinion and most likely a lot of people won't agree with it. So anyway firstly I'm not a fan of PHP, it's just generally awful language (see http://phpsadness.com/ and look at PHP src :D), I know it because I've been writing it for like 7+ years but now 2-3 years I'm PHP-less and happy about it :) Next it looks like you aren't using any PHP framework but self-developed one which gives you more work than is needed and obviously it's less battletested. But overall code itself is nice and pretty, correctly uses MVC pattern. Bad things are that you don't have separate public directory for frontend and anyone can access PHP files directly, view templates for example ( https://pod.movim.eu/app/views/admin.tpl) it's not a big deal, but still not good idea (running version https://pod.movim.eu/VERSION). Then in some places HTML tags and entities are used directly rather than proper Unicode which isn't a good idea and it means that string isn't later escaped and if it gets mixed together with user-input or translation strings there's a place for XSS. The worse thing probably is that sanitization is based on regexp blacklists/filters, I'm talking about StringHelper.php, I didn't look how it's actually used, but still even without trying I'm pretty sure it would be possible to find XSS there, why? because Rails framework over 5 years have had ~20 XSS vulnerabilities and it's extremely good framework used by dozens of projects and reviewed regularly, and it's even based on whitelists, but still uses regexps for that which isn't good and I wonder why no one does proper SGML parsing which they should. Just take a look at sanitizer.rb <https://github.com/rails/rails/blob/master/actionview/lib/action_view/vendor/html-scanner/html/sanitizer.rb#L72> to see how non-trivial it is. Anyway the whole idea of sanitization is wrong, you should just escape all text and don't try to guess which tags you should render. I suggest any web developer to read OWASP <https://www.owasp.org> from A to Z it's a must for any web developer. Then there's `?>` PHP end tags used at end of various files which are useless and can introduce problems like famous "headers already sent" warning. So seems that's about it with my quick look, but I might have forgotten to mention some things. Another thing I don't like is that AGPL is used, I really dislike all GPL family, but that's just me and I rather prefer copyfree <http://copyfree.org> so if there's similar projects then I'll rather contribute to MIT than any GPL variant :P And I'm not a fan of Bazaar nor LaunchPad but that's not the worst thing (someone should ban CVS and SVN :D) So to sum up about Movim, good parts: - Good idea - Quite decent code, MVC used correctly - Localization support - Pretty website - Open Source - Active development - SCM is used and bad: - Not enough known, marketed - PHP is used - No PHP framework used but self-developed one - Some questionable and potentially vulnerable code in some places - Scripts and files accessible directly - Not my favorite (un)license But yeah keep it up and continue developing it ;) I might use it some day...

sounds great :) well, about frameworks I've tried really a lot of them over years and I was satisfied only with FuelPHP <http://fuelphp.com/> which I think is the best one IMO, but I haven't heard about Laravel, seems to be really new. About FuelPHP I really like idea of HMVC and it's very useful. But as I said I haven't done anything with PHP for few years now. But generally you've to choose framework when starting a project as it's usually not easy to change it later. Yeah I forgot to mention that it's really good that you're using Composer and have good object orientated code structure. About internationalization it depends which parts you want to cover. If translated strings then look at transifex.com <http://www.transifex.com> and crowdin.net they both offer free solutions to open source projects and are quite good. If you want localized date and time formats then use CLDR <http://cldr.unicode.org/> you can either write a script to directly get data from them or use some already made libraries for PHP, I don't know for PHP, but for Ruby there's ruby-cldr <https://github.com/svenfuchs/ruby-cldr> and twitter-cldr-rb <https://github.com/twitter/twitter-cldr-rb>, for example ruby-cldr can export CLDR data to yaml files and then you could parse and use those in PHP. (twitter-cldr also uses that same exported data from ruby-cldr) 2014-07-07 20:44 GMT+03:00 edhelas <edhelas@movim.eu>:

Hi Davis,

Thank you very much for this awesome feedback, having constructives criticizes like this helps me a lot :) I'll try to explain my choices for some of them.

I agree with your first comment, there's clearly a lack of communication. But, I'm currently working on the 0.8 release and on a fundraising (on Kickstarter) before the end of summer. I'll create a thread on Reddit and talk about it theses next couple of weeks ;)

For the video, I don't have skills to do a nice looking one. But if you have tips, do not hesitate to share them with me.

For PHP, the choice was made a couple of years ago and the aim was to install Movim on quite all the servers (I love Ruby on Rails but deploying a RoR application can be quite difficult for some administrators). We also tried to built Movim on top of differents PHP frameworks (Zend, Symfony and more recently Laravel). The thing is that Movim works in a really special way (the connexion is kept open with the XMPP server using BOSH threw long polling requests so I have to do session synchronisation en prevent session-lock… all in PHP) so it cannot be ported easily on a "classical MVC" framework. We also use our own internal widget system with event handling (when a specific XMPP stanza is handled).

I'll take a look at the sanitizer.rb file and try to find a proper way to sanitize the strings, maybe use an external library for that ;)

Having a public/ folder is also planned for the 1.0 version but I need to refactor a couple of stuffs in the app to make it works properly.

I'm also using the PSR standard (http://www.php-fig.org/) especially for the library loading (using composer) and the logger. I've already moved parts of Movim to independant libraries to modularize the project ;)

I'm trying to move from Bazaar to Git but I mave a couple of issues when I convert the commit-history tree. Also I'm looking for a proper way to handle the internationalisation (Launchpad has a ship-in system for that).

We are also one of the most advanced XMPP client, with a really nice implementation of the standard (all the currently implemented XEP are listed here : http://wiki.movim.eu/en:dev:protocol_implementations). I'm working with the XMPP Standard Fundation to standardise and improve the XMPP protocol.

Thanks again !

edhelas

On lun., juil. 7, 2014 at 7:18 , Dāvis Mosāns <davispuh@gmail.com> wrote:

2014-07-07 10:11 GMT+03:00 edhelas <edhelas@movim.eu>:

...

Hi everyone,

I'm working on the Movim project since 2008, our aim is to create a full social network on top of the XMPP protocol. As I see again, the guys of the Tox project are trying to reinvent the wheel… again. Now, to do IM, we have Skype, BBM, Line, WhatsApp, MSN, QQ, AIM, ICQ, IRC, XMPP, Facebook Messenger…

Same for the social networks as Davis said (PumpIO, TentIO…)

I really think that we need to focus on an existent standard and improve it, and for me XMPP seem to be the perfect protocol for all theses things : - Standard IM + chatroom - Video/Audio conferencing (with Jingle, we are using it with WebRTC on Movim) - Pubsub (for newsfeeds, blogging) - Geolocation - Vcard4 support - SASL2 authentication - OTR support - Full encryption between the servers (https://xmpp.net/list.php) - and so on…

XMPP can do a lot more than just IM, it's a full social-communication protocol it just need to be implemented, tested and debugged :)

Tim

On lun., juil. 7, 2014 at 6:00 , Dāvis Mosāns <davispuh@gmail.com> wrote:

2014-07-06 23:28 GMT+03:00 rysiek <rysiek@hackerspace.pl>:

...

Dnia niedziela, 6 lipca 2014 22:25:59 piszesz:

...

hmm, I wonder are there any such open protocol specification created? I know about XMPP, but nothing more...

Well, there's the Diaspora protocol: https://wiki.diasporafoundation.org/Federation_protocol_overview

And... StatusNet/OStatus, PumpIO, TentIO, ActivityStreams, BuddyCloud (XMPP- based, I guess), and quite a few others I don't really remember. Some of them are related, all are incompatible. And all the devs are showing strong symptoms of the NIH syndrome.

Which is absurd.

-- Pozdr rysiek

that indeed is stupid and so no one have solved it yet... for social network or basically any IM/chat/etc to be usable it must have majority of people (eg. your friends) users there, otherwise without people they are totally useless so currently we're stuck with no-so-great applications/protocols only because everyone already are on them like Facebook and Skype. On that mailing list there were discussion about a polyglot protocol/application which could support all networks so users wouldn't be forced to migrate which I think is essential because a lot of people won't bother. There was mention to Sockethub <http://sockethub.org/>which seems quite cool, only for a bit different use case I would say. Another thing I would like to mention is BitlBee <http://bitlbee.org> it is a gateway between various IM/chat networks and IRC so you can chat with friends on Facebook using your favorite IRC client, or post a tweet on your Twitter and use various other protocols. It even supports OTR.

okay so I've quickly reviewed Movim, idea is really good and it seems to be nice, but I haven't yet tried to run it, will do that someday. It looks like you haven't really marketed it good enough because this is first time I hear about it despite it being an somewhat old project. For example Tox is pretty new but it's already quite popular and I keep hearing about it every few months. I would suggest to post more on various social sites, forums and just let people know it exists (eg. post to Reddit) Another thing I would suggest is add a video to website of example usage so people could see how it is actually used, explain various features and such as users might not immediately discover some features.

Now I'll tell a few things, but that's only my personal opinion and most likely a lot of people won't agree with it. So anyway firstly I'm not a fan of PHP, it's just generally awful language (see http://phpsadness.com/ and look at PHP src :D), I know it because I've been writing it for like 7+ years but now 2-3 years I'm PHP-less and happy about it :) Next it looks like you aren't using any PHP framework but self-developed one which gives you more work than is needed and obviously it's less battletested. But overall code itself is nice and pretty, correctly uses MVC pattern. Bad things are that you don't have separate public directory for frontend and anyone can access PHP files directly, view templates for example ( https://pod.movim.eu/app/views/admin.tpl) it's not a big deal, but still not good idea (running version https://pod.movim.eu/VERSION). Then in some places HTML tags and entities are used directly rather than proper Unicode which isn't a good idea and it means that string isn't later escaped and if it gets mixed together with user-input or translation strings there's a place for XSS. The worse thing probably is that sanitization is based on regexp blacklists/filters, I'm talking about StringHelper.php, I didn't look how it's actually used, but still even without trying I'm pretty sure it would be possible to find XSS there, why? because Rails framework over 5 years have had ~20 XSS vulnerabilities and it's extremely good framework used by dozens of projects and reviewed regularly, and it's even based on whitelists, but still uses regexps for that which isn't good and I wonder why no one does proper SGML parsing which they should. Just take a look at sanitizer.rb <https://github.com/rails/rails/blob/master/actionview/lib/action_view/vendor/html-scanner/html/sanitizer.rb#L72> to see how non-trivial it is. Anyway the whole idea of sanitization is wrong, you should just escape all text and don't try to guess which tags you should render. I suggest any web developer to read OWASP <https://www.owasp.org> from A to Z it's a must for any web developer. Then there's `?>` PHP end tags used at end of various files which are useless and can introduce problems like famous "headers already sent" warning. So seems that's about it with my quick look, but I might have forgotten to mention some things. Another thing I don't like is that AGPL is used, I really dislike all GPL family, but that's just me and I rather prefer copyfree <http://copyfree.org> so if there's similar projects then I'll rather contribute to MIT than any GPL variant :P And I'm not a fan of Bazaar nor LaunchPad but that's not the worst thing (someone should ban CVS and SVN :D)

So to sum up about Movim, good parts:

- Good idea - Quite decent code, MVC used correctly - Localization support - Pretty website - Open Source - Active development - SCM is used

and bad:

- Not enough known, marketed - PHP is used - No PHP framework used but self-developed one - Some questionable and potentially vulnerable code in some places - Scripts and files accessible directly - Not my favorite (un)license

But yeah keep it up and continue developing it ;) I might use it some day...