Tox.im

[Cathal Garvey]

> A lot of protocols are text based, often implemented using C strings and
> that's what I mean, you can't embed a JSON with binary data containing NUL
> there (because NUL will terminate that string), so you handle it like
> typical binary file and then what's the point of JSON to use in first
> place, because I don't see how it can be any better than any other proper
> binary data. Such binary JSON gives only overhead but no advantages.

You can also shove NUL bytes into any other transfer encoding to make
headaches for the recipient, it's not an encoding-specific problem. It's
up to the author of a decoder to account for their language's issues. In
C, you'd have to treat the data as binary, not a string, to avoid NUL
termination, and only decode (safely) that which is explicitly described
as a UTF8 string by leading with "u"; if that contains premature NULs,
it's an error condition.

On 09/07/14 23:27, Dāvis Mosāns wrote:
> 2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <l at odewijk.nl>:
> 
>> 2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <davispuh at gmail.com>:
>>
>> Also what about NUL bytes? I bet most parsers are implemented in C/C++
>>> using typical char * null-terminated string, how'll pass this JSON to
>>> someone? because well NUL...
>>
>>
>> At the parser level you would find a "b" character that's not between
>> brackets, signalling a binary header is coming. A binary header is actually
>> just the number of bytes that follow in binary format. The following bytes
>> are then a binary file, to be assigned to a string as if it were a
>> variable. We have a binarybuffer in javascript, that sort of thing. It
>> would contain the NULL byte if you like it to.
>>
>> [...]
>>
>> I think it's much more serious that you have to serve the BJSON completely
>> as a binary file. It's not like you can dump it onto a webpage anymore. You
>> can't slip it into your normal HTTP text transfer bodies either, has to
>> work with attachments. Attachments should be secure when facing malformed
>> transfers* etc. Inconvenient, but not the end.
>>
>>
> A lot of protocols are text based, often implemented using C strings and
> that's what I mean, you can't embed a JSON with binary data containing NUL
> there (because NUL will terminate that string), so you handle it like
> typical binary file and then what's the point of JSON to use in first
> place, because I don't see how it can be any better than any other proper
> binary data. Such binary JSON gives only overhead but no advantages.
> 
> 
> 2014-07-09 7:25 GMT+03:00 Bill Stewart <billstewart at pobox.com>:
> 
>> I haven't used it in years, but I was always quite fond of XDR
>> https://en.wikipedia.org/wiki/External_Data_Representation
>> Sun's External Data Representation coding from the 80s, RFC-1014.
>> Defines a bunch of variable types, and gives you tools for packing and
>> unpacking them.
>>
>>
> It's actually pretty good, but there are reasons why Protobuf was created
> and used instead. The main benefit of Protobuf is that it's easily
> extendable and can have optional fields. If you add or remove optional
> fields to server all old clients will still work like nothing have changed.
> But with XDR you can't do that unless you add another layer on top of it,
> but that's more work comparing to just taking Protobuf and using it. Also
> currently Protobuf is much more popular and have more libraries available
> for dozens of langauges.
> 
> 
> 2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski" <
> cyberkiller8 at gmail.com>:
> 
>>
>> It's really nice that so many of you got into the spirit and start
>> thinking about how to change xmpp to make it something new, but what are
>> you achieving here? It will end up being a new protocol, incompatible
>> with existing xmpp, it will take a few years to finish the spec, then
>> another 10+ years until any meaningful applications start using it (if
>> at all)... So yeah, except being "coder porn" it does nothing to help
>> the problem here and now.
>>
>>
> The goal would be to create smaller overhead and thus be more performance
> effective. Also it doesn't have to be incompatible. It could be
> incorporated in XMPP so that new applications could use it but other's just
> use same legacy XMPP and everything keeps working fine and people wouldn't
> know what's happening under the hood, nor they would care. And I think it
> would be trivial to convince people to use and enable this "Binary" XMPP
> mode (if it's implemented in their client) which makes their chat client
> app to use 100x times less bandwidth and 50x times less CPU time (spent in
> parsing), thus your phone's battery would last longer. And yes XML overhead
> is that big.
> 
> 
>> A technically pretty proto won't help, today’s world has a huge problem
>> with taking anything new. Better to stay with existing stuff, make it
>> maybe less efficient because of it, but it will be here fast, when it's
>> needed. Plus being less efficient is a no issue today, with fast
>> machines (you can use compression on the fly, yes really :-P ), loads of
>> storage, broadband connections (even the 3G data caps are getting larger
>> and larger each year), etc. People are sending gigabytes of binary files
>> in base64 each day in email messages, so why even care? ;-)
>>
>>
> I guess you don't know that nothing is ever fast or good enough. People
> will always want things faster. What about real-time video call in 4k @ 60
> FPS ? It's unreal to imagine this in XMPP unless some really good binary
> protocol is used so that it's not your software that creates a bottleneck,
> but if it does then your software is bad and why would I use it over other
> that can do it, the one that was designed for it, for example see Elemental
> Demonstrates 4K HEVC Video at 60 fps in London
> <http://www.streamingmediaglobal.com/Articles/Editorial/Featured-Articles/Elemental-Demonstrates-4K-HEVC-Video-at-60-fps-in-London-93707.aspx>
> 
> Anyway, I must admit that I haven't studied XMPP enough to know how good or
> bad it is, but always should try to minimize any overhead, basically you
> want to process as little as possible.
> 
> Here straight from wiki, weaknesses:
> 
>    - Does not support Quality of Service (QoS)
>       - XMPP does not have the ability to set the timing flow of messages,
>       preventing XMPP from becoming practical for many embedded distributed
>       realtime, Machine-to-Machine, or IoT applications.
>    - High overhead for embedded applications
>       - As a text based protocol, XMPP has a relatively high computing and
>       network overhead.
>    - In-band binary data transfer is inefficient
>       - Binary data must be first base64 encoded before it can be
>       transmitted in-band. Therefore any significant amount of binary
> data (e.g.,
>       file transfers) is best transmitted out-of-band, using in-band
> messages to
>       coordinate. The best example of this is the Jingle
>       <http://en.wikipedia.org/wiki/Jingle_%28protocol%29> XMPP Extension
>       Protocol, XEP-0166 <http://xmpp.org/extensions/xep-0166.html>. This
>       issue are being adressed by the experimental XEP-0322: Efficient XML
>       Interchange (EXI) Format <http://xmpp.org/extensions/xep-0322.html>.
> 
> 
> that sounds really really bad. But it's not all lost, Jingle
> <http://xmpp.org/about-xmpp/technology-overview/jingle/> actually seems
> good as it have option to switch to Real-time Transport Protocol (RTP)
> <http://en.wikipedia.org/wiki/Real-time_Transport_Protocol> and then it's
> just pure binary stream with minimal overhead. And looks like they are
> aware of these issues as EXI is being developed, but still while it's a big
> step forward, it will never beat pure binary protocol.
> 
> Also from wiki, this is good idea:
> 
> A perhaps more efficient transport for real-time messaging is WebSocket
> <http://en.wikipedia.org/wiki/WebSocket>, a web technology providing for
> bi-directional, full-duplex communications channels over a single TCP
> connection. Experimental implementations of XMPP over WebSocket exist, and
> a (now-expired) Internet-Draft documenting this approach was published at
> the IETF but not yet standardized.
> 
> 
> In my opinion the bottom line is - a small addition to existing xmpp has
>> a far larger chance of being widely adopted (by applications and by the
>> users) than a completely new protocol. And despite how awesome coder one
>> might be - you won't be able to write all those implementations yourself
>> or convince the masses to switch (again!).
>>
>>
> Maybe yes, maybe no. I think if you've written specification in very clear
> and understandable way and if you've reference implementation library which
> everyone could just link against and if your protocol does it better than
> current existing solutions then I don't see why it wouldn't get adapted.
> Besides you don't need it implemented everywhere, you need it so that it's
> in application you use and you could contribute there yourself.
> 
> 
> 2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <l at odewijk.nl>:
> 
>> [...]
>>
>> There is no masses using XMPP. Masses of coders, maybe, and they will use
>> the best tool for the job.
>>
>> All the extensions have succeeded in making any XMPP app lacking in
>> usability. I sure haven't found any nice XMPP clients, nice enough to
>> compare with native clients.
>>
>  That's true indeed, currently there aren't any decent XMPP client (atleast
> I'm not aware of any). I mean from user's usability point (UX/UI). There
> are good either proprietary clients (eg. Skype) or good open source clients
> (eq. Quassel) that doesn't support XMPP :D
> 
> In fact I'm willing to bet everyone in the western world uses FB, Google
>> chat and MSN (slackers and slowpokes). They all have limited XMPP
>> implementations, they native clients do more. And there's no good app for
>> interacting with XMPP.
>>
> About which Western wold you're talking about? I don't know, but I would
> assume that in Europe, Skype would be one of the most popular clients.
> Atleast here MSN never was a thing and everyone have always been using
> Skype and almost everyone still does. FB isn't really used that much (here
> we've better alternative). And about Google Talk only some people are aware
> that it even exists. I know that in Russia it's ICQ and in China it's QQ
> that's dominating there. Anyway I think they don't implement XMPP because
> it's too much work for too small benefit. Also then how they would compete
> with others if they would be limited with XMPP if it doesn't do something
> that they want to do. So own protocol might be safer bet for a lot of
> companies. By the way Google Talk dropped XMPP support
> <https://www.eff.org/deeplinks/2013/05/google-abandons-open-standards-instant-messaging>
> .
> 
> 
> I actually really hoped that Tox would succeed so that I could drop Skype
> and convince others to do same :D But now I'm not so sure if there will be
> some replacement in nearest future. Actually I think one of reason why Tox
> created new protocol and didn't used XMPP is that generally XMPP consists
> of client-server architecture and P2P support is only with extension which
> isn't widely known nor implemented much. Also XMPP seems way too
> complicated than it should be.
> 
> Pidgin really isn't good, it's just the only one out there. And it is still
>> in the MSN era. I've switched to Office 2013 from Libre/OpenOffice and it
>> really is in a different league all together. And it sucks that it is. But
>> what can we do?
>>
> I personally don't like Pidgin too. And about Office and LibreOffice they
> both have their own advantages and disadvantages but I wouldn't say that
> any of them would be significantly better or worse than other. Also it's
> nothing to do with open souirce, it's just we've more users that want
> everything to be perfect without any effort for free than go and help and
> contribute to projects.
> 
> 
> Ok, so anyway I've come to conclusion that I'm not sure anymore whether
> it's better to try to fix XMPP or just create new much simpler and better
> protocol. But in any case we really should summarize all our points about
> what's bad in XMPP and send them to XMPP group and see if they're willing
> to fix them and accept our proposals.
> 
> If we want to create a new protocol, then I've few ideas about it. Firstly
> it's really a shame that a lot of things are getting reinvented over and
> over again. So the main goal would be don't reinvent stuff but reuse as
> much as possible from existing things. It would help a lot as there's
> libraries for already everything and would just have to combine them. Next,
> think about high-level differences between instant message, group message,
> offline chat message, email, SMS, MMS, mailing list. What are differences?
> Actually if you think about it, there are no differences at all, it's the
> same message being delivered in various ways. But why single protocol
> couldn't handle it all? In fact it could. We're already using browser for
> all of that. It's just that there's no specification to unify it all in
> single protocol. Imagine if we wouldn't need a separate application for
> IRC, for email nor for other chats. It all could be handled by one general
> protocol. And why stop here, actually calls, sound and video are also
> exactly same. There's no really difference between text or sound/video,
> it's just a data and it depends only how you interpret it. Also security,
> there's only one option, just encrypt it all before sending. It's just so
> simple.
> 
> This next generation protocol would be a superset of typical media
> protocols and thus it wouldn't matter anymore what other people use. You
> would just use some middle-ware protocol layer that would translate that
> foreign protocol to this next-gen protocol and you simply use your favorite
> client. From user's point of view it doesn't matter at all which protocol
> is used under the hood.
> 
> So the main features and design goals of this protocol should be:
> 
>    - Support for peer-to-peer and client-server architectures over both TCP
>    and UDP
>    - Lightweight, minimal overhead, generally be just pass-thru
>    - Efficient encoding, basically binary streams
>    - Encryption by default with option for OTR
>    - Data stream itself can contain anything: text, images, sound, video,
>    animations, screen sharing (and even remote desktop control would be
>    possible)
>    - Dozens of events: joined, left, started typing and so on
>    - One to one and one to many streams
>    - maybe more things
> 
> 
> So how that all would be possible? It's extremely simple. In client-server
> architecture, open connection to server, establish TLS (for example) now
> send this next-gen protocol header message (to specify what kind of data
> you're sending) and then send your data, it could be for example a raw
> Theora <http://www.theora.org/> stream (directly from your camera) or just
> a text message. Then server just relays it to all parties.
> 
> I don't think it could be any simpler and nothing will beat this in terms
> of performance. You've just some basic dependencies and what would be the
> reason to complicate this? It's trivial to implement as there's already
> libraries for those. In case of offline messages server would just store
> them and in fact it could do same for sound and videos too. You know Vine,
> Snapchat? what about YouTube, Soundcloud, basically it could support them
> all. IMO this is millions worth idea :D
> 
> 
> It's worth looking at WebSockets <https://www.websocket.org/>, WebRTC
> <http://www.webrtc.org> and probably other standards.
> 
> 
> I really don't understand why currently it all have been made so
> complicated because it isn't.
> 

-- 
T: @onetruecathal, @IndieBBDNA
P: +353876363185
W: http://indiebiotech.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x988B9099.asc
Type: application/pgp-keys
Size: 6176 bytes
Desc: not available
URL: <http://lists.cpunks.org/pipermail/cypherpunks/attachments/20140710/243194fb/attachment-0001.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 901 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cpunks.org/pipermail/cypherpunks/attachments/20140710/243194fb/attachment-0001.sig>