July 2018 - cypherpunks-legacy

Re: [tahoe-dev] erasure coding makes files more fragile, not less
by Brian Warner 06 Jul '18

06 Jul '18

On 3/27/12 12:06 PM, Zooko Wilcox-O'Hearn wrote: > In fact, I think approximately 90% of all files that have ever been > stored on a Tahoe-LAFS grid have died. (That's excluding all of the > files of all of the customers of allmydata.com, which went out of > business.) That.. is a pretty broad and potentially disingenuous statement, and feels unsupported. I think I know what you mean, but it's a bit like saying "everybody dies" or "all programs crash eventually": maybe true, but kinda useless, and kinda deceptive, or at least distracting. [BTW folks: Zooko and I talk about this stuff all the time, and we know each other's opinions pretty well, so please don't misinterpret my words as indicating anger or annoyance. We're old pals, and this is a well-worn comfortable argument.] The metric that I'd find useful is what percentage of files *that people actively tried to keep around* were lost. Tahoe is a system for multiplying the durability of your servers, but it's not magic, and if you start with lousy unmaintained servers, then you aren't likely to have good results. > I came up with this provocative slogan (I know Brian loves my > provocative slogans): "erasure coding makes files more fragile, not > less". Ah, you do know how to provoke me :). That's like saying "seat belts, airbags, and helmets kill people", invoking haunting images of demonic safety gear stalking the last remaining humans through the forest, seatbelts to snare, airbags to suffocate, and helmets to keep watch for the desperate counterattack. Sometimes (I'm reminded of your alien toaster example) you imply things like "we should outlaw seatbelts, and mandate that car seats must be attached to the front bumper, so people feel scared enough to drive slower", which, although it might reduce the rate of car crashes, is not a workable solution. And what I think you should mean instead is "seat belts certainly save lives, but we should pay attention to whether people might be tempted to drive faster because of the feeling-of-safety they provide, and think about how to mitigate that". > The idea behind that is that erasure coding lulls people into a false > sense of security. That's the important part (if it's even true), and provocative soundbites which omit it are delivering the wrong message. > If K=N=1, or even if K=1 and N=2 (which is the same fault tolerance as > RAID-1), then people understand that they need to constantly monitor > and repair problems as they arise. You know, I'm not sure that's actually true. I have a feeling that any folks who lose data in a Tahoe grid (and I'm not accepting that claim yet: I haven't personally heard many examples of loss, although you talk to more users than I do) would be just as likely to lose data in a RAID array, or in a single-disk server. I.e. to study this properly, I'd want to separate the user population into "careful sysadmins" and "casual end-users", and examine failure rates (and experiences) in the two camps separately. > But if K=3 and N=10, then the beautiful combinatorial math tells you > that your file has lots of "9's" of reliability. The beautiful > combinatorial math lies! Hrm, we've had this argument before and I'm never sure where to go it. Yes, the math in our provisioning/reliability tool describes a somewhat unrealistic model with the usual because-it-makes-the-math-easier assumptions (Poisson processes, independent identically-distributed failures). Should we get rid of it? No, I think it still has value. Should we add some warning stickers that say "human error and non-independent failure modes will probably limit how close you can get to these numbers"? Sure. If people ignore those stickers and believe the fairy-tale math and drive too fast and crash and burn, should we throw out the math? No, I think the tools are still useful to people who understand the limits of the model. > If almost all of the files that have ever been stored on Tahoe-LAFS > have died, this implies one of two things: (ugh, just the way you phrase that claim makes it sound like Tahoe is a fundamentally flawed technology and any file that comes into contact with it catches the plague and falls deathly ill. How about "any potential file loss in Tahoe must come about because of one of the following:" instead?) > 1. The "reliability" of the storage servers must have been below K/N. > I.e. if a file was stored with 3-of-10 encoding, but if each storage > server had a 75% chance of dying, then the file would be *more* likely > to die due to the erasure coding, rather than less likely to die, > because a 75% chance of dying, a.k.a. a 25% chance of staying alive, > is worse than the 30% number of shares required to recover the file. Wait wait, the details are somewhat correct but the conclusion is wrong and the premise is off-base. Yes, k>1 on servers with <50% reliability is worse than k=1 on those same servers: bad servers are bad, relying upon more of them is worse. 3-of-10 on bad servers is worse than 1-of-10. But 1-of-10 is way better than 1-of-2 or 1-of-1. And 3-of-10 is better than 3-of-3. 3-of-10 on good=25% servers has an 80.7% chance of failure, nearly the same as 1-of-1's 75% chance of failure. 2-of-10 on good=25% gets you a 24.4% chance of failure, way better than 75%, and 1-of-10 is down to 5.6% failure[1]. So it's not erasure-coding/replication that's causing the problem, it's the combination of k>1 and horrifically bad servers. I'd rewrite your conclusion to be that the reliability must have been below 50%, not below k/N. I think we've always assumed that servers will have better than 50% reliability (you'd never pay a hosting provider for anything worse than that). Tahoe is a tool for making a great grid out of good servers, not for making a good grid out of lousy servers. If you're stuck with that, use 1-of-N and hope for the best. (If our goal had been to support lots-of-lousy-servers, we'd probably have built something else: implement replication and automatic repair first, then get around to things like mutable files and a web interface. The result would be inefficient on good servers.) > 2. The behavior of storage servers must not have been *independent*. Sure. But let's add some other possibilities, some of which we can improve with code, some of which depend upon sysadmins doing their jobs and grid members honoring commitments to their clients: 3: people got bored, wandered away, took their servers with them 4: hosts got rebooted and servers didn't automatically come back up 5: failed servers weren't replaced 6: files weren't repaired When there's no incentive to keep your server running (which could be as simple as knowing that other people will know when it's been offline), servers tend to go away, and using replication or FEC to mitigate that is expensive (the old problem of not knowing whether the server is coming back or not, therefore needing to treat it as permanent, triggering immediate repair, and eventually the repair bandwidth is so high that you can't use the grid for actual work). We need better OS-integration code to make it easy to get a server to come up on each reboot. On OS-X that means a LaunchAgent or something. On Debian/etc it means an init.d or upstart job. (we have this problem all the time with buildslaves: it's pretty easy to get one running by hand, but the energy barrier between that and having a real every-reboot service is high enough that a lot of folks don't bother). I've been saying forever that it's too hard/slow/inconvenient to get periodic repair to run automatically (cron jobs are soo gross, and suffer from the same energy-barrier problem). And I've been planning (and failing to complete) to move this functionality into the Tahoe client for nearly as long (#643, #483). I really think that having automatic repair of everything reachable from your rootcap(s) is necessary to get close to the enticing durability promise that erasure-coding provides. > My conclusion: if you care about the longevity of your files, forget > about erasure coding and concentrate on monitoring. My conclusion: we can't make serious claims about the benefits/etc of erasure coding until we've eliminated the confounders, by fixing known problems that hurt reliability and then collecting actual data[2]. I think that claiming "erasure coding makes files more fragile" is wrong. Getting good reliability out of a Tahoe grid requires just as much attention and effort as any other storage technology: it can't conjure reliability out of nothing. But I still believe that, for the same effort, you'll get much more durability out of Tahoe than out of simple replication/RAID (when Tahoe has the same level of automation as those other tools, which it doesn't yet). And I agree that building monitoring tools, and especially the automatic repair agent, is just as important as Zooko says. Note that monitoring by itself isn't enough: you need to take action when it's required, either on the small scale (repair) or on the large (replacing servers). But good tools to tell you when action is needed is the first step. yay provocation! :-) cheers, -Brian [1]: 3-of-10 on p=25%, chance of failure is the sum of three cases: Num(good servers)=0: 25%^10 = 5.6% Num(good servers)=1: 10*25%^1*75%^9 = 18.8% Num(good servers)=2: 10*9*25%^2*75%^8 = 56.3% 2-of-10 on p=25%: sum of the first two terms (Num=0,Num=1) [2]: this was in the plans for the "repair agent" at Allmydata before it shut down: that would have been a good place to collect long-term statistics on drive failure, share loss, repair bandwidth, and file decay curves. The idea was to collect that data for a year or two, including failures due to motherboards breaking and upgrades going wrong, and then use it to justify a more deliberate choice of k and N (minimizing cost while still meeting the reliability goals). This project lost a lot of momentum when we lost the centralized place to do that research, and the paycheck to build the supporting tools. _______________________________________________ tahoe-dev mailing list tahoe-dev(a)tahoe-lafs.org http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

[tahoe-dev] Plans for 1.9.2, 1.10, and 1.11 releases
by David-Sarah Hopwood 06 Jul '18

06 Jul '18

Hello Tahoe-LAFS users and hackers! I'm the release manager for the Tahoe 1.9.2 and 1.10 releases. Version 1.9.2 will be a bugfix release focussed mainly on fixing regressions in mutable file support. The mutable publish and retrieve code was changed extensively in 1.9.0 to support MDMF, and this unfortunately introduced a few bugs. Several of these have already been fixed, but the mutable code needs further testing and review to make sure it is as reliable as possible. There is also an important regression in immutable upload to be fixed: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1212#comment:41>. 1.9.2 will be released from the darcs branch that you can obtain using: darcs get --lazy https://tahoe-lafs.org/source/tahoe-lafs/1.9.2 tahoe-1.9.2 or browse at <https://tahoe-lafs.org/trac/tahoe-lafs/browser/1.9.2>. I hope we can release 1.9.2 by the end of May, although that will depend on how soon we fix the remaining open tickets: <https://tahoe-lafs.org/trac/tahoe-lafs/query?status=!closed&milestone=1.9.2> Version 1.10 will include Brian's long-awaited introducer improvements, using Ed25519-signed announcements, that will be the basis for future interesting features like multiple introducers and accounting. It will also include a "move" operation for the web interface and any other additions that are ready in time for the release, but we won't wait very long to release it after 1.9.2. It will be released from git trunk, and we'll probably switch off the darcs-git bridge shortly afterwards. The feature set and release schedule of version 1.11 is less certain, but it could include: - improvements to share placement to meet the servers-of-happiness criterion in more cases; - extending servers-of-happiness to mutable files; - some form of accounting; - a read-only mode for Tahoe gateways; - support for AES+XSalsa encryption (which could in principle be more secure against timing attacks than AES alone); - a 'tahoe mount' command that simplifies mounting a Tahoe filesystem via sshfs; - merging the S3 backend support (and possibly other cloud service support) developed by Least Authority Enterprises; - extending the drop-upload feature to Windows; - merging the patches for Tor and I2P into the main release. -- David-Sarah Hopwood b% _______________________________________________ tahoe-dev mailing list tahoe-dev(a)tahoe-lafs.org https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

[IP] CA State bill could cripple P2P
by David Farber 06 Jul '18

06 Jul '18

------ Forwarded Message From: Dewayne Hendricks <dewayne(a)warpspeed.com> Reply-To: <dewayne(a)warpspeed.com> Date: Wed, 19 Jan 2005 01:38:48 -0800 To: Dewayne-Net Technology List <dewayne-net(a)warpspeed.com> Subject: [Dewayne-Net] CA State bill could cripple P2P State bill could cripple P2P By John Borland <http://news.com.com/State+bill+could+cripple+P2P/2100-1028_3 -5540937.html> Story last modified Tue Jan 18 17:55:00 PST 2005 A bill introduced in California's Legislature last week has raised the possibility of jail time for developers of file-swapping software who don't stop trades of copyrighted movies and songs online. The proposal, introduced by Los Angeles Sen. Kevin Murray, takes direct aim at companies that distribute software such as Kazaa, eDonkey or Morpheus. If passed and signed into law, it could expose file-swapping software developers to fines of up to $2,500 per charge, or a year in jail, if they don't take "reasonable care" in preventing the use of their software to swap copyrighted music or movies--or child pornography. Peer-to-peer software companies and their allies immediately criticized the bill as a danger to technological innovation, and as potentially unconstitutional. "State Sen. Murray did not choose to seek out the facts before introducing misguided legislation that effectively would make criminals out of many companies that bring jobs and economic growth to California," Mike Weiss, CEO of Morpheus parent StreamCast Networks, said in a statement. "This bill is an attack on innovation itself and tax-paying California-based businesses like StreamCast depend on that freedom to innovate." The bill comes as much of the technology world is waiting for the Supreme Court to rule on the legal status of file-swapping technology. Federal courts have twice ruled that peer-to-peer software companies are not legally responsible for the illegal actions of people using their products. Hollywood studios and record companies appealed those decisions to the nation's top court, which is expected to rule on the issue this summer. In the meantime, entertainment companies' push for federal legislation on file-swapping issue has been put temporarily on the back burner. A controversial bill that would have put more legal responsibility on the peer-to-peer developers failed to pass at the end of last year's congressional session. California has taken a lead among states in putting pressure on the file-swapping world. Attorney General Bill Lockyer was a key figure last year in pushing for more state-level legal scrutiny of the companies' actions, and Gov. Arnold Schwarzenegger has sought to ban illegal downloading on any state computers, including those owned by the state university systems. [snip] Archives at: <http://Wireless.Com/Dewayne-Net> Weblog at: <http://weblog.warpspeed.com> ------ End of Forwarded Message ------------------------------------- You are subscribed as eugen(a)leitl.org To manage your subscription, go to http://v2.listbox.com/member/?listname=ip Archives at: http://www.interesting-people.org/archives/interesting-people/ ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net [demime 1.01d removed an attachment of type application/pgp-signature]

1 0

Potential Rumsfield replacement (Robert Gates) was director of
by Peter Langston 06 Jul '18

06 Jul '18

voting company Reply-To: psl(a)acm.org User-Agent: Thunderbird 1.5.0.8 (Windows/20061025) Forwarded-by: David Michael <harp(a)olympus.net> [another reason not to confirm him....dm] Permission to excerpt or reprint granted, with link to http://www.blackboxvoting.org Rumsfield replacement (Robert Gates) was director of voting company by Bev Harris Defense Secretary Donald H. Rumsfield will resign, reportedly to be replaced by former CIA director Robert Gates. Gates was on the board of directors of VoteHere, a strange little company that was the biggest elections industry lobbyist for the Help America Vote Act (HAVA). VoteHere spent more money than ES&S, Diebold, and Sequoia combined to help ram HAVA through. And HAVA, of course, was a bill sponsored by convicted Abramoff pal Bob Ney and K-street lobbyist buddy Steny Hoyer. HAVA put electronic voting on steroids. You can find copies of the VoteHere lobbying forms here: http://sopr.senate.gov/cgi-win/m_opr_viewer.exe?DoFn=0 I can't get them to save to pdf, perhaps you can. Enter search terms in both "registrant" and "client" fields and put in terms "Rhoads" "Livingston" and "Votehere" (one at a time.). Then look at the gravy train while it was in the process of derailing American democracy. I first became acquainted with VoteHere when I met a source, Dan Spillane, who is the wonderful guy that identified the Diebold source code modules for me after I found the Diebold files. He is the person who introduced me, and subsequently everyone else, to the odd role of The Election Center and R. Doug Lewis in the elections industry. Spillane also filled me in on The Livingston Group, VoteHere lobbyists, run by Bob Livingston -- the fellow that Hustler publisher Larry Flynt outed during the Bill Clinton blow job days. Larry Flynt offered a million dollars to anyone who could locate a Republican congressman committing adultery, and out popped peccadilloes by Livingston. Livingston couldn't live that one down, so he resigned his post as House Speaker-Elect and became a lobbyist -- but that's not all! He also launched a group called "Center for Democracy" which was going to "monitor elections." This group also featured several good old boys from the tobacco industry and some mining companies. Former VoteHere test engineer Dan Spillane was looking into all this because he had been fired after he questioned the certification process on a touch-screen system in which he had identified 250 flaws. It was way back in November 2002 that Spillane told me, "The voting machine industry is a house of cards. And the certification and testing process is the bottom card in the house of cards." BUT DON'T RUN OUT OF THE ROOM TO TAKE A SHOWER YET. There's more. VoteHere was a company shilling cryptographic solutions and filled with NSA types (another director was Admiral Bill Owens, another crony of Rummy, Perle and Wolfowitz). For some reason this company claims it was unable to prevent itself from being hacked. In this alleged hack, VoteHere claims that someone stole their source code. Said source code was offered to me in October 2003, an obvious attempt at entrapment which I refused. Nevertheless, VoteHere claimed to the media that its master security experts had supposedly "tracked" the hacker and had identified the hacker as an activist in the election reform community. For some reason, it was decided that I should be investigated in connection with this "hack" of VoteHere -- nevermind that I can't remember how to change the password on my own laptop. Therefore I was interviewed by the Secret Service several times about this. Curiously, they never seemed to ask any questions about VoteHere, only my role in finding the Diebold files and publishing the Diebold memos. This nonsense eventually culminated in a gag order and a letter from the U.S. Attorney to appear in front of a federal grand jury with information on all the visitors to the Black Box Voting Web site. (As if they couldn't get that in less dramatic ways in post-Patriot Act America). Attorney Lowell Finley (now with http://www.VoterAction.org ) went to bat for me on this. A reporter named George Howland from the Seattle Weekly also got wind of it. When it hit the press, and with Lowell Finley's help, their harassment of me stopped. VoteHere never sold any voting machines that I can find, but apparently did set up some deals to embed its cryptography into some voting systems. We found memos in the Diebold trash about VoteHere's crypto-crap, and Maryland Director of Elections Linda Lamone shows up in VoteHere-related letters. Sequoia Voting Systems signed an agreement with VoteHere, but it's not clear to me whether they ever did anything about it. Robert Gates stepped away from VoteHere shortly before he showed up in Chapter 8 of my book, Black Box Voting, in a short bit about the VoteHere company history. You can read that here: http://www.blackboxvoting.org/bbv_chapter-8.pdf I don't know about you, but I'd rather use a paper, pencil, and count by hand at the polling place than have former CIA director Robert Gates fooling around with my vote. But that's just me. -- Bev Harris Founder, Black Box Voting P.S. Since the HBO special, I have plenty of moral support, but even after the Secret Service interviews and all the rest of the nonsense my husband and I have had to put up with, there are others who have had it rougher. I'd like you to take a moment to visit this Web site -- not affiliated with Black Box Voting -- to meet one of the heroic citizens in this movement who has faced the most brutal retaliation of all: Stephen Heller. If you saw the HBO film "Hacking Democracy" you may remember a scene where I am chastising Diebold for lying about correcting problems with its product. I refer to "Release Notes." Those notes came from a source. Stephen Heller is being threatened with up to five years in prison for allegedly leaking me those documents. Kevin Shelley then decertified Diebold, and recommended criminal prosecution of Diebold. Diebold was never prosecuted, but Stephen Heller is being prosecuted RIGHT NOW. I hope you will donate to his defense. If not for citizens like him, where would your vote be now? To contribute to Stephen Heller's defense fund: http://www.hellerlegaldefensefund.com/ * * * * * Black Box Voting is a nonprofit, nonpartisan 501c(3) elections watchdog group supported entirely by citizen donations. We refuse funds from any vendor or vested interest. To support Black Box Voting: click to http://www.blackboxvoting.org/donate.html or send to: Black Box Voting 330 SW 43rd St Suite K PMB 547 Renton WA 98055 * * * * * ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

1 0

[liberationtech] "Pirate" wifi on NYC subway: now open-source!
by Nathan of Guardian 06 Jul '18

06 Jul '18

Some of you may have heard about a temporary wifi network that was run on New York Subway's L train last month. If not, you can catch up here: http://laughingsquid.com/l-train-notwork-pirate-wi-fi-for-new-york-city-sub… What made this effort unique is that this "mesh" didn't try to allow people to get online to the plain old Internet, but it instead conceived of entire new apps and content distribution mechanisms that were appropriate to the context of people crammed in a subway car on their smartphones. They also considered issues of how to make people aware of this network, and how to physically put it in place each morning they deployed it, including how to not look suspicious to the very serious NYC police anti-terrorist surveillance system. One of the primary developers of the project is Sean McIntyre, a student of mine at NYU ITP. While I cannot take credit for the idea, I did give him permission to use this work as his semester project, and I can definitely say he will easily be getting a passing grade! I have cc'd him on this email if you are interested in getting touch about deploying your own "notwork". *** Open-source project: https://github.com/boxysean/LTrainNotwork Press release: http://wemakecoolsh.it/#2300081/L-Train-Notwork-Press-Release Video on the project here: http://vimeo.com/32149926 Tech doc describing the hardware and configuration: https://github.com/boxysean/LTrainNotwork/blob/master/techdoc.md *** and if you are in the New York area, here is this from Sean: "We're having a meeting this Thursday, Dec 15 from 7-10 PM off the Morgan L in Brooklyn to chat about the project. The goal of the meeting will be to release this idea and help develop ideas in the same realm. Email me off list if you are interested attending." Best, Nathan _______________________________________________ liberationtech mailing list liberationtech(a)lists.stanford.edu Should you need to change your subscription options, please go to: https://mailman.stanford.edu/mailman/listinfo/liberationtech If you would like to receive a daily digest, click "yes" (once you click above) next to "would you like to receive list mail batched in a daily digest?" You will need the user name and password you receive from the list moderator in monthly reminders. Should you need immediate assistance, please contact the list moderator. Please don't forget to follow us on http://twitter.com/#!/Liberationtech ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

Re: [tahoe-dev] erasure coding makes files more fragile, not less
by Brian Warner 06 Jul '18

06 Jul '18

On 3/27/12 12:06 PM, Zooko Wilcox-O'Hearn wrote: > In fact, I think approximately 90% of all files that have ever been > stored on a Tahoe-LAFS grid have died. (That's excluding all of the > files of all of the customers of allmydata.com, which went out of > business.) That.. is a pretty broad and potentially disingenuous statement, and feels unsupported. I think I know what you mean, but it's a bit like saying "everybody dies" or "all programs crash eventually": maybe true, but kinda useless, and kinda deceptive, or at least distracting. [BTW folks: Zooko and I talk about this stuff all the time, and we know each other's opinions pretty well, so please don't misinterpret my words as indicating anger or annoyance. We're old pals, and this is a well-worn comfortable argument.] The metric that I'd find useful is what percentage of files *that people actively tried to keep around* were lost. Tahoe is a system for multiplying the durability of your servers, but it's not magic, and if you start with lousy unmaintained servers, then you aren't likely to have good results. > I came up with this provocative slogan (I know Brian loves my > provocative slogans): "erasure coding makes files more fragile, not > less". Ah, you do know how to provoke me :). That's like saying "seat belts, airbags, and helmets kill people", invoking haunting images of demonic safety gear stalking the last remaining humans through the forest, seatbelts to snare, airbags to suffocate, and helmets to keep watch for the desperate counterattack. Sometimes (I'm reminded of your alien toaster example) you imply things like "we should outlaw seatbelts, and mandate that car seats must be attached to the front bumper, so people feel scared enough to drive slower", which, although it might reduce the rate of car crashes, is not a workable solution. And what I think you should mean instead is "seat belts certainly save lives, but we should pay attention to whether people might be tempted to drive faster because of the feeling-of-safety they provide, and think about how to mitigate that". > The idea behind that is that erasure coding lulls people into a false > sense of security. That's the important part (if it's even true), and provocative soundbites which omit it are delivering the wrong message. > If K=N=1, or even if K=1 and N=2 (which is the same fault tolerance as > RAID-1), then people understand that they need to constantly monitor > and repair problems as they arise. You know, I'm not sure that's actually true. I have a feeling that any folks who lose data in a Tahoe grid (and I'm not accepting that claim yet: I haven't personally heard many examples of loss, although you talk to more users than I do) would be just as likely to lose data in a RAID array, or in a single-disk server. I.e. to study this properly, I'd want to separate the user population into "careful sysadmins" and "casual end-users", and examine failure rates (and experiences) in the two camps separately. > But if K=3 and N=10, then the beautiful combinatorial math tells you > that your file has lots of "9's" of reliability. The beautiful > combinatorial math lies! Hrm, we've had this argument before and I'm never sure where to go it. Yes, the math in our provisioning/reliability tool describes a somewhat unrealistic model with the usual because-it-makes-the-math-easier assumptions (Poisson processes, independent identically-distributed failures). Should we get rid of it? No, I think it still has value. Should we add some warning stickers that say "human error and non-independent failure modes will probably limit how close you can get to these numbers"? Sure. If people ignore those stickers and believe the fairy-tale math and drive too fast and crash and burn, should we throw out the math? No, I think the tools are still useful to people who understand the limits of the model. > If almost all of the files that have ever been stored on Tahoe-LAFS > have died, this implies one of two things: (ugh, just the way you phrase that claim makes it sound like Tahoe is a fundamentally flawed technology and any file that comes into contact with it catches the plague and falls deathly ill. How about "any potential file loss in Tahoe must come about because of one of the following:" instead?) > 1. The "reliability" of the storage servers must have been below K/N. > I.e. if a file was stored with 3-of-10 encoding, but if each storage > server had a 75% chance of dying, then the file would be *more* likely > to die due to the erasure coding, rather than less likely to die, > because a 75% chance of dying, a.k.a. a 25% chance of staying alive, > is worse than the 30% number of shares required to recover the file. Wait wait, the details are somewhat correct but the conclusion is wrong and the premise is off-base. Yes, k>1 on servers with <50% reliability is worse than k=1 on those same servers: bad servers are bad, relying upon more of them is worse. 3-of-10 on bad servers is worse than 1-of-10. But 1-of-10 is way better than 1-of-2 or 1-of-1. And 3-of-10 is better than 3-of-3. 3-of-10 on good=25% servers has an 80.7% chance of failure, nearly the same as 1-of-1's 75% chance of failure. 2-of-10 on good=25% gets you a 24.4% chance of failure, way better than 75%, and 1-of-10 is down to 5.6% failure[1]. So it's not erasure-coding/replication that's causing the problem, it's the combination of k>1 and horrifically bad servers. I'd rewrite your conclusion to be that the reliability must have been below 50%, not below k/N. I think we've always assumed that servers will have better than 50% reliability (you'd never pay a hosting provider for anything worse than that). Tahoe is a tool for making a great grid out of good servers, not for making a good grid out of lousy servers. If you're stuck with that, use 1-of-N and hope for the best. (If our goal had been to support lots-of-lousy-servers, we'd probably have built something else: implement replication and automatic repair first, then get around to things like mutable files and a web interface. The result would be inefficient on good servers.) > 2. The behavior of storage servers must not have been *independent*. Sure. But let's add some other possibilities, some of which we can improve with code, some of which depend upon sysadmins doing their jobs and grid members honoring commitments to their clients: 3: people got bored, wandered away, took their servers with them 4: hosts got rebooted and servers didn't automatically come back up 5: failed servers weren't replaced 6: files weren't repaired When there's no incentive to keep your server running (which could be as simple as knowing that other people will know when it's been offline), servers tend to go away, and using replication or FEC to mitigate that is expensive (the old problem of not knowing whether the server is coming back or not, therefore needing to treat it as permanent, triggering immediate repair, and eventually the repair bandwidth is so high that you can't use the grid for actual work). We need better OS-integration code to make it easy to get a server to come up on each reboot. On OS-X that means a LaunchAgent or something. On Debian/etc it means an init.d or upstart job. (we have this problem all the time with buildslaves: it's pretty easy to get one running by hand, but the energy barrier between that and having a real every-reboot service is high enough that a lot of folks don't bother). I've been saying forever that it's too hard/slow/inconvenient to get periodic repair to run automatically (cron jobs are soo gross, and suffer from the same energy-barrier problem). And I've been planning (and failing to complete) to move this functionality into the Tahoe client for nearly as long (#643, #483). I really think that having automatic repair of everything reachable from your rootcap(s) is necessary to get close to the enticing durability promise that erasure-coding provides. > My conclusion: if you care about the longevity of your files, forget > about erasure coding and concentrate on monitoring. My conclusion: we can't make serious claims about the benefits/etc of erasure coding until we've eliminated the confounders, by fixing known problems that hurt reliability and then collecting actual data[2]. I think that claiming "erasure coding makes files more fragile" is wrong. Getting good reliability out of a Tahoe grid requires just as much attention and effort as any other storage technology: it can't conjure reliability out of nothing. But I still believe that, for the same effort, you'll get much more durability out of Tahoe than out of simple replication/RAID (when Tahoe has the same level of automation as those other tools, which it doesn't yet). And I agree that building monitoring tools, and especially the automatic repair agent, is just as important as Zooko says. Note that monitoring by itself isn't enough: you need to take action when it's required, either on the small scale (repair) or on the large (replacing servers). But good tools to tell you when action is needed is the first step. yay provocation! :-) cheers, -Brian [1]: 3-of-10 on p=25%, chance of failure is the sum of three cases: Num(good servers)=0: 25%^10 = 5.6% Num(good servers)=1: 10*25%^1*75%^9 = 18.8% Num(good servers)=2: 10*9*25%^2*75%^8 = 56.3% 2-of-10 on p=25%: sum of the first two terms (Num=0,Num=1) [2]: this was in the plans for the "repair agent" at Allmydata before it shut down: that would have been a good place to collect long-term statistics on drive failure, share loss, repair bandwidth, and file decay curves. The idea was to collect that data for a year or two, including failures due to motherboards breaking and upgrades going wrong, and then use it to justify a more deliberate choice of k and N (minimizing cost while still meeting the reliability goals). This project lost a lot of momentum when we lost the centralized place to do that research, and the paycheck to build the supporting tools. _______________________________________________ tahoe-dev mailing list tahoe-dev(a)tahoe-lafs.org http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

[tahoe-dev] Plans for 1.9.2, 1.10, and 1.11 releases
by David-Sarah Hopwood 06 Jul '18

06 Jul '18

Hello Tahoe-LAFS users and hackers! I'm the release manager for the Tahoe 1.9.2 and 1.10 releases. Version 1.9.2 will be a bugfix release focussed mainly on fixing regressions in mutable file support. The mutable publish and retrieve code was changed extensively in 1.9.0 to support MDMF, and this unfortunately introduced a few bugs. Several of these have already been fixed, but the mutable code needs further testing and review to make sure it is as reliable as possible. There is also an important regression in immutable upload to be fixed: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1212#comment:41>. 1.9.2 will be released from the darcs branch that you can obtain using: darcs get --lazy https://tahoe-lafs.org/source/tahoe-lafs/1.9.2 tahoe-1.9.2 or browse at <https://tahoe-lafs.org/trac/tahoe-lafs/browser/1.9.2>. I hope we can release 1.9.2 by the end of May, although that will depend on how soon we fix the remaining open tickets: <https://tahoe-lafs.org/trac/tahoe-lafs/query?status=!closed&milestone=1.9.2> Version 1.10 will include Brian's long-awaited introducer improvements, using Ed25519-signed announcements, that will be the basis for future interesting features like multiple introducers and accounting. It will also include a "move" operation for the web interface and any other additions that are ready in time for the release, but we won't wait very long to release it after 1.9.2. It will be released from git trunk, and we'll probably switch off the darcs-git bridge shortly afterwards. The feature set and release schedule of version 1.11 is less certain, but it could include: - improvements to share placement to meet the servers-of-happiness criterion in more cases; - extending servers-of-happiness to mutable files; - some form of accounting; - a read-only mode for Tahoe gateways; - support for AES+XSalsa encryption (which could in principle be more secure against timing attacks than AES alone); - a 'tahoe mount' command that simplifies mounting a Tahoe filesystem via sshfs; - merging the S3 backend support (and possibly other cloud service support) developed by Least Authority Enterprises; - extending the drop-upload feature to Windows; - merging the patches for Tor and I2P into the main release. -- David-Sarah Hopwood b% _______________________________________________ tahoe-dev mailing list tahoe-dev(a)tahoe-lafs.org https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

Re: [tor-relays] Call for discussion: turning funding into more exit relays
by Andrew Lewman 06 Jul '18

06 Jul '18

On Thu, 26 Jul 2012 16:05:53 +0000 kupo(a)damnfbi.tk wrote: > We should probably talk further then since I'm _in_ Iceland atm and > would also like to see a high capacity node here. > May I ask for your reasoning though? A lot of people on both sides of > the pond have believed that IMMI <https://en.wikipedia.org/wiki/IMMI> > has been passed here already when it has in fact not (yet). I'm in > touch with those trying to pass it and it comes up for major review > in september. Have you tried talking to DataCell > <http://www.datacell.com/>? I talked to Datacell roughly a year ago. They were fine with an exit relay, but at the time were distracted by suing Visa. The only issue was pure cost. Traffic leaving Iceland costs a lot. I wasn't prepared to spend ISK300,000 per month for a 100 mbps exit relay. Maybe times have changed and traffic from Iceland is not so expensive anymore. -- Andrew http://tpo.is/contact pgp 0x6B4D6475 _______________________________________________ tor-relays mailing list tor-relays(a)lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

Re: [liberationtech] Modern FIDONET for net disable countries?
by Jacob Appelbaum 06 Jul '18

06 Jul '18

Eric S Johnson: > For the record, Burma/Myanmar (MM) has very little cybercensorship > now (the previous censorship started loosening up in about August > 2011 and was basicallybnot entirely, but mostlybdropped by the end of > October 2011). > Hi, I've just returned from Burma in the last month. There is total surveillance in Burma on the commercial YTP ISP. They censor plenty of sites and their BlueCoat proxy devices fail to deliver content for unblocked sites often. The other networks seem rather under surveillance as well and they also have censorship. We (OONI) have data from my trip there and it includes all of the major networks. We'll write it up and publish all of it soon. > Ibve been visiting MM since the mid-aughts and have never encountered > FidoNet there. Havenbt seen it since Africa, mid-nineties. But that > doesnbt mean it wasnbt/isnbt there. > I haven't seen FidoNet but I did see lots of WiMax, VSAT, GSM/CDMA and even some discussion of X.25, etc. > > > (Vietnam still has a measurable amount of online censorship, but itbs > not nearly as heavy-handed as Chinabs or Iranbs. Itbs more like > Ethiopiabs. > Ethiopia has extremely sophisticated censorship: https://blog.torproject.org/blog/ethiopia-introduces-deep-packet-inspection https://blog.torproject.org/blog/update-censorship-ethiopia > Some of Cambodiabs ~30 ISPs censor a half-dozen sites, but thatbs > hardly serious (the largest, Vietnamese-controlled, ISP doesnbt!).) > The serious problem is the infrastructure even if today it is only used on a few sites. All the best, Jake -- Unsubscribe, change to digest, or change password at: https://mailman.stanford.edu/mailman/listinfo/liberationtech ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE

1 0

[osint] Missing uniforms revive 9/11 fears in Canada
by MARQUEZ Thierry 06 Jul '18

06 Jul '18

http://www.hindustantimes.com/news/181_1141489,00050001.htm Missing uniforms revive 9/11 fears in Canada Gurmukh Singh Vancouver, December 7 The 9/11 has cast its shadow over Canada as hundreds of uniforms, including badges, worn by screeners at 89 airports across Canada have gone missing, setting alarm bells ringing over air travel safety in North America. Numbering 1,127, the uniforms and badges have been stolen or gone missing over a period of nine months. Canadian and American aviation authorities fear the uniforms and badges might be used by terrorists to gain entry into airports and hijack planes. Hundreds of planes fly into the US every day from Canadian destinations. Transport minister Jean Lapierre sought a report from the Canadian Air Transport Security Authority (CATSA), which looks after security at the nation's airports, on Saturday. The minister was booed in Parliament when he tried to assure the House that all possible steps were being taken to check the misuse of the uniforms. Airports had been alerted and the private firms conducting screening at airports asked to double-check the ID of all employees. [Non-text portions of this message have been removed] ------------------------ Yahoo! Groups Sponsor --------------------~--> $4.98 domain names from Yahoo!. Register anything. http://us.click.yahoo.com/Q7_YsB/neXJAA/yQLSAA/TySplB/TM --------------------------------------------------------------------~-> -------------------------- Want to discuss this topic? Head on over to our discussion list, discuss-osint(a)yahoogroups.com. -------------------------- Brooks Isoldi, editor bisoldi(a)intellnet.org http://www.intellnet.org Post message: osint(a)yahoogroups.com Subscribe: osint-subscribe(a)yahoogroups.com Unsubscribe: osint-unsubscribe(a)yahoogroups.com *** FAIR USE NOTICE. This message contains copyrighted material whose use has not been specifically authorized by the copyright owner. OSINT, as a part of The Intelligence Network, is making it available without profit to OSINT YahooGroups members who have expressed a prior interest in receiving the included information in their efforts to advance the understanding of intelligence and law enforcement organizations, their activities, methods, techniques, human rights, civil liberties, social justice and other intelligence related issues, for non-profit research and educational purposes only. We believe that this constitutes a 'fair use' of the copyrighted material as provided for in section 107 of the U.S. Copyright Law. If you wish to use this copyrighted material for purposes of your own that go beyond 'fair use,' you must obtain permission from the copyright owner. For more information go to: http://www.law.cornell.edu/uscode/17/107.shtml Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/osint/ <*> To unsubscribe from this group, send an email to: osint-unsubscribe(a)yahoogroups.com <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/ --- end forwarded text -- ----------------- R. A. Hettinga <mailto: rah(a)ibuc.com> The Internet Bearer Underwriting Corporation <http://www.ibuc.com/> 44 Farquhar Street, Boston, MA 02131 USA "... however it may deserve respect for its usefulness and antiquity, [predicting the end of the world] has not been found agreeable to experience." -- Edward Gibbon, 'Decline and Fall of the Roman Empire'

1 0