- cypherpunks - lists.cpunks.org

Sorry for calling you old Jim
by rooty 06 Mar '20

06 Mar '20

Hi Jim I wanted to apologise for calling you old. At 81 years young Viagra has given new life - Your friend rooty

1 0

Trump tells Republicans he wont' extend surveillance law without significant reform.
by jim bell 06 Mar '20

06 Mar '20

https://www.nationalreview.com/news/trump-tells-republicans-he-wont-extend-… Good if it happens, but.... Jim Bell

1 0

"Earn it" act is sneak attack on encryption.
by jim bell 06 Mar '20

06 Mar '20

https://www.wired.com/story/earn-it-act-sneak-attack-on-encryption/

1 0

Mali GPU open source drivers
by Zenaan Harkness 06 Mar '20

06 Mar '20

Noting this: https://www.collabora.com/news-and-blog/blog/2019/12/20/opening-up-mali-t72… brings to mind the following thoughts: - open source gets to parity, it just take s a while when reverse engineering is required - once parity is met, superiority in performance, cross platform support and stability, are all close on the heels - once this is achieved, closed source drivers are no longer competitive - at this point, corporations have various incentives/motives to use the floss drivers, rather than the old, deficient proprietary drivers (no more development/test/etc cost per se) and finally: - so those who cock blocked release of specs (thereby requiring said reverse engineering) ought be publicly named and shamed, for the great cost they effectively imposed upon the community of floss developers

1 0

"There are FIVE variants of coronavirus. Only ONE showed up in China..." and...
by Razer 06 Mar '20

06 Mar '20

Wait for it... "There are FIVE variants of coronavirus. Only ONE showed up in China..." and the deadliest variety seems to have landed in... wait for it... Iran. Pepe Escobar: "I had already referenced the crucial Global Times report - but this brings it to a whole new level, incorporating Chinese, Japanese and Taiwanese scientific analysis. The killer paragraphs are these: "The geographical location with the greatest diversity of virus strains must be the original source because a single strain cannot emerge from nothing. He demonstrated that only the US has all the five known strains of the virus (while Wuhan and most of China have only one, as do Taiwan and South Korea, Thailand and Vietnam, Singapore, and England, Belgium and Germany), constituting a thesis that the haplotypes in other nations may have originated in the US. Korea and Taiwan have a different haplotype of the virus than China, perhaps more infective but much less deadly, which would account for a death rate only 1/3 that of China. Neither Iran nor Italy were included in the above tests, but both countries have now deciphered the locally prevalent genome and have declared them of different varieties from those in China, which means they did not originate in China but were of necessity introduced from another source. It is worth noting that the variety in Italy has approximately the same fatality rate as that of China, three times as great as other nations, while the haplotype in Iran appears to be the deadliest with a fatality rate of between 10% and 25%. (7) (8) (9)" So to recap: there are FIVE variants of coronavirus. Only ONE showed up in China - as well as in South Korea and Taiwan. Less deadly variant in S. Korea than in China. Variants in Italy and Iran DID NOT originate in China. Deadliest variant is in Iran. If all that is confirmed beyond a shadow of a doubt, imagine the repercussions." https://www.globalresearch.ca/china-coronavirus-shocking-update/5705196 Rr

2 1

Egypt missiles: from provoking all its neighbours, soon to directly provoke Iran - what could possibly go wrong?
by Zig the N.g 06 Mar '20

06 Mar '20

Did I say Egypt? I meant the Chinese ^B^B actually Israel. Israel kitting up with (US Pegasus) refueling air tanker jets, so as to be able to actively drop bombs on Iran, rather than just "all immediate neighbours". Antagonising all your neighbours with literal bomb drops and assassinations - what could possibly go wrong? US to Sell Pegasus Tanker Jets to Israel, Allowing Its Fighters to Reach Iran's Territory https://sputniknews.com/military/202003051078486176-us-to-sell-pegasus-tank…

1 0

Cryptocurrency: Is Bitcoin Mixing a Crime?
by grarpamp 05 Mar '20

05 Mar '20

> https://news.bitcoin.com/criminalize-onchain-privacy-mixer/ https://news.bitcoin.com/how-to-mix-your-bitcoins-using-coinjoin/ Privacy is not crime, it's a fundamental innate action of human nature, thus by necessity and nature, it's rightly translating itself from the physical world of human interaction, to the digital... wherever humans go, so too does privacy keeping efforts among them. And as any search for distributed non-custodial mix join shuffle fusion DEX market privacy coin tech all running over anonymous overly networks would show, the first generation centralized custodial mixing swap services are being rapidly abandoned, as they should be.

2 1

Re: In Surprise Move, Pope Francis Shills For Renewed Connection With God
by Cecilia Tanaka 05 Mar '20

05 Mar '20

Juan really, really Loves Jesus. Tee hee. On Sun, Feb 23, 2020 at 08:57:53PM +1100, Zig the N.g wrote: > In a surprise move today, the beloved [by Juan] Pope Francis made an international call for renewed devotion to God and prayer to Christ. > > The Pope has repeatedly ... hey wait, it looks like there's a new sin, literary: > > Pope Francis The Taxman > https://www.zerohedge.com/political/pope-francis-taxman > https://antoniusaquinas.com/2020/02/18/pope-francis-the-taxman/ > > On the heels of calling for a Marxist economic conference this coming March (“Economy of Francesco”), Jorge Bergoglio (a.k.a. “Pope Francis”) has once again opined on financial matters. The purported head of the Catholic Church has now designated “tax cuts” as sinful behavior on a par, apparently, with stealing, lying, and adultery: > > Today’s structures of sin include repeated tax cuts for the richest people, often justified in the name of investment and development. > > Bergoglio did not mention what category of sin advocacy for tax cuts falls under – venial or mortal. > > Maybe the details of how such a policy ranks in offending Divine Justice will be hammered out at the upcoming Economy of Francesco Commie confab! > > In Bergoglio’s collectivist mind, those who try and keep their wealth from the ravenous demands of the State are somehow denying the poor their just due: > > Every year hundreds of billions of dollars, which should be paid in taxes to fund health care and education, accumulate in tax haven accounts, thus impeding the possibility of the dignified and sustained development of all social agents. > ... > > > Glad we cleared that up. > > Ahem ... in a surprise move today, the beloved [by Juan] Pope Francis made an international declaration that repeated tax cuts for the rich is a sin, either mortal or venal (as yet undetermined - Not Sure (TM)(C)(R)). > > This literal papal declaration is considered by many as an indirect declaration of support for Bernie Sanders' presidency run. > > Undisclosed sources from the Vatican assured Bernie campaign staffers that Indulgences may be sold until the conclusion of the primaries, with all proceeds going to the Bernie campaign.

2 2

Quantifying Independently Reproducible Machine Learning
by coderman 05 Mar '20

05 Mar '20

https://thegradient.pub/independently-reproducible-machine-learning/ Peer review has been an integral part of scientific research for [more than 300 years](https://blogs.scientificamerican.com/information-culture/the-birth-o…. But even before peer review was introduced, reproducibility was a primary component of the scientific method. One of the first reproducible experiments was presented by Jabir Ibn Haiyan in 800 CE. In the past few decades, many domains have encountered high profile cases of non-reproducible results. The [American Psychological Association has struggled with authors failing to make data available](https://psycnet.apa.org/doi/10.1037/0003-066X.61.7.726). A 2011 study found that only [6% of medical studies could be fully reproduced](https://doi.org/10.1038%2Fnrd3439-c1). In 2016, a survey of researchers from many disciplines found that most had [failed to reproduce one of their previous papers](https://doi.org/10.1038%2F533452a). Now, we hear warnings that Artificial Intelligence (AI) and Machine Learning (ML) [face their own reproducibility crises](https://science.sciencemag.org/content/359/6377/725). This leads us to ask: is it true? It would seem hard to believe, as ML permeates every smart-device and intervenes evermore in our daily lives. From helpful hints on how to [act like a polite human over email](https://ai.googleblog.com/2018/05/smart-compose-using-neural-network…, to Elon Musk’s [promise](https://www.wired.com/story/elon-musk-tesla-full-self-driving-2019… of self-driving cars next year, it seems like machine learning is indeed reproducible. How reproducible is the latest ML research, and can we begin to quantify what impacts its reproducibility? This question served as motivation for my [NeurIPS 2019 paper](https://arxiv.org/abs/1909.06674). Based on a combination of masochism and stubbornness, over the past eight years I have attempted to implement various ML algorithms from scratch. This has resulted in a ML library called [JSAT](https://github.com/EdwardRaff/JSAT). My investigation in reproducible ML has also relied on personal notes and records hosted on [Mendeley](https://www.mendeley.com/) and Github. With these data, and clearly no instinct for preserving my own sanity, I set out to quantify and verify reproducibility! As I soon learned, I would be engaging in [meta-science](https://en.wikipedia.org/wiki/Metascience), the study of science itself. What is Reproducible Machine Learning? One does not simply follow the description in the paper. https://abstrusegoose.com/588 Before we dive in, it is important to define what we mean by reproducible. Ideally, full reproducibility means that simply reading a scientific paper should give you all the information you need to 1) set up the same experiments, 2) follow the same approach, and then 3) obtain similar results. If we can get all the way to step 3 based solely on information present in the paper, we might call that independent reproducibility. In this example, our result is reproducible because we are able to get the same result, and independent because we have done so in an effort completely independent of the original publication. But as our friend from the comic above might tell us, simply following the content of the paper isn’t always sufficient. If we can’t get to step 3 by using only the information in the paper (or from cited prior work), we would determine that the paper is not independently reproducible. Some may wonder, why make this distinction between reproducibility and independent reproducibility? Almost all of AI and ML research is based on computer code. We don’t require the burden of expensive and labor-intensive chemical synthesis, waiting for bacteria in a petri dish to mature, or pesky human trials. It should be easy to simply get code from the authors, run that on the same data, and get the same results! If you have never had to read a researcher's code before... you are doing pretty OK in life. Good job. http://phdcomics.com/comics/archive.php?comicid=1689 Our aversion to using or asking for the authors code is more than fear of working with undocumented research-grade code. [Chris Drummond](https://www.researchgate.net/profile/Chris_Drummond) has [described the approach](http://cogprints.org/7691/7/ICMLws09.pdf) of using an author’s code as replicability, and made a very salient argument that replication is desirable, but not sufficient for good science. A paper is supposed to be the scientific distillation of the work, representing what we have learned and now understand to enable these new results. If we can’t reproduce the results of a paper without the authors code, it may suggest that the paper itself didn’t successfully capture the important scientific contributions. This is before we consider the possibility that there may be bugs in the code that actually benefit the results, or any number of other possible discrepancies between code and paper. Another [great example](http://proceedings.mlr.press/v97/bouthillier19a/bouthillier19a.pdf) from ICML this past year showed that even if we can replicate the results of a paper, slightly altering the experimental setup could have dramatically different results. For these reasons, we don’t want to consider the authors code, as this could be a source of bias. We want to focus on the question of reproducibility, without wading into the murky waters of replication. What Makes a ML Paper Reproducible? Feature Important My Reaction Hyperparameters ✅ 👍 Easy to Read ✅ 👍 Equations per Page ✅ 🤔 Empirical vs Rigor ✅ 🤨 Pseudo Code ✅ 🤯 Replies to Questions ✅ 🤷 Include Toy Problems ❌ 😭 Year Published ❌ 😌 Open Source Code ❌ 😱Some of the features that were/were not related with reproducibility, that I found the most interesting. I reviewed every paper I have attempted to implement up to 2017, and filtered out papers based on two criteria: if the attempt would be biased by having looked at released source code, or if there was a personal relationship with the authors. For each paper, I recorded as much information as I could to create a quantifiable set of features. Some were completely objective (how many authors where on the paper), while others highly subjective (does the paper look intimidating)? The goal of this analysis was to get as much information as possible about things that might impact a paper’s reproducibility. This left me with 255 attempted papers, and 162 successful reproductions. Each paper was distilled to a set of 26 features, and statistical testing was done to determine which were significant. In the table to the right I've put what I think are the most interest and important results, along with my initial reactions. Some of the results where unsurprising. For example, the number of authors shouldn’t have any particular importance to a paper’s reproducibility, and it did not have a significant relationship. Hyperparameters are the knobs we can adjust to change an algorithms behavior, but are not learned by the algorithm itself. Instead, we humans must set their values (or devise a clever way to pick them). Whether or not a paper detailed the hyperparameters used was found to be significant, and we can intuit why. If you don’t tell the reader what the settings where, the reader has to guess. That takes work, time, and is error prone! So, some of our results have given credence to the ideas the community has already been pursuing in order to make papers more reproducible. What is important is that we can now quantify why these are good things to be pursuing. Other findings follow basic logic, such as the finding that papers that are easier to read are easier to reproduce, likely because they are easier to understand. I implore you to read the paper for a deeper discussion, but there are a few additional results that I think are particularly interesting; either because they challenge our assumptions about what we “know” a good paper is or lead to some surprising conclusions. All of these results are nuanced more than I can unpack in this article, but are worth mentioning if for nothing else but to stimulate a deeper conversation, and hopefully spur further research to answer these questions. Finding 1: Having fewer equations per page makes a paper more reproducible. Math is like catnip for reviewers. They just can't help themselves. https://xkcd.com/982/ It appears to be the case because the most readable papers use the fewest equations. We often see papers that have many equations and derivations listed, for any number of reasons. It appears that a careful and judicious use of equations makes things easier to read, primarily because you can use math selectively to communicate more effectively. This result clashes with the incentive structure of getting a paper published. On more than one occasion, reviewers have asked me to include more math in a paper. It may be that the math itself makes a paper more scientific or grounded in objectivity. While more specification may seem to be better, it is not synonymous with reproducibility. This is a cultural issue we need to address as a community. Finding 2: Empirical papers may be more reproducible than theory-oriented papers. There is considerable debate within the community about where and how much rigor needs to be normalized in the community. This is done under the guide that as a community, our focus should be on getting the best results for a given bench mark. Yet in optimizing for bench marks, we risk losing our understanding of what is actually happening and why these methods work. The inclusion of theoretical work and formal proofs do not cover all aspects of what might be meant by the term rigor. Given the common belief that elaborate mathematical proofs ensure a better understanding of a given method, it is interesting to see that greater mathematical specification isn’t necessarily making research easier to reproduce. The important point here is that papers containing a mix of theory and empirical emphasis have the same overall reproduction rates as purely empirical papers. An empirical bent can be helpful from the reproducibility perspective, but [could also hamper progress](https://openreview.net/pdf?id=rJWF0Fywf) by creating perverse incentives and unintended side effects. Finding 3: Sharing code is not a panacea We have already touched upon the idea that reproduction via released code is not the same thing as reproduction done independently. Is this a difference without distinction? It is not! My results indicate that the open sourcing of code is at best a weak indicator of reproducibility. As conferences begin to more strongly encourage code submission and examination as part of the review process, I believe this is a crucial point. As a community, we need to understand what our goals are with such efforts and what we are actually accomplishing. Careful thought and consideration should go into this distinction if we ever make code submission mandatory, and the guidance we give reviewers to evaluate such code. I find this result particularly noteworthy in terms of other people's reactions. While presenting at NeurIPS, many people commented on it. Half of them were certain that releasing would have been correlated with reproducibility , and the other half felt it obvious that the non-relationship would emerge. This strong contrast in opinions that were all deeply held is a perfect example of why I wanted to do this study. We don't really know until we sit down and measure it! Finding 4: Having detailed pseudo code is just as reproducible as having no pseudo code. Step-Code: Concise, but requires context from other parts of the paper to decipher.Standard-Code: Relatively detailed, can be almost self contained. Usually mathematical notation.Code-Like: Almost always self contained, easy to convert to code. This finding challenged my assumptions of what constituted a good paper, but made more sense as I thought about the results. Somewhere in the paper, the process must be described. A computer scientist by training, I always preferred a type of description called pseudo code. But pseudo code can take many different forms. I categorized the papers it into four groups: None, Step-Code, Standard-Code, and Code-Like. I have some representative samples of these to the right from some widely reproduced papers that may or may not have been in this study! I was shocked when Standard-Code and Code-Like has roughly equal reproduction rates, and floored to discover that None at all was just as good! However, cogent writing is just as effective in communicating a process. What was not as effective was so-called Step code, where a bulleted list of steps is listed, with each step referring to another section of the paper. Step code actually makes reading and understanding the paper harder, as the reader must now jump back and forth between different sections, rather than following a single sequential flow. Finding 5: Creating simplified example problems do not appear to help with reproducibility. This was another surprising result that I am still coming to grips with. I’ve always valued writers who can take a complex idea and boil it down to a simpler and more digestible form. I have likewise appreciated papers that create so-called toy problems. Toy problems which exemplify some property in a way that is easily visualized and turned into experiments. Subjectively, I always found simplified examples useful for understanding what a paper is trying to accomplish. Reproducing the toy problem was a useful tool in creating a smaller test case I could use for debugging. From an objective standpoint, simplified examples appear to provide no benefit for making a paper more reproducible. In fact, they do not even make papers more readable! I still struggle to understand and explain this result. This is exactly why it is important for us as a community to quantify these questions. If we do not do the work of quantifiction, we will never know that our work is tackling the issues most relevant to the research problem at hand. Finding 6: Please, check your email The last result I want to discuss is that replying to questions has a huge impact on a paper’s reproducibility. This result was expected, as not all papers rarely contain a perfect description of their methods. I emailed 50 different authors with questions regarding how to reproduce their results. In the 24 cases where I never got a reply, I was able to reproduce their results only once (a 4% success rate). For the remaining 26 cases in which the author did respond, I was able to successfully reproduce 22 of the papers (an 85% success rate). I think this result is most interesting for what it implies about the publication process itself. What if we allowed published papers to be updated over time, without it becoming some kind of “new” publication? This way, authors could incorporate common feedback and questions into the original paper. This is already possible when papers are [posted on the arXiv](https://arxiv.org/); this should be the case for conference venue publications as well. These are things that could potentially advance science by increasing reproducibility, but only if we allow them to happen. What Have We Learned? Experts call this "hyperparameter tuning". https://xkcd.com/1838/ This work was inspired by the headline, “Artificial intelligence faces reproducibility crisis”. Is this headline hype or does it point to a systematic problem in the field? After completing this effort, my inclination is that there is room for improvement, but that we in the AI/ML field are doing a better job than most disciplines. A 62% success rate is higher than many meta-analyses from other sciences, and I suspect my 62% number is lower than reality. Others who are more familiar with research areas outside of my areas of expertise might be able to succeed where I have failed. Therefore, I consider the 62% estimate to be a lower bound. One thing I want to make very clear: none of these results should be taken as a definitive statement on what is and what is not reproducible. There are a huge number of potential biases that may impact these results. Most obvious is that these 255 attempts at reproduction were all done by a single person. There are no community standards for internal consistency between meta-analysts. What I find easy to reproduce may be difficult for others, and vice-versa. For example, I couldn’t reproduce any of the Bayesian or fairness-based papers I attempted, but I don’t believe that these fields are irreproducible. My personal biases, in terms of background, education, resources, interests, and more, are all inseparable from the results obtained. That said, I think this work provides strong evidence for a number of our communities’ current challenges while validating many reproducibility efforts currently under way in the community. The biggest factors are that we cannot take all of our assumptions about so-called reproducible ML at face value. These assumptions need to be tested, and I hope more than anything that this work will inspire others to begin quantifying and collecting this data for themselves. As a community, we are in a very unique position to perform meta-science on ourselves. The cost of replication is so much lower for us than for any other field of science. What we learn here could have impacts that extend beyond AI & ML to other subfields of Computer Science. More than anything, I think this work reinforces how difficulties of evaluating the reproducibility of research. Considering each feature in isolation is a fairly simple way to approach this analysis. This analysis has already delivered a number of potential insights, unexpected results, and complexities. However, it does not begin to consider correlations among papers based on authors, and representing the data as a graph, or even just looking at non-linear interactions of the current features! This is why I’ve attempted to make [much of the data publicly available](https://github.com/EdwardRaff/Quantifying-Independently-Reproduc… so that others can perform a deeper analysis. Finally, it has been pointed out to me that I may have created the most unreproducible ML research ever. But in reality, it leads to a number of issues regarding how we do the science of meta-science, to study how we implement and evaluate our research. With that, I hope I’ve encouraged you to read my paper for further details and discussion. Think about how your own work fits into the larger picture of human knowledge and science. As the avalanche of new AI and ML research continues to grow, our ability to leverage and learn from all this work will be highly dependent on our ability to distill ever more knowledge down to a digestible form. At the same time, our process and systems must result in reproducible work that does not lead us astray. I have more work I would like to do in this space, and I hope you will join me. [Dr. Edward Raff](https://www.edwardraff.com/) is a Chief Scientist at Booz Allen Hamilton, Visiting Professor at the University of Maryland, Baltimore County (UMBC), and author of the [JSAT](https://github.com/EdwardRaff/JSAT) machine learning library. Dr. Raff leads the machine learning research team at Booz Allen, while also supporting clients who have advanced ML needs. He received his BS and MS in Computer Science from Purdue University, and his PhD from UMBC. You can follow him on [Twitter @EdwardRaffML](https://twitter.com/EdwardRaffML).

1 0

A new model and dataset for long-range memory
by coderman 05 Mar '20

05 Mar '20

https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_me… ''' We find the Compressive Transformer has state-of-the-art performance in the modelling of natural language for two widely-used long-range benchmarks, WikiText-103 and Enwik8, compared to [published results](http://nlpprogress.com/english/language_modeling.html) that do not use additional sources of training data. We also show it can be used effectively to model speech, handles rare words especially well, and can be used within a reinforcement learning agent to solve a memory task. ''' Article: This blog introduces a new long-range memory model, the [Compressive Transformer](https://arxiv.org/abs/1911.05507), alongside a new benchmark for book-level language modelling, [PG19](https://github.com/deepmind/pg19). We provide the conceptual tools needed to understand this new research in the context of recent developments in memory models and language modelling. Throughout our lives, we build up memories that are retained over a diverse array of timescales, from minutes to months to years to decades. When reading a book, we can recall characters who were introduced many chapters ago, or in an earlier book in a series, and reason about their motivations and likely actions in the current context. We can even put the book down during a busy week, and pick up from where we left off without forgetting the plotline. We do not achieve such feats by storing every detail of sensory input we receive about the world throughout our lifetimes. [Our brains select, filter, and integrate](https://www.ncbi.nlm.nih.gov/pubmed/28641107) input stimuli based on factors of relevance, surprise, perceived danger, and repetition. In other words, we compress lifelong experience to a set of salient memories which help us understand the past, and better anticipate the future. A major goal of AI researchers is discovering ways of implementing such abilities in computational systems and benchmarks which require complex reasoning over long time-spans. Memory systems for artificial neural networks have advanced considerably in the past two decades. In this post, we look to past advances to explore why this is such a difficult task and consider how natural language modelling could offer an effective means of designing better long range memory systems? We reflect on the necessity for better compressive memory architectures, and sparse memory access mechanisms, to work towards the goal of incorporating lifelong reasoning in our computational systems. A brief history of memory in deep learning > There is no memory or retentive faculty based on lasting impression. What we designate as memory is but increased responsiveness to repeated stimuli. [Nikola Tesla](https://deepmind.com/research?filters=%7B%22authors%22:%5B%226136023… One of the earliest and most widely-used memory architectures in present day is a recurrent neural network (RNN) called the [Long Short-Term-Memory](https://www.bioinf.jku.at/publications/older/2604.pdf) (LSTM). The LSTM maintains a compact memory in the form of a vector of numbers, which it accesses and modifies with gated read, write, and forget operations. It was originally developed on a suite of synthetic tasks that involved learning logical operations on a stream of bits. However, it has since become a ubiquitous model of sequential data: from recognising handwritten notes to predicting the early onset of kidney injury. One weakness of the LSTM, and of many contemporary RNNs, is capacity. They are designed so that each unit of memory can influence every other unit in memory with a learnable weight. But this results in a computationally inefficient system: the number of learnable parameters in the model grows quadratically with the memory size. For example, an LSTM with a memory of size 64KB results in parameters of size 8GB. Circumventing this memory capacity bottleneck has been an active research area. [Figure 1. Long-range reasoning is crucial to general intelligence. Here, an agent remembers the existence and location of a key over a long period of time, and recalls this information when a treasure chest is discovered – prompting the agent to return to the remembered location to retrieve the key.]Figure 1. Long-range reasoning is crucial to general intelligence. Here, an agent remembers the existence and location of a key over a long period of time, and recalls this information when a treasure chest is discovered – prompting the agent to return to the remembered location to retrieve the key. Researchers at DeepMind proposed a novel architecture, the [Differentiable Neural Computer](https://deepmind.com/blog/article/differentiable-neural-computers) (DNC), which augments an LSTM with a much larger memory matrix to address these deficits. The DNC uses an attention operation to read from this memory matrix. In visual attention, our eyes are drawn by pertinent objects in a visual scene–for example, one might typically spend more time observing a friend’s face during an emotional conversation than on noticing their shoes. Here, memory models can attend to particular events/data in the past. This attention operation requires a fixed number of parameters, independent of the memory size, and so the memory capacity of the model can be significantly increased. Alongside the DNC, recurrent neural networks with an additional attention mechanism were showing promise in the domains of [translation](https://arxiv.org/abs/1409.0473) and [question answering](https://arxiv.org/abs/1410.3916). These models were able to reason over time using two memory structures: a small and compact LSTM memory and a large external memory. However, more recently researchers at Google Brain Team proposed the Transformer which removes the LSTM, and only uses attention to transmit [information across time](https://arxiv.org/abs/1706.03762). [Figure 2. A visualisation of the neural network’s attention for English to French translation. Source: Attention and Augmented Recurrent Neural Networks, Olah & Carter, 2016]Figure 2. A visualisation of the neural network’s attention for English to French translation. Source: Attention and Augmented Recurrent Neural Networks, Olah & Carter, 2016 The Transformer was originally shown to significantly outperform recurrent neural networks for machine translation. However it has since been applied to a range of applications in natural language processing, from question answering, document summarisation, sentiment classification and the modelling of natural language – a task that has seen particular exciting developments over the past year. Modelling natural language Finding machine learning tasks which both drive the development of better memory architectures and push us further towards artificial general intelligence is challenging. Statistical language modelling is one such task that we believe could be valuable for both purposes. Language models work by sequentially predicting the next word in a stream of text. They can be used to model existing texts and also to generate novel texts. As they get better at modelling the past, their predictions become more accurate, and the texts they generate become more realistic. In Claude Shannon’s seminal article “[A Mathematical Theory of Communication](https://onlinelibrary.wiley.com/doi/10.1002/j.1538-7305.1948… published in 1948, which founded the field of information theory, he discussed primitive language models and illustrated how adding more context improves the quality and realism of generated text. He does this by introducing the most simple model of English text, which has no contextual modelling at all – a character-level model which treats each character independently. By sampling characters with their relative frequencies (8% of the time for ‘a’, 1.5% for ‘b’ etc.) we arrive with a nonsensical string : XFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD. However, he remarks at the improvement in sample quality if one instead models the probability of words independently. Now the modelled context is approximately 7X larger (the average number of characters in a word): REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE. By modelling the probability of word pairs, a further 2X in context length, even more realistic text emerges: THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED In other words, an increase in the length of context leads to an improvement in the quality of text generated. Shannon remarks on the quality of his produced samples and conjectures that natural text samples may emerge from a sufficiently complex statistical model, “The particular sequence of ten words “attack on an English writer that the character of this” is not at all unreasonable. It appears then that a sufficiently complex stochastic process will give a satisfactory representation of a discrete source”. One criticism of language modelling as a task for long-range reasoning is that models can capture a large portion of their predictions from the local context. Neural language models have traditionally ignored the wider context, focusing mostly on the short term. For example, in 2017 [Dailuk et al.](https://arxiv.org/abs/1702.04521) found their neural language model rarely attends beyond the preceding five words. However in the past year large Transformer models have been shown to make use of hundreds of words of context to generate ever-more realistic text with a longer range of coherence. A demo from [OpenAI’s GPT-2](https://openai.com/blog/better-language-models/), a 1.5B parameter Transformer, indicate that the model is able to generate realistic text and retain key entities (e.g. Dr Jorge Pérez and unicorns) across multiple paragraphs: The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist. Transferring knowledge Such samples would likely astound Shannon, 70 years on from his early language model experiments. However the real benefit of powerful neural language models – and their relevance to the goal of AGI – is their ability to transfer knowledge to a suite of tasks. In the process of learning how to model text, neural language models appear to build up a knowledge-base of associations, and a plethora of skills. For instance, researchers at OpenAI showed that GPT-2 can be applied to natural-language processing tasks such as question answering, paraphrasing, or sentiment analysis with surprisingly good performance – especially for a model that has never been explicitly trained to perform such tasks. When large Transformer language models are fine-tuned on particular tasks such as question answering, the resulting performance is significantly better than models that were designed and trained solely for question answering. Google’s prominent natural language model, [BERT](https://arxiv.org/abs/1810.04805), achieves state-of-the-art performance on a wide array of NLP benchmarks, and is now [a part of Google Search](https://www.blog.google/products/search/search-language-understanding-bert/). And more recently, it was shown that GPT-2 can learn to play rudimentary chess by training it on strings of [game moves](https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-game/). Benchmarking language models A popular long-range language model benchmark is [WikiText-103](https://www.salesforce.com/products/einstein/ai-research/the-…, which is comprised of English-language Wikipedia articles, and was developed by researchers at [Salesforce AI](https://arxiv.org/abs/1609.07843). Articles are around 3,600 words on average, which, at the time of creation, was far beyond the memory window of state-of-the-art models. However researchers at Google recently showed that a Transformer variant called the TransformerXL – which maintains a memory of past network activations and recently obtained state-of-the-art results on WikiText-103 – can make use of contexts spanning over [one thousand words](https://arxiv.org/abs/1901.02860). This raises the question: will models soon saturate these benchmarks? As such, we’ve compiled and released a new, longer-range language model benchmark based on books. A new dataset for long-term memory research To support growing interest in long-range sequence models, we are releasing a new language modelling benchmark, [PG-19](https://github.com/deepmind/pg19), which is derived from books in the [Project Gutenberg online library](https://www.gutenberg.org/). Books provide a rich context for the development of long-range memory models. We selected a subset of approximately 28,000 books from Project Gutenberg published before 1919. Unlike prior language modeling dataset releases, we apply very little pre-processing to the text. For example, we do not limit the vocabulary size of the data or censor numbers, to avoid the filtering of useful information. PG-19 is over double the size of prior language modelling benchmarks, such as the [Billion Word Benchmark](https://ai.google/research/pubs/pub41880), and contains text that is over 10X longer in context than the prior long-range language model benchmark, WikiText-103. We provide a comparative table of existing language modelling benchmarks, below: Compressive Transformer Alongside a new benchmark, we propose a long-range memory model called the [Compressive Transformer](https://arxiv.org/abs/1911.05507). We take inspiration from the role of sleep in the formation of [consolidated episodic memories](https://www.ncbi.nlm.nih.gov/pubmed/28641107). Sleep is known to be crucial for memory, and it’s thought that sleep serves to compress and consolidate memories, thereby improving reasoning abilities for memory tasks. In the Compressive Transformer, granular memories akin to episodic memories are collected online as the model passes over a sequence of inputs; over time, they are eventually compacted. The Compressive Transformer uses attention to select information from the past, like the Transformer. It maintains a short-term memory of past activations, in the same style as the recently-proposed [TransformerXL](https://arxiv.org/abs/1901.02860). Where the TransformerXL discards past activations when they become older, the Compressive Transformer instead compacts them into a compressed memory. The compression is performed by a neural network guided by an auxiliary loss that guides it to keep around task-relevant information. It can learn to filter out irrelevant memories, as well as combine memories so that the salient information is preserved and retrievable over a longer period of time. [Figure 3. Compressive Transformer: a long-range sequence attentive sequence model which characterises the past with a granular short-term memory with a coarse compressed memory.]Figure 3. Compressive Transformer: a long-range sequence attentive sequence model which characterises the past with a granular short-term memory with a coarse compressed memory. We find the Compressive Transformer has state-of-the-art performance in the modelling of natural language for two widely-used long-range benchmarks, WikiText-103 and Enwik8, compared to [published results](http://nlpprogress.com/english/language_modeling.html) that do not use additional sources of training data. We also show it can be used effectively to model speech, handles rare words especially well, and can be used within a reinforcement learning agent to solve a memory task. We find the Compressive Transformer produces the largest performance gain in modelling long-context book text from the PG-19 benchmark. The model’s conditional samples can be used to write book-like extracts. Below we show a sample that is fed a paragraph of text to be used as context, taken from “The Patrol of the Sun Dance” by Ralph Connor, which the model has not previously seen. Context from The Patrol of the Sun Dance Trail by Ralph Connor Along the tote-road, which ran parallel to the steel, a man, dark of skin, slight but wiry, came running, his hard panting, his streaming face, his open mouth proclaiming his exhaustion. At a little trail that led to the left he paused, noted its course toward the flaunting flag, turned into it, then struggled up the rocky hillside till he came to the wooden shack, with a deep porch running round it, and surrounded by a rustic fence which enclosed a garden whose neatness illustrated a characteristic of the British soldier. The runner passed in through the gate and up the little gravel walk and began to ascend the steps... Continuation by the Compressive Transformer ...As though by mutual consent, all the three members of the Superintendent Force were standing on the veranda watching his progress, a man came strolling slowly down the path, stopping often to gaze curiously about him, then stopping once more when he saw the four standing on the veranda. His face was dark, his eyes prominent, and the lines that furrowed his forehead showed marks of many woes, yet his step was alert and he began to mount the steps with steady strides. The man’s attention was diverted by the unexpected appearance of the two policemen; a third man came slipping up, grumbling under his breath, and stopped long enough to look doubtfully around him, then slipped away again. In the meantime, the four on the veranda continued to watch the approach of the new arrival. In spite of himself, the Superintendent Force became conscious of a curious feeling–uneasiness that was almost fear–in the presence of that strange man. He would not allow himself to admit the fact, yet he felt it in his bones. But to the watchers, the plain, everyday features of that stranger and his coming, seemed only just what the Seven White Shee owed him–their weight, their hurry, their blast... The Compressive Transformer is able to produce narrative in a variety of styles, from multi-character dialogue, first-person diary entries, or third-person prose. Although the model does not have an understanding of language that’s grounded in the real world, or the events that take place in it – by capturing longer-range correlations, we see the emergence of more coherent text. The future of memory architectures As we strive to create agents that operate over days, weeks or even years, it will be impractical to compute over all raw input data at each timestep. Even with the current growth in computing power, we will need to develop compressive and sparse architectures for memory to build representations and reason about actions. Models which are able to capture relevant correlations across days, months, or years’ worth of experience are on the horizon. We believe the route to more powerful reasoning over time will emerge from better selective attention of the past, and more effective mechanisms to compress it. As we explore ideas in this space, we need tasks and datasets that span longer and longer time intervals. The PG-19 dataset can help researchers move in this direction, presenting textual data in the longest form that we typically consume as humans: full-length books. We hope that its release will spur interest in new models that compress the past in order to predict the future and act effectively in the present. Read more - [Compressive Transformer paper](https://arxiv.org/abs/1911.05507) - [PG-19 Benchmark](https://github.com/deepmind/pg19)

1 0