- cypherpunks - lists.cpunks.org

Re: [openzfs/zfs] mentions to 'child gangs' in the source (#10554)
by Zenaan Harkness 12 Jul '20

12 Jul '20

On Sat, Jul 11, 2020 at 11:46:46AM -0700, misterbigstuff wrote: > considering the trend for removing problematic language, is there another term that could be applied to gang blocks that could make discussion of such topics easier and less trigger-prone? This PC posturing is reactionary, and not at all sincere (by the hidden instigators originating this plan upon us all), - it is attempts to control our language, and more yet.. Controlling language is a temporary "effect producing intention" dasigned to demonstrate power, to cause submission to a hidden will, and nothing else, and under the guise of social signalling. Controlling language, or rather, imposing the will of a minority (not the stated minority of "those triggered" either) upon the majority in regard to the words we use, is also an attempt to control thoughts. As Professor of psychology Dr. Jordan Peterson has pointed out quite succinctly, we "think" in words, and so such attempts to control the words we use, is an attempt to control our thoughts. And the reason this can only be temporary is that all concepts still exist (child, gang, etc) for eternity, and in time we will if we succumb to this hidden intention, begin to use new words for the old concepts, but in the meantime we will be poorer of thought and definitively testing the waters of society wide, violent revolution; I suggest this is unwise. As temporary as it may be, the destructive power it unleashes is usually unseen until too late. Some folks really do not know that they are being lead in their superficial social signalling desires, for hidden and very dark intentions. If you are unfamiliar, please begin to read Solzhenitsyn's "Gulag Archipelago" which was recently translated into English and available for download, and was for decades outlawed in Russia. We owe it to ourselves to not fall into these same traps again. If you struggle with the basics of good and evil, as many have and some still do, consider the following: I have a conscience. By my conscience I know good, and evil. By my will, I choose to do good, and not evil. Pretending evil "will no longer exist if certain words are no longer used" is obviously not true, and not a solution, and carrying very real and historically precedented potential consequences. It is not the words we use which are the measure of us, it is how we treat one another. Let's be kind, but firm in the face of those who need help to cope in this world and direct them in weys to get the support and help they may genuinely need.

1 1

The Next Killer Crypto App: Decentralized Monetized Organizing
by jim bell 12 Jul '20

12 Jul '20

[Jim Bell's note: I cannot find the name of the person who wrote this.] https://lbry.tv/@DigitalCashNetwork:c/activism:b?r=Gd7GBo8adnW7dSEBDc2pPP8Y… The Next Killer Crypto App: Decentralized Monetized Organizing "At the start of 2015, inspired by the Bitcoin revolution, I started an activism project called the Rights Brigade where I organized various pro-freedom activities across the state of New Hampshire, compensated by Bitcoin donations. Our highlight was informing jurors of their right to nullify bad laws, to the tune of many thousands informed at 10 of the 11 courthouses in the state, some days with five simultaneous operations around the state involving dozens of activists. I ran into some organizational headaches, most notably in the difficulty of sending payments to long ugly cryptographic hashes and the lack of decentralized recurring payments. When I heard Dash was working on exactly these things, I put the activism on hold and went on to work full-time towards the advancement of these crypto technologies. Now, in 2020, we're much closer to solving these organizing pain points, but we're still not quite there. "Seeing what one guy alone could do five years ago with almost no budget and some really inferior tools at his disposal, I can easily see just how powerful for the world a streamlined, all-in-one, monetized, decentralized organizing app would be. To say that it could change the world would be a massive understatement. Here's what we need, what the perfect solution would look like, and where we stand now. "What's "Organizing"? "What I mean by "organizing" is, essentially, collaborating, communicating, sharing resources, and executing plans of action. Basically, any situation where a bunch of people are trying to get stuff done on a voluntary and collaborative basis (i.e. they aren't on the payroll of a company or other entity that can simply tell them what to do). Here's a few of the key tools needed to make this sort of thing happen. [end of partial quote] Jim Bell's comments follow: Maybe this could be described (in part) as "paid-flash-mobs"? I suspect there was a lot of much-less technologically advanced activity going on in the "Occupy Wall Street" and other protests.

2 1

The Virtue of Science, the foundation of the '1st world'
by Punk-BatSoup-Stasi 2.0 12 Jul '20

12 Jul '20

https://en.wikipedia.org/wiki/Compulsory_sterilization No dobut Technology can be applied to everything, to make everything 'efficient' and 'automatic'.

1 0

brain imaging: TEDx: The most important lesson from 83,000 brain scans
by Zenaan Harkness 12 Jul '20

12 Jul '20

"Did you know that psychiatrists are the only medical specialists that virtually _never_ look at the organ they treat." ----- Forwarded message from Gil May <gilmay97(a)gmail.com> ----- From: Gil May <gilmay97(a)gmail.com> Date: Mon, 6 Jul 2020 20:37:19 +1000 Subject: The most important lesson from 83,000 brain scans A MUST SEE —EXTREMELY INTERESTING AND UNDERSTANDABLE https://www.youtube.com/watch?v=esPRsT-lmw8 The most important lesson from 83,000 brain scans| Daniel Amen | TEDxOrangeCoast ----- End forwarded message -----

2 1

Decrypt: Ethereum 2.0 delayed until 2021? Vitalik Buterin disagrees
by jim bell 11 Jul '20

11 Jul '20

Decrypt: Ethereum 2.0 delayed until 2021? Vitalik Buterin disagrees. https://decrypt.co/?p=35210

1 0

Joe Biden if president, will be replaced with his vice president
by Zig the N.g 10 Jul '20

10 Jul '20

For those who missed the obvious, Joe Biden if president, will be replaced with his vice president, whoever she is. And in that case, Joe "Biden Collection" will either step down willingly (probably glad to be done with the whole circus), or if for some odd "failure to comprehend why he was allowed to be nominated in the first place" reason, will be revoved, one way or another. Normally one would presume that a competent candidate for the US presidency would understand such basic principles... Judd Gregg: The Coming Biden Coup Judd Gregg, op-ed via The Hill, https://thehill.com/opinion/campaign/505924-judd-gregg-the-coming-biden-coup https://www.zerohedge.com/political/judd-gregg-coming-biden-coup

2 1

Reports of a Silicon Valley/Military Divide Have Been Greatly Exaggerated
by coderman 08 Jul '20

08 Jul '20

https://techinquiry.org/SiliconValley-Military/ Reports of a Silicon Valley/Military Divide Have Been Greatly Exaggerated Jack Poulson <jack(a)techinquiry.org> July 7, 2020 Overview We performed an in-depth analysis of all public US federal (sub)contracting [data](https://www.fpds.gov/) over the last four and a half years to estimate the rankings of tech companies, both in and out of Silicon Valley, as contractors with the military, law enforcement, and diplomatic arms of the United States. Inspired by [Mijente](https://mijente.net/)'s [groundbreaking report](https://mijente.net/2018/10/whos-behind-ice-the-tech-companies-fuel… on the US tech companies supporting the [Immigration and Customs Enforcement](https://en.wikipedia.org/wiki/U.S._Immigration_and_Customs_Enf… (ICE) agency of the [Department of Homeland Security](https://en.wikipedia.org/wiki/United_States_Department_of_Homelan… (DHS), we also ranked contractors with the [Justice Department](https://en.wikipedia.org/wiki/United_States_Department_of_Justi…, [Department of Defense](https://en.wikipedia.org/wiki/United_States_Department_of_Defense) (DoD), [State Department](https://en.wikipedia.org/wiki/United_States_Department_of_State), [Agency for International Development](https://en.wikipedia.org/wiki/United_States_Agency_for_Interna… (AID), and [Agency for Global Media](https://en.wikipedia.org/wiki/U.S._Agency_for_Global_Media) (AGM). We hope to address the alarmist claims from the [Chairman of the Joint Chiefs of Staff](https://www.nationaldefensemagazine.org/articles/2018/11/18/dunford-… and [corporate executives](https://www.bloomberg.com/opinion/articles/2018-06-06/google-s-… that a tech company -- namely, Google -- electing to not directly contribute to weapons systems is ["treasonous"](https://www.bloomberg.com/news/articles/2019-07-15/thiel-urges-u-s-probe-of-google-s-seemingly-treasonous-acts) and part of a divide between Silicon Valley and the US military that is a ["national-security threat"](https://www.theatlantic.com/ideas/archive/2018/12/growing-gulf-between-silicon-valley-and-washington/577963/). [1, 2, 3] We separately address: - The implication that defense contracting is rare in Silicon Valley -- the capacity question, [4] and - That the Department of Defense appears to only [publicly criticize](https://www.cnn.com/2019/03/14/politics/dunford-china-google/ind… U.S. tech companies [facilitating](https://www.hrw.org/news/2010/01/27/china-microsoft-issues-we… [authoritarian](https://theintercept.com/2019/03/20/rodrigo-duterte-ibm-surv… [suppression](http://cyberlaw.stanford.edu/blog/2015/03/china%E2%80%99s-gold… of [dissent](https://www.buzzfeednews.com/article/meghara/linkedin-censorship-p… as a means of demanding that they [contract](https://www.cnbc.com/2019/04/06/microsoft-hololens-2-army-plans-t… with the Department of Defense -- the human rights question. The Alleged Silicon Valley/Military Divide: Capacity Our analysis shows a diversity of contracting postures (see Tables [2](https://techinquiry.org/SiliconValley-Military/#table-2) and [3](https://techinquiry.org/SiliconValley-Military/#table-3)) not a systemic divide from Washington. Within a substantial list of namebrand tech companies, only Facebook, Apple, and Twitter look to be staying out of major military and law enforcement contracts. [5, 6] The accusations of treason stemmed from Google's 2018 release of its [worker](https://www.bloomberg.com/news/articles/2018-06-21/google-engineers… [AI Principles](https://ai.google/principles); because one of the principles involved a commitment to not building weapons systems, Google promised to not renew its contract providing custom-built drone object tracking AI for a [Joint Artificial Intelligence Center](https://venturebeat.com/2020/05/23/the-pentagons-joint-ai-center-wa… (JAIC) pathfinder project (Maven). [7, 8] Months later, the company cited potential conflicts with their AI Principles, along with their [missing](https://www.zdnet.com/article/google-heres-why-were-pulling-out-of… government certifications, as their motivations for walking away from the [Joint Enterprise Defense Infrastructure](https://www.bloomberg.com/news/articles/2018-10-08/google-d… cloud competition. The missing certifications, as well as the near-certainty that the award would go to Amazon or Microsoft, indicated that Google was not so much exiting federal cloud contracting as limiting embarrassment from a loss. This intuition was later born out through the company's deep involvement with the Department of Defense's efforts to include Artificial Intelligence / Machine Learning in the modernization of its battle networks: the [Defense Innovation Initiative](https://thediplomat.com/2014/11/a-tale-of-two-offset-strategies…. [9] Google's former CEO, Eric Schmidt, has chaired two of its components since their inception: the [Defense Innovation Board](https://en.wikipedia.org/wiki/Defense_Innovation_Advisory_Board) (DIB) and the [National Security Commission on AI](https://www.nscai.gov/about/faq) (NSCAI). And, in May, Google Cloud [poached](https://www.defenseone.com/technology/2020/05/defense-innovation-b… the Executive Director of the DIB and then landed a [secure cloud contract](https://www.prnewswire.com/news-releases/defense-innovation-unit-… through its sibling, the [Defense Innovation Unit](https://en.wikipedia.org/wiki/Defense_Innovation_Unit) (DIU). [10, 11] On balance, Google's position became supporting the DoD's cloud and cybersecurity while avoiding direct contributions to weapons systems. [12] Federal procurement data also suggests, contrary to popular [narrative](https://www.theguardian.com/world/2017/jul/30/palantir-peter-thi…, that [Palantir](https://www.buzzfeed.com/badge/insidepalantir) and [Anduril](https://theintercept.com/2019/03/09/anduril-industries-project-mav… are, financially speaking, modest players in the defense contracting space relative to Hewlett Packard (especially its [spinoff](https://www.nasdaq.com/articles/dxc-completes-usps-spin-merger-tra…, [Perspecta](https://en.wikipedia.org/wiki/Vencore)) IBM, Microsoft, Dell Technologies, AT&T, Verizon, and Amazon. Even [Accenture](https://en.wikipedia.org/wiki/Accenture), through [Accenture Federal Services](https://techinquiry.org/lobbying/vendor/accenture%20federal%20ser…, and [Johns Hopkins University](https://en.wikipedia.org/wiki/Johns_Hopkins_University), through its [Applied Physics Laboratory](https://en.wikipedia.org/wiki/Applied_Physics_Laboratory), rank substantially higher within the studied DoD and DHS agencies. [13] We conclude that the US weapons and intelligence community dramatically overreacted to a prominent tech company democratically deciding to not contribute to weapons systems. [14] In July 2019, as part of the [Aspen Security Forum](https://aspensecurityforum.org/), Defense Innovation Unit director Michael Brown [stated](https://aspensecurityforum.org/wp-content/uploads/2019/07/Technolog… that coverage of Maven was "overblown" before pointing out that many tech companies, namely Amazon and Microsoft, understand that workplaces should not be democracies and that > "the place to exercise [concern over weapons systems] is at the ballot box and we need to support the government." [15, 16, 17] The namebrand tech companies, universities, and tech-oriented traditional defense contractors, by and large, appear to be willing and able to modernize US battle networks. [18] The Alleged Silicon Valley/Military Divide: Human Rights We argue that two of the primary tech defense contractors, [Microsoft](https://techinquiry.org/SiliconValley-Military/#microsoft_overvi… and [IBM](https://techinquiry.org/SiliconValley-Military/#ibm_overview), helped normalize their industry's suppression of human rights in exchange for market access. Despite the human rights [messaging](https://www.reaganfoundation.org/programs-events/events-calendar… of Microsoft's head lawyer and [lobbyist](https://www.bizjournals.com/seattle/blog/techflash/2011/06/micros…, the company has proactively [suppressed](https://www.nytimes.com/2019/01/23/business/china-microsoft-bin… dissent in Bing for more than a decade, and its subsidiary, LinkedIn, is [infamous](https://www.buzzfeednews.com/article/meghara/linkedin-censorship-… for doing the same; in early 2010 Bill Gates [pointedly criticized](https://www.theguardian.com/technology/2010/jan/25/bill-gates-w… Google for -- as it turns out, [temporarily](https://theintercept.com/2018/09/21/google-suppresses-memo-rev… -- taking a principled stand for human rights. [19] IBM's ["safe city"](https://www-03.ibm.com/press/us/en/pressrelease/38152.wss) products are known to have involved a video surveillance system for strongman Rodrigo Duterte in [Davao City](https://theintercept.com/2019/03/20/rodrigo-duterte-ibm-surveillance/). And IBM's CEO having [directly led](https://en.wikipedia.org/wiki/IBM_and_the_Holocaust) sales of cataloguing equipment for Nazi Germany, which directly contributed to the Holocaust, is [still defended](https://twitter.com/oliviasolon/status/1080584006201094144) by the company. Similarly, [Perspecta](https://en.wikipedia.org/wiki/Vencore), a close Hewlett Packard affiliate/spin-off that dominates our chart of tech defense contractors, incorporates components of both [Computer Sciences Corporation](https://en.wikipedia.org/wiki/Computer_Sciences_Corporation), which [helped charter](https://web.archive.org/web/20120601081413/http://www.reprieve.org… CIA rendition flights to secret prisons, and QinetiQ North America. [QinetiQ](https://en.wikipedia.org/wiki/Qinetiq), the [privatization](http://timshorrock.com/forthcoming-spies-for-hire/) of England's former [Defense Evaluation and Research Agency](https://en.wikipedia.org/wiki/Defence_Evaluation_and_Research_Agenc…, has had former CIA Chief who [personally authorized](https://www.nytimes.com/2014/12/13/us/politics/ex-chief-leads-v… his agency's use of torture on its [board](https://www.theguardian.com/business/2006/oct/24/usnews.money). [20] Another tech defense contractor, [Cisco](https://techinquiry.org/SiliconValley-Military/#cisco_overview), has been [in court](https://hrlf.net/doe-vs-cisco-systems/) since 2013 for not only [helping](http://cyberlaw.stanford.edu/blog/2015/03/china%E2%80%99s-golden-s… custom-build China's ["Golden Shield"](https://en.wikipedia.org/wiki/Golden_Shield_Project) project -- commonly known as the ["Great Firewall of China"](https://www.theguardian.com/books/2019/apr/27/the-great-firewall-of-china-by-james-griffiths-review-how-to-control-the-internet) -- but even [purpose-building](https://www.eff.org/deeplinks/2016/04/ciscos-latest-attem… a censorship, video surveillance, and "forced conversion" module for suppressing a dissident religious minority. Some of Cisco's internal marketing materials mentioning this work even [leaked](https://www.wired.com/2008/05/leaked-cisco-do/) the day before a Senate human rights hearing. And despite Google's 2010-2018 [public](https://www.spiegel.de/international/business/google-co-founder-on-pulling-out-of-china-it-was-a-real-step-backward-a-686269.html) [stance](https://publicpolicy.googleblog.com/2010/01/new-approach-to-china.h… against suppressing dissent, Google's former CEO, and primary interface to the DoD, Eric Schmidt, has [defended](https://www.bbc.com/news/technology-48262148) complying with authoritarian demands [since 2006](https://www.infoworld.com/article/2654650/google-ceo-on-censoring---w…. [21] Companies and executives which comply with authoritarian demands are only openly criticized by the US national security community if they do not significantly interface with the DoD. When they are willing to partner with the DoD, as is the case with [Eric Schmidt](https://www.cnbc.com/2019/05/14/former-google-ceo-eric-schmidt-adv… and [Reid Hoffman](https://innovation.defense.gov/Media/Biographies/Bio-Display/Artic…, they are [welcomed](https://www.nscai.gov/about/commissioners) as [leaders](https://innovation.defense.gov/Members/). Data Availability Alongside this report, we are officially releasing the initial version of our procurement and lobbying explorer: techinquiry.org/lobbying/. In addition to providing summaries of over one hundred thousand companies' procurement and lobbying behavior, and lists of heavy-hitter vendors for over 100 government agencies, it includes custom [nearest-neighbor](https://techinquiry.org/SiliconValley-Military/#embedding… utilities for finding lesser-known analogues of contractors. Our manual curation of the noisy FPDS data is also made public in [JavaScript Object Notation](https://en.wikipedia.org/wiki/JSON) (JSON) via: - More than 9000 [company name normalizations](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/b…, - Roughly 5000 [corporate parentage links](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc0e4…, - More than 5000 [explanatory websites](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc…, and - More than 650 [contracting/requesting agency normalizations](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/b…. And since this work was sparked by a need to understand federal procurement data well enough to properly [submit](https://docs.google.com/document/d/17lM9zZMDlDwpxF79qU1NFYsWPApunVr… [Freedom of Information](https://en.wikipedia.org/wiki/Freedom_of_Information_Act_(Unit… requests for contracts, we are also releasing the responses to our requests. They involve three tech companies (two based in Silicon Valley): - A [redacted contract](https://techinquiry.org/FOIA/OSD-F-0128-ResponsiveDocuments.pdf), with Virginia-based [Transvoyant](https://theintercept.com/2016/04/14/in-undisclosed-cia-investm…, for the Air Force to test Transvoyant's software for a "Precise Predictive Logistics (P2L) Pilot" meant to produce a "LOGistics Common Operating Picture (LOGCOP)". The deliverables include: - "Situational (unit readiness, weapon system status) and battlespace (intelligence, political climates, adversary actions, weather) awareness in real-time and continuous" [emphasis ours], - "Visibility of missions (centralized repairs, supply chain, distribution networks) and operations (contingency and steady state) globally", and - "[The] ability to plan for operations now or anytime in the future, allows planners to adapt to dynamic situations that arise during currents operations and develop courses of actions (COAs)." - A [redacted contract](https://techinquiry.org/FOIA/OSD-F-0129-ResponsiveDocuments.pdf), with [Palantir USG, Inc.](https://techinquiry.org/lobbying/vendor/palantir%20usg%2C%20inc./?use…, for the Air Force to test Palantir's Gotham software for "Supply Chain Visualization and Analytics" to produce a "LOGistics Common Operating Picture (LOGCOP)". - The last response involves [Pivotal Software](https://en.wikipedia.org/wiki/Pivotal_Software) -- a [Dell Technologies](https://techinquiry.org/SiliconValley-Military/#dell_overview) subsidiary -- which is the largest recipient of funds from the Defense Innovation Unit. Pivotal's work on [modernizing](https://www.airforce-technology.com/features/pivotal-innovativ… the Air Force's Air Operations Centers has been widely publicized, often alongside the tongue-in-cheek hashtag ["#agileAF"](https://media.defense.gov/2019/May/01/2002126684/-1/-1/0/VIGNETTE%205%20-%20KESSEL%20RUN,%20THE%20FUTURE%20OF%20DEFENSE%20ACQUISITION%20IS%20). While this FOI request, along with the other two, was submitted in September of 2019, the responsive documents for Pivotal are expected to arrive in the first half of July. Once they arrive, we will link to them here. Lastly, we make public our lists of annotated subcontract award summaries, which we describe in detail in a [later section](https://techinquiry.org/SiliconValley-Military/#indirect-contracti…. Analyzing Direct Contracting This project began with the submission of [a large number](https://docs.google.com/document/d/17lM9zZMDlDwpxF79qU1NFYsWPApunVr… of [Freedom of Information](https://en.wikipedia.org/wiki/Freedom_of_Information_Act_(Unit… (FOI) requests, through the [Office of the Secretary of Defense](https://en.wikipedia.org/wiki/Office_of_the_Secretary_of_Defense), for contracts relating to Silicon Valley's role in the [Defense Innovation Initiative](https://thediplomat.com/2014/11/a-tale-of-two-offset-strategies…. Intermediate responses made clear that at least a passing familiarity with the [Federal Procurement Data System](https://en.wikipedia.org/wiki/Federal_procurement_data_system) was required. And due to the extended wait times, high redaction rates, and low response rates, we decided that a detailed study of the [hundreds of gigabytes](https://files.usaspending.gov/database_download/) of federal procurement awards might prove intrinsically valuable, in addition to improving our ability to target future FOI requests. While [Tech Inquiry](https://techinquiry.org/) is entirely made up of tech workers, roughly our first year was spent pointedly avoiding the common pitfall of assuming that change is best achieved by throwing up a website and writing software. But such an acknowledgement of the centrality of organizing and communications should not prevent detailed curation and analysis of datasets that allow us to [study up](https://medium.com/swlh/studying-up-reorienting-the-field-of-algorithmi…, such as databases of [corporate registrations](https://opencorporates.com/), [corporate lobbying and political contributions](https://opensecrets.org/), or, in the case of this report, military and law enforcement contracting. Given that the author previously built recommendation systems for a major tech company, one of our first questions was whether the FPDS award data was rich enough to allow for the automatic retrieval of answers to questions of the form: "Which companies contract like Palantir?". As it turns out, the answer is yes -- we discuss our system for producing nearest neighbors of FPDS vendors in a [later section](https://techinquiry.org/SiliconValley-Military/#embeddings). But, through our own usage of our tool, we found the following, more basic, questions to typically be more important: - What are all of the names a particular company is listed under? - Who are the parent and child companies -- if any -- of this company? - What is an informative website associated with this company? - What is this company's address and latitude/longitude? And is there an associated phone number? - Which agencies does a particular company primarily contract with? And how much money have they received from each? - Which companies (perhaps including all of their subsidiaries) receive the most money from a particular agency? - What are the most recent awards received by a particular company? - What are the biggest awards ever received by a particular company? - What is the parent award, if any, of a particular award? Or, from the opposite perspective: what are the subawards? - Which awards contained a particular phrase (e.g., "facial recognition" or "aws govcloud")? And what vendors and agencies were they associated with? - Who are the subcontractors of a particular company? And how much money did they receive from subcontracting? Of all of these questions, the last turns out to be the most critical -- and, by far, the most laborious -- for understanding tech defense contracting, and we delay its discussion to a later section. The others can be fairly quickly answered in each instance through the curation of a[Postgres](https://en.wikipedia.org/wiki/PostgreSQL) database of [geocoded](https://www.geocod.io/) FPDS award data with an associated interface (in our case, techinquiry.org/lobbying/) But it is helpful for us to take a step back and briefly review the context of US federal procurement data before diving into the structure of the associated data feed and how we extract answers to our above questions from it. Overview of the Federal Procurement Data System and its Curation For an in-depth overview of the roots of U.S. federal procurement, dating back to the Revolutionary War, we highly recommend [The U.S. Federal Procurement System: An Introduction](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3063559) by Christopher R. Yukins. But this report only involves the [Federal Procurement Data System](https://en.wikipedia.org/wiki/Federal_Procurement_Data_System) (FPDS), which, since 2010, has been [run by IBM](https://fcw.com/articles/2010/02/18/integrate-acquisition-databases.as… through the [General Services Administration](https://en.wikipedia.org/wiki/General_Services_Administrati…. At roughly 1:30AM ET each day, anywhere from 10,000 to 100,000 new award modification summaries are made available through an [ATOM feed](https://www.fpds.gov/wiki/index.php/ATOM_Feed_FAQ) provided by the official source for FPDS data, fpds.gov. [22] While FPDS is the definitive source for US federal procurement data, it is known to have numerous shortcomings, such as inconsistencies and and inaccuracies in award amounts, slow and incomplete uploads from contract officers (including 90 day delays for DoD procurement), and corrections frequently taking place years after the signing data. The [Government Accountability Office](https://en.wikipedia.org/wiki/Government_Accountability_Office) (GAO) issued [a report](https://www.gao.gov/products/GAO-20-75) in late 2019 covering many such shortcomings. One of the more delicate issues is that FPDS integrates the proprietary [Data Universal Numbering System](https://en.wikipedia.org/wiki/Data_Universal_Numbering_System) (DUNS), an assignment of a nine digit number to each business entity. Beyond accuracy and precision issues associated with DUNS, bulk dissemination of FPDS data is [alleged](https://www.fpds.gov/help/DnB.htm) to violate the intellectual property of the owner of the data, [Dun & Bradstreet](https://en.wikipedia.org/wiki/Dun_%26_Bradstreet). [23] As a result, we avoid overreliance on DUNS information (which includes some information on parent companies) and manually maintain our own vendor name [normalizations](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/b… and corporate [parentage](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/f… hierarchies. Anatomy of an FPDS Award To give a concrete example of what type of data exists in the FPDS ATOM feed, a boiled down version of the JSON equivalent of the October 2019 JEDI award, which went to Microsoft, is shown in [Figure 1](https://techinquiry.org/SiliconValley-Military/#fpds-jedi-snippet). { "title": "New IDC HQ003420D0001 awarded to MICROSOFT CORPORATION for the amount of $0", "content": { "IDV": { "contractID": { "IDVID": { "PIID": "HQ003420D0001", "modNumber": "0" } }, "relevantContractDates": { "signedDate": "2019-10-25 00:00:00", "effectiveDate": "2019-10-25 00:00:00", "lastDateToOrder": "2030-10-24 00:00:00" }, "dollarValues": { "obligatedAmount": "0.00", "baseAndAllOptionsValue": "10000000000.00" }, "totalDollarValues": { "totalObligatedAmount": "0.00", "totalBaseAndAllOptionsValue": "10000000000.00" }, "purchaserInformation": { "contractingOfficeAgencyID": { "@name": "WASHINGTON HEADQUARTERS SERVICES (WHS)", "@departmentName": "DEPT OF DEFENSE" }, "contractingOfficeID": { "@name": "WASHINGTON HEADQUARTERS SERVICES" }, "fundingRequestingAgencyID": { "@name": "DEPT OF DEFENSE", "@departmentName": "DEPT OF DEFENSE" }, "fundingRequestingOfficeID": { "@name": "DOD CIO" } }, "contractMarketingData": { "websiteURL": " HTTPS://WWW.CLOUD.MIL ", "whoCanUse": "DEFENSE", "emailAddress": " support(a)cloud.mil ", "maximumOrderLimit": "1000000000.00", }, "contractData": { "contractActionType": { "@description": "IDC" }, "descriptionOfContractRequirement": "ENTERPRISE LEVEL, COMMERCIAL INFRASTRUCTURE AS A SERVICE (IAAS) AND PLATFORM AS A SERVICE (PAAS) TO SUPPORT DEPARTMENT OF DEFENSE BUSINESS AND MISSION OPERATIONS." }, "vendor": { "vendorHeader": { "vendorName": "MICROSOFT CORPORATION" }, "vendorSiteDetails": { "vendorLocation": { "streetAddress": "1 MICROSOFT WAY", "state": { "@name": "WASHINGTON", } "ZIPCode": { "@city": "REDMOND", "#text": "980528300" }, "countryCode": { "@name": "UNITED STATES" }, "phoneNo": "5713532701" }, "vendorDUNSInformation": { "DUNSNumber": "081466849", "globalParentDUNSNumber": "081466849", "globalParentDUNSName": "MICROSOFT CORPORATION" } } }, "competition": { "extentCompeted": { "@description": "FULL AND OPEN COMPETITION" }, "solicitationProcedures": { "@description": "NEGOTIATED PROPOSAL/QUOTE" }, "numberOfOffersReceived": "7" }, "transactionInformation": { "createdBy": " ANGELA.DEREN.HQ0034(a)WHS.MIL ", "createdDate": "2019-10-29 13:24:08", "lastModifiedBy": "HABERLACHJ", "lastModifiedDate": "2019-10-31 16:12:48", "status": { "@description": "FINAL" } } } } } Figure 1: A small, relevant subset of a JSON translation of the official FPDS XML (via the python [xmltodict](https://github.com/martinblech/xmltodict) module) for the original award to Microsoft for the [Joint Enterprise Defense Infrastructure](https://en.wikipedia.org/wiki/Joint_Enterprise_Defense_Infr… (JEDI) contract. We can read off a number of useful conclusions from this -- again, truncated -- snippet: - The [Procurement Instrument IDentifier](https://www.fpds.gov/help/Procurement_Instrument_Identifier.htm) (PIID) for JEDI is HQ0034-20-D-0001. - The original JEDI award was signed, and also made effective, on October 25, 2019. Whereas the last modification to this original award was October 31, 2019 (and an email address associated with the modifier is provided). - There is no legally [obligated amount](https://fedspendingtransparency.github.io/whitepapers/amount/), whereas the potential value of the contract, if all options become exercised, is $10B. - The award was contracted through the [Washington Headquarters Service](https://en.wikipedia.org/wiki/Washington_Headquarters_Services) but was requested by the DoD's [Chief Information Officer](https://dodcio.defense.gov/) (as it happens, later awards moved the requesting agency down to the [Defense Information Systems Agency](https://en.wikipedia.org/wiki/Defense_Information_Systems_Agency)) - The official description of the award is to provide: > Enterprise level, commercial Infrastructure as a Service (IAAS) and Platform as a Service (PAAS) to support Department of Defense business and mission operations. - Microsoft's listed address is: > 1 Microsoft Way > Redmond, WA 98052-8300 and their listed phone number is (571) 353-2701. While the address of large companies, such as Microsoft, is readily available, aggregated lists for obscure contractors are somewhat rare. This consistent source for company addresses is behind our ability to show maps of geocoded headquarters for the vast majority of federal contractors. - Microsoft's associated DUNS number is 081466849, and it has no listed parent company (technically, the award lists its parent as itself). - The award is claimed to have undergone a full, open competition that received 7 offers. We underline that, on a typical day, between twenty and thirty thousand new award summaries are made public via the[FPDS](https://fpds.gov) ATOM feed. Three Measures of Contract Value One of the significant complications in analyzing federal procurement is that there is no straight-forward answer for how to measure money-flow. One of the best such examples is the above [JEDI award](https://techinquiry.org/SiliconValley-Military/#fpds-jedi-snippet), which reports a minimum, or ["obligated"](https://fedspendingtransparency.github.io/whitepapers/amount/) award amount of zero, and a ["base and all options"](https://fedspendingtransparency.github.io/whitepapers/amount/) -- or "potential" -- value of ten billion dollars. That is to say, the minimum amount of the award was set to zero, and the maximum amount was set to ten billion, and the actual value could be anywhere in the middle. This was pointedly [explained](https://breakingdefense.com/2020/05/exclusive-dods-cio-makes-cas… by the Department of Defense's Chief Information Officer, [Dana Deasy](https://www.defense.gov/Our-Story/Biographies/Biography/Article/2051…: > "People think JEDI is a 10-year, $10 billion contract. It’s not – not necessarily. While that’s the maximum value and duration of the contract, the Pentagon has the option to terminate it after two years. There’s another end-it-or-extend-it decision three years later, and a third three years after that. The minimum the winning contractor is guaranteed to get? Just $1 million over two years." To understand a possible source for the million dollar floor on the contract value, we incorporate two further complications: - Awards often cannot be analyzed in isolation -- they can have both modification awards (using the same PIID) and subawards (which reference the original award's PIID). - There are not just lower and upper bounds on contract values -- the "obligated" amount and "potential", a.k.a. ["base and all options"](https://fedspendingtransparency.github.io/whitepapers/amount/) value -- but also a typically more accurate lower bound, called the "current", or ["base and exercised options"](https://fedspendingtransparency.github.io/whitepapers/amount/) value. Figure 2: A subaward to the original JEDI award showing all three contract value types as one million dollars. One of the several subawards to the original JEDI award is shown in [Figure 2](https://techinquiry.org/SiliconValley-Military/#fpds-jedi-subaward), with each of the three measures of contract value set to the same mysterious one million dollar floor value mentioned by Deasy. But we emphasize that one should not expect FPDS award data to always be correct: a 2017 Government Accountability Office [report](https://www.gao.gov/products/GAO-18-138) claimed that less than one percent of awards are fully consistent! The takeaway should be that, when FPDS award data is consistent, the running totals of the contract values should satisfy the ordering: In several extreme cases, typically involving DoD contracts run through the GSA's [Federal Acquisition Service](https://www.gsa.gov/about-us/organization/federal-acquisition-serv… (FAS), both the "obligated" and "current" contract values are zero, while the "potential" value is almost exactly one trillion dollars. We therefore argue that it is a serious, qualitative mistake to pick any one of these three contract values as representative. Our approach is to recognize the noise and inconsistency in FPDS data and instead focus on companies' rankings within government agencies when sorted by contract values. Which contract value should we sort by to determine rankings? We do so for all three, and then take the most significant of the three results -- that is, the highest ranking -- as a numerical indicator of influence of a company on a particular agency. To give a specific example: we measured IBM as the 2nd largest vendor with the DHS's [Customs and Border Protection](https://en.wikipedia.org/wiki/U.S._Customs_and_Border_Protectio… by obligated amount, 3rd largest by current values, and 19th largest by potential values. As a result, we set IBM's influence ranking with CBP as 2nd. By focusing on per-agency rankings, rather than dollar amounts, we help avoid coming to misleading conclusions due to systemic idiosyncrasies and imprecision in agency data. Normalizing Company Names Most questions that we would like to investigate involve aggregating large numbers of awards for each vendor, which therefore requires adoption of some form of company identifier. An obvious approach would be to set this to the DUNS number, but, for the accuracy and intellectual property reasons described above, we instead use a specific spelling of the business name (potentially qualified with the state of incorporation) and manually curate a list of corrections from encountered misspellings (possibly restricted to its pairing with a particular DUNS numbers). While our JEDI award example properly provided Microsoft's (lowercased) name as "microsoft corporation", this is far from always the case. Our [current normalization map](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc0e4ed… includes corrections from the encountered misspellings: "microsoft corporation sitz in", "microsoft corporation sitz in redmond corporation", and "microsoftcorporation". To demonstrate why we also allow for the restriction of vendor name modifications to particular DUNS numbers, we take the -- exceedingly complicated -- example of the [major](https://www.thenation.com/article/archive/five-corporations-now-domi… defense contractor [Science Applications International Corporation](https://en.wikipedia.org/wiki/Science_Applications_Internation… (SAIC). In 2013, the company decided that it could avoid legal barriers arising from conflicts between different contracts by [splitting](https://www.washingtonpost.com/business/capitalbusiness/one-year… off its legacy national security unit. Rather than making the reasonable choice of giving the spun-off company a new name, because the spin-off would be maintaining legacy contracts, they gave the spin-off the name SAIC and renamed the parent company [Leidos](https://en.wikipedia.org/wiki/Leidos) (taken from the middle of the word "kaleidoscope"). As a result of the SAIC corporate shell game, knowing when to rename an occurrence of "SAIC" to "Leidos" requires knowledge of an extensive list of DUNS numbers (it turns out, more than 60 in this case) to build custom normalization rules around. These rules, and more than 9000 others, can be found in our [vendor normalization map](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc0e4ed…. Normalizing Requesting and Contracting Agencies As was the case for vendor name spellings, we can substantially improve the accuracy and consistency of the listings for the contracting and requesting agencies for an award. Our above approach can be adopted through the analogy: DUNS is to vendor name as office name is to agency name. That is, we build a normalization map that corrects listed agency names and provide overriding exceptions for specific pairings of offices and agencies. For example, in the Microsoft JEDI award of [Figure 1](https://techinquiry.org/SiliconValley-Military/#fpds-jedi-snippet), the requesting agency identifier sets both the "@name" (which typically refers to the agency level) and "@departmentName" (which typically refers to the parent department of the agency) to "DEPT OF DEFENSE". A reasonable default rule is to normalize such a pairing to a unique string associated with the DoD -- we standardized on "Defense". But we have the side information that the requesting office name was given as "DOD CIO", which refers to the DoD's Chief Information Officer. We therefore override our agency normalization map to normalize to "Defense: OCIO" when the agency was simply listed as the DoD but the office name is "DOD CIO". This rule, along with more than 650 others, can be found in our current [agency normalization map](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc0e4ed…. We note that there exist interesting agencies, such as the [National Security Agency](https://en.wikipedia.org/wiki/National_Security_Agency) and the [National Reconnaissance Office](https://en.wikipedia.org/wiki/National_Reconnaissance_Office), which we have only observed through the requesting office rather than the requesting agency. And several other agencies, such as the [Defense Intelligence Agency](https://en.wikipedia.org/wiki/Defense_Intelligence_Agency) (DIA), are often obscured as simply being the Department of Defense in the agency field -- in the case of the DIA, we often made use of the office being an obvious reference to the [Defense Attache System](https://en.wikipedia.org/wiki/Defense_Attach%C3%A9_System). We corrected many such improper coarsenings up to the Department of Defense via a manual review of the list of all occurring pairings of agencies and offices. Mapping Out Parent Companies and Subsidiaries The defense contracting space is notorious for its frequent mergers, acquisitions, spin-offs, and even more exotic exchanges, such as the [spin-merger](https://www.forbes.com/sites/joecornell/2017/04/04/hewlett-pac…. For example, when we colloquially refer to "Hewlett Packard", we mean a network of four affiliates: the personal computer company [HP Inc.](https://en.wikipedia.org/wiki/HP_Inc.), the enterprise IT provider [Hewlett Packard Enterprise](https://en.wikipedia.org/wiki/Hewlett_Packard_Enterprise) (HPE), HPE's B2B spin-merge [DXC Technology](https://en.wikipedia.org/wiki/DXC_Technology), DXC's spin-off of its public sector segment, [Perspecta](https://en.wikipedia.org/wiki/Perspecta), and all of their subsidiaries. Two of HPE's major subsidiaries include: - [Cray Inc.](https://en.wikipedia.org/wiki/Cray), the historically preeminent supercomputer manufacturer that has been [critical](https://www.nsa.gov/about/cryptologic-heritage/historical-figures… to the NSA's signals intelligence and continues to prove foundational to [Department of Energy](https://en.wikipedia.org/wiki/United_States_Department_of_Energy) [supercomputers](https://www.cray.com/company/news-and-media/doe-el-capitan-…, and - [HPE Government, LLC](https://techinquiry.org/lobbying/vendor/hpe%20government%2C%20llc), which includes the remnants of high-performance graphics company [Silicon Graphics, Inc.](https://en.wikipedia.org/wiki/Silicon_Graphics) (SGI). When we speak of the sum of contracts between HPE and a particular agency -- for example, the [Defense Information Systems Agency](https://en.wikipedia.org/wiki/Defense_Information_Systems_Agency) -- we could either be referring to the contracts awarded directly to HPE (an "exclusive" count), or all awards to HPE, HPE Government, Cray, and its other subsidiaries (an "inclusive" count). For this report, we focus on inclusive counts. As mentioned in our discussion of the [JEDI award](https://techinquiry.org/SiliconValley-Military/#fpds-jedi-snippet), FPDS [DUNS](https://en.wikipedia.org/wiki/Data_Universal_Numbering_System) information sometimes provides a hint for a single parent company. But, for the same reasons which we avoid overdependence on DUNS information for unique vendor identifiers, as well as the fact that many companies have multiple parents (e.g., all [joint ventures](https://en.wikipedia.org/wiki/Joint_venture)) we manually curate our own JSON map of parent companies. Our [current map](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/fc0e4ed… contains more than 5000 parent links, which, in combination with our vendor and agency normalization maps, allows us to automate inclusive award sums for a large number of pairings of vendors and agencies. As another layer of complication, we consider HP, HPE, and Perspecta to be affiliates, not parents or subsidiaries of each other. And so when we compute influence rankings of the nebulous "Hewlett Packard" network, we use the maximum influence ranking between HP, HPE, and Perspecta. In almost all cases, this corresponds to Perspecta's influence rank. Logistics of Data Retrieval and Indexing At this point, we are ready to describe the construction of a table of [direct contracting influence](https://techinquiry.org/SiliconValley-Military/#table-1) rankings, but we take the opportunity to add in the logistical details of efficiently retrieving, searching, and summarizing the federal procurement data. As mentioned in the FPDA ATOM feed [FAQ](https://www.fpds.gov/wiki/index.php/ATOM_Feed_FAQ), each pull from the ATOM feed is limited to retrieving only ten records, but up to ten simultaneous requests are allowed -- which combines to up to 100 simultaneous award requests. Given that 30,000 to 100,000 award modifications are not unusual, performing multithreaded retrievals leads to very significant savings (often, from an hour down to a few minutes). And while the FAQ recommends 9AM ET retrievals, we find that the data is typically available around 1:30AM ET. One can get an idea of the storage requirements for mirroring and indexing the entire FPDS database by reading the [USASpending.gov](https://usaspending.gov) [database guide](https://files.usaspending.gov/database_download/): more than 1.5TB of hard disk space are required for their system. Getting access to this much space on [DigitalOcean](https://www.digitalocean.com/pricing/) would cost $500 per month, which is out of the price range of our small, grassroots non-profit. We therefore adopt a two-tiered approach: we store an entire copy of the raw FPDS data (converted into JSON via python's [xmltodict](https://github.com/martinblech/xmltodict) module) on a private workstation and export salient subsets to CSV for loading into a [Postgres](https://en.wikipedia.org/wiki/PostgreSQL) database with [full-text](https://www.postgresql.org/docs/9.5/textsearch.html) search indices. We then expose an interface to the indices via our public website, techinquiry.org/lobbying/, which is a straight-forward combination of [Express.js](https://en.wikipedia.org/wiki/Express.js), [jQuery](https://pugjs.org/api/getting-started.html), [Pug](https://pugjs.org/api/getting-started.html), and [Leaflet](https://en.wikipedia.org/wiki/Leaflet_(software)). Addresses for companies are retrieved from the procurement awards and geocoded via a combination of [geocodio](https://www.geocod.io/) (for U.S. addresses) and [Nominatim](https://wiki.openstreetmap.org/wiki/Nominatim) (for international addresses). Our lists of "similar contractors" for vendors are generated by extracting nearest neighbors from the results of the embedding processed, which we execute on a private workstation, described in a [later section](https://techinquiry.org/SiliconValley-Military/#embeddings). Building a picture of direct contracting influence We can now combine all of the previously discussed FPDS curation mechanisms to compute maximum ranks of direct financial flow between various tech-related companies and U.S. federal agencies -- which we treat as proxies for influence/impact. Brief [summaries](https://techinquiry.org/SiliconValley-Military/#company-summarie… of many of the studied companies are available in a section of the appendix. Agencies For the sake of brevity, we will simply list the agencies of focus rather than providing short summaries. One exception is the Justice Department's Offices, Boards, and Divisions (OBD) unit, which is less widely documented. We recommend the book [Badges without Borders](https://www.ucpress.edu/book/9780520295629/badges-without-borders) (or the associated [interview](https://historicly.substack.com/p/history-of-the-modern-police-w… with [Historic.ly](https://historicly.substack.com/)) which details the history of the international, often explicitly anti-communist, policing programs once run through USAID's [Office of Public Safety](https://en.wikipedia.org/wiki/Office_of_Public_Safety). These programs transitioned into the Justice Department's [International Criminal Training Assistance Program](https://en.wikipedia.org/wiki/ICITAP) (ICITAP) under the OBD agency. There are [numerous](https://techinquiry.org/lobbying/search/?text=ICITAP&useModifiedD… recent awards from this program to [Science Applications International Corporation](https://en.wikipedia.org/wiki/Science_Applications_Internation… and its subsidiary, [Engility Holdings](https://privacyinternational.org/sites/default/files/2018-07/Teac…, for international policing in Indonesia, the Kingdom of Saudi Arabia, and Pakistan. For the [Justice Department](https://en.wikipedia.org/wiki/United_States_Department_of_Justi… as a whole, we investigate rankings within: - [The Federal Bureau of Investigation](https://en.wikipedia.org/wiki/Federal_Bureau_of_Investigatio… (FBI), [24] - [The Drug Enforcement Administration](https://en.wikipedia.org/wiki/Drug_Enforcement_Administrati… (DEA), - [Offices, Boards, and Divisions](https://www.justice.gov/jmd/doing-business-offices-boards-and-di… (OBD), and - [The Federal Bureau of Prisons](https://en.wikipedia.org/wiki/Federal_Bureau_of_Prisons) (BoP). From the[Treasury Department](https://en.wikipedia.org/wiki/United_States_Department_of_the_T…, we only include the [Internal Revenue Service](https://en.wikipedia.org/wiki/Internal_Revenue_Service) (IRS). Several members of the [Department of Homeland Security](https://en.wikipedia.org/wiki/United_States_Department_of_Homelan… are of interest: - [The U.S. Secret Service](https://en.wikipedia.org/wiki/United_States_Secret_Service) (SS), - [Immigration and Customs Enforcement](https://en.wikipedia.org/wiki/U.S._Immigration_and_Customs_Enf… (ICE), - [Customs and Border Protection](https://en.wikipedia.org/wiki/U.S._Customs_and_Border_Protectio… (CBP), - [U.S. Citizenship and Immigration Services](https://en.wikipedia.org/wiki/United_States_Citizenship_and_Immig… (CIS), - [The Transportation Security Administration](https://en.wikipedia.org/wiki/Transportation_Security_Admin… (TSA), and - [The Science and Technology Directorate](https://en.wikipedia.org/wiki/DHS_Science_and_Technology_Direc… (S&T). The largest number of agencies are from the [Department of Defense](https://en.wikipedia.org/wiki/United_States_Department_of_Defense): - [The U.S. Air Force](https://en.wikipedia.org/wiki/United_States_Air_Force), - [The U.S. Navy](https://en.wikipedia.org/wiki/United_States_Navy), - [The U.S. Army](https://en.wikipedia.org/wiki/United_States_Army), - [The U.S. Special Operations Command](https://en.wikipedia.org/wiki/United_States_Special_Operations_Com… (SOCOM), - [The Defense Intelligence Agency](https://en.wikipedia.org/wiki/Defense_Intelligence_Agency) (DIA), - [The Defense Logistics Agency](https://en.wikipedia.org/wiki/Defense_Logistics_Agency) (DLA) and [The U.S. Transportation Command](https://en.wikipedia.org/wiki/United_States_Transportation_Command) (TRANSCOM), - [The Washington Headquarters Service](https://en.wikipedia.org/wiki/Washington_Headquarters_Services) (WHS), - [The Defense Information Systems Agency](https://en.wikipedia.org/wiki/Defense_Information_Systems_Agency) (DISA), - [The Defense Advanced Research Projects Agency](https://en.wikipedia.org/wiki/DARPA) (DARPA), - [The National Geospatial-Intelligence Agency](https://en.wikipedia.org/wiki/National_Geospatial-Intelligence_Agen… (NGA), - [The National Security Agency](https://en.wikipedia.org/wiki/National_Security_Agency) (NSA) and [U.S. Cyber Command](https://en.wikipedia.org/wiki/United_States_Cyber_Command) (CYBERCOM). Lastly, we include rankings within several independent agencies: - [The State Department](https://en.wikipedia.org/wiki/United_States_Department_of_State), - [The U.S. Agency for International Development](https://en.wikipedia.org/wiki/United_States_Agency_for_Interna… (AID), - [The U.S. Agency for Global Media](https://en.wikipedia.org/wiki/U.S._Agency_for_Global_Media)(AGM); formerly known as the Broadcasting Board of Governors (BBG), and - [The General Services Administration](https://en.wikipedia.org/wiki/General_Services_Administrati…'s Federal Acquisition Service (FAS). FBI DEA OBD BoP IRS SS ICE CBP CIS TSA DHS S&T AirForce Navy Army SOCOM DIA DLA / TRANSCOM WHS DISA DARPA NGA NSA State AID AGM FAS HP / Perspecta 21 311 31 828 7 3 10 6 6 41 13 82 7 25 302 35 225 365 10 20 80 635 165 5 Deloitte 8 51 6 172 2 27 14 24 152 7 42 125 21 97 13 94 16 38 297 47 17 12 925 15 MITRE 2 24 285 469 1 23 49 34 80 109 2 16 85 26 1 6471 100 1 155 247 451 955 IBM 24 75 3 2483 3 286 88 2 1118 10 221 137 118 27 143 61 62 23 18 19 241 25 302 18 Accenture 10 100 17 412 1 602 56 28 18 8 92 99 49 2438 62 40 126 80 34 3 13 197 33 AT&T 99 37 1 248 6 7 95 18 171 294 219 109 107 60 126 85 2019 24 9 891 32 70 26 20 15 38 Microsoft 128 105 33 1528 1463 68 146 66 37 75 255 102 253 113 364 441 1 2 57 561 490 448 Verizon 32 28 23 296 12 6 163 133 26 359 234 132 72 257 76 1134 595 1 204 47 553 2073 39 59 Dell 56 241 343 8396 53 20 4 28 33 30 123 38 252 47 64 59 307 63 26 1612 28 539 446 73 14 Johns Hopkins 5550 144 261 43 36 183 23 231 45 861 5 104 13 29 5 428 71 4814 IDA 17,896 62 1 1629 1150 Cisco 3250 2580 301 309 175 717 6 98 Palantir 44 3265 25 84 27 10,015 763 468 14 946 1208 1531 Oracle 1873 931 294 266 292 684 825 388 642 2270 111 315 337 40 142 175 Anduril 163 1255 5788 100 49 NVIDIA 5671 95 Intel 5504 11,060 91,055 102 12,104 Apple 1593 809 7400 1356 509 5380 19,124 20,847 13,639 3538 18,757 2039 1473 392 156 Google 8951 76,708 17,248 2227 2908 255 12,353 Amazon 1176 1096 6559 733 3219 13,141 7472 19,578 1208 730 1013 1895 986 1314 841 Facebook 8902 3054 1498 Twitter 1009 Table 1: Corporate influence rankings by contracts directly between various tech-related companies and U.S. agencies between the beginning of 2016 and July 4, 2020. Measurements were generated by combining all awards listed in the Federal Procurement Data System (FPDS) which were signed or modified within the time window. In order to smooth out known inconsistencies and FPDS data, companies were ranked by each of the three ways to measure contract values (obligated, current, and potential) and their influence was set to the most significant (that is, smallest) of the three ranks. These influence rankings were then binned as either: top 10 (), top 25, (), top 50, (), top 100, (), top 250, (), top 500 (), top 1000 (), or below (). If a company was not observed contracting with an agency, the square was left white. The numerical influence rankings are shown on mouse-over. Even though we have restricted the analysis behind the influence rankings in [Table 1](https://techinquiry.org/SiliconValley-Military/#table-1) to direct contracts -- which misses, for example, a high-profile $25M (base and all options value) [award](https://www.fedscoop.com/transportation-command-aws-cloud/) between [TRANSCOM](https://en.wikipedia.org/wiki/United_States_Transportation_Comman… and Amazon through [ECS Federal](https://ecstech.com/) -- we can interpret the results as underestimates. Given this caveat, we immediately notice that the Hewlett Packard network of companies noticeably dominate, with top-10 influence rankings in 8 of the 26 agencies. [25] Equally as surprising is that consulting companies [Accenture](https://en.wikipedia.org/wiki/Accenture) and [Deloitte Consulting](https://techinquiry.org/lobbying/vendor/deloitte%20consulting%2… -- which contributed much of the award money to parent company Deloitte -- were more influential via direct contracting than essentially every tech company except HP and IBM. Next after the management consulting companies is the not-for-profit [MITRE Corporation](https://en.wikipedia.org/wiki/Mitre_Corporation), which was built specifically to manage [Federally Funded Research and Development Centers](https://en.wikipedia.org/wiki/Federally_funded_research_and_develo… (FFRDCs); its high rankings should thus serve more as a benchmark than a surprise. We underline that it ranked first with the DIA, NGA, and DHS S&T, and second with the FBI. If we restricted our attention to the ["Big Five"](https://en.wikipedia.org/wiki/Big_Tech) tech companies: Google, Apple, Facebook, Amazon, and Microsoft, then a purely direct contracting analysis would suggest that Microsoft is the only significant defense contractor -- which, again, misses the numerous multi-million dollar Amazon Web Services awards passed through subcontracts (such as [ECS Federal](https://ecstech.com/) and [JHC Technology](https://www.jhctechnology.com/)) The question of how to account for more commoditized relationships -- namely, Intel, NVIDIA, and Apple hardware sales -- is more complicated, and we will partially address it in the next section. We also notice the minor influence rankings of Palantir and Anduril relative to: management consulting companies, HP, Dell, telecoms, and even the Johns Hopkins [Applied Physics Laboratory](https://en.wikipedia.org/wiki/Applied_Physics_Laboratory) (which is responsible for the vast majority of awards to Johns Hopkins). Palantir's highest influence was within the U.S. Special Operations Command, who [publicly](https://www.defenseone.com/technology/2016/07/war-over-soon-be-ou… praised Palantir's software during their years-long struggle to win the [DCGS-A](https://en.wikipedia.org/wiki/Distributed_Common_Ground_System) contract (and a subsequent ["capability drop"](https://www.fedscoop.com/plantir-army-contract-global-network-system-data/)). Anduril's biggest influences are observed as through DARPA, the Washington Headquarters Service, and Customs and Border Protection; [direct inspection](https://techinquiry.org/lobbying/vendor/anduril%20industries%2C… of their awards reveals, for example, a $250M (base and all options value) award through CBP for "autonomous surveillance towers" and a $100M (base and all options value) award requested by DARPA and contracted through the Air Force for "advance[d] battle management anduril phase 3 idiq". [26, 27] A few of the agency columns are worth dedicated discussion. For example, Microsoft's second place influence rank within the Defense Information Systems Agency is entirely due to the $10B (base and all options value) JEDI award -- only [Leidos](https://en.wikipedia.org/wiki/Leidos) ranked higher -- and so the award being transferred to Amazon due to a [bid protest](https://www.cnbc.com/2019/11/22/amazon-files-suit-protesting-micro… would give it the same position. The other top-ten influencers within DISA, unsurprisingly, included several communication infrastructure companies: Verizon, Cisco, and AT&T, as well as HP (IBM was 18th, and Dell was 26th). The column combining the National Security Agency and the U.S. Cyber Command -- which we note only has a modest amount of reported awards -- contains top-ten rankings for Accenture (3rd) and Johns Hopkins (5th), but known NSA contractors [Dell](https://www.reuters.com/article/us-usa-security-snowden-dell/snowden-… and [AT&T](https://www.nytimes.com/2015/08/16/us/politics/att-helped-nsa-spy-on-… were only 28th and 70th, respectively. The FBI rankings, like many others, show the professional services companies, Accenture and Deloitte, as heavier-hitters than tech giants. Nevertheless, Perspecta and IBM are both in the top 25, [Verizon](http://www.documentcloud.org/documents/1342118-stratfor-computer-f… and Palantir are in the top 50, and Dell and AT&T are in the top 100. As we will see in the next section, Palantir and Anduril's results are not significantly changed by incorporating subcontracts -- indeed, Anduril's are not changed at all -- and so, if they are setting the bar for what it means for a tech company to be a defense contractor, then HP, IBM, Microsoft, Dell, Cisco, AT&T, and Verizon are more than meeting it, even without incorporating their subcontracting passthroughs. And that Intel chips are a core component of most HP and Dell machines suggests we should already be able to justify its addition it into the list. Incorporating Conservative Subcontracting Passthrough Estimates We now discuss our approach for incorporating conservative subcontracting estimates into the direct contracting influence rankings of the last section (summarized in [Table 1](https://techinquiry.org/SiliconValley-Military/#table-1)) Our approach was simple, yet exceedingly laborious: - Generate a list of awards whose descriptions include key terms (we built a custom API to return such lists in a minimal JSON format). - Manually remove all awards not attributable to the target company after direct inspection. - Expand out to the entire award series of those that remain and annotate each such award with an estimated passthrough percentage. - Augment the three types of total award amounts from the direct award analysis by adding in the products of the subcontract values and their estimated passthrough rates. - Re-rank. In practice, we combine steps (2-4) so that we need only perform a single pass through the original list. And we first sort the awards by maximum absolute value so that the most impactful items are handled first. This process was repeated for ten companies, and, when restricted to our agencies of study, resulted in: Keywords Kept subcontracts* Major intermediaries Microsoft "microsoft" or "azure" or "windows licenses" or "windows server" 6860 subcontracts ([443 cloud](https://techinquiry.org/SiliconValley-Military/subcontracts/microsof…, [6417 non-cloud](https://techinquiry.org/SiliconValley-Military/subcontracts/micr…) Dell, [CDW Corporation](https://techinquiry.org/awards/vendor/cdw%20corporation/), [Insight Enterprises](https://techinquiry.org/awards/vendor/insight%20enterprises%20…, and [Minburn Technology Group](https://techinquiry.org/awards/vendor/minburn%20technology%20group,%… Amazon "amazon" or "aws" or "govcloud" [477 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/a… [Four Points Technology](https://techinquiry.org/awards/vendor/four%20points%20technolog…, [JHC Technology](https://techinquiry.org/awards/vendor/jhc%20technology,%20inc./), and [ECS Federal](https://techinquiry.org/awards/vendor/ecs%20federal,%20llc/) Google "google" [384 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/g… [The Daston Corporation](https://techinquiry.org/awards/vendor/daston%20corporation,%20…, [DLT Solutions](https://techinquiry.org/lobbying/vendor/dlt%20solutions%2C%20llc…, [Eyak Technology](https://techinquiry.org/awards/vendor/eyak%20technology,%20llc/), and [Dnutch Associates](https://techinquiry.org/awards/vendor/dnutch%20associates%20inc… Facebook "facebook" [172 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/f… [Chaise Management Group](https://techinquiry.org/awards/vendor/chaise%20management%20group,%2…, [Sage Communications](https://techinquiry.org/awards/vendor/sage%20communications…, and [ZilYen](https://techinquiry.org/awards/vendor/zilyen%20inc/) (now doing business as [Forge Branding](https://www.forgeresults.com/)) NVIDIA "nvidia" or "tesla" [165 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/n… [General Dynamics Mission Systems](https://techinquiry.org/awards/vendor/general%20dynamics%20mission…, [Iron Bow Technologies](https://techinquiry.org/awards/vendor/iron%20bow%20technologi…, [FCN, Inc.](https://techinquiry.org/awards/vendor/fcn,%20inc./), and [DH Technologies](https://techinquiry.org/awards/vendor/dh%20technologies,%20ll… Twitter "twitter" [43 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/t… [Sage Communications](https://techinquiry.org/awards/vendor/sage%20communications…, [ZilYen](https://techinquiry.org/awards/vendor/zilyen%20inc/) (now doing business as [Forge Branding](https://www.forgeresults.com/)) and [Chaise Management Group](https://techinquiry.org/awards/vendor/chaise%20management%20group,%2… Palantir "palantir" or "gotham" [26 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/p… [i3 Federal](https://techinquiry.org/awards/vendor/i3%20federal,%20llc/), [Pat V. Mack](https://techinquiry.org/awards/vendor/pat%20v.%20mack,%20inc./), [Sava Workforce Solutions](https://techinquiry.org/awards/vendor/pat%20v.%20mack,%20inc./), and [Affigent](https://techinquiry.org/awards/vendor/affigent,%20llc) IDA "institute for defense analyses" [7 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/i… telecoms (e.g., AT&T) MITRE "mitre" [6 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/m… MIT: each award was for "mitre-lincoln laboratory research&development" Johns Hopkins "johns hopkins" [2 subcontracts](https://techinquiry.org/SiliconValley-Military/subcontracts/j… Anduril "anduril" 0 subcontracts *: We emphasize that these counts of the number of manually annotated contracts -- which we colloquially described as the number of "kept subcontracts" -- contain a small percentage of awards that simply make reference to the company in question. In such cases, the percentage of money estimated to be passed through to the company is set to zero. Such awards are kept in the list to help demonstrate that a simple keyword search is insufficient. As part of the subcontract review process, we noticed that, during the 2005-2011 time period that an [Alaska Native](https://en.wikipedia.org/wiki/Alaska_Native_corporation) subsidiary, [Eyak Technology](https://techinquiry.org/awards/vendor/eyak%20technology,%20llc/), was operating a [kickback scheme](https://www.justice.gov/opa/pr/federal-contractors-eyak-technology-… relating to its billion-dollar prime contract with the U.S. Army Corps of Engineers, it was also serving as a supplier of Google technology to the U.S. Army through numerous awards in 2009. We also discovered that Amazon, through [JHC Technology](https://techinquiry.org/awards/vendor/jhc%20technology,%20inc./), has received several million dollars in AWS cloud contracts through the [Federal Bureau of Prisons](https://en.wikipedia.org/wiki/Federal_Bureau_of_Prisons). It also became clear that many of the NVIDIA [subcontracted awards](https://techinquiry.org/SiliconValley-Military/subcontracts/nvidia.… were for the acquisition of their [DGX](https://en.wikipedia.org/wiki/Nvidia_DGX) compact [General-Purpose Graphics Processing Unit](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_p… (GPGPU) supercomputers. Recipients included: The Army, Navy, Air Force, Washington Headquarters Service, and, most surprisingly, even Veterans Affairs. While we would have preferred to have performed similar analyses for the remaining companies, we remind the reader that such analyses only increase ranks. Thus, incorporating any missed subcontracts for HP, Deloitte, IBM, AT&T, Verizon, and Dell would only make them more dominant. The opportunities for significant qualitative change are with extending our subcontract analysis to Appl, Cisco, Oracle, and Intel. FBI DEA OBD BoP IRS SS ICE CBP CIS TSA DHS S&T AirForce Navy Army SOCOM DIA DLA / TRANSCOM WHS DISA DARPA NGA NSA State AID AGM FAS HP / Perspecta 21 311 32 829 7 3 10 6 6 41 13 83 7 25 303 35 226 366 10 20 81 636 168 5 Deloitte 8 51 6 172 2 28 14 24 153 7 42 125 21 97 13 94 16 38 297 48 17 12 928 15 MITRE* 2 24 285 470 1 24 49 34 81 109 2 16 85 26 1 6471 100 1 155 248 455 956 IBM 24 75 3 2484 3 286 88 2 1119 10 222 137 118 27 144 61 62 23 18 19 241 25 305 18 Accenture 10 101 18 413 1 602 56 28 18 8 93 99 49 2439 62 40 126 80 34 3 13 197 33 AT&T 100 37 1 248 6 7 95 18 172 294 220 109 108 60 127 85 2020 24 9 892 32 70 26 20 15 38 Microsoft* 73 98 8 1216 19 21 97 41 27 29 150 60 106 56 19 244 1 1 29 198 31 270 Verizon 32 28 24 296 12 6 163 133 26 359 235 133 73 258 76 1135 596 1 204 47 553 2074 40 59 Dell 56 241 343 8398 54 20 4 28 34 31 123 38 253 47 65 59 308 63 26 1613 29 539 447 74 14 Amazon* 555 709 150 731 620 207 47 474 2611 4827 9133 742 133 269 709 463 7 787 987 437 826 Johns Hopkins* 5552 144 261 44 36 183 23 231 46 862 5 104 13 29 5 428 71 4814 IDA* 17897 62 1 1630 1151 Cisco 3252 2581 301 309 175 718 7 100 Palantir* 41 3266 26 85 374 27 9455 763 460 14 947 1210 1532 Oracle 1875 933 294 267 293 684 825 388 642 2271 111 316 337 40 145 175 NVIDIA* 1104 3292 8097 2055 10,362 7238 18,956 168 26,173 1764 95 6969 3527 7990 Anduril* 163 1255 5789 100 49 Apple 1595 809 7401 1357 509 5382 19,126 20,849 13,641 3539 18,758 2040 1475 393 159 Google* 336 1197 1829 2545 745 755 963 5412 6859 6692 1928 540 1651 1363 793 390 123 2101 Facebook* 887 52,865 2807 2715 96 Twitter* 983 7833 254 Table 2: Corporate influence rankings by a combination of direct contracts and subcontracting passthroughs between various tech-related companies and U.S. agencies between the beginning of 2016 and July 4, 2020 (subcontract analysis only included up to June 20). Direct contract measurements are generated in the same manner as for Table 1. Conservative subcontract passthroughs were estimated and incorporated for companies annotated with '*' through manual review of thousands of FPDS awards whose descriptions mentioned the company or one of its major products (e.g., "aws govcloud"). Due to the intense time demands of manually reviewing tens of thousands of awards, several companies were not augmented with subcontracting passthrough estimates. These influence rankings were then binned as either: top 10 (), top 25, (), top 50, (), top 100, (), top 250, (), top 500 (), top 1000 (), or below (). If a company was not observed contracting with an agency, the square was left white. The numerical influence rankings are shown on mouse-over. FBI DEA OBD BoP IRS SS ICE CBP CIS TSA DHS S&T AirForce Navy Army SOCOM DIA DLA / TRANSCOM WHS DISA DARPA NGA NSA State AID AGM FAS HP / Perspecta 21 311 32 829 7 3 10 6 6 41 13 83 7 25 303 35 226 366 10 20 81 636 168 5 IBM 24 75 3 2484 3 286 88 2 1119 10 222 137 118 27 144 61 62 23 18 19 241 25 305 18 Microsoft* 73 98 8 1216 19 21 97 41 27 29 150 60 106 56 19 244 1 1 29 198 31 270 Dell 56 241 343 8398 54 20 4 28 34 31 123 38 253 47 65 59 308 63 26 1613 29 539 447 74 14 Amazon* 555 709 150 731 620 207 47 474 2611 4827 9133 742 133 269 709 463 7 787 987 437 826 Cisco 3252 2581 301 309 175 718 7 100 Palantir* 41 3266 26 85 374 27 9455 763 460 14 947 1210 1532 Oracle 1875 933 294 267 293 684 825 388 642 2271 111 316 337 40 145 175 NVIDIA* 1104 3292 8097 2055 10,362 7238 18,956 168 26,173 1764 95 6969 3527 7990 Anduril* 163 1255 5789 100 49 Apple 1595 809 7401 1357 509 5382 19,126 20,849 13,641 3539 18,758 2040 1475 393 159 Google* 336 1197 1829 2545 745 755 963 5412 6859 6692 1928 540 1651 1363 793 390 123 2101 Facebook* 887 52,865 2807 2715 96 Twitter* 983 7833 254 Table 3: A restriction of [Table 2](https://techinquiry.org/SiliconValley-Military/#table-2) to tech companies. The results of incorporating our [subcontracting estimations](https://techinquiry.org/SiliconValley-Military/#table-subcontr… are demonstrated in [Table 2](https://techinquiry.org/SiliconValley-Military/#table-2) and its restriction to tech companies, [Table 3](https://techinquiry.org/SiliconValley-Military/#table-3). The most significant qualitative difference is with Amazon, whose large numbers of Justice, DHS, and DoD cloud contracts are almost entirely through intermediaries, such as [Four Points Technology](https://techinquiry.org/awards/vendor/four%20points%20technolog…, [JHC Technology](https://techinquiry.org/awards/vendor/jhc%20technology,%20inc./), and [ECS Federal](https://techinquiry.org/awards/vendor/ecs%20federal,%20llc/) (who was also the prime contractor for Google's [Maven](https://www.nytimes.com/2018/04/04/technology/google-letter-ceo-pent… contracts). Another significant change is that Microsoft moves into the top 10 influencers within the Justice Department's Offices, Boards, and Divisions. As we explained in a previous section, this little-known agency is the current home for U.S. international policing program [ICITAP](https://en.wikipedia.org/wiki/ICITAP). We also notice that Google has moved into the top 500 contractors with the FBI (through supplying [FISMA](https://blog.rsisecurity.com/overview-of-the-fisma-certification-and… [Google Apps for Government](https://googleblog.blogspot.com/2010/07/introducing-google-apps…) Google, Facebook, and Twitter's contracting with the U.S. Agency for Global Media -- which is formerly known as the Broadcasting Board of Governors, which itself grew out of the propaganda-focused [Information Agency](https://en.wikipedia.org/wiki/United_States_Information_Agency) -- as well as USAID, and, to a lesser degree, The State Department, became more pronounced. As did minor amounts of contracts with the TSA. The conservative estimation of relative financial flow between major tech companies and various military, prosecutory, law enforcement, and diplomatic organizations in [Table 3](https://techinquiry.org/SiliconValley-Military/#table-3) makes clear that each of these companies is playing at least a minor support role to the U.S. government. And, in the cases of: HP, IBM, Microsoft, Dell, Amazon, Cisco, Palantir, Oracle, NVIDIA, and Anduril, these roles are significant. In light of this data, continuing to claim that Silicon Valley has abandoned Washington would be disingenuous -- even if one technically excluded Microsoft, Amazon, and IBM. Generating Contractor "Embeddings" and Nearest Neighbors The original motivation for mirroring the entire U.S. federal procurement database was to answer the question: > "Are the description text fields and contracting/requesting agencies in procurement data enough to generate leads for companies similar to a given one?" More specifically, the goal was to generate glue between otherwise siloed company profiles through the[nearest neighbors](https://cloud.google.com/solutions/machine-learning/recommendati… resulting from the left factor of a weighted low-rank approximation to the co-occurrence score matrix between terms from the contract description, the vendor receiving the award, and the contracting and requesting agencies. Variants of such an approach are at the heart of many commercial recommendation systems -- which have been frequently criticized as being both [unacceptably opaque](https://www.cnet.com/news/mozilla-is-sharing-youtube-horror-stories… and detrimentally [engagement driven](https://thenextweb.com/google/2019/06/14/youtube-recommendations-to…. We repurpose a basic variant here for the purpose of exploring federal contracting. Given that the resulting recommendations only involve federally mandated records of corporate entities, and our site is not monetarily incentivized by engagement (we are a non-profit and we do not sell ads), we do not forsee any analogues of the typical failure modes. We assert no conclusions about the resulting nearest neighbors, other than that they often contain companies which contract in similar areas to the generating company. They are simply useful for expanding one's breadth of knowledge of companies. Our Approach to Generating Similar Contractor Lists We built such a system on top of [SciPy](https://en.wikipedia.org/wiki/SciPy): a stand-alone Alternating Weighted Least Squares (AWLS) embedding and nearest neighbor extraction [utility](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/3da… and a [driver](https://gitlab.com/tech-inquiry/gov-contract-embeddings/-/blob/3dac… specific to the Federal Procurement Data System. After the co-occurrence score matrix,, is formed -- which, to be clear, is where most the "art" of this model resides [28] -- the objects of interest are the rows of the tall-skinny matrix in the low-rank matrix , which approximates the co-occurrence matrix, in the sense of approximately, locally minimizing given a nonnegative weight matrix with a particular sparse plus rank-one structure, represents the [entrywise product](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)), is the entrywise square-root, and is the [Frobenius norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm). More specifically, the weight matrix is required to be of the form where: - is a binary matrix with the same sparsity pattern as the co-occurrence score matrix,- , - is a length-- column vector of ones, - - , and - The 'th entry of vector is analogous to the [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) weighting for item represented by the - 'th column. We standardized on, , a rank of 150, and 10 alternating iterations. And to keep memory usage and runtimes in check, the rows of the co-occurrence matrix were restricted to the dominant 50,000 terms, 150,000 vendors, and 25,000 agencies. Likewise, the columns were restricted to the dominant 75,000 terms, 150,000 vendors, and 100,000 agencies. Given such a structure for the weight matrix, it was [shown](http://yifanhu.net/PUB/cf.pdf) by Hu, Koren, and Volinsky in 2008 -- incidentally, while working at two companies studied in this report, AT&T and Yahoo! -- that each factor update could be formed in linear time by precomputing a certain 'background' [Gramian](https://en.wikipedia.org/wiki/Gramian_matrix) and sparsely updating it to solve the normal equations for each row's update. After the final iteration of the minimization process is completed, we normalize each row of the matrix to have unit Euclidean norm and then extract 20 to 30 of its nearest neighbors, in a cosine-similarity sense. Examples of Generated Neighbors Each of the "vendor" pages on the website associated with this report, techinquiry.org/lobbying/, contains a list of "Similar Contractors" within the "US Federal Contracting" tab that are generated with the algorithm described above. We show a few of the examples from our most recently trained model, which was trained on roughly the last year and a half of procurement data. Cellebrite Our first example is that of [Cellebrite](https://en.wikipedia.org/wiki/Cellebrite), a "mobile forensics" company which has [allegedly](https://web.archive.org/web/20110423031449/http://redtape.msnbc.… been used by Michigan State Police to conduct unlawful searches. It has also been [reported](https://www.vice.com/en_us/article/aekqjj/cellebrite-sold-phone-h… that Cellebrite sells its software to the governments of Turkey, the United Arab Emirates, and Russia. The five closest neighbors produced for [Cellebrite Inc.](https://techinquiry.org/lobbying/vendor/cellebrite%20inc./), a subsidiary of [Cellebrite DI Ltd.](https://techinquiry.org/lobbying/vendor/cellebrite di ltd./), were: - [Cellebrite DI Ltd.](https://techinquiry.org/lobbying/vendor/cellebrite%20di%20ltd.) - [Grayshift, LLC](https://techinquiry.org/lobbying/vendor/grayshift%2C%20llc) - [Pen-Link, Ltd.](https://techinquiry.org/lobbying/vendor/pen-link%2C%20ltd.) - [Avail Forensics LLC](https://techinquiry.org/lobbying/vendor/avail%20forensics%20llc) - [Berla Corporation](https://techinquiry.org/lobbying/vendor/berla%20corporation) We note that these five neighbors were chosen out of the set of more than 130,000 companies kept by the truncation procedure described above. Given that the parent company was picked up as the nearest neighbor, and the other four are also forensics companies, we believe that these are quality results. Palantir Technologies We return to investigate the answers to our original question: "Which companies contract like Palantir [Technologies]?". Our top-five results are: - [XFinion Inc.](https://techinquiry.org/lobbying/vendor/xfinion%20inc.) - [Ardent Management Consulting, Inc.](https://techinquiry.org/lobbying/vendor/ardent%20management%20consult… - [VIRE Consulting, Inc.](https://techinquiry.org/lobbying/vendor/vire%20consulting%2C%20inc.) - [iWorks Corporation](https://techinquiry.org/lobbying/vendor/iworks%20corporation) - [Alethix, LLC](https://techinquiry.org/lobbying/vendor/alethix%2C%20llc) Anduril Industries Similarly, the top-five results for [Anduril](https://techinquiry.org/lobbying/vendor/anduril industries%2C inc./) are currently: - [Orbital Insight, Inc.](https://techinquiry.org/lobbying/vendor/orbital%20insight%2C%20inc.) - [Solid State Scientific Corporation](https://techinquiry.org/lobbying/vendor/solid%20state%20scient… - [Omni Fed LLC](https://techinquiry.org/lobbying/vendor/omni%20fed%20llc) - [G2 Ops, Inc.](https://techinquiry.org/lobbying/vendor/g2%20ops%2C%20inc.) - [Entheleon Technologies, Inc.](https://techinquiry.org/lobbying/vendor/entheleon%20technologies%2C%2… [Orbital Insight](https://orbitalinsight.com/) markets itself with the blurb: > "Access the most current visibility, intelligence, and transparency of the world’s physical activity ... all on one secure, private geospatial data platform. and[Solid State Scientific Corporation](https://www.sbir.gov/sbc/solid-state-scientific-corporation-0) appears to focus on applying machine learning to Air Force problems, much like Anduril. Microsoft Corporation Given that our nearest neighbors lists are generated entirely from contracting behavior, they produce lists of companies which contract similarly to the given company, rather than companies which are, in a vague sense, generally thought of as similar. This distinction becomes clear when we look at a very large tech company like Microsoft. The most recently produced top-five list of vendors who contract similarly to [Microsoft](https://techinquiry.org/lobbying/vendor/microsoft corporation/) was: - [Four Inc.](https://techinquiry.org/lobbying/vendor/four%20inc.) - [Dynamic Systems, Inc.](https://techinquiry.org/lobbying/vendor/dynamic%20systems%2C%20inc.) - [Forcepoint Federal LLC](https://techinquiry.org/lobbying/vendor/forcepoint%20federal%20llc) - [Emergent LLC](https://techinquiry.org/lobbying/vendor/emergent%2C%20llc) - [Accelera Solutions, Inc.](https://techinquiry.org/lobbying/vendor/accelera%20solutions%2C%20inc… According to Forcepoint Federal's own[brochure](https://www.forcepoint.com/sites/default/files/resources/file…, they are a subsidiary of weapons manufacturer Raytheon formerly known as Websense. Future Extensions of Embeddings There exist namebrand tech companies -- for example, Google -- which do very little direct federal contracting, but perform a large amount of registered lobbying. We can therefore expect to improve the quality of "similar companies" recommendations by extending from our current procurement-only approach to one which includes Senate [OPR LD-1/LD-2](https://www.senate.gov/legislative/Public_Disclosure/database_do… and [LD-203](https://www.senate.gov/legislative/Public_Disclosure/contributions_… filings. We have already included said filings in our website, but have not yet worked them into the embedding/neighbor generation pipeline. Conclusions and Future Work We have demonstrated a framework for converting the Federal Procurement Database System (FPDS) into a set of rankings meant to indicate the degree of (financial) influence of a company, including its subsidiaries, within a particular government agency -- including some which are only indirectly indicated in procurement data. When we applied this methodology to major tech companies, and augmented the award amounts with conservative estimates of subcontract passthrough amounts, we demonstrated that recent narratives decrying a massive divide between Silicon Valley and the military are anecdotal and qualitatively false. We also demonstrated a recommendation system for automatically generating lists of similar contractors for the vast majority of -- more than 130,000 -- U.S. federal contractors. We plan to extend these "embeddings" to incorporate U.S. registered lobbying data, and provide a similar map of tech company lobbying influence, in a future report. One of the fundamental missing features of our website is a means of accepting user-contributed corrections and additions -- ideally coupled with citations that could be verified before their inclusion. We hope to begin experimenting with such interfaces alongside our incorporation of more datasets (e.g., Canadian and European procurement records). Lastly, as a means of both connecting our work to that of Mijente's ["Who's Behind ICE?"](https://mijente.net/2018/10/whos-behind-ice-the-tech-companies-fueling-deportations/) report and demonstrating the breadth of our database's coverage, we provide a [subsection](https://techinquiry.org/SiliconValley-Military/#expanding-whos-… of the appendix that links to our profile page for each company mentioned in their report. A similar [map](https://techinquiry.org/SiliconValley-Military/#expanding-diu) is provided from a collection of contractors with the Defense Innovation Unit -- and its spin-out, [Kessel Run](https://kesselrun.af.mil/) -- that we curated. An autocomplete interface is also provided for each via the "DIU" and "Who's Behind ICE?" radio options on techinquiry.org/lobbying/. We have also incorporated the [Project on Government Oversight](https://en.wikipedia.org/wiki/Project_On_Government_Oversight)'s [Pentagon Revolving Door](https://www.pogo.org/database/pentagon-revolving-door/) and [Federal Contractor Misconduct](https://www.contractormisconduct.org/) databases but refrain from listing them in this report. Acknowledgements The author would like to thank Irene Knapp ([@ireneista](https://twitter.com/ireneista)) for detailed help with this project's database ([Postgres](https://en.wikipedia.org/wiki/PostgreSQL)) and the web server's operating system configuration ([NixOS](https://en.wikipedia.org/wiki/NixOS)) He would also like to separately thank Liz O'Sullivan ([@lizjosullivan](https://twitter.com/lizjosullivan)) and Shauna Gordon-McKeon ([@shauna_gm](https://twitter.com/shauna_gm/)) for detailed suggestions that significantly improved an early draft of this document. Lastly, he would like to thank Cornell's [Center for Applied Mathematics](https://www.cam.cornell.edu/events/cam-colloquium-jack-poulson…, and the [Balsillie School of International Affairs](https://www.balsillieschool.ca/), for hosting him to talk about very early versions of this work. Conflict of Interest Disclosure This work was entirely self-funded. The author acknowledges that they were [formerly](https://www.nytimes.com/2019/04/23/opinion/google-privacy-china.h… an employee at Google, as was [Irene Knapp](https://www.nytimes.com/interactive/2020/02/18/magazine/google-revol…. Likewise, Liz O'Sullivan [formerly](https://www.aclu.org/blog/national-security/targeted-killing/i-qu… worked at Clarafai. Appendix Company Summaries - [HP Inc.](https://en.wikipedia.org/wiki/HP_Inc.) / [Hewlett Packard Enterprise Company](https://en.wikipedia.org/wiki/Hewlett_Packard_Enterprise) (HPE) / [Perspecta Inc.](https://en.wikipedia.org/wiki/Perspecta): Each of these three companies has roots in the (now defunct) [Hewlett-Packard Company](https://en.wikipedia.org/wiki/Hewlett-Packard), which [formed](https://www.dxc.technology/about_us/ds/140019-our_history) its HP Enterprise Services division after acquiring the remnants of [Ross Perot](https://en.wikipedia.org/wiki/Ross_Perot)'s [Electronic Data Systems](https://en.wikipedia.org/wiki/Electronic_Data_Systems) in 2009; HP formally split into a personal computer business (HP) and an enterprise IT company (HPE) in 2015. The following year, HPE acquired the remains of the high-performance graphics company, [SGI](https://en.wikipedia.org/wiki/Silicon_Graphics_International). And in 2017, HPE spun off its enterprise services segment, and merged it with the [Computer Sciences Corporation](https://en.wikipedia.org/wiki/Computer_Sciences_Corporation), to form [DXC Technology](https://en.wikipedia.org/wiki/DXC_Technology). About six months later, DXC spun out its public sector business and merged it with [Vencore](https://en.wikipedia.org/wiki/Vencore), which had previously acquired the North American division of a privatized British intelligence agency, [QinetiQ](https://en.wikipedia.org/wiki/Qinetiq), to form Perspecta. In mid 2019, HPE [acquired](https://www.hpe.com/us/en/newsroom/press-release/2019/05/hpe-to-a… supercomputer manufacturer [Cray](https://en.wikipedia.org/wiki/Cray). Rather than summing the award amounts from HP, HPE, and Perspecta, we individually rank them and then take the highest ranking result (in almost all cases, that result is from Perspecta). - [Deloitte](https://en.wikipedia.org/wiki/Deloitte): Deloitte Touche Tohmatsu Limited is a British accounting firm considered one of the "Big Four". In 2002, Deloitte [acquired](https://www.theglobeandmail.com/report-on-business/andersen-deloi… both the British and Canadian branches of [Arthur Andersen](https://en.wikipedia.org/wiki/Arthur_Andersen), which was convicted of destroying documents in the [Enron scandal](https://en.wikipedia.org/wiki/Enron_scandal). Deloitte has subsequently been under investigation by the UK's [Financial Reporting Council](https://en.wikipedia.org/wiki/Financial_Reporting_Council) for [failing](https://www.belfasttelegraph.co.uk/business/uk-world/deloitte-failed-to-challenge-bosses-at-tech-giant-autonomy-before-sale-38582296.html) in its role as an external auditor of, [Autonomy](https://en.wikipedia.org/wiki/HP_Autonomy), which was the largest software company in the UK at the time of its 2011 acquisition by Hewlett-Packard. Almost immediately after Autonomy was acquired by HP, most of its value was written down as a loss, partly due to said accounting irregularities (in 2017, HP sold Autonomy to [Micro Focus](https://en.wikipedia.org/wiki/Micro_Focus), and, in 2019, Autonomy's former CFO was given a [five year](https://www.telegraph.co.uk/technology/2019/05/13/autonomy-finance-chief-sushovan-hussain-handed-five-year-jail/) prison sentence in the US for fraud). - [Accenture plc](https://en.wikipedia.org/wiki/Accenture): Accenture was the consulting division of [Arthur Andersen](https://en.wikipedia.org/wiki/Arthur_Andersen) from the 1950s until it became a separate unit called Andersen Consulting in 1989, then an independent company in 2000, and then officially became Accenture at the beginning of 2001. After the Enron scandal became public at the end of 2001, Accenture issued a [statement](https://newsroom.accenture.com/news/accenture-response-to-media-… clarifying this timeline of its separation from Arthur Andersen. In 2002, the Government Accountability Office issued a [report](https://www.gao.gov/new.items/d03194r.pdf) critiquing Accenture as one of the four publicly traded federal contractors incorporated in a tax haven -- alongside [McDermott International](https://en.wikipedia.org/wiki/McDermott_International), [Foster Wheeler](https://en.wikipedia.org/wiki/Foster_Wheeler) (now [John Wood Group](https://en.wikipedia.org/wiki/John_Wood_Group)) and [Tyco International](https://en.wikipedia.org/wiki/Tyco_International) (now [Johnson Controls](https://en.wikipedia.org/wiki/Johnson_Controls)) In April of 2019, the DHS's Customs and Border Protection [terminated](https://www.cnn.com/2019/04/05/politics/cbp-terminate-hiring-co… a contract with Accenture's subsidiary, Accenture Federal Services, over having paid 297 million dollars for the company to process just 58 job applicants. - [IBM](https://en.wikipedia.org/wiki/IBM): International Business Machines Corporation (IBM) is one of the largest employers in the world, and its [IBM Cloud](https://en.wikipedia.org/wiki/IBM_Cloud) platform is a major competitor for federal cloud contracts. Some of IBM's inventions include the [relational database](https://en.wikipedia.org/wiki/Relational_database) and the [Structured Query Language](https://en.wikipedia.org/wiki/SQL), which are both employed in this project. But the company has a history of profiting from severe human rights abuses. IBM's most famous founder, [Thomas J. Watson](https://en.wikipedia.org/wiki/Thomas_J._Watson), personally [spearheaded](https://en.wikipedia.org/wiki/IBM_and_the_Holocaust) the company's role in [Nazi Germany](https://en.wikipedia.org/wiki/Nazi_Germany)'s cataloguing systems for exterminating its Jewish population. More recently, IBM's international sales of ["safe city"](https://www-03.ibm.com/press/us/en/pressrelease/38152.wss) technology have involved the construction of video surveillance networks for strongman [Rodrigo Duterte](https://theintercept.com/2019/03/20/rodrigo-duterte-ibm-surveillan…. In 2011, IBM acquired military intelligence software company [i2 Group](https://en.wikipedia.org/wiki/I2_Limited), which received $10M from Palantir after alleging [fraudulent](https://www.reuters.com/article/idUS121866071120110217) theft of its intellectual property. Conflicts with IBM's i2 intensified over Palantir's attempted replacement of the Army's intelligence integration software, [DCGS-A](https://en.wikipedia.org/wiki/Distributed_Common_Ground_System) -- members of the Army, and Congress, [suggested](https://www.defenseone.com/technology/2016/07/war-over-soon-be-o… DCGS-A weaknesses were the cause of the U.S. [bombing](https://en.wikipedia.org/wiki/Kunduz_hospital_airstrike) of a Doctors without Borders hospital in Kunduz. Palantir ultimately [won](https://www.defensenews.com/land/2019/03/29/palantir-who-successfully-… DCGS-A. - [AT&T Inc.](https://en.wikipedia.org/wiki/AT%26T): As chronicled in [The Idea Factory](https://en.wikipedia.org/wiki/The_Idea_Factory), AT&T played a foundational role in the development of computing and communication equipment, beginning with its deployment in the early twentieth century of vacuum tubes as a means of amplifying long-distance telephone call signals. The company's long-running monopoly was partially due to its large-scale construction of US [communication networks](https://publisher.abc-clio.com/9781851097371/) during both World Wars. One of the positive byproducts of their monopoly was a golden era of innovation at its [Bell Labs](https://en.wikipedia.org/wiki/Bell_Labs), whose end loosely coincided with the 1984 [antitrust decree](https://en.wikipedia.org/wiki/Breakup_of_the_Bell_System) breaking up the Bell System (the pieces have since reassembled into: AT&T, Verizon, and CenturyLink). The past two decades have revealed a darker side to AT&T's close relationship with the US government: numerous [documents](https://www.nytimes.com/2015/08/16/us/politics/att-helped-nsa-sp… and [whistleblowers](https://www.wired.com/2013/06/nsa-whistleblower-klein/) have shown that the company has been eager to comply with large-scale [warrantless surveillance](https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs… and [call monitoring](https://www.pbs.org/newshour/politics/report-att-cooperated-ext…. - [Microsoft Corporation](https://en.wikipedia.org/wiki/Microsoft): One of the "Big Five" tech companies, it famously won the -- still [heavily contested](https://www.washingtonpost.com/business/2020/02/10/amazon-depose… -- [Joint Enterprise Defense Infrastructure](https://en.wikipedia.org/wiki/Joint_Enterprise_Defense_Infr… (JEDI) cloud competition with its [Azure](https://en.wikipedia.org/wiki/Microsoft_Azure) platform. Along with Google and Yahoo (now Verizon), it cofounded the [Global Network Initiative](https://en.wikipedia.org/wiki/Global_Network_Initiative) (GNI) in 2008, partially to avoid [Congressional intervention](https://www.forbes.com/sites/larrydownes/2011/03/30/why-no-on… in response to [numerous](https://www.hrw.org/reports/2006/china0806/5.htm) international [human rights](https://www.computerweekly.com/news/2240076194/Microsoft-pulls-blog… [violations](https://www.nytimes.com/2007/11/07/technology/07yahoo.html). Despite its GNI membership, Microsoft followed Google's [lead](https://money.cnn.com/2006/01/27/news/international/davos_3_fortune/) by complying with Chinese government [censorship](https://www.wired.com/story/us-companies-help-censor-internet-c… and surveillance demands when it launched its search engine, Bing, in China in 2009. Microsoft further took stewardship of the [dissent suppression](https://www.scmp.com/news/china/article/2180620/linkedin-adher… mechanisms in professional networking site LinkedIn after acquiring it in 2016. A 2010 [report](https://www.giswatch.org/en/sexual-rights/privatising-censorship-on… had similarly revealed that Bing was extensively suppressing LBGT-related content in Arab countries. And, after Google [pulled back](https://publicpolicy.googleblog.com/2010/01/new-approach-to-china.htm… from proactively suppressing dissent in its Chinese localization of its Search product in early 2010, Microsoft's founder, Bill Gates, personally [attacked](https://bits.blogs.nytimes.com/2010/01/25/microsofts-bill-gates-d… Google's principled stand as hypocritical. - [Verizon Communications Inc.](https://en.wikipedia.org/wiki/Verizon_Communications): Verizon has its [roots](https://www.verizon.com/about/our-company/history-and-timeline) in one of the "Baby Bells" resulting from the 1984 breakup of AT&T: Bell Atlantic. It expanded into Silicon Valley through the purchase of [AOL](https://techcrunch.com/2015/05/12/verizon-aol-4-4b/) in 2015, and then the non-[Alibaba](https://en.wikipedia.org/wiki/Alibaba_Group) assets of [Yahoo!](https://www.nytimes.com/2016/07/25/business/yahoo-sale.html) in 2016, and the combined the two into a subsidiary known as Oath, which then rebranded as [Verizon Media Group](https://www.theverge.com/2018/11/5/18064172/verizon-oath-media-group…. In 2006, Nobel Peace Prize laureate [Liu Xiaobo](https://en.wikipedia.org/wiki/Liu_Xiaobo) wrote an [open letter](https://www.hrw.org/reports/2006/china0806/14.htm) to Yahoo!'s then chairman, Jerry Yang, arguing that the company had caused the arrest of journalist Shi Tao as part of their negotiations for their stake in Alibaba. As was widely [reported](https://www.newyorker.com/news/amy-davidson/the-n-s-a-verizon-sca… in 2013, Verizon's Business Network Services division had been turning over "all call detail records" to the National Security Agency. Similar "warrantless wiretapping" had previously been [reported](https://www.wired.com/2010/01/fbi-att-verizon-violated-wiretappin…. We also note that subsidiary [Verizon Business](http://www.documentcloud.org/documents/1342118-stratfor-computer-… performed an analysis of the hacking of intelligence consulting company Stratfor that is ["key"](https://gizmodo.com/dojs-new-wikileaks-indictment-has-significant-convenie-1844166678) to the Department of Justice's current indictment of WikiLeaks. - [Dell Technologies, Inc.](https://en.wikipedia.org/wiki/Dell_Technologies): Dell Technologies is the result of the 2016 [merger](https://www.nytimes.com/2018/02/02/technology/dell-ipo-vmware-merge… between [Dell](https://en.wikipedia.org/wiki/Dell) and enterprise storage company [EMC Corporation](https://www.theregister.com/2009/08/30/richard_egan_obituary/) (now [Dell EMC](https://en.wikipedia.org/wiki/Dell_EMC)) before going public at the end of 2018, it was the largest privately held company in the world. Reuters [reported](https://www.reuters.com/article/us-usa-security-snowden-dell/snow… that whistleblower Edward Snowden began downloading documents while working as a contractor for Dell, and Snowden's [memoir](https://en.wikipedia.org/wiki/Permanent_Record_(autobiography)) stated that he arrived at Dell through its acquisition of [Perot Systems](https://en.wikipedia.org/wiki/Perot_Systems) (which was sold to [NTT Data](https://en.wikipedia.org/wiki/NTT_Data) in 2016). [Pivotal Software](https://en.wikipedia.org/wiki/Pivotal_Software), a subsidiary of virtualization company [VMWare](https://en.wikipedia.org/wiki/VMware), which is in turn majority owned by Dell Technologies, has built [drone strike](https://www.youtube.com/watch?v=u5tn76xjAfc&feature=youtu.b) and [fuel allocation](https://www.fastcompany.com/40588729/the-air-force-learned-to-c… tools in collaboration with the Defense Innovation Unit spin-off, [Kessel Run](https://kesselrun.af.mil/). These contracts are the subject of our near year-long [FOI](https://en.wikipedia.org/wiki/Freedom_of_Information_Act_(United_State… process that should complete in the coming weeks. - [Cisco Systems, Inc.](https://en.wikipedia.org/wiki/Cisco_Systems): A networking company founded by two Stanford computer scientists and named after San Francisco, Cisco was the most valuable company in the world just before the dot-com bubble burst. In the early 2000s, Cisco sold the Chinese government routers that served as the [backbone](http://cyberlaw.stanford.edu/blog/2015/03/china%E2%80%99s-golden-… of its ["Great Firewall"](https://en.wikipedia.org/wiki/Golden_Shield_Project) -- [allegedly](https://hrlf.net/doe-vs-cisco-systems/) going so far as [designing](https://www.eff.org/deeplinks/2016/04/ciscos-latest-attempt-dodg… a custom video surveillance and censorship module to target a dissident religious group. And, in 2013, [documents](https://www.techdirt.com/articles/20140518/17433327281/cisco-goe… released by whistleblower Edward Snowden detailed the NSA's process for implanting malware into Cisco products, leading to substantial declines in international revenue. - [Palantir Technologies, Inc.](https://en.wikipedia.org/wiki/Palantir_Technologies): Founded in Palo Alto in 2003 by [Peter Thiel](https://en.wikipedia.org/wiki/Peter_Thiel) as an [extension](https://www.forbes.com/sites/andygreenberg/2013/08/14/agent-of-i… of PayPal's fraud detection tools into military and law enforcement contracting, the company ended up paying $10M to IBM's intelligence subsidiary, [i2 Group](https://en.wikipedia.org/wiki/I2_Limited), to settle [allegations](https://en.wikipedia.org/wiki/Peter_Thiel) that Palantir fraudulently stole i2's intellectual property. As noted above, Palantir intensely competed -- and ultimately [won](https://www.defensenews.com/land/2019/03/29/palantir-who-successfully-… -- against a consortium of defense contractors (including IBM's i2) to replace the Army's [DCGS-A](https://en.wikipedia.org/wiki/Distributed_Common_Ground_System) intelligence synthesis tool with its [Gotham](https://en.wikipedia.org/wiki/Palantir_Technologies#Palantir_Gotham) software (our FOI for Palantir's [DIU contract](https://techinquiry.org/FOIA/OSD-F-0129-ResponsiveDocuments.pdf) to build a LOGistics Common Operating Picture (LOGCOP) is for analogous functionality). In the early days of the company, Gotham was presented as a tool for data-driven civic engagement -- which are disturbingly similar to some of the goals of the [website](https://techinquiry.org/lobbying/) behind this report -- via their (now defunct) procurement and lobbying analysis website [AnalyzeThe.US](https://web.archive.org/web/20131113211219/https://analyzeth…. One of their other products, Metropolis -- which they ultimately replaced with [Foundry](https://www.quora.com/What-are-the-main-differences-between-the-Pa… -- was used by [JPMorgan](https://en.wikipedia.org/wiki/JPMorgan_Chase) to [spy](https://www.bloomberg.com/features/2018-palantir-peter-thiel/) on its employees' emails, phonecalls, web-browsing patterns, and GPS locations to attempt to predict disgruntled employees ("insider threats"). And, as [reported](https://mijente.net/2019/05/palantir-arresting-families/) by LatinX organizing group [Mijente](https://mijente.net/), Palantir, through its contracts with ICE's [Homeland Security Investigations](https://www.ice.gov/hsi) (HSI) unit, has "played a key role" in the Trump administration's accelerated arrest and deportation of immigrants. One can find a detailed explanation of the history of the company's products from their CTO [here](https://www.quora.com/What-are-the-main-differences-between-the-Palan…. - [Oracle Corporation](https://en.wikipedia.org/wiki/Oracle_Corporation): Oracle, was incorporated in 1977 and named after the founders' work with the CIA -- who was their [first customer](https://www.sfgate.com/bayarea/article/Oracle-s-coziness-with-gov… -- on the [Project ORACLE](https://www.cia.gov/library/readingroom/document/cia-rdp80-01794r00… data storage system. As detailed by [numerous](https://www.nytimes.com/2002/01/31/opinion/IHT-a-single-national-… [sources](https://www.penguinrandomhouse.com/books/157388/the-naked-crowd-by…, surveillance work remained critical to the company's growth, with $2.5B of its [2003 revenue](https://paleofuture.gizmodo.com/larry-ellisons-oracle-started-as-a… coming from federal contracts. Oracle's intense growth led to a major downfall in the integrity of the U.S. Justice Department when the former Attorney General, [John Ashcroft](https://en.wikipedia.org/wiki/John_Ashcroft), left office in 2005 to form a lobbying shop, The Ashcroft Group, LLC (now [TAG Holdings LLC](https://tagholdings.com/)) which accepted [$220,000](https://www.tmcnet.com/usubmit/2006/jan/1275410.htm) from Oracle to help [reverse](https://www.chicagotribune.com/news/ct-xpm-2005-11-19-0511190256-s… an anti-trust hold on an acquisition that he had put in place. In 2012, Oracle was [banned](https://www.informationweek.com/regulations/feds-banish-oracle-from… from a General Service Administration procurement vehicle after paying $200M to the GSA the previous year to settle an overbilling lawsuit through the [False Claims Act](https://en.wikipedia.org/wiki/False_Claims_Act) (five years earlier, Oracle's [PeopleSoft](https://en.wikipedia.org/wiki/PeopleSoft) paid $99M to the GSA due to another overbilling case). Nevertheless, Oracle has been a [litigious](https://www.nextgov.com/cio-briefing/2020/01/oracles-jedi-appeal… participant in the ongoing Joint Enterprise Defense Infrastructure bid protests, having hoped that its [Oracle Cloud](https://en.wikipedia.org/wiki/Oracle_Cloud) platform would be at least partially adopted rather than the award going entirely to one of its rivals. - [Anduril Industries, Inc.](https://www.anduril.com/): Roughly three years after Palmer Luckey's virtual reality company, [Oculus](https://en.wikipedia.org/wiki/Oculus_VR), was acquired by Facebook in 2014, Facebook was [ordered](https://www.zdnet.com/article/facebooks-oculus-ordered-to-pay-500-… to pay $500M to [ZeniMax Media](https://en.wikipedia.org/wiki/ZeniMax_Media) for his failure to comply with a non-disclosure agreement -- so they [fired](https://www.wsj.com/articles/why-did-facebook-fire-a-top-executive-h… him. He was subsequently [inspired](https://www.wired.com/story/palmer-luckey-anduril-border-wall/) by Peter Thiel's [Founders Fund](https://en.wikipedia.org/wiki/Founders_Fund) to transition into weapons contracting. One of the first projects of his new company, Anduril, was a contract with the Department of Homeland Security building surveillance systems for the U.S.-Mexico border. And, after Google publicly cancelled its work building object tracking AI's on top of drone surveillance footage for the Air Force's Joint Artificial Intelligence Center -- called Project Maven -- Anduril was [one of](https://theintercept.com/2019/03/09/anduril-industries-project-maven-pa… the companies to take over the work. Other work has included [counter-UAS](https://www.prnewswire.com/news-releases/anduril-industries-an… and [satellite surveillance](https://www.usaspending.gov/#/award/CONT_AWD_FA880820C0006_97… systems.

1 0

minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster
by Zenaan Harkness 08 Jul '20

08 Jul '20

Anyone here able to answer this annoying buffer issue on bulk copies? ----- Forwarded message from Zenaan Harkness <zenaan(a)freedbms.net> ----- From: Zenaan Harkness <zenaan(a)freedbms.net> To: debian-user(a)lists.debian.org Date: Wed, 8 Jul 2020 17:52:06 +1000 Subject: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster On Wed, Jul 08, 2020 at 08:26:18AM +0200, Martin Reissner wrote: > For the sake of completeness, I seem to have solved it after some more > research and it turned out to be systemd, as following bug that was > reported for Centos7 seems to be applying to Debian Buster as well: > > https://github.com/systemd/systemd/issues/9276 > > Luckily the workaround mentioned in: > > https://github.com/systemd/systemd/issues/9276#issuecomment-442514543 > > applies here as well and by setting this to the desired swappiness value > and rebooting the system so far it seems to work as before and swapping > out is only done if it can't be avoided. Your thread brings to mind an "$AGE_OF_LINUX_KERNEL + 1 year" years old Linux kernel bona fydee, bulk copy bug bear: Doing a bulk copy e.g. using cp or rsync, to copy GiBs far greater than available RAM from one disk to another, results in ... <sinister musak please> Total Annihilation of Desktop Latency <sinister laugh> This happens to this day even on the latest Linux kernels on Sid such as 5.7.0-1-amd64 to pick an entirely random example... and has been literally bugging me since the beginnings of my Debian journey ~20 years ago. Note that in this extremely humble user's experience, even ionice --class Idle ... is not even able to quell the Memory Beast's all consuming hunger games for all possible RAM and then some, in it's insatiable appetite for all the page cache in... <more sinister musak please> The Entire Universe <sinister laugh> What needs to happen ™©® What is needed is an option somewhere, strictly abided by, where "For the following command" only say 1MiB of buffer and Page Cache, in TOTAL, is able to be used by that command ever, until its completion. In this way my 16GiBs of RAM will __finally__ be left to its rightful owners Firefox, mutt, and my XOrg's cursor, since to me as an extremely humble desktop user, these are the only things in the world that I ever care about, and I esPECIALLY want to NOT care that a 1,700 GiB movie disk backup takes an extra 5 or 10 % longer than the NINE hours it already takes across its USB 2 connection! Seriously, is there a way to stop bulk copies from eternally flushing my $Desktop's cached pages down the drain minute after blessful minute, hour after gratitude filled hour? The heavens will truly be pleased .. ----- End forwarded message -----

2 1

P*ssy Grabber in Chief once complimented his niece
by Zig the N.g 08 Jul '20

08 Jul '20

So America's püssy grabber in chief once complimented his niece, apparently telling her many years ago ‘Holy S***, Mary. You're Stacked’. Since the locker room tapes so severely damaged Trump's reputation, there's no doubt that this explosive book by his own niece revealing these lurid compliments is just going to totally ruin any possible chance of Trump's re-election. Or something... ‘Holy S***, Mary. You're Stacked’: Niece Claims Trump Ogled Her Body in Forthcoming Tell-All Book https://sputniknews.com/us/202007081079824916-holy-s-mary-youre-stacked-nie… The tantalising details of the soon-to-be released book by Mary Trump, the current US President’s niece, have been making headlines, as the tell-all volume will ostensibly feature an in-depth look into Trump family dynamics. ... [How absolutely lurid!] [Or as the Millennialls would say, "How absolutely!"]

1 0

DHT algorithm cheat sheet?
by Zenaan Harkness 08 Jul '20

08 Jul '20

Anyone got personal experience with DHT re-hashing/ re-Distributing the table as nodes join/leave, and in particular any problem you had to solve or work around? https://en.wikipedia.org/wiki/Distributed_hash_table

4 4