De-anonymizing Programmers via Code Stylometry

De-anonymizing Programmers via Code Stylometry Aylin Caliskan-Islam Drexel University Arvind Narayanan Princeton University Richard Harang U.S. Army Research Laboratory Clare Voss U.S. Army Research Laboratory Andrew Liu University of Maryland Fabian Yamaguchi University of Goettingen Rachel Greenstadt Drexel University Abstract Source code authorship attribution is a significant privacy threat to anonymous code contributors. However, it may also enable attribution of successful attacks from code left behind on an infected system, or aid in resolving copyright, copyleft, and plagiarism issues in the programming fields. In this work, we investigate machine learning methods to de-anonymize source code authors of C/C++ using coding style. Our Code Stylometry Feature Set is a novel representation of coding style found in source code that reflects coding style from properties derived from abstract syntax trees. Our random forest and abstract syntax tree-based approach attributes more authors (1,600 and 250) with significantly higher accuracy (94% and 98%) on a larger data set (Google Code Jam) than has been previously achieved. Furthermore, these novel features are robust, difficult to obfuscate, and can be used in other programming languages, such as Python. We also find that (i) the code resulting from difficult programming tasks is easier to attribute than easier tasks and (ii) skilled programmers (who can complete the more difficult tasks) are easier to attribute than less skilled programmers. 1 Introduction Do programmers leave fingerprints in their source code? That is, does each programmer have a distinctive “coding style”? Perhaps a programmer has a preference for spaces over tabs, or while loops over for loops, or, more subtly, modular rather than monolithic code. These questions have strong privacy and security implications. Contributors to open-source projects may hide their identity whether they are Bitcoin’s creator or just a programmer who does not want her employer to know about her side activities. They may live in a regime that prohibits certain types of software, such as censorship circumvention tools. For example, an Iranian pro- grammer was sentenced to death in 2012 for developing photo sharing software that was used on pornographic websites [31]. The flip side of this scenario is that code attribution may be helpful in a forensic context, such as detection of ghostwriting, a form of plagiarism, and investigation of copyright disputes. It might also give us clues about the identity of malware authors. A careful adversary may only leave binaries, but others may leave behind code written in a scripting language or source code downloaded into the breached system for compilation. While this problem has been studied previously, our work represents a qualitative advance over the state of the art by showing that Abstract Syntax Trees (ASTs) carry authorial ‘fingerprints.’ The highest accuracy achieved in the literature is 97%, but this is achieved on a set of only 30 programmers and furthermore relies on using programmer comments and larger amounts of training data [12, 14]. We match this accuracy on small programmer sets without this limitation. The largest scale experiments in the literature use 46 programmers and achieve 67.2% accuracy [10]. We are able to handle orders of magnitude more programmers (1,600) while using less training data with 92.83% accuracy. Furthermore, the features we are using are not trivial to obfuscate. We are able to maintain high accuracy while using commercial obfuscators. While abstract syntax trees can be obfuscated to an extent, doing so incurs significant overhead and maintenance costs. Contributions. First, we use syntactic features for code stylometry. Extracting such features requires parsing of incomplete source code using a fuzzy parser to generate an abstract syntax tree. These features add a component to code stylometry that has so far remained almost completely unexplored. We provide evidence that these features are more fundamental and harder to obfuscate. Our complete feature set consists of a comprehensive set of around 120,000 layout-based, lexical, and syntactic features. With this complete feature set we are able to achieve a significant increase in accuracy compared to previous work. Second, we show that we can scale our method to 1,600 programmers without losing much accuracy. Third, this method is not specific to C or C++, and can be applied to any programming language. We collected C++ source of thousands of contestants from the annual international competition “Google Code Jam”. A bagging (portmanteau of “bootstrap aggregating”) classifier - random forest was used to attribute programmers to source code. Our classifiers reach 98% accuracy in a 250-class closed world task, 93% accuracy in a 1,600-class closed world task, 100% accuracy on average in a two-class task. Finally, we analyze various attributes of programmers, types of programming tasks, and types of features that appear to influence the success of attribution. We identified the most important 928 features out of 120,000; 44% of them are syntactic, 1% are layout-based and the rest of the features are lexical. 8 training files with an average of 70 lines of code is sufficient for training when using the lexical, layout and syntactic features. We also observe that programmers with a greater skill set are more easily identifiable compared to less advanced programmers and that a programmer’s coding style is more distinctive in implementations of difficult tasks as opposed to easier tasks. The remainder of this paper is structured as follows. We begin by introducing applications of source code authorship attribution considered throughout this paper in Section 2, and present our AST-based approach in Section 3. We proceed to give a detailed overview of the experiments conducted to evaluate our method in Section 4 and discuss the insights they provide in Section 5. Section 6 presents related work, and Section 7 concludes. 2 Jam competition, as we discuss in Section 4.1. Doubtless there will be additional challenges in using our techniques for digital forensics or any of the other real-world applications. We describe some known limitations in Section 5. Programmer De-anonymization. In this scenario, the analyst is interested in determining the identity of an anonymous programmer. For example, if she has a set of programmers who she suspects might be Bitcoin’s creator, Satoshi, and samples of source code from each of these programmers, she could use the initial versions of Bitcoin’s source code to try to determine Satoshi’s identity. Of course, this assumes that Satoshi did not make any attempts to obfuscate his or her coding style. Given a set of probable programmers, this is considered a closedworld machine learning task with multiple classes where anonymous source code is attributed to a programmer. This is a threat to privacy for open source contributors who wish to remain anonymous. Ghostwriting Detection. Ghostwriting detection is related to but different from traditional plagiarism detection. We are given a suspicious piece of code and one or more candidate pieces of code that the suspicious code may have been plagiarized from. This is a well-studied problem, typically solved using code similarity metrics, as implemented by widely used tools such as MOSS [6], JPlag [25], and Sherlock [24]. For example, a professor may want to determine whether a student’s programming assignment has been written by a student who has previously taken the class. Unfortunately, even though submissions of the previous year are available, the assignments may have changed considerably, rendering code-similarity based methods ineffective. Luckily, stylometry can be applied in this setting—we find the most stylistically similar piece of code from the previous year’s corpus and bring both students in for gentle questioning. Given the limited set of students, this can be considered a closed-world machine learning problem. Software Forensics. In software forensics, the analyst assembles a set of candidate programmers based on previously collected malware samples or online code repositories. Unfortunately, she cannot be sure that the anonymous programmer is one of the candidates, making this an open world classification problem as the test sample might not belong to any known category. Copyright Investigation. Theft of code often leads to copyright disputes. Informal arrangements of hired programming labor are very common, and in the absence of a written contract, someone might claim a piece of code was her own after it was developed for hire and delivered. A dispute between two parties is thus a two-class classification problem; we assume that labeled code from both programmers is available to the forensic expert. Motivation Throughout this work, we consider an analyst interested in determining the programmer of an anonymous fragment of source code purely based on its style. To do so, the analyst only has access to labeled samples from a set of candidate programmers, as well as from zero or more unrelated programmers. The analyst addresses this problem by converting each labeled sample into a numerical feature vector, in order to train a machine learning classifier, that can subsequently be used to determine the code’s programmer. In particular, this abstract problem formulation captures the following five settings and corresponding applications (see Table 1). The experimental formulations are presented in Section 4.2. We emphasize that while these applications motivate our work, we have not directly studied them. Rather, we formulate them as variants of a machine-learning (classification) problem. Our data comes from the Google Code 2 Authorship Verification. Finally, we may suspect that a piece of code was not written by the claimed programmer, but have no leads on who the actual programmer might be. This is the authorship verification problem. In this work, we take the textbook approach and model it as a two-class problem where positive examples come from previous works of the claimed programmer and negative examples come from randomly selected unrelated programmers. Alternatively, anomaly detection could be employed in this setting, e.g., using a one-class support vector machine [see 30]. As an example, a recent investigation conducted by Verizon [17] on a US company’s anomalous virtual private network traffic, revealed an employee who was outsourcing her work to programmers in China. In such cases, training a classifier on employee’s original code and that of random programmers, and subsequently testing pieces of recent code, could demonstrate if the employee was the actual programmer. In each of these applications, the adversary may try to actively modify the program’s coding style. In the software forensics application, the adversary tries to modify code written by her to hide her style. In the copyright and authorship verification applications, the adversary modifies code written by another programmer to match his own style. Finally, in the ghostwriting application, two of the parties may collaborate to modify the style of code written by one to match the other’s style. Application De-anonymization Ghostwriting detection Software forensics Copyright investigation Authorship verification Learner Multiclass Multiclass Multiclass Two-class Two/One-class Comments Closed world Closed world Open world Closed world Open world different features to represent both syntax and structure of program code (Section 3.2) and finally, we train a random forest classifier for classification of previously unseen source files (Section 3.3). In the following sections, we will discuss each of these steps in detail and outline design decisions along the way. The code for our approach is made available as open-source to allow other researchers to reproduce our results1 . 3.1 To date, methods for source code authorship attribution focus mostly on sequential feature representations of code such as byte-level and feature level n-grams [8, 13]. While these models are well suited to capture naming conventions and preference of keywords, they are entirely language agnostic and thus cannot model author characteristics that become apparent only in the composition of language constructs. For example, an author’s tendency to create deeply nested code, unusually long functions or long chains of assignments cannot be modeled using n-grams alone. Addressing these limitations requires source code to be parsed. Unfortunately, parsing C/C++ code using traditional compiler front-ends is only possible for complete code, i.e., source code where all identifiers can be resolved. This severely limits their applicability in the setting of authorship attribution as it prohibits analysis of lone functions or code fragments, as is possible with simple n-gram models. As a compromise, we employ the fuzzy parser Joern that has been designed specifically with incomplete code in mind [32]. Where possible, the parser produces abstract syntax trees for code fragments while ignoring fragments that cannot be parsed without further information. The produced syntax trees form the basis for our feature extraction procedure. While they largely preserve the information required to create n-grams or bagof-words representations, in addition, they allow a wealth of features to be extracted that encode programmer habits visible in the code’s structure. As an example, consider the function foo as shown in Figure 1, and a simplified version of its corresponding abstract syntax tree in Figure 2. The function contains a number of common language constructs found in many programming languages, such as if-statements (line 3 and 7), return-statements (line 4, 8 and 10), and function call expressions (line 6). For each of these constructs, the abstract syntax tree contains a corresponding node. While the leaves of the tree make classical syntactic features such as keywords, identifiers and operators accessible, inner nodes represent operations showing Evaluation Section 4.2.1 Section 4.2.1 Section 4.2.2 Section 4.2.3 Section 4.2.4 Table 1: Overview of Applications for Code Stylometry We emphasize that code stylometry that is robust to adversarial manipulation is largely left to future work. However, we hope that our demonstration of the power of features based on the abstract syntax tree will serve as the starting point for such research. 3 Fuzzy Abstract Syntax Trees De-anonymizing Programmers One of the goals of our research is to create a classifier that automatically determines the most likely author of a source file. Machine learning methods are an obvious choice to tackle this problem, however, their success crucially depends on the choice of a feature set that clearly represents programming style. To this end, we begin by parsing source code, thereby obtaining access to a wide range of possible features that reflect programming language use (Section 3.1). We then define a number of 1 https://github.com/calaylin/CodeStylometry 3 Function int foo CompoundStmt If RelExpr (<) x Figure 1: Sample Code Listing Or UnaryOp (-) RelExpr (>) 1 x Feature Condition Return EqExpr (!=) ret 0 UnaryOp (-) 1 Else int Return ret Assign(=) ret 1 Call bar Args MAX Definition WordUnigramTF Term frequency of word unigrams in source code ln(numkeyword/ Log of the number of occurrences of keylength) word divided by file length in characters, where keyword is one of do, else-if, if, else, switch, for or while ln(numTernary/ Log of the number of ternary operators dilength) vided by file length in characters ln(numTokens/ Log of the number of word tokens divided length) by file length in characters ln(numComments/ Log of the number of comments divided by length) file length in characters ln(numLiterals/ Log of the number of string, character, and length) numeric literals divided by file length in characters ln(numKeywords/ Log of the number of unique keywords length) used divided by file length in characters ln(numFunctions/ Log of the number of functions divided by length) file length in characters ln(numMacros/ Log of the number of preprocessor direclength) tives divided by file length in characters nestingDepth Highest degree to which control statements and loops are nested within each other branchingFactor Branching factor of the tree formed by converting code blocks of files into nodes avgParams The average number of parameters among all functions stdDevNumParams The standard deviation of the number of parameters among all functions avgLineLength The average length of each line stdDevLineLength The standard deviation of the character lengths of each line Feature Extraction Analyzing coding style using machine learning approaches is not possible without a suitable representation of source code that clearly expresses program style. To address this problem, we present the Code Stylometry Feature Set (CSFS), a novel representation of source code developed specifically for code stylometry. Our feature set combines three types of features, namely lexical features, layout features and syntactic features. Lexical and layout features are obtained from source code while the syntactic features can only be obtained from ASTs. We now describe each of these feature types in detail. 3.2.1 Return Decl x Figure 2: Corresponding Abstract Syntax Tree how these basic elements are combined to form expressions and statements. In effect, the nesting of language constructs can also be analyzed to obtain a feature set representing the code’s structure. 3.2 0 Condition If Lexical and Layout Features We begin by extracting numerical features from the source code that express preferences for certain identifiers and keywords, as well as some statistics on the use of functions or the nesting depth. Lexical and layout features can be calculated from the source code, without having access to a parser, with basic knowledge of the programming language in use. For example, we measure the number of functions per source line to determine the programmer’s preference of longer over shorter functions. Furthermore, we tokenize the source file to obtain the number of occurrences of each token, so called word unigrams. Table 2 gives an overview of lexical features. In addition, we consider layout features that represent code-indentation. For example, we determine whether the majority of indented lines begin with whitespace or tabulator characters, and we determine the ratio of whitespace to the file size. Table 3 gives a detailed description of these features. Count dynamic* 7 1 1 1 1 1 1 1 1 1 1 1 1 1 *About 55,000 for 250 authors with 9 files. Table 2: Lexical Features 3.2.2 Syntactic Features The syntactic feature set describes the properties of the language dependent abstract syntax tree, and keywords. Calculating these features requires access to an abstract syntax tree. All of these features are invariant to changes in source-code layout, as well as comments. Table 4 gives an overview of our syntactic features. We obtain these features by preprocessing all C++ source files in the dataset to produce their abstract syntax trees. 4 Feature Definition ln(numTabs/length) Log of the number of tab characters divided by file length in characters Log of the number of space characters divided by file length in characters Log of the number of empty lines divided by file length in characters, excluding leading and trailing lines between lines of text The ratio between the number of whitespace characters (spaces, tabs, and newlines) and non-whitespace characters A boolean representing whether the majority of code-block braces are preceded by a newline character A boolean representing whether the majority of indented lines begin with spaces or tabs ln(numSpaces/length) ln(numEmptyLines/ length) whiteSpaceRatio newLineBefore OpenBrace tabsLeadLines particular type of node. The inverse document frequency is calculated by dividing the number of authors in the dataset by the number of authors that use that particular node. Consequently, we are able to capture how rare of a node it is and weight it more according to its rarity. The maximum depth of an abstract syntax tree reflects the deepest level a programmer nests a node in the solution. The average depth of the AST nodes shows how nested or deep a programmer tends to use particular structural pieces. And lastly, term frequency of each C++ keyword is calculated. Each of these features is written to a feature vector to represent the solution file of a specific author and these vectors are later used in training and testing by machine learning classifiers. Count 1 1 1 1 1 1 Table 3: Layout Features 3.3 Using the feature set presented in the previous section, we can now express fragments of source code as numerical vectors, making them accessible to machine learning algorithms. We proceed to perform feature selection and train a random forest classifier capable of identifying the most likely author of a code fragment. An abstract syntax tree is created for each function in the code. There are 58 node types in the abstract syntax tree (see Appendix A) produced by Joern [33]. Feature Definition MaxDepthASTNode ASTNodeBigramsTF ASTNodeTypesTF Maximum depth of an AST node Term frequency AST node bigrams Term frequency of 58 possible AST node type excluding leaves ASTNodeTypesTFIDF Term frequency inverse document frequency of 58 possible AST node type excluding leaves ASTNodeTypeAvgDep Average depth of 58 possible AST node types excluding leaves cppKeywords Term frequency of 84 C++ keywords CodeInASTLeavesTF Term frequency of code unigrams in AST leaves CodeInASTLeaves Term frequency inverse document freTFIDF quency of code unigrams in AST leaves CodeInASTLeaves Average depth of code unigrams in AvgDep AST leaves Classification Count 1 dynamic* 58 3.3.1 58 Due to our heavy use of unigram term frequency and TF/IDF measures, and the diversity of individual terms in the code, our resulting feature vectors are extremely large and sparse, consisting of tens of thousands of features for hundreds of classes. The dynamic Code stylometry feature set, for example, produced close to 120,000 features for 250 authors with 9 solution files each. In many cases, such feature vectors can lead to overfitting (where a rare term, by chance, uniquely identifies a particular author). Extremely sparse feature vectors can also damage the accuracy of random forest classifiers, as the sparsity may result in large numbers of zerovalued features being selected during the random subsampling of the features to select a best split. This reduces the number of ‘useful’ splits that can be obtained at any given node, leading to poorer fits and larger trees. Large, sparse feature vectors can also lead to slowdowns in model fitting and evaluation, and are often more difficult to interpret. By selecting a smaller number of more informative features, the sparsity in the feature vector can be greatly reduced, thus allowing the classifier to both produce more accurate results and fit the data faster. We therefore employed a feature selection step using WEKA’s information gain [26] criterion, which evaluates the difference between the entropy of the distribution of classes and the entropy of the conditional distribution of classes given a particular feature: 58 84 dynamic** dynamic** dynamic** *About 45,000 for 250 authors with 9 files. **About 7,000 for 250 authors with 9 files. **About 4,000 for 150 authors with 6 files. **About 2,000 for 25 authors with 9 files. Table 4: Syntactic Features The AST node bigrams are the most discriminating features of all. AST node bigrams are two AST nodes that are connected to each other. In most cases, when used alone, they provide similar classification results to using the entire feature set. The term frequency (TF) is the raw frequency of a node found in the abstract syntax trees for each file. The term frequency inverse document frequency (TFIDF) of nodes is calculated by multiplying the term frequency of a node by inverse document frequency. The goal in using the inverse document frequency is normalizing the term frequency by the number of authors actually using that Feature Selection IG(A, Mi ) = H(A) − H(A|Mi ) 5 (1) where A is the class corresponding to an author, H is Shannon entropy, and Mi is the ith feature of the dataset. Intuitively, the information gain can be thought of as measuring the amount of information that the observation of the value of feature i gives about the class label associated with the example. To reduce the total size and sparsity of the feature vector, we retained only those features that individually had non-zero information gain. (These features can be referred to as IG-CSFS throughout the rest of the paper.) Note that, as H(A|Mi ) ≤ H(A), information gain is always non-negative. While the use of information gain on a variable-per-variable basis implicitly assumes independence between the features with respect to their impact on the class label, this conservative approach to feature selection means that we only use features that have demonstrable value in classification. To validate this approach to feature selection, we applied this method to two distinct sets of source code files, and observed that sets of features with non-zero information gain were nearly identical between the two sets, and the ranking of features was substantially similar between the two. This suggests that the application of information gain to feature selection is producing a robust and consistent set of features (see Section 4 for further discussion). All the results are calculated by using CSFS and IG-CSFS. Using IG-CSFS on all experiments demonstrates how these features generalize to different datasets that are larger in magnitude. One other advantage of IGCSFS is that it consists of a few hundred features that result in non-sparse feature vectors. Such a compact representation of coding style makes de-anonymizing thousands of programmers possible in minutes. 3.3.2 of features can be found using the out of bag (oob) error estimate, or the error estimate derived from those samples not selected for training on a given tree. During classification, each test example is classified via each of the trained decision trees by following the binary decisions made at each node until a leaf is reached, and the results are then aggregated. The most populous class can be selected as the output of the forest for simple classification, or classifications can be ranked according to the number of trees that ‘voted’ for a label when performing relaxed attribution (see Section 4.3.4). We employed random forests with 300 trees, which empirically provided the best trade-off between accuracy and processing time. Examination of numerous oob values across multiple fits suggested that (logM) + 1 random features (where M denotes the total number of features) at each split of the decision trees was in fact optimal in all of the experiments (listed in Section 4), and was used throughout. Node splits were selected based on the information gain criteria, and all trees were grown to the largest extent possible, without pruning. The data was analyzed via k-fold cross-validation, where the data was split into training and test sets stratified by author (ensuring that the number of code samples per author in the training and test sets was identical across authors). k varies according to datasets and is equal to the number of instances present from each author. The cross-validation procedure was repeated 10 times, each with a different random seed. We report the average results across all iterations in the results, ensuring that they are not biased by improbably easy or difficult to classify subsets. Random Forest Classification 4 We used the random forest ensemble classifier [7] as our classifier for authorship attribution. Random forests are ensemble learners built from collections of decision trees, each of which is grown by randomly sampling N training samples with replacement, where N is the number of instances in the dataset. To reduce correlation between trees, features are also subsampled; commonly (logM) + 1 features are selected at random (without replacement) out of M, and the best split on these (logM) + 1 features is used to split the tree nodes. The number of selected features represents one of the few tuning parameters in random forests: increasing the number of features increases the correlation between trees in the forest which can harm the accuracy of the overall ensemble, however increasing the number of features that can be chosen at each split increases the classification accuracy of each individual tree making them stronger classifiers with low error rates. The optimal range of number Evaluation In the evaluation section, we present the results to the possible scenarios formulated in the problem statement and evaluate our method. The corpus section gives an overview of the data we collected. Then, we present the main results to programmer de-anonymization and how it scales to 1,600 programmers, which is an immediate privacy concern for open source contributors that prefer to remain anonymous. We then present the training data requirements and efficacy of types of features. The obfuscation section discusses a possible countermeasure to programmer de-anonymization. We then present possible machine learning formulations along with the verification section that extends the approach to an open world problem. We conclude the evaluation with generalizing the method to other programming languages and providing software engineering insights. 6 4.1 (ii) Within each year, we ordered the corpus files by the round in which they were written, and by the problem within a round, as all competitors proceed through the same sequence of rounds in that year. As a result, we performed stratified cross validation on each program file by the year it was written, by the round in which the program was written, by the problems solved in the round, and by the author’s highest round completed in that year. Some limitations of this dataset are that it does not allow us to assess the effect of style guidelines that may be imposed on a project or attributing code with multiple/mixed programmers. We leave these interesting questions for future work, but posit that our improved results with basic stylometry make them worthy of study. Corpus One concern in source code authorship attribution is that we are actually identifying differences in coding style, rather than merely differences in functionality. Consider the case where Alice and Bob collaborate on an open source project. Bob writes user interface code whereas Alice works on the network interface and backend analytics. If we used a dataset derived from their project, we might differentiate differences between frontend and backend code rather than differences in style. In order to minimize these effects, we evaluate our method on the source code of solutions to programming tasks from the international programming competition Google Code Jam (GCJ), made public in 2008 [2]. The competition consists of algorithmic problems that need to be solved in a programming language of choice. In particular, this means that all programmers solve the same problems, and hence implement similar functionality, a property of the dataset crucial for code stylometry analysis. The dataset contains solutions by professional programmers, students, academics, and hobbyists from 166 countries. Participation statistics are similar over the years. Moreover, it contains problems of different difficulty, as the contest takes place in several rounds. This allows us to assess whether coding style is related to programmer experience and problem difficulty. The most commonly used programming language was C++, followed by Java, and Python. We chose to investigate source code stylometry on C++ and C because of their popularity in the competition and having a parser for C/C++ readily available [32]. We also conducted some preliminary experimentation on Python. A validation dataset was created from 2012’s GCJ competition. Some problems had two stages, where the second stage involved answering the same problem in a limited amount of time and for a larger input. The solution to the large input is essentially a solution for the small input but not vice versa. Therefore, collecting both of these solutions could result in duplicate and identical source code. In order to avoid multiple entries, we only collected the small input versions’ solutions to be used in our dataset. The programmers had up to 19 solution files in these datasets. Solution files have an average of 70 lines of code per programmer. To create our experimental datasets that are discussed in further detail in the results section; (i) We first partitioned the corpus of files by year of competition. The “main” dataset includes files drawn from 2014 (250 programmers). The “validation” dataset files come from 2012, and the “multi-year” dataset files come from years 2008 through 2014 (1,600 programmers). 4.2 Applications In this section, we will go over machine learning task formulations representing five possible real-world applications presented in Section 2. 4.2.1 Multiclass Closed World Task This section presents our main experiment—deanonymizing 250 programmers in the difficult scenario where all programmers solved the same set of problems. The machine learning task formulation for de-anonymizing programmers also applies to ghostwriting detection. The biggest dataset formed from 2014’s Google Code Jam Competition with 9 solution files to the same problem had 250 programmers. These were the easiest set of 9 problems, making the classification more challenging (see Section 4.3.6). We reached 91.78% accuracy in classifying 250 programmers with the Code Stylometry Feature Set. After applying information gain and using the features that had information gain, the accuracy was 95.08%. We also took 250 programmers from different years and randomly selected 9 solution files for each one of them. We used the information gain features obtained from 2014’s dataset to see how well they generalize. We reached 98.04% accuracy in classifying 250 programmers. This is 3% higher than the controlled large dataset’s results. The accuracy might be increasing because of using a mixed set of Google Code Jam problems, which potentially contains the possible solutions’ properties along with programmers’ coding style and makes the code more distinct. We wanted to evaluate our approach and validate our method and important features. We created a dataset from 2012’s Google Code Jam Competition with 250 programmers who had the solutions to the same set of 9 problems. We extracted only the features that had positive information gain in 2014’s dataset that was used as 7 Where V j (i) = 1 if the jth tree voted for class i and 0 otherwise, and |T | f denotes the total number of trees in forest f . Note that by construction, ∑i P(Ci ) = 1 and P(Ci ) ≥ 0 ∀ i, allowing us to treat P(Ci ) as a probability measure. There was one correct classification made with 13.7% confidence. This suggests that we can use a threshold between 13.7% and 15% confidence level for verification, and manually analyze the classifications that did not pass the confidence threshold or exclude them from results. We picked an aggressive threshold of 15% and to validate it, we trained a random forest classifier on the same set of 30 programmers 270 code samples. We tested on 150 different files from the programmers in the training set. There were 6 classifications below the 15% threshold and two of them were misclassified. We took another set of 420 test files from 30 programmers that were not in the training set. All the files from the 30 programmers were attributed to one of the 30 programmers in the training set since this is a closed world classification task, however, the highest confidence level in these classifications was 14.7%. The 15% threshold catches all the instances that do not belong to the programmers in the suspect set, gets rid of 2 misclassifications and 4 correct classifications. Consequently, when we see a classification with less than a threshold value, we can reject the classification and attribute the test instance to an unknown suspect. the main dataset to implement the approach. The classification accuracy was 96.83%, which is higher than the 95.07% accuracy obtained in 2014’s dataset. The high accuracy of validation results in Table 5 show that we identified the important features of code stylometry and found a stable feature set. This feature set does not necessarily represent the exact features for all possible datasets. For a given dataset that has ground truth information on authorship, following the same approach should generate the most important features that represent coding style in that particular dataset. A = #programmers, F = max #problems completed N = #problems included in dataset (N ≤ F) A = 250 from 2014 A = 250 from 2012 A = 250 all years F = 9 from 2014 F = 9 from 2014 F ≥ 9 all years N=9 N=9 N=9 Average accuracy after 10 iterations with IG-CSFS features 95.07% 96.83% 98.04% Table 5: Validation Experiments 4.2.2 Mutliclass Open World Task The experiments in this section can be used in software forensics to find out the programmer of a piece of malware. In software forensics, the analyst does not know if source code belongs to one of the programmers in the candidate set of programmers. In such cases, we can classify the anonymous source code, and if the majority number of votes of trees in the random forest is below a certain threshold, we can reject the classification considering the possibility that it might not belong to any of the classes in the training data. By doing so, we can scale our approach to an open world scenario, where we might not have encountered the suspect before. As long as we determine a confidence threshold based on training data [30], we can calculate the probability that an instance belongs to one of the programmers in the set and accordingly accept or reject the classification. We performed 270 classifications in a 30-class problem using all the features to determine the confidence threshold based on the training data. The accuracy was 96.67%. There were 9 misclassifications and all of them were classified with less than 15% confidence by the classifier. The class probability or classification confidence that source code fragment C is of class i is calculated by taking the percentage of trees in the random forest that voted for that particular class, as follows2: P(Ci ) = ∑ j V j (i) |T | f 4.2.3 Two-class Closed World Task Source code author identification could automatically deal with source code copyright disputes without requiring manual analysis by an objective code investigator. A copyright dispute on code ownership can be resolved by comparing the styles of both parties claiming to have generated the code. The style of the disputed code can be compared to both parties’ other source code to aid in the investigation. To imitate such a scenario, we took 60 different pairs of programmers, each with 9 solution files. We used a random forest and 9-fold cross validation to classify two programmers’ source code. The average classification accuracy using CSFS set is 100.00% and 100.00% with the information gain features. 4.2.4 Two-class/One-class Open World Task Another two-class machine learning task can be formulated for authorship verification. We suspect Mallory of plagiarizing, so we mix in some code of hers with a large sample of other people, test, and see if the disputed code gets classified as hers or someone else’s. If it gets classified as hers, then it was with high probability really written by her. If it is classified as someone else’s, it really was someone else’s code. This could be an open (2) 8 world problem and the person that originally wrote the code could be a previously unknown programmer. This is a two-class problem with classes Mallory and others. We train on Mallory’s solutions to problems a, b, c, d, e, f, g, h. We also train on programmer A’s solution to problem a, programmer B’s solution to problem b, programmer C’s solution to problem c, programmer D’s solution to problem d, programmer E’s solution to problem e, programmer F’s solution to problem f, programmer G’s solution to problem g, programmer H’s solution to problem h and put them in one class called ABCDEFGH. We train a random forest classifier with 300 trees on classes Mallory and ABCDEFGH. We have 6 test instances from Mallory and 6 test instances from another programmer ZZZZZZ, who is not in the training set. These experiments have been repeated in the exact same setting with 80 different sets of programmers ABCDEFGH, ZZZZZZ and Mallorys. The average classification accuracy for Mallory using the CSFS set is 100.00%. ZZZZZZ’s test instances are classified as programmer ABCDEFGH 82.04% of the time, and classified as Mallory for the rest of the time while using the CSFS. Depending on the amount of false positives we are willing to accept, we can change the operating point on the ROC curve. These results are also promising for use in cases where a piece of code is suspected to be plagiarized. Following the same approach, if the classification result of the piece of code is someone other than Mallory, that piece of code was with very high probability not written by Mallory. 4.3 Additional Insights 4.3.1 Scaling scales. We are able to de-anonymize 1,600 programmers using 32GB memory within one hour. Alternately, we can use 40 trees and get nearly the same accuracy (within 0.5%) in a few minutes. Figure 3: Large Scale De-anonymization 4.3.2 Training Data and Features We selected different sets of 62 programmers that had F solution files, from 2 up to 14. Each dataset has the solutions to the same set of F problems by different sets of programmers. Each dataset consisted of programmers that were able to solve exactly F problems. Such an experimental setup makes it possible to investigate the effect of programmer skill set on coding style. The size of the datasets were limited to 62, because there were only 62 contestants with 14 files. There were a few contestants with up to 19 files but we had to exclude them since there were not enough programmers to compare them. The same set of F problems were used to ensure that the coding style of the programmer is being classified and not the properties of possible solutions of the problem itself. We were able to capture personal programming style since all the programmers are coding the same functionality in their own ways. Stratified F-fold cross validation was used by training on everyone’s (F − 1) solutions and testing on the F th problem that did not appear in the training set. As a result, the problems in the test files were encountered for the first time by the classifier. We used a random forest with 300 trees and (logM)+1 features with F-fold stratified cross validation, first with the Code Stylometry Feature Set (CSFS) and then with the CSFS’s features that had information gain. Figure 4 shows the accuracy from 13 different sets of 62 programmers with 2 to 14 solution files, and consequently 1 to 13 training files. The CSFS reaches an optimal training set size at 9 solution files, where the classifier trains on 8 (F − 1) solutions. In the datasets we constructed, as the number of files increase and problems from more advanced rounds are included, the average line of code (LOC) per file also increases. The average lines of code per source code in the dataset is 70. Increased number of lines of code might have a positive effect on the accuracy but at the same time it reveals programmer’s choice of program We collected a larger dataset of 1,600 programmers from various years. Each of the programmers had 9 source code samples. We created 7 subsets of this large dataset in differing sizes, with 250, 500, 750, 1,000, 1,250, 1,500, and 1,600 programmers. These subsets are useful to understand how well our approach scales. We extracted the specific features that had information gain in the main 250 programmer dataset from this large dataset. In theory, we need to use more trees in the random forest as the number of classes increase to decrease variance, but we used fewer trees compared to smaller experiments. We used 300 trees in the random forest to run the experiments in a reasonable amount of time with a reasonable amount of memory. The accuracy did not decrease too much when increasing the number of programmers. This result shows that information gain features are robust against changes in class and are important properties of programmers’ coding styles. The following Figure 3 demonstrates how well our method 9 detectable change in the performance of the classifier for this sample. The results are summarized in Table 6. We took the maximum number of programmers, 20, that had solutions to 9 problems in C and obfuscated the code (see example in Appendix B) using a much more sophisticated open source obfuscator called Tigress [1]. In particular, Tigress implements function virtualization, an obfuscation technique that turns functions into interpreters and converts the original program into corresponding bytecode. After applying function virtualization, we were less able to effectively de-anonymize programmers, so it has potential as a countermeasure to programmer de-anonymization. However, this obfuscation comes at a cost. First of all, the obfuscated code is neither readable nor maintainable, and is thus unsuitable for an open source project. Second, the obfuscation adds significant overhead (9 times slower) to the runtime of the program, which is another disadvantage. The accuracy with the information gain feature set on the obfuscated dataset is reduced to 67.22%. When we limit the feature set to AST node bigrams, we get 18.89% accuracy, which demonstrates the need for all feature types in certain scenarios. The accuracy on the same dataset when the code is not obfuscated is 95.91%. Figure 4: Training Data length in implementing the same functionality. On the other hand, the average line of code of the 7 easier (76 LOC) or difficult problems (83 LOC) taken from contestants that were able to complete 14 problems, is higher than the average line of code (68) of contestants that were able to solve only 7 problems. This shows that programmers with better skills tend to write longer code to solve Google Code Jam problems. The mainstream idea is that better programmers write shorter and cleaner code which contradicts with line of code statistics in our datasets. Google Code Jam contestants are supposed to optimize their code to process large inputs with faster performance. This implementation strategy might be leading to advanced programmers implementing longer solutions for the sake of optimization. We took the dataset with 62 programmers each with 9 solutions. We get 97.67% accuracy with all the features and 99.28% accuracy with the information gain features. We excluded all the syntactic features and the accuracy dropped to 88.89% with all the non-syntactic features and 88.35% with the information gain features of the non-syntactic feature set. We ran another experiment using only the syntactic features and obtained 96.06% with all the syntactic features and 96.96% with the information gain features of the syntactic feature set. Most of the classification power is preserved with the syntactic features, and using non-syntactic features leads to a significant decline in accuracy. 4.3.3 Obfuscator Programmers Stunnix Stunnix Tigress Tigress 20 20 20 20 Lang Results w/o Obfuscation C++ 98.89% C++ 98.89*% C 93.65% C 95.91*% *Information gain features Results w/ Obfuscation 100.00% 98.89*% 58.33% 67.22*% Table 6: Effect of Obfuscation on De-anonymization 4.3.4 Relaxed Classification The goal here is to determine whether it is possible to reduce the number of suspects using code stylometry. Reducing the set of suspects in challenging cases, such as having too many suspects, would reduce the effort required to manually find the actual programmer of the code. In this section, we performed classification on the main 250 programmer dataset from 2014 using the information gain features. The classification was relaxed to a set of top R suspects instead of exact classification of the programmer. The relaxed factor R varied from 1 to 10. Instead of taking the highest majority vote of the decisions trees in the random forest, the highest R majority vote decisions were taken and the classification result was considered correct if the programmer was in the set of top R highest voted classes. The accuracy does not improve much after the relaxed factor is larger than 5. Obfuscation We took a dataset with 9 solution files and 20 programmers and obfuscated the code using an off-the-shelf C++ obfuscator called stunnix [3]. The accuracy with the information gain code stylometry feature set on the obfuscated dataset is 98.89%. The accuracy on the same dataset when the code is not obfuscated is 100.00%. The obfuscator refactored function and variable names, as well as comments, and stripped all the spaces, preserving the functionality of code without changing the structure of the program. Obfuscating the data produced little 10 programming languages by implementing the layout and lexical features as well as using a fuzzy parser. Lang. Python Python Programmers 23 229 Classification IG 87.93% 79.71% 53.91% 39.16% Top-5 99.52% 75.69% Top-5 IG 96.62 55.46 Figure 5: Relaxed Classification with 250 Programmers Table 7: Generalizing to Other Programming Languages 4.3.5 4.3.6 Generalizing the Method Features derived from ASTs can represent coding styles in various languages. These features are applicable in cases when lexical and layout features may be less discriminating due to formatting standards and reliance on whitespace and other ‘lexical’ features as syntax, such as Python’s PEP8 formatting. To show that our method generalizes, we collected source code of 229 Python programmers from GCJ’s 2014 competition. 229 programmers had exactly 9 solutions. Using only the Python equivalents of syntactic features listed in Table 4 and 9-fold cross-validation, the average accuracy is 53.91% for top-1 classification, 75.69% for top-5 relaxed attribution. The largest set of programmers to all work on the same set of 9 problems was 23 programmers. The average accuracy in identifying these 23 programmers is 87.93% for top-1 and 99.52% for top-5 relaxed attribution. The same classification tasks using the information gain features are also listed in Table 7. The overall accuracy in datasets composed of Python code are lower than C++ datasets. In Python datasets, we only used syntactic features from ASTs that were generated by a parser that was not fuzzy. The lack of quantity and specificity of features accounts for the decreased accuracy. The Python dataset’s information gain features are significantly fewer in quantity, compared to C++ dataset’s information gain features. Information gain only keeps features that have discriminative value all on their own. If two features only provide discriminative value when used together, then information gain will discard them. So if a lot of the features for the Python set are only jointly discriminative (and not individually discriminative), then the information gain criteria may be removing features that in combination could effectively discriminate between authors. This might account for the decrease when using information gain features. While in the context of other results in this paper the results in Table 7 appear lackluster, it is worth noting that even this preliminary test using only syntactic features has comparable performance to other prior work at a similar scale (see Section 6 and Table 9), demonstrating the utility of syntactic features and the relative ease of generating them for novel programming languages. Nevertheless, a CSFS equivalent feature set can be generated for other Software Engineering Insights We wanted to investigate if programming style is consistent throughout years. We found the contestants that had the same username and country information both in 2012 and 2014. We assumed that these are the same people but there is a chance that they might be different people. In 2014, someone else might have picked up the same username from the same country and started using it. We are going to ignore such a ground truth problem for now and assume that they are the same people. We took a set of 25 programmers from 2012 that were also contestants in 2014’s competition. We took 8 files from their submissions in 2012 and trained a random forest classifier with 300 trees using CSFS. We had one instance from each one of the contestants from 2014. The correct classification of these test instances from 2014 is 96.00%. The accuracy dropped to 92.00% when using only information gain features, which might be due to the aggressive elimination of pairs of features that are jointly discriminative. These 25 programmers’ 9 files from 2014 had a correct classification accuracy of 98.04%. These results indicate that coding style is preserved up to some degree throughout years. To investigate problem difficulty’s effect on coding style, we created two datasets from 62 programmers that had exactly 14 solution files. Table 8 summarizes the following results. A dataset with 7 of the easier problems out of 14 resulted in 95.62% accuracy. A dataset with 7 of the more difficult problems out of 14 resulted in 99.31% accuracy. This might imply that more difficult coding tasks have a more prevalent reflection of coding style. On the other hand, the dataset that had 62 programmers with exactly 7 of the easier problems resulted in 91.24% accuracy, which is a lot lower than the accuracy obtained from the dataset whose programmers were able to advance to solve 14 problems. This might indicate that, programmers who are advanced enough to answer 14 problems likely have more unique coding styles compared to contestants that were only able to solve the first 7 problems. To investigate the possibility that contestants who are able to advance further in the rounds have more unique coding styles, we performed a second round of experiments on comparable datasets. We took the dataset with 11 12 solution files and 62 programmers. A dataset with 6 of the easier problems out of 12 resulted in 91.39% accuracy. A dataset with 6 of the more difficult problems out of 12 resulted in 94.35% accuracy. These results are higher than the dataset whose programmers were only able to solve the easier 6 problems. The dataset that had 62 programmers with exactly 6 of the easier problems resulted in 90.05% accuracy. that coding style is reflected more through difficult programming tasks. This might indicate that programmers come up with unique solutions and preserve their coding style more when problems get harder. On the other hand, programmers with a better skill set have a prevalent coding style which can be identified more easily compared to contestants who were not able to advance as far in the competition. This might indicate that as programmers become more advanced, they build a stronger coding style compared to novices. There is another possibility that maybe better programmers start out with a more unique coding style. Effects of Obfuscation. A malware author or plagiarizing programmer might deliberately try to hide his source code by obfuscation. Our experiments indicate that our method is resistant to simple off-the-shelf obfuscators such as stunnix, that make code look cryptic while preserving functionality. The reason for this success is that the changes stunnix makes to the code have no effect on syntactic features, e.g., removal of comments, changing of names, and stripping of whitespace. In contrast, sophisticated obfuscation techniques such as function virtualization hinder de-anonymization to some degree, however, at the cost of making code unreadable and introducing a significant performance penalty. Unfortunately, unreadability of code is not acceptable for open-source projects, while it is no problem for attackers interested in covering their tracks. Developing methods to automatically remove stylometric information from source code without sacrificing readability is therefore a promising direction for future research. Limitations. We have not considered the case where a source file might be written by a different author than the stated contestant, which is a ground truth problem that we cannot control. Moreover, it is often the case that code fragments are the work of multiple authors. We plan to extend this work to study such datasets. To shed light on the feasibility of classifying such code, we are currently working with a dataset of git commits to open source projects. Our parser works on code fragments rather than complete code, consequently we believe this analysis will be possible. Another fundamental problem for machine learning classifiers are mimicry attacks. For example, our classifier may be evaded by an adversary by adding extra dummy code to a file that closely resembles that of another programmer, albeit without affecting the program’s behavior. This evasion is possible, but trivial to resolve when an analysts verifies the decision. Finally, we cannot be sure whether the original author is actually a Google Code Jam contestant. In this case, we can detect those by a classify and then verify approach as explained in Stolerman et al.’s work [30]. Each classification could go through a verification step A = #programmers, F = max #problems completed N = #problems included in dataset (N ≤ F) A = 62 F = 14 N=7 F=7 N=7 N=7 F = 12 N=6 F=6 N=6 N=6 Average accuracy after 10 iterations while using CSFS 99.31% 95.62%2 91.24%1 94.35% 91.39%2 90.05%1 Average accuracy after 10 iterations while using IG CSFS 99.38% 98.62%2 96.77%1 96.69% 95.43%2 94.89%1 1 Drop in accuracy due to programmer skill set. 2 Coding style is more distinct in more difficult tasks. Table 8: Effect of Problem Difficulty on Coding Style 5 Discussion In this section, we discuss the conclusions we draw from the experiments outlined in the previous section, limitations, as well as questions raised by our results. In particular, we discuss the difficulty of the different settings considered, the effects of obfuscation, and limitations of our current approach. Problem Difficulty. The experiment with random problems from random authors among seven years most closely resembles a real world scenario. In such an experimental setting, there is a chance that instead of only identifying authors we are also identifying the properties of a specific problem’s solution, which results in a boost in accuracy. In contrast, our main experimental setting where all authors have only answered the nine easiest problems is possibly the hardest scenario, since we are training on the same set of eight problems that all the authors have algorithmically solved and try to identify the authors from the test instances that are all solutions of the 9th problem. On the upside, these test instances help us precisely capture the differences between individual coding style that represent the same functionality. We also see that such a scenario is harder since the randomized dataset has higher accuracy. Classifying authors that have implemented the solution to a set of difficult problems is easier than identifying authors with a set of easier problems. This shows 12 to eliminate instances where the classifier’s confidence is below a threshold. After the verification step, instances that do not belong to the set of known authors can be separated from the dataset to be excluded or for further manual analysis. 6 tural features to achieve higher accuracies at larger scales and the first to study how code obfuscation affects code stylometry. There has also been some code stylometry work that focused on manual analysis and case studies. Spafford and Weeber [29] suggest that use of lexical features such as variable names, formatting and comments, as well as some syntactic features such as usage of keywords, scoping and presence of bugs could aid in source code attribution but they do not present results or a case study experiment with a formal approach. Gray et al. [15] identify three categories in code stylometry: the layout of the code, variable and function naming conventions, types of data structures being used and also the cyclomatic complexity of the code obtained from the control flow graph. They do not mention anything about the syntactic characteristics of code, which could potentially be a great marker of coding style that reveals the usage of programming language’s grammar. Their case study is based on a manual analysis of three worms, rather than a statistical learning approach. Hayes and Offutt [16] examine coding style in source code by their consistent programmer hypothesis. They focused on lexical and layout features, such as the occurrence of semicolons, operators and constants. Their dataset consisted of 20 programmers and the analysis was not automated. They concluded that coding style exists through some of their features and professional programmers have a stronger programming style compared to students. In our results in Section 4.3.6, we also show that more advanced programmers have a more identifying coding style. There is also a great deal of research on plagiarism detection which is carried out by identifying the similarities between different programs. For example, there is a widely used tool called Moss that originated from Stanford University for detecting software plagiarism. Moss [6] is able to analyze the similarities of code written by different programmers. Rosenblum et al. [27] present a novel program representation and techniques that automatically detect the stylistic features of binary code. Related Work Our work is inspired by the research done on authorship attribution of unstructured or semi-structured text [5, 22]. In this section, we discuss prior work on source code authorship attribution. In general, such work (Table 9) looks at smaller scale problems, does not use structural features, and achieves lower accuracies than our work. The highest accuracies in the related work are achieved by Frantzeskou et al. [12, 14]. They used 1,500 7-grams to reach 97% accuracy with 30 programmers. They investigated the high-level features that contribute to source code authorship attribution in Java and Common Lisp. They determined the importance of each feature by iteratively excluding one of the features from the feature set. They showed that comments, layout features and naming patterns have a strong influence on the author classification accuracy. They used more training data (172 line of code on average) than us (70 lines of code). We replicated their experiments on a 30 programmer subset of our C++ data set, with eleven files containing 70 lines of code on average and no comments. We reach 76.67% accuracy with 6-grams, and 76.06% accuracy with 7-grams. When we used a 6 and 7-gram feature set on 250 programmers with 9 files, we got 63.42% accuracy. With our original feature set, we get 98% accuracy on 250 programmers. The largest number of programmers studied in the related work was 46 programmers with 67.2% accuracy. Ding and Samadzadeh [10] use statistical methods for authorship attribution in Java. They show that among lexical, keyword and layout properties, layout metrics have a more important role than others which is not the case in our analysis. There are also a number of smaller scale, lower accuracy approaches in the literature [9, 11, 18–21, 28], shown in Table 9, all of which we significantly outperform. These approaches use a combination of layout and lexical features. The only other work to explore structural features is by Pellin [23], who used manually parsed abstract syntax trees with an SVM that has a tree based kernel to classify functions of two programmers. He obtains an average of 73% accuracy in a two class classification task. His approach explained in the white paper can be extended to our approach, so it is the closest to our work in the literature. This work demonstrates that it is non-trivial to use ASTs effectively. Our work is the first to use struc- Related Work Pellin [23] MacDonell et al.[21] Frantzeskou et al.[14] Burrows et al. [9] Elenbogen and Seliya [11] Kothari et al. [18] Lange and Mancoridis [20] Krsul and Spafford [19] Frantzeskou et al. [14] Ding and Samadzadeh [10] This work This work This work This work # of Programmers 2 7 8 10 12 12 20 29 30 46 8 35 250 1,600 Results 73% 88.00% 100.0% 76.78% 74.70% 76% 75% 73% 96.9% 67.2% 100.00% 100.00% 98.04% 92.83% Table 9: Comparison to Previous Results 13 7 Conclusion and Future Work References [1] The tigress diversifying c virtualizer, http://tigress.cs.arizona.edu. Source code stylometry has direct applications for privacy, security, software forensics, plagiarism, copyright infringement disputes, and authorship verification. Source code stylometry is an immediate concern for programmers who want to contribute code anonymously because de-anonymization is quite possible. We introduce the first principled use of syntactic features along with lexical and layout features to investigate style in source code. We can reach 94% accuracy in classifying 1,600 authors and 98% accuracy in classifying 250 authors with eight training files per class. This is a significant increase in accuracy and scale in source code authorship attribution. In particular, it shows that source code authorship attribution with the Code Stylometry Feature Set scales even better than regular stylometric authorship attribution, as these methods can only identify individuals in sets of 50 authors with slightly over 90% accuracy [see 4]. Furthermore, this performance is achieved by training on only 550 lines of code or eight solution files, whereas classical stylometric analysis requires 5,000 words. Additionally, our results raise a number of questions that motivate future research. First, as malicious code is often only available in binary format, it would be interesting to investigate whether syntactic features can be partially preserved in binaries. This may require our feature set to be improved in order to incorporate information obtained from control flow graphs. Second, we would also like to see if classification accuracy can be further increased. For example, we would like to explore whether using features that have joint information gain alongside features that have information gain by themselves improve performance. Moreover, designing features that capture larger fragments of the abstract syntax tree could provide improvements. These changes (along with adding lexical and layout features) may provide significant improvements to the Python results and help generalize the approach further. Finally, we would like to investigate whether code can be automatically normalized to remove stylistic information while preserving functionality and readability. 8 [2] Google code jam, https://code.google.com/codejam, 2014. [3] Stunnix, http://www.stunnix.com/prod/cxxo/, November 2014. [4] A BBASI , A., AND C HEN , H. Writeprints: A stylometric approach to identity-level identification and similarity detection in cyberspace. ACM Trans. Inf. Syst. 26, 2 (2008), 1–29. [5] A FROZ , S., B RENNAN , M., AND G REENSTADT, R. Detecting hoaxes, frauds, and deception in writing style online. In Security and Privacy (SP), 2012 IEEE Symposium on (2012), IEEE, pp. 461–475. [6] A IKEN , A., ET AL . Moss: A system for detecting software plagiarism. University of California–Berkeley. See www. cs. berkeley. edu/aiken/moss. html 9 (2005). [7] B REIMAN , L. Random forests. Machine Learning 45, 1 (2001), 5–32. [8] B URROWS , S., AND TAHAGHOGHI , S. M. Source code authorship attribution using n-grams. In Proc. of the Australasian Document Computing Symposium (2007). [9] B URROWS , S., U ITDENBOGERD , A. L., AND T URPIN , A. Application of information retrieval techniques for source code authorship attribution. In Database Systems for Advanced Applications (2009), Springer, pp. 699–713. [10] D ING , H., AND S AMADZADEH , M. H. Extraction of java program fingerprints for software authorship identification. Journal of Systems and Software 72, 1 (2004), 49–57. [11] E LENBOGEN , B. S., AND S ELIYA , N. Detecting outsourced student programming assignments. Journal of Computing Sciences in Colleges 23, 3 (2008), 50–57. [12] F RANTZESKOU , G., M AC D ONELL , S., S TAMATATOS , E., AND G RITZALIS , S. Examining the significance of high-level programming features in source code author classification. Journal of Systems and Software 81, 3 (2008), 447–460. [13] F RANTZESKOU , G., S TAMATATOS , E., G RITZALIS , S., C HASKI , C. E., AND H OWALD , B. S. Identifying authorship by byte-level n-grams: The source code author profile (scap) method. International Journal of Digital Evidence 6, 1 (2007), 1–18. [14] F RANTZESKOU , G., S TAMATATOS , E., G RITZALIS , S., AND K ATSIKAS , S. Effective identification of source code authors using byte-level information. In Proceedings of the 28th International Conference on Software Engineering (2006), ACM, pp. 893–896. Acknowledgments This material is based on work supported by the ARO (U.S. Army Research Office) Grant W911NF-14-10444, the DFG (German Research Foundation) under the project DEVIL (RI 2469/1-1), and AWS in Education Research Grant award. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the ARO, DFG, and AWS. [15] G RAY, A., S ALLIS , P., AND M AC D ONELL , S. Software forensics: Extending authorship analysis techniques to computer programs. [16] H AYES , J. H., AND O FFUTT, J. Recognizing authors: an examination of the consistent programmer hypothesis. Software Testing, Verification and Reliability 20, 4 (2010), 329–356. [17] I NOCENCIO , R. U.s. programmer outsources own job to china, surfs cat videos, January 2013. 14 A [18] KOTHARI , J., S HEVERTALOV, M., S TEHLE , E., AND M AN CORIDIS , S. A probabilistic approach to source code authorship identification. In Information Technology, 2007. ITNG’07. Fourth International Conference on (2007), IEEE, pp. 243–248. Appendix: Keywords and Node Types AdditiveExpression AndExpression Argument [19] K RSUL , I., AND S PAFFORD , E. H. Authorship analysis: Identifying the author of a program. Computers & Security 16, 3 (1997), 233–257. ArgumentList ArrayIndexing AssignmentExpr BitAndExpression BlockStarter BreakStatement Callee CallExpression CastExpression [20] L ANGE , R. C., AND M ANCORIDIS , S. Using code metric histograms and genetic algorithms to perform author identification for software forensics. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (2007), ACM, pp. 2082–2089. CastTarget CompoundStatement Condition ConditionalExpression ContinueStatement DoStatement ElseStatement EqualityExpression ExclusiveOrExpression Expression ExpressionStatement ForInit ForStatement FunctionDef GotoStatement Identifier IdentifierDecl IdentifierDeclStatement IdentifierDeclType IfStatement IncDec IncDecOp InclusiveOrExpression InitializerList Label MemberAccess MultiplicativeExpression OrExpression Parameter ParameterList ParameterType PrimaryExpression PtrMemberAccess RelationalExpression ReturnStatement ReturnType ShiftExpression Sizeof SizeofExpr SizeofOperand Statement SwitchStatement UnaryExpression UnaryOp UnaryOperator [21] M AC D ONELL , S. G., G RAY, A. R., M AC L ENNAN , G., AND S ALLIS , P. J. Software forensics for discriminating between program authors using case-based reasoning, feedforward neural networks and multiple discriminant analysis. In Neural Information Processing, 1999. Proceedings. ICONIP’99. 6th International Conference on (1999), vol. 1, IEEE, pp. 66–71. [22] NARAYANAN , A., PASKOV, H., G ONG , N. Z., B ETHENCOURT, J., S TEFANOV, E., S HIN , E. C. R., AND S ONG , D. On the feasibility of internet-scale author identification. In Security and Privacy (SP), 2012 IEEE Symposium on (2012), IEEE, pp. 300– 314. [23] P ELLIN , B. N. Using classification techniques to determine source code authorship. White Paper: Department of Computer Science, University of Wisconsin (2000). WhileStatement [24] P IKE , R. The sherlock plagiarism detector, 2011. Table 10: Abstract syntax tree node types [25] P RECHELT, L., M ALPOHL , G., AND P HILIPPSEN , M. Finding plagiarisms among a set of programs with jplag. J. UCS 8, 11 (2002), 1016. Table 10 lists the AST node types generated by Joern that were incorporated to the feature set. Table 11 shows the C++ keywords used in the feature set. [26] Q UINLAN , J. Induction of decision trees. Machine learning 1, 1 (1986), 81–106. [27] ROSENBLUM , N., Z HU , X., AND M ILLER , B. Who wrote this code? identifying the authors of program binaries. Computer Security–ESORICS 2011 (2011), 172–189. alignas alignof and and_eq asm auto bitand bitor bool break case catch char char16_t char32_t class compl const constexpr const_cast continue decltype default delete do double dynamic_cast else enum explicit export extern false float for friend goto if inline int long mutable namespace new noexcept not not_eq nullptr operator or or_eq private protected public register [31] W IKIPEDIA. Saeed Malekpour, 2014. [Online; accessed 04November-2014]. reinterpret_cast return short signed sizeof static static_assert static_cast struct switch [32] YAMAGUCHI , F., G OLDE , N., A RP, D., AND R IECK , K. Modeling and discovering vulnerabilities with code property graphs. In Proc of IEEE Symposium on Security and Privacy (S&P) (2014). template this thread_local throw true try typedef typeid typename union unsigned using virtual void volatile [33] YAMAGUCHI , F., W RESSNEGGER , C., G ASCON , H., AND R IECK , K. Chucky: Exposing missing checks in source code for vulnerability discovery. In Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security (2013), ACM, pp. 499–510. wchar_t while xor xor_eq [28] S HEVERTALOV, M., KOTHARI , J., S TEHLE , E., AND M AN CORIDIS , S. On the use of discretized source code metrics for author identification. In Search Based Software Engineering, 2009 1st International Symposium on (2009), IEEE, pp. 69–78. [29] S PAFFORD , E. H., AND W EEBER , S. A. Software forensics: Can we track code to its authors? Computers & Security 12, 6 (1993), 585–595. [30] S TOLERMAN , A., OVERDORF, R., A FROZ , S., AND G REEN STADT, R. Classify, but verify: Breaking the closed-world assumption in stylometric authorship attribution. In IFIP Working Group 11.9 on Digital Forensics (2014), IFIP. Table 11: C++ keywords 15 B Appendix: Original vs Obfuscated Code Figure 6: A code sample X Figure 6 shows a source code sample X from our dataset that is 21 lines long. After obfuscation with Tigress, sample X became 537 lines long. Figure 7 shows the first 13 lines of the obfuscated sample X. Figure 7: Code sample X after obfuscation 16