In this article we will create simple examples to search remote GitHub using search API REST endpoints through Java code. In earlier article, we went through Tree API. Here is link for reference.
Examples in this article:
- Search for files with given file name & extension in given organization.
- Search for code by content i.e. method name in given repository by language .
- Search for open pull requests by date.
- Search for commits by author and date range.
For testing purpose, we will be using below organisation, repository or author in our search queries.
- Organization – Apache
- Repository – Apache commons-lang
- Author – Gary Gregory (As per Apache commons-lang commit history, this user has most commits. Kudos to the great work !)
GitHub Search API
We will be using search API for searching GitHub. Here is the documentation for GitHub Search API. Search API is dependent on search ‘query’ which needs to be sent as URL request parameter as shown in below sample.
https://api.github.com/search/code?q=filename:WordUtil+extension:java+org:apache
‘q’ is key of query parameter. Query syntax is generally “<qualifier>:<value>”. Query supports very extensive syntax which can be found in Query Syntax Documentation. As we go through examples, we will also use different qualifiers & see how searches can be very useful.
Lets Code
We will use Apache commons HTTP Client library with Apache Fluent httpcomponents library for making REST calls to GitHub API endpoints. We will be using GSON for JSON response parsing.
Here is the code to make the REST call which we will use in all below examples.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
/** * This method will make a REST GET call for this URL using Apache http client & * fluent library. * * Then parse response using GSON & return parsed Map. */ private static Map makeRESTCall(String restUrl, String acceptHeaderValue) throws ClientProtocolException, IOException { Request request = Request.Get(restUrl); if (acceptHeaderValue != null && !acceptHeaderValue.isBlank()) { request.addHeader("Accept", acceptHeaderValue); } Content content = request.execute().returnContent(); String jsonString = content.asString(); // System.out.println("content = " + jsonString); // To print response JSON, using GSON. Any other JSON parser can be used here. Map jsonMap = gson.fromJson(jsonString, Map.class); return jsonMap; } private static Map makeRESTCall(String restUrl) throws ClientProtocolException, IOException { return makeRESTCall(restUrl, null); } |
Search files by file name & extension
This can be achieved using ‘Search Code API‘. We will use these qualifiers in query
- filename – Name of the file to search. We will search for “WordUtil”
- extension – Extension of the file to search. We will search “.java” files.
- org – GitHub organization to search in.
Here is documentation for complete list of qualifiers for Code API Query.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
private static String GITHUB_API_BASE_URL = "https://api.github.com/"; private static String GITHUB_API_SEARCH_CODE_PATH = "search/code?q="; private static void searchFileByFileName() throws ClientProtocolException, IOException { /* * Search files by file name * * 1) Search for file name containing "WordUtil", 2) File extension = "java" 3) * File from any repo of organization "apache" */ String codeFileQuery = "filename:WordUtil+extension:java+org:apache"; Map fileNameSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_CODE_PATH + codeFileQuery); System.out.println("Total number or results = " + fileNameSearchResult.get("total_count")); gson.toJsonTree(fileNameSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out.println("\tFile: " + r.getAsJsonObject().get("name") + "\n\t\t | Repo: " + r.getAsJsonObject().get("repository").getAsJsonObject().get("html_url") + "\n\t\t | Path: " + r.getAsJsonObject().get("path"))); } |
Here is the output which shows searched file names along with repository & path of the file.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Total number or results = 4.0 File: "WordTokenUtil.java" | Repo: "https://github.com/apache/ctakes" | Path: "ctakes-core/src/main/java/org/apache/ctakes/core/util/WordTokenUtil.java" File: "RareWordUtil.java" | Repo: "https://github.com/apache/ctakes" | Path: "ctakes-gui/src/main/java/org/apache/ctakes/gui/dictionary/util/RareWordUtil.java" File: "WordCountUtil.java" | Repo: "https://github.com/apache/avro" | Path: "lang/java/mapred/src/test/java/org/apache/avro/mapred/WordCountUtil.java" File: "WordCountUtil.java" | Repo: "https://github.com/apache/avro" | Path: "lang/java/trevni/avro/src/test/java/org/apache/trevni/avro/WordCountUtil.java" |
Search files by code/content with text matching
This is also achieved using ‘Search Code‘ API. We will use these qualifiers in this query
- word to query – This doesn’t need any qualifier key. We will search for code/method “containsAny” in this example.
- in – Where to search. In this case files.
- language – Which language code to search. IN this case we ill search above code in Java code.
- repo – GitHub repository to search in.
You can notice that we have also passed a request header “Accept” with value of “application/vnd.github.v3.text-match+json”. This is to activate text matching functionality which will tell us exact line number & column number in file where search matched along with actual fragment of matched content.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
private static void searchCodeByContent() throws ClientProtocolException, IOException { /* * Search Code by content in file * * 1) Search for word (method name) = "containsAny", 2) Search in files, 3) * Search in file with extension ".java", 4) Search in Repository = * https://github.com/apache/commons-lang */ String codeContentQuery = "containsAny+in:file+language:java+repo:apache/commons-lang"; Map contentSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_CODE_PATH + codeContentQuery, "application/vnd.github.v3.text-match+json"); // System.out.println( // " Response = \n<API RESPONSE START>\n " + gson.toJson(contentSearchResult) + // "\n<API RESPONSE END>\n"); System.out.println("Total number or results = " + contentSearchResult.get("total_count")); gson.toJsonTree(contentSearchResult).getAsJsonObject().get("items").getAsJsonArray().forEach(r -> { System.out.println("\tFile: " + r.getAsJsonObject().get("name") + "\n\t\t | Repo: " + r.getAsJsonObject().get("repository").getAsJsonObject().get("html_url") + "\n\t\t | Path: " + r.getAsJsonObject().get("path")); r.getAsJsonObject().get("text_matches").getAsJsonArray() .forEach(t -> System.out.println("\t\t| Matched line: " + t.getAsJsonObject().get("fragment"))); }); } |
Here is the output which shows name of file in which content if found along with repository URL & path of the file. Due to the header that we passed in the request for text matching, we are also able to get fragments of matched line.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
Total number or results = 5.0 File: "StringUtilsContainsTest.java" | Repo: "https://github.com/apache/commons-lang" | Path: "src/test/java/org/apache/commons/lang3/StringUtilsContainsTest.java" | Matched line: "(StringUtils.containsAny(null, (char[]) null));\n assertFalse(StringUtils.containsAny(null, new" | Matched line: " char[0]));\n assertFalse(StringUtils.containsAny(null, 'a', 'b'));\n\n assertFalse" File: "CharSetUtilsTest.java" | Repo: "https://github.com/apache/commons-lang" | Path: "src/test/java/org/apache/commons/lang3/CharSetUtilsTest.java" | Matched line: " testContainsAny_StringString() {\n assertFalse(CharSetUtils.containsAny(null, (String) null));\n assertFalse" | Matched line: "(CharSetUtils.containsAny(null, \"\"));\n\n assertFalse(CharSetUtils.containsAny(\"\", (String) null" File: "CharSetUtils.java" | Repo: "https://github.com/apache/commons-lang" | Path: "src/main/java/org/apache/commons/lang3/CharSetUtils.java" | Matched line: " specified string.</p>\n *\n * <pre>\n * CharSetUtils.containsAny(null, *) = false" | Matched line: "\n * CharSetUtils.containsAny(\"\", *) = false\n * CharSetUtils.containsAny(*, null" File: "StringUtils.java" | Repo: "https://github.com/apache/commons-lang" | Path: "src/main/java/org/apache/commons/lang3/StringUtils.java" | Matched line: "</b>\n * - index-of any of a set of Strings</li>\n * <li><b>ContainsOnly/ContainsNone/ContainsAny" | Matched line: " CharSequenceUtils.indexOf(seq, searchChar, 0) >= 0;\n }\n\n // ContainsAny" File: "StringEscapeUtils.java" | Repo: "https://github.com/apache/commons-lang" | Path: "src/main/java/org/apache/commons/lang3/StringEscapeUtils.java" | Matched line: " ( StringUtils.containsAny(quoteless, CSV_SEARCH_CHARS) ) {\n // deal with escaped quotes; ie" |
Search for open pull requests for given branch
This can be achieved using ‘Search Issues/Pull Requests API‘. We will use these qualifiers in the query.
- Word in title – This doesn’t need qualifier key. We will search for PR with ‘number’ word in their title.
- type – Since we are looking for pull requests, we will give value as ‘pr’.
- state – Since we are interested in only open PRs, we give value as ‘open’
- base – This is the branch in which PR is intended to be merged. We give it as ‘master’
- repo – GitHub repository to search in.
- sort & order – This defines how results should be sorted.
Here is documentation for complete list of qualifiers for Search Issues/PRs API Query.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
private static String GITHUB_API_BASE_URL = "https://api.github.com/"; private static String GITHUB_API_SEARCH_ISSUES_PATH = "search/issues"; private static void searchPullRequests() throws ClientProtocolException, IOException { /* * Search pull requests * * 1) Search in repo = "apache/commons-lang", 2) Type as Pull Requests 3) Only * open pull requests 4) Pull requests which are to be merged in master branch * 5) Sort by created date-time in ascending order. */ String pullRequestsQuery = "?q=number+repo:apache/commons-lang+type:pr+state:open+base:master&sort=created&order=asc"; Map pullRequestsSearchResult = makeRESTCall( GITHUB_API_BASE_URL + GITHUB_API_SEARCH_ISSUES_PATH + pullRequestsQuery); System.out.println("Total number or results = " + pullRequestsSearchResult.get("total_count")); gson.toJsonTree(pullRequestsSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out.println("\tTitle: " + r.getAsJsonObject().get("title") + "\n\t\t | By User: " + r.getAsJsonObject().get("user").getAsJsonObject().get("login") + "\n\t\t | Path: " + r.getAsJsonObject().get("pull_request").getAsJsonObject().get("html_url"))); } |
Here is the output which shows titles of the PRs matched along with the user details & path of the pull requests.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Total number or results = 5.0 Title: "LANG-341: Add number to byte array methods to org.apache.commons.lang3.Conversion" | By User: "yurelle" | Path: "https://github.com/apache/commons-lang/pull/219" Title: "Add methods allowing masking of Strings" | By User: "greenman18523" | Path: "https://github.com/apache/commons-lang/pull/332" Title: "LANG-1400: Add StringUtils.mask() function" | By User: "stokito" | Path: "https://github.com/apache/commons-lang/pull/335" Title: "Add a check to StringUtils.repeat() for large length repeat value" | By User: "Turan91" | Path: "https://github.com/apache/commons-lang/pull/362" Title: "LANG-1369: Formatted and Paramaterized Exception Classes" | By User: "belugabehr" | Path: "https://github.com/apache/commons-lang/pull/414" |
Search commits by author & date range
This can be achieved using ‘Search Commits API‘. As of the date of writing this article, Search Commit API is only available for developers preview & might not be good for production use.
We will use these qualifiers in query
- author – GitHub author name whose commits need to be searched.
- commiter-date – Date range to search commits in. This can include < or > etc. for date.
Here is documentation for complete list of qualifiers for Search Commits API Query.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
private static void searchCommits() throws ClientProtocolException, IOException { /* * Search commits * * ">" url encoded as "%3e" */ String commitsQuery = "?q=author:garydgregory+committer-date:%3e2019-08-01"; Map commitsSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_COMMITS_PATH + commitsQuery, "application/vnd.github.cloak-preview"); System.out.println("Total number or results = " + commitsSearchResult.get("total_count")); gson.toJsonTree(commitsSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out .println("\n\t | Message: " + r.getAsJsonObject().get("commit").getAsJsonObject().get("message") + "\n\t | Date: " + r.getAsJsonObject().get("commit").getAsJsonObject().get("committer") .getAsJsonObject().get("date"))); } |
Here is the output showing commit message along with the date.
1 2 3 4 5 6 7 8 9 10 |
Total number or results = 3.0 | Message: "Drop Oracle JDK 8." | Date: "2019-08-14T13:25:23.000-07:00" | Message: "Drop Oracle JDK 8." | Date: "2019-08-14T13:24:52.000-07:00" | Message: "[TEXT-170] Add String lookup for host names and IP addresses (#118)\n\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\nDo not use hardcoded IP addresses.\r\n\r\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\nDo not use hardcoded IP addresses.\r\n\r\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\nFix Checkstyle issues.\r\n\r\n* Fix version number in Javadoc.\r\n\r\n* Update Javadoc per PR review.\r\n\r\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\nUpdate test per PR review.\r\n\r\n* [TEXT-170] Add String lookup for host names and IP addresses\r\n(DnsStringLookup).\r\n\r\nUpdate test per PR review." | Date: "2019-08-04T11:26:52.000-04:00" |
Complete Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
package com.itsallbinary.gitapi; import java.io.IOException; import java.net.URISyntaxException; import java.util.Map; import org.apache.http.client.ClientProtocolException; import org.apache.http.client.fluent.Content; import org.apache.http.client.fluent.Request; import com.google.gson.Gson; import com.google.gson.GsonBuilder; public class GitHubAPI_Search_Example { private static Gson gson; private static String GITHUB_API_BASE_URL = "https://api.github.com/"; private static String GITHUB_API_SEARCH_CODE_PATH = "search/code?q="; private static String GITHUB_API_SEARCH_ISSUES_PATH = "search/issues"; private static String GITHUB_API_SEARCH_COMMITS_PATH = "search/commits"; public static void main(String[] args) throws IOException, URISyntaxException { // Using GSON to parse or print response JSON. gson = new GsonBuilder().setPrettyPrinting().create(); searchFileByFileName(); searchCodeByContent(); searchPullRequests(); searchCommits(); } private static void searchCommits() throws ClientProtocolException, IOException { /* * Search commits * * ">" url encoded as "%3e" */ String commitsQuery = "?q=author:garydgregory+committer-date:%3e2019-08-01"; Map commitsSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_COMMITS_PATH + commitsQuery, "application/vnd.github.cloak-preview"); System.out.println("Total number or results = " + commitsSearchResult.get("total_count")); gson.toJsonTree(commitsSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out .println("\n\t | Message: " + r.getAsJsonObject().get("commit").getAsJsonObject().get("message") + "\n\t | Date: " + r.getAsJsonObject().get("commit").getAsJsonObject().get("committer") .getAsJsonObject().get("date"))); } private static void searchPullRequests() throws ClientProtocolException, IOException { /* * Search pull requests * * 1) Search in repo = "apache/commons-lang", 2) Type as Pull Requests 3) Only * open pull requests 4) Pull requests which are to be merged in master branch * 5) Sort by created date-time in ascending order. */ String pullRequestsQuery = "?q=number+repo:apache/commons-lang+type:pr+state:open+base:master&sort=created&order=asc"; Map pullRequestsSearchResult = makeRESTCall( GITHUB_API_BASE_URL + GITHUB_API_SEARCH_ISSUES_PATH + pullRequestsQuery); System.out.println("Total number or results = " + pullRequestsSearchResult.get("total_count")); gson.toJsonTree(pullRequestsSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out.println("\tTitle: " + r.getAsJsonObject().get("title") + "\n\t\t | By User: " + r.getAsJsonObject().get("user").getAsJsonObject().get("login") + "\n\t\t | Path: " + r.getAsJsonObject().get("pull_request").getAsJsonObject().get("html_url"))); } private static void searchCodeByContent() throws ClientProtocolException, IOException { /* * Search Code by content in file * * 1) Search for word (method name) = "containsAny", 2) Search in files, 3) * Search in file with extension ".java", 4) Search in Repository = * https://github.com/apache/commons-lang */ String codeContentQuery = "containsAny+in:file+language:java+repo:apache/commons-lang"; Map contentSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_CODE_PATH + codeContentQuery, "application/vnd.github.v3.text-match+json"); // System.out.println( // " Response = \n<API RESPONSE START>\n " + gson.toJson(contentSearchResult) + // "\n<API RESPONSE END>\n"); System.out.println("Total number or results = " + contentSearchResult.get("total_count")); gson.toJsonTree(contentSearchResult).getAsJsonObject().get("items").getAsJsonArray().forEach(r -> { System.out.println("\tFile: " + r.getAsJsonObject().get("name") + "\n\t\t | Repo: " + r.getAsJsonObject().get("repository").getAsJsonObject().get("html_url") + "\n\t\t | Path: " + r.getAsJsonObject().get("path")); r.getAsJsonObject().get("text_matches").getAsJsonArray() .forEach(t -> System.out.println("\t\t| Matched line: " + t.getAsJsonObject().get("fragment"))); }); } private static void searchFileByFileName() throws ClientProtocolException, IOException { /* * Search files by file name * * 1) Search for file name containing "WordUtil", 2) File extension = "java" 3) * File from any repo of organization "apache" */ String codeFileQuery = "filename:WordUtil+extension:java+org:apache"; Map fileNameSearchResult = makeRESTCall(GITHUB_API_BASE_URL + GITHUB_API_SEARCH_CODE_PATH + codeFileQuery); System.out.println("Total number or results = " + fileNameSearchResult.get("total_count")); gson.toJsonTree(fileNameSearchResult).getAsJsonObject().get("items").getAsJsonArray() .forEach(r -> System.out.println("\tFile: " + r.getAsJsonObject().get("name") + "\n\t\t | Repo: " + r.getAsJsonObject().get("repository").getAsJsonObject().get("html_url") + "\n\t\t | Path: " + r.getAsJsonObject().get("path"))); } private static void test() throws ClientProtocolException, IOException { } /** * This method will make a REST GET call for this URL using Apache http client & * fluent library. * * Then parse response using GSON & return parsed Map. */ private static Map makeRESTCall(String restUrl, String acceptHeaderValue) throws ClientProtocolException, IOException { Request request = Request.Get(restUrl); if (acceptHeaderValue != null && !acceptHeaderValue.isBlank()) { request.addHeader("Accept", acceptHeaderValue); } Content content = request.execute().returnContent(); String jsonString = content.asString(); // System.out.println("content = " + jsonString); // To print response JSON, using GSON. Any other JSON parser can be used here. Map jsonMap = gson.fromJson(jsonString, Map.class); return jsonMap; } private static Map makeRESTCall(String restUrl) throws ClientProtocolException, IOException { return makeRESTCall(restUrl, null); } } |
Complete code is also committed & available in GitHub repository.
Полезно