In this article, we will go through very simple example of GitHub REST API to programmatically browse remote GitHub repository without cloning into local.
Example in this article:
- Make RESTFul call to GitHub content REST API end point for a test repository.
- Iterate through list of files from test repository.
- Get content of one of the file & print content of the file.
- Download file in local with the content of the remote file.
GitHub contents API
GitHub provides access to its repository & its content through REST endpoints. Here is the link for Documentation for content API.
- Base URL for GitHub API – https://api.github.com
- Path for Contents API – /repos/:owner/:repo/contents/:path (GET)
- Parameter – ref (Branch/Tag/Commit)
We will use Test repository for this example. This is a public repository so authentication won’t be needed.
Test Repo Web URL = https://github.com/Ravikharatmal/test
- “:owner” will be “RaviKharatmal”
- “:repo” will be “test”.
- We will not provide “:path” so that we can get list of all files in this repository.
- We will pass “ref” param as “develop” so that we will get files from “develop” branch.
Note that this API has restrictions. File list from the response is not recursive, so use Tree API to get recursive file list. Contents API also has limit 1000 files & files upto 1 MB so if you need all files or bigger files then also Tree API must be used.
Lets Code
For this example, to keep it simple & short, we will use few libraries as given below. But you are free to chose any other library or code to do below things.
- REST Client library – We will use Spring Framework’s RestTemplate since it provides very easy way to call REST end point.
- GSON – We use GSON library to parse & pretty print the response of the REST call.
- Apache commons IO – We will sue this to fetch file or content from the remote URLs.
Here are the dependencies for above libraries.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>2.1.7.RELEASE</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-web</artifactId> <version>5.1.9.RELEASE</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.10.0.pr1</version> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.8.5</version> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.6</version> </dependency> |
Here is the java code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
package com.itsallbinary.gitapi; import java.io.File; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.nio.charset.Charset; import java.util.List; import java.util.Map; import org.apache.commons.io.FileUtils; import org.apache.commons.io.IOUtils; import org.springframework.web.client.RestTemplate; import com.google.gson.Gson; import com.google.gson.GsonBuilder; public class GitHubAPI_GetContent_Example { public static void main(String[] args) throws IOException, URISyntaxException { /* * Call GitHub REST API - https://developer.github.com/v3/repos/contents/ * * Using Spring's RestTemplate to simplify REST call. Any other REST client * library can be used here. */ RestTemplate restTemplate = new RestTemplate(); List<Map> response = restTemplate.getForObject( "https://api.github.com/repos/{owner}/{repo}/contents?ref={branch}", List.class, "RaviKharatmal", "test", "develop"); // To print response JSON, using GSON. Any other JSON parser can be used here. Gson gson = new GsonBuilder().setPrettyPrinting().create(); System.out.println("<JSON RESPONSE START>\n" + gson.toJson(response) + "\n<JSON RESPONSE END>\n"); // Iterate through list of file metadata from response. for (Map fileMetaData : response) { // Get file name & raw file download URL from response. String fileName = (String) fileMetaData.get("name"); String downloadUrl = (String) fileMetaData.get("download_url"); System.out.println("File Name = " + fileName + " | Download URL = " + downloadUrl); // We will only fetch read me file for this example. if (downloadUrl != null && downloadUrl.contains("README")) { /* * Get file content as string * * Using Apache commons IO to read content from the remote URL. Any other HTTP * client library can be used here. */ String fileContent = IOUtils.toString(new URI(downloadUrl), Charset.defaultCharset()); System.out.println("\nfileContent = <FILE CONTENT START>\n" + fileContent + "\n<FILE CONTENT END>\n"); /* * Download read me file to local. * * Using Apache commons IO to create file from remote content. Any other library * or code can be written to get content from URL & create file in local. */ File file = new File("github-api-downloaded-" + fileName); FileUtils.copyURLToFile(new URL(downloadUrl), file); } } } } |
Here is the output of the code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
00:24:13.167 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET https://api.github.com/repos/RaviKharatmal/test/contents?ref=develop 00:24:13.237 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[application/json, application/*+json] 00:24:13.668 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK 00:24:13.676 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.util.List<?>] <JSON RESPONSE START> [ { "name": ".gitignore", "path": ".gitignore", "sha": "5f2dbe11df91c879164ae1150f5577ac7263c1da", "size": 277, "url": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop", "html_url": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore", "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da", "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore", "type": "file", "_links": { "self": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop", "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da", "html": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore" } }, { "name": "README.md", "path": "README.md", "sha": "949311fee79bfc0102fc9015d46a283e0b252d3a", "size": 75, "url": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop", "html_url": "https://github.com/Ravikharatmal/test/blob/develop/README.md", "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a", "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md", "type": "file", "_links": { "self": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop", "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a", "html": "https://github.com/Ravikharatmal/test/blob/develop/README.md" } }, { "name": "fileInDevelop.txt", "path": "fileInDevelop.txt", "sha": "7ae2e2119a7ef448dd397b6049c55f329096ab8e", "size": 44, "url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop", "html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt", "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e", "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt", "type": "file", "_links": { "self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop", "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e", "html": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt" } }, { "name": "fileInMaster.txt", "path": "fileInMaster.txt", "sha": "a744f3268f141a5a35a666d96fbe5dd174e784c4", "size": 23, "url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop", "html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt", "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4", "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt", "type": "file", "_links": { "self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop", "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4", "html": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt" } }, { "name": "some-dir", "path": "some-dir", "sha": "e7c5c0e0f9175fa416c13358d75886e13965652e", "size": 0, "url": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop", "html_url": "https://github.com/Ravikharatmal/test/tree/develop/some-dir", "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e", "type": "dir", "_links": { "self": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop", "git": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e", "html": "https://github.com/Ravikharatmal/test/tree/develop/some-dir" } } ] <JSON RESPONSE END> File Name = .gitignore | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore File Name = README.md | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md fileContent = <FILE CONTENT START> # test test for JGit & GitHub API. Some sample content for test purpose. <FILE CONTENT END> File Name = fileInDevelop.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt File Name = fileInMaster.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt File Name = some-dir | Download URL = null |
Complete code is also committed & available in GitHub repository.
Further reading
Program will create file as shown below with the content from remote read me file.
Hi – Very nice article. How to pass authentication information to a repo which is locked?