In this article, we will go through very simple example of GitHub REST API to programmatically browse remote GitHub repository without cloning into local.

Example in this article:

Make RESTFul call to GitHub content REST API end point for a test repository.
Iterate through list of files from test repository.
Get content of one of the file & print content of the file.
Download file in local with the content of the remote file.

GitHub contents API

GitHub provides access to its repository & its content through REST endpoints. Here is the link for Documentation for content API.

Base URL for GitHub API – https://api.github.com
Path for Contents API – /repos/:owner/:repo/contents/:path (GET)
Parameter – ref (Branch/Tag/Commit)

We will use Test repository for this example. This is a public repository so authentication won’t be needed.

Test Repo Web URL = https://github.com/Ravikharatmal/test

“:owner” will be “RaviKharatmal”
“:repo” will be “test”.
We will not provide “:path” so that we can get list of all files in this repository.
We will pass “ref” param as “develop” so that we will get files from “develop” branch.

Note that this API has restrictions. File list from the response is not recursive, so use Tree API to get recursive file list. Contents API also has limit 1000 files & files upto 1 MB so if you need all files or bigger files then also Tree API must be used.

GitHub REST API | Tree API to get remote repo files list & metadata recursively programmatically without cloning in local

Lets Code

For this example, to keep it simple & short, we will use few libraries as given below. But you are free to chose any other library or code to do below things.

REST Client library – We will use Spring Framework’s RestTemplate since it provides very easy way to call REST end point.
GSON – We use GSON library to parse & pretty print the response of the REST call.
Apache commons IO – We will sue this to fetch file or content from the remote URLs.

Here are the dependencies for above libraries.

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter</artifactId>
	<version>2.1.7.RELEASE</version>
</dependency>
<dependency>
	<groupId>org.springframework</groupId>
	<artifactId>spring-web</artifactId>
	<version>5.1.9.RELEASE</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-databind</artifactId>
	<version>2.10.0.pr1</version>
</dependency>
<dependency>
	<groupId>com.google.code.gson</groupId>
	<artifactId>gson</artifactId>
	<version>2.8.5</version>
</dependency>
<dependency>
	<groupId>commons-io</groupId>
	<artifactId>commons-io</artifactId>
	<version>2.6</version>
</dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter</artifactId>

<version>2.1.7.RELEASE</version>

</dependency>

<groupId>org.springframework</groupId>

<artifactId>spring-web</artifactId>

<version>5.1.9.RELEASE</version>

</dependency>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-databind</artifactId>

</dependency>

<groupId>com.google.code.gson</groupId>

</dependency>

<groupId>commons-io</groupId>

<artifactId>commons-io</artifactId>

</dependency>

Here is the java code.

package com.itsallbinary.gitapi;

import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.charset.Charset;
import java.util.List;
import java.util.Map;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.springframework.web.client.RestTemplate;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class GitHubAPI_GetContent_Example {

	public static void main(String[] args) throws IOException, URISyntaxException {

		/*
		 * Call GitHub REST API - https://developer.github.com/v3/repos/contents/
		 * 
		 * Using Spring's RestTemplate to simplify REST call. Any other REST client
		 * library can be used here.
		 */
		RestTemplate restTemplate = new RestTemplate();
		List<Map> response = restTemplate.getForObject(
				"https://api.github.com/repos/{owner}/{repo}/contents?ref={branch}", List.class, "RaviKharatmal",
				"test", "develop");

		// To print response JSON, using GSON. Any other JSON parser can be used here.
		Gson gson = new GsonBuilder().setPrettyPrinting().create();
		System.out.println("<JSON RESPONSE START>\n" + gson.toJson(response) + "\n<JSON RESPONSE END>\n");

		// Iterate through list of file metadata from response.
		for (Map fileMetaData : response) {

			// Get file name & raw file download URL from response.
			String fileName = (String) fileMetaData.get("name");
			String downloadUrl = (String) fileMetaData.get("download_url");
			System.out.println("File Name = " + fileName + " | Download URL = " + downloadUrl);

			// We will only fetch read me file for this example.
			if (downloadUrl != null && downloadUrl.contains("README")) {

				/*
				 * Get file content as string
				 * 
				 * Using Apache commons IO to read content from the remote URL. Any other HTTP
				 * client library can be used here.
				 */
				String fileContent = IOUtils.toString(new URI(downloadUrl), Charset.defaultCharset());
				System.out.println("\nfileContent = <FILE CONTENT START>\n" + fileContent + "\n<FILE CONTENT END>\n");

				/*
				 * Download read me file to local.
				 * 
				 * Using Apache commons IO to create file from remote content. Any other library
				 * or code can be written to get content from URL & create file in local.
				 */
				File file = new File("github-api-downloaded-" + fileName);
				FileUtils.copyURLToFile(new URL(downloadUrl), file);
			}
		}
	}
}

package com.itsallbinary.gitapi;

import java.io.File;

import java.io.IOException;

import java.net.URI;

import java.net.URISyntaxException;

import java.net.URL;

import java.nio.charset.Charset;

import java.util.List;

import java.util.Map;

import org.apache.commons.io.FileUtils;

import org.apache.commons.io.IOUtils;

import org.springframework.web.client.RestTemplate;

import com.google.gson.Gson;

import com.google.gson.GsonBuilder;

public class GitHubAPI_GetContent_Example {

public static void main(String[] args) throws IOException, URISyntaxException {

* Call GitHub REST API - https://developer.github.com/v3/repos/contents/

* Using Spring's RestTemplate to simplify REST call. Any other REST client

* library can be used here.

RestTemplate restTemplate = new RestTemplate();

List<Map> response = restTemplate.getForObject(

"https://api.github.com/repos/{owner}/{repo}/contents?ref={branch}", List.class, "RaviKharatmal",

"test", "develop");

// To print response JSON, using GSON. Any other JSON parser can be used here.

Gson gson = new GsonBuilder().setPrettyPrinting().create();

System.out.println("<JSON RESPONSE START>\n" + gson.toJson(response) + "\n<JSON RESPONSE END>\n");

// Iterate through list of file metadata from response.

for (Map fileMetaData : response) {

// Get file name & raw file download URL from response.

String fileName = (String) fileMetaData.get("name");

String downloadUrl = (String) fileMetaData.get("download_url");

System.out.println("File Name = " + fileName + " | Download URL = " + downloadUrl);

// We will only fetch read me file for this example.

if (downloadUrl != null && downloadUrl.contains("README")) {

* Get file content as string

* Using Apache commons IO to read content from the remote URL. Any other HTTP

* client library can be used here.

String fileContent = IOUtils.toString(new URI(downloadUrl), Charset.defaultCharset());

System.out.println("\nfileContent = <FILE CONTENT START>\n" + fileContent + "\n<FILE CONTENT END>\n");

* Download read me file to local.

* Using Apache commons IO to create file from remote content. Any other library

* or code can be written to get content from URL & create file in local.

File file = new File("github-api-downloaded-" + fileName);

FileUtils.copyURLToFile(new URL(downloadUrl), file);

}

Here is the output of the code.

00:24:13.167 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET https://api.github.com/repos/RaviKharatmal/test/contents?ref=develop
00:24:13.237 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[application/json, application/*+json]
00:24:13.668 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK
00:24:13.676 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.util.List<?>]
<JSON RESPONSE START>
[
  {
    "name": ".gitignore",
    "path": ".gitignore",
    "sha": "5f2dbe11df91c879164ae1150f5577ac7263c1da",
    "size": 277,
    "url": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop",
    "html_url": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore",
    "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da",
    "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore",
    "type": "file",
    "_links": {
      "self": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop",
      "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da",
      "html": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore"
    }
  },
  {
    "name": "README.md",
    "path": "README.md",
    "sha": "949311fee79bfc0102fc9015d46a283e0b252d3a",
    "size": 75,
    "url": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop",
    "html_url": "https://github.com/Ravikharatmal/test/blob/develop/README.md",
    "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a",
    "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md",
    "type": "file",
    "_links": {
      "self": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop",
      "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a",
      "html": "https://github.com/Ravikharatmal/test/blob/develop/README.md"
    }
  },
  {
    "name": "fileInDevelop.txt",
    "path": "fileInDevelop.txt",
    "sha": "7ae2e2119a7ef448dd397b6049c55f329096ab8e",
    "size": 44,
    "url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop",
    "html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt",
    "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e",
    "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt",
    "type": "file",
    "_links": {
      "self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop",
      "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e",
      "html": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt"
    }
  },
  {
    "name": "fileInMaster.txt",
    "path": "fileInMaster.txt",
    "sha": "a744f3268f141a5a35a666d96fbe5dd174e784c4",
    "size": 23,
    "url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop",
    "html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt",
    "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4",
    "download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt",
    "type": "file",
    "_links": {
      "self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop",
      "git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4",
      "html": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt"
    }
  },
  {
    "name": "some-dir",
    "path": "some-dir",
    "sha": "e7c5c0e0f9175fa416c13358d75886e13965652e",
    "size": 0,
    "url": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop",
    "html_url": "https://github.com/Ravikharatmal/test/tree/develop/some-dir",
    "git_url": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e",
    "type": "dir",
    "_links": {
      "self": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop",
      "git": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e",
      "html": "https://github.com/Ravikharatmal/test/tree/develop/some-dir"
    }
  }
]
<JSON RESPONSE END>

File Name = .gitignore | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore
File Name = README.md | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md

fileContent = <FILE CONTENT START>
# test

test for JGit & GitHub API.

Some sample content for test purpose.

<FILE CONTENT END>

File Name = fileInDevelop.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt
File Name = fileInMaster.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt
File Name = some-dir | Download URL = null

100

101

102

103

00:24:13.167 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET https://api.github.com/repos/RaviKharatmal/test/contents?ref=develop

00:24:13.237 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[application/json, application/*+json]

00:24:13.668 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK

00:24:13.676 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.util.List<?>]

[

{

"name": ".gitignore",

"path": ".gitignore",

"sha": "5f2dbe11df91c879164ae1150f5577ac7263c1da",

"size": 277,

"url": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop",

"html_url": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore",

"git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da",

"download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore",

"type": "file",

"_links": {

"self": "https://api.github.com/repos/Ravikharatmal/test/contents/.gitignore?ref\u003ddevelop",

"git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/5f2dbe11df91c879164ae1150f5577ac7263c1da",

"html": "https://github.com/Ravikharatmal/test/blob/develop/.gitignore"

}

{

"name": "README.md",

"path": "README.md",

"sha": "949311fee79bfc0102fc9015d46a283e0b252d3a",

"size": 75,

"url": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop",

"html_url": "https://github.com/Ravikharatmal/test/blob/develop/README.md",

"git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a",

"download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md",

"type": "file",

"_links": {

"self": "https://api.github.com/repos/Ravikharatmal/test/contents/README.md?ref\u003ddevelop",

"git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/949311fee79bfc0102fc9015d46a283e0b252d3a",

"html": "https://github.com/Ravikharatmal/test/blob/develop/README.md"

}

{

"name": "fileInDevelop.txt",

"path": "fileInDevelop.txt",

"sha": "7ae2e2119a7ef448dd397b6049c55f329096ab8e",

"size": 44,

"url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop",

"html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt",

"git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e",

"download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt",

"type": "file",

"_links": {

"self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInDevelop.txt?ref\u003ddevelop",

"git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/7ae2e2119a7ef448dd397b6049c55f329096ab8e",

"html": "https://github.com/Ravikharatmal/test/blob/develop/fileInDevelop.txt"

}

{

"name": "fileInMaster.txt",

"path": "fileInMaster.txt",

"sha": "a744f3268f141a5a35a666d96fbe5dd174e784c4",

"size": 23,

"url": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop",

"html_url": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt",

"git_url": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4",

"download_url": "https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt",

"type": "file",

"_links": {

"self": "https://api.github.com/repos/Ravikharatmal/test/contents/fileInMaster.txt?ref\u003ddevelop",

"git": "https://api.github.com/repos/Ravikharatmal/test/git/blobs/a744f3268f141a5a35a666d96fbe5dd174e784c4",

"html": "https://github.com/Ravikharatmal/test/blob/develop/fileInMaster.txt"

}

{

"name": "some-dir",

"path": "some-dir",

"sha": "e7c5c0e0f9175fa416c13358d75886e13965652e",

"size": 0,

"url": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop",

"html_url": "https://github.com/Ravikharatmal/test/tree/develop/some-dir",

"git_url": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e",

"type": "dir",

"_links": {

"self": "https://api.github.com/repos/Ravikharatmal/test/contents/some-dir?ref\u003ddevelop",

"git": "https://api.github.com/repos/Ravikharatmal/test/git/trees/e7c5c0e0f9175fa416c13358d75886e13965652e",

"html": "https://github.com/Ravikharatmal/test/tree/develop/some-dir"

}

]

File Name = .gitignore | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/.gitignore

File Name = README.md | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/README.md

fileContent = <FILE CONTENT START>

# test

test for JGit & GitHub API.

Some sample content for test purpose.

File Name = fileInDevelop.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInDevelop.txt

File Name = fileInMaster.txt | Download URL = https://raw.githubusercontent.com/Ravikharatmal/test/develop/fileInMaster.txt

File Name = some-dir | Download URL = null

Complete code is also committed & available in GitHub repository.

Its All Binary – Coding Posts, Examples, Projects & More

Site for innovative & creative blog posts, Code examples, Java release history, Other projects & much more.

GitHub REST API | Get remote repo files list & download file content programmatically without cloning in local

Example in this article:

GitHub contents API

Lets Code

Further reading

One Reply to “GitHub REST API | Get remote repo files list & download file content programmatically without cloning in local”

Leave a Reply Cancel reply