2

Background

I have a Jenkins build pipeline that dynamically spins up a Windows VM, clones a repo containing utilities for our release process, and then uses a python script in that repo to clone a separate private Github repo containing the actual source code and then build it. The Jenkins pipeline is also responsible for providing the github personal access token to the VM.

Previously we had a single dedicated machine for doing this (instead of dynamically-provisioned VMs) and that machine had a private key pair that it used for cloning the private repos via SSH.

Problem

Our IT department does not want us to an SSH key that would have to be shared between a dynamic number of VMs and is requiring that we switch to HTTPS using an access token.

While I can easily clone the repos using git clone https://{TOKEN}@github.com/{org}/{repo}, I run into problems when I try to update the submodules because the .gitmodules file is configured to use SSH.

What I've Tried

I have had some success by:

  1. Running git config --global url."https://github.com/".insteadOf "[email protected]:" to configure git to substitute the SSH URLs with the equivalent HTTPS URL.
  2. After doing this I can clone the repo which prompts me with a GUI to input the credentials. I put in my access token and the repo clones successfully. (Not sure if it matters but the current credential.credentialStore in my gitconfig is dpapi but I'm not tied to this.)
  3. Run git submodule update --init from within the cloned repo, which works successfully since the access token is now stored from step 2.

The problem with this method is that I have to interact with the GUI interface to input the access token, which only works when I am doing this manually. I need this to be an automated process but I cannot figure out how to add the access token as a credential before running the clone operation.

Limitations

  • I have to use HTTPS, I cannot use SSH; this is mandated by our IT department.
  • I can't rewrite .gitmodules to use HTTPS URLs because this will break it for our devs who will still be using SSH to authenticate.
  • I can't move the logic from the python script into the Jenkins pipeline directly, this is simply out of scope at the moment.

TL;DR

How can I preset the Github access token in my credentials or environment before running the initial clone operation, such that it doesn't require me to type/paste it manually? I'm also open to another approach for making sure that the repos containing the code can be seamlessly cloned and have its submodules updated.

Update

Here's what eventually (mostly) worked for me.

  1. I changed the credential helper to store to prevent having to interact with the GUI element.
  2. It turned out this was being overridden in the gitconfig file in the Git/etc directory to use wincredmain so I switched the helper there instead of in the global config.
  3. Now I can preload the credentials using git credential approve and setting protocol=https, host=github.com and username and password. This still required manual interaction but it was all from the command line which was a step in the right direction.
  4. I can write the credentials manually to %UserProfile%\.git-credentials in order to reproduce the results of doing step 3 interactively. However, this adds a CRLF to the end of the credential line in the file which causes git to ignore it. I have to manually convert the EOLs to Unix-style (just LF). I'd love a solution to this but I will post it as a separate question.
Improve this question

2 Answers 2

Reset to default
3

Our IT department does not want us to an SSH key that would have to be shared between a dynamic number of VMs and is requiring that we switch to HTTPS using an access token.

Find a way to tell your IT department that using a pre-set access token that would have to be shared between a dynamic number of VMs is literally the same thing as using an SSH key that would have to be shared between a dynamic number of VMs. It is still a static credential that has to be copied around (and can be stolen).

See git-credential(1) for the CLI interface to Git's credential helpers.

Use git credential fill to retrieve the password or token from whichever helper is set up (providing a key=value query via stdin), then on the other systems git credential approve (with the retrieved data as stdin) to store it into whichever helper is set up.

(echo host=github.com; echo protocol=https) | git credential fill

cat creds.txt | git credential approve

(You can configure your jenkins environment to use a more suitable credential.helper – e.g. 'cache' for temporary in-memory storage, or 'store' for plain, unencrypted file-based storage like Christopher suggests – instead of the usual GUI-based 'wincred' or whatever you have.)

It is also possible to directly invoke credential helpers one by one, although the subcommands are different: get/store instead of fill/approve, but otherwise the usage is the same. For example, git credential-wincred get.

cat creds.txt | git credential-store store

Note that the credential.credentialStore setting doesn't switch helpers overall – it's a parameter specifically for the "Git Credential Manager" helper.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

The trick is to use a Git credential helper to store your GitHub token before you do any cloning. That way, Git just grabs the token automatically and you don’t get prompted for anything. You can either write the token to ~/.git-credentials (just make sure to clean it up after your build for security), or embed the token in the clone URL if you’re in a pinch. Also, your insteadOf config is spot on for rewriting SSH URLs to HTTPS, just make sure you set it before you do any submodule stuff. Hope that helps!

Improve this answer

1 Comment

Thanks, I tried writing to %UserProfile%\.git-credentials which I believe is the Windows location for the credentials file, but it didn't seem to work. Do I literally just write the token to the file as its own line? If not, what is the proper syntax? The VM will be deleted after the build is done so the cleanup isn't an issue :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.