How to scp to ec2 instance via ssm agent using boto3 and send file [closed]

The answer given by @Nathan Williams is confusing (scp file.txt ec2-user@i-04843lr540028e96a).

When using scp you use SSH as protocol to copy, so you have to setup a username/password or user name ssh keys to copy a file. This scp command doesn't work unless you share keys, specify a region and a profile. The complete command would be something like:

scp -i keyfile file.txt ec2-user@i-04843lr540028e96a --region xxx --profile myprofile 

If you configured your default profile and region you don't have to put them in the command.

I think that what most people is looking for is just an easy way to transfer a file to an ec2 instance only using SSM, like ssm cp file instance-name, which as far as I could research doesn't exist. @Matteo statement is correct, why do I need ssh keys if the whole point of SSM is to get rid of it? Basically what you do is to use SSM as a kind of proxy so you can reach you EC2 machines without having to specify the actual IP address, (may be the instance doesn't have a public IP, or it does and in that case you have to whitelist your source IP in a Security Group), so you reach port 22 of the EC2 instance over SSM by just specifying the i-id but you authenticate over SSH (key and user). SCP works over SSH so you still need a KEY to use it. Again I think that most people was expecting was just plain ssm cp file instance.

If you have SSH over SSM setup, you can just use normal scp, like so:

scp file.txt ec2-user@i-04843lr540028e96a

If it isn't working, make sure you have:

• Session Manager plugin installed locally
• Your key pair on the instance and locally (you will need to define it in your ssh config, or via the -i switch)
• SSM agent on the instance (installed by default on Amazon Linux 2)
• An instance role attached to the instance that allows Session Manager (it needs to be there at boot, so if you just attached, reboot)

Reference: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

If you need more detail, give me more info on your setup, and I'll try and help.

If you have ssm already setup, why do you need to use boto3 and send-file?

aws ssm start-session --target i-xxxx \
 --document-name AmazonEKS-ExecuteNonInteractiveCommand \
 --parameters 'command="cat remotefile"' | tail -n +3 | head -n -3  > file

Where i-xxx is your instance ID, remotefile is the name of the remote file and file is the name it will be called when you get it. Obviously ssm-user will need to be able to read remotefile. If not, you could probably add some sudo magic in there too.

No need to setup ssh keys or S3 buckets or ...whatever.....

The tail -n +3| head -n -3 is needed because I can't see a way to persuade ssm not to print a blank line and "Starting session..." or "Exiting session..." on each connection.

That only works up to about 250kb filesize for me. I don't know if it was corrupting binary or not because my file was bigger.

Alternative:

aws ssm start-session --target i-xxxx \
 --document-name AWSFleetManager-GetFileContent \
 --parameters 'Path=remotefile,PrintInteractiveStatements=No' | tail -n +4 | head -n -3 > file

Only works for text files I think. It nuked my tar file. It might be just doing a unix2dos conversion on the way through.

Ran the file through base64 first and it was happy.

NBs:

• If you're on OSX, the head command might not work, you can use a text editor instead to remove the last few lines.
• Both options require you to press a key when the transfer finishes.

You can use an S3 bucket as proxy. The only thing required is to give EC2 permission to access the S3. This way you don't have to use a SSH protocol to copy files between machines.

All steps are explained here

Even not the OP's requested boto3 approach but here are the vanilla CLI steps:

1. Use ssm-agent to connect to the instance.
2. become ubuntu/ec2-user
3. edit ~/.ssh/authorized_keys and add a line for your personal SSH key. Make sure the permissions are 0600.
4. On your local machine edit ~/.ssh/config and add something like

host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

5. export the required environment variables, e.g. export AWS_PROFILE=your-profile and export AWS_REGION=eu-central-1.

6. verify you can log in via SSH by running ssh i-0123456789 -l ubuntu

7. tar.gz the content up to make it faster as the transfer is slow.

8. SCP the archive of the machine: scp ubuntu@i-0123456789:/home/ubuntu/content.tar.gz ~/Downloads/

What worked for me was to upload the file to S3 first, and then get it from there.

High level plan:

1. install AWS CLI if not already present

2. find out the current identity that the CLI will use

3. locate/create an S3 bucket where you want to upload the file

4. upload the file to the S3 bucket

5. download the file from S3 bucket

1. Install AWS CLI

Try running aws from the command line: if you get an error, follow the instructions at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

2. Find out current IAM identity

aws sts get-caller-identity

You should get something like this:

{
    "UserId": "AROAXKJ7HLGGCEYWXB3UV:i-08cc8005ac8681f86",
    "Account": "123456789012345",
    "Arn": "arn:aws:sts::123456789012345:assumed-role/awsddcdisprod01-CloudWatchAgent/i-0987654321"
}

Copy the value of the Arn attribute:

arn:aws:sts::123456789012345:assumed-role/awsddcdisprod01-CloudWatchAgent/i-0987654321

3. S3 Bucket

Open the AWS Console, into the S3 Buckets section and ope the "Permissions" settings for the bucket you want to upload to (or create a new one first), scroll to "Bucket policy" section and click "Edit", and copy paste a JSON like this, where in the Principal/AWS key you must enter the ARN from the previous step, and replace <BUCKET-NAME> with the actual bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TempUploadFromSSM",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:sts::123456789012345:assumed-role/awsddcdisprod01-CloudWatchAgent/i-0987654321"
      },
      "Action": [
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET-NAME>",
        "arn:aws:s3:::<BUCKET-NAME>/temp/*"
      ]
    }
  ]
}

4. Upload file

Back to the SSM console of the EC2 instance, use a command like this:

aws s3 cp logs.zip s3://<BUCKET-NAME>/temp/

(replace <BUCKET-NAME> with the bucket you changed in step 3)

5. Download file

Use any S3 client or the AWS Web Console itself to download the file you just uploaded.

6. Bonus step

Celebrate! 🥳