How to transfer files between AWS S3 and AWS EC2

Introduction

EC2 and S3 are regarded as some of the most extensively used services that AWS offers. At times, situations arise where we need to transfer files between the EC2 instance and the S3 bucket.

This article will be a quick primer on how to configure an EC2 instance to communicate with an S3 bucket to transfer files between the two resources. In this article we will use an Instance Profile role for EC2 instance to grant permissions to access our S3 bucket.

Transfer files between AWS S3 and AWS EC2

Prerequisites

1. An EC2 instance should exist where we would be performing actions on the S3 bucket (Documentation)
2. Ensure that AWS CLI is installed on the EC2 instance (Documentation)
3. An S3 bucket with some files to use for the transfer (Documentation)

Adding Instance Profile to grant permissions to EC2 instances

Perform the following steps to add an instance profile to the EC2 instance to grant required permissions to the instance:

1. Log in to the AWS console

2. Navigate to the EC2 details page

3. Select the instance we want to add the instance profile (IAM role) to, click on the Actions drop-down menu, and click on the “Modify IAM role” option

   

4. Select an existing IAM role from the drop-down menu (or create a new one if needed) and click on the “Save” button

   

Adding required S3 permissions to the IAM Instance Profile role

Perform the following steps to add required permission to the IAM instance profile role to allow the instance to perform actions on the S3 bucket

1. Navigate to the IAM details page and click on the roles menu and select the role that was attached to the EC2 instance

   

2. Click on the “Add permissions” button and select the “Attach policies” option

   

3. Filter and attach the “AmazonS3ReadOnlyAccess” policy from the list and click on the “Attach policies” button

   

Copying files from the S3 bucket to the EC2 instance

Perform the following steps to copy a file from an S3 bucket to the EC2 instance:

1. SSH into the EC2 instance

2. Run aws sts get-caller-identity to confirm that the EC2 instance has the correct role attached and AWS CLI is working properly

   

3. Run aws s3 <S3_Object_URI> <Local_File_Path> to copy files from S3 bucket to the EC2 instance

   

Notes

We have added permission to the instance profile, AmazonS3ReadOnlyAccess, which only allows the instance to read files from an S3 bucket and copy them to the instance. The instance can also write objects to S3 buckets, as well as perform other S3 management tasks such as creating new buckets, deleting buckets, etc. by editing the permission attached to the IAM Instance Profile Role.

Additionally, we could have used IAM user credentials with AWS CLI from the instance to perform the same operations on the S3 bucket. While that is another potential method when working with AWS EC2 instances, using IAM Instance Profile Roles is the best practice to grant permissions to the EC2 instance to be able to perform actions on AWS resources, not just limited to S3 buckets.

Conclusion

In this article we saw how we can configure an EC2 instance to use an IAM Instance Profile role, granting it permissions to fetch objects from an S3 bucket to the instance. We looked at how to add the appropriate policy to the instance profile role for the EC2 instance to be able to perform actions on the S3 bucket.

To reiterate, using instance profiles is the best practice method to grant permissions to the EC2 instance instead of using IAM user CLI credentials. Lastly, the actions that are permitted to the EC2 instance via the profile role can be extended by adding additional permissions by either adding existing or custom-managed IAM policies or adding an inline policy to the role.