How to integrate AWS s3 bucket with AWS ec2 instance (Centos & Ubuntu)

In this tutorial we can check how to mount S3 bucket on your EC@ instance.

S3FS is a FUSE (File System in User Space) will mount Amazon S3 as a local file system. S3FS has an ability to manipulate Amazon S3 bucket in many useful ways. If you wish to access your Amazon S3 bucket without mounting it on your server, you can use s3cmd command line utility to manage S3 bucket.

What is an Amazon S3 bucket?

Amazon S3 is a cloud based web service interface that you can used to store and retrieve any amount of data. To upload your data, first you need to create an S3 bucket. S3 bucket doesn't need any region.

Creating a Bucket

S3 provides an API for creating and managing buckets. You can create a maximum of 100 buckets from your AWS console. When you create a bucket, you need to provide a name and AWS region where you want to create the bucket. In each bucket, you can store any number of objects. You can use your AWS account root credentials to create a bucket, but it is not recommended. Instead  just create an IAM user and add full permission to that user on S3 bucket. You can access your S3 bucket from your Amazon S3 console.

Please follow the below steps to mount s3 bucket on your server.

1) Remove Existing Packages

Before installing any package, first you need to check if you have any existing fuse or S3FS on your server. If it is already existing, then remove it from your server to avoid further conflicts. Use the following command to check if you have any existing fuse or S3FS on your server

CentOS users:

$ yum remove fuse fuse-s3fs

Ubuntu Users:

$ apt-get remove fuse

2) Install Packages

Install all dependency packages for fuse and s3cmd using the below command.

CentOS users:

$ yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap  

yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Ubuntu Users:

$ apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support

3) Download and Compile Fuse

Move to /usr/src then download and compile fuse source code. After compiling, add fuse to kernel. In my case the latest version of fuse is fuse-3.0.0

$ cd /usr/src/

$ wget https://github.com/libfuse/libfuse/releases/download/fuse-3.0.0/fuse-3.0.0.tar.gz

$ tar xzf fuse-3.0.0.tar.gz

$ cd fuse-3.0.0

$ ./configure –prefix=/usr/local

$ make && make install

$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

$ ldconfig

$ modprobe fuse

4) Download and compile S3FS

Navigate to /usr/src, Download and compile s3fs source code.

$ git clone https://github.com/s3fs-fuse/s3fs-fuse.git

$ cd s3fs-fuse

$ ./autogen.sh

$ ./configure

$ make

$ make install

5) Setup Access Key

Both access key and secret key of your s3 AWS account is required for configuring S3FS. Replace the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.

$ vi /etc/passwd-s3fs

AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY

Make sure that the file has proper permission.

$ chmod 600 /etc/passwd-s3fs

6) Mount S3 Bucket

You can run the below command to mount s3fs.

$ s3fs mybucket /path/to/mountpoint -o passwd_file=/etc/passwd-s3fs

You can also mount the s3 bucket on boot by following below commands.

$ mkdir /tmp/cache

$ mkdir /path/to/mountpoint

$ chmod 777 /tmp/cache /path/to/mountpoint

$ vi /etc/fstab

s3fs# /path/to/mountpoint fuse allow_other,use_cache=/tmp/cache,uid=userid,gid=groupid 0 0

$ mount -a

Congratulations you have successfully mounted s3 bucket on your server.