[AWS Experiment] Use AWS DataSync and Storage Gateway to migrate local NFS shares

This lab demonstrates how to use AWS DataSync and AWS Storage Gateway file gateway to migrate data from a local Network File System (NFS) server to Amazon Simple Storage Service (Amazon S3).

Article directory

Experiment overview

AnyCompany makes heavy use of Network File System (NFS) file shares to conduct day-to-day business. The chief information officer (CIO) is concerned that the data on the local NFS file server is not adequately backed up or protected from a disaster at the primary data center. The budget is tight. She would rather invest money to improve her business than purchase expensive backup or replication solutions that also require significant staff time to deploy and maintain. She asks you to provide a solution that meets the following requirements:

Protect data on local NFS file shares from corporate data center disasters
Economical and cost-effective
Easy to deploy and maintain
Existing data can be quickly migrated
Future data can be replicated off-site with minimal interaction with the support team

After extensive research, you decide to use AWS DataSyncAWS Storage Gateway File Gateway for a proof of concept.

DataSync meets many of these requirements. The service:

Copy existing data from a local NFS file server to Amazon Simple Storage Service (Amazon S3) for secure, redundant storage
Comes with a built-in scheduling mechanism that allows you to perform periodic data transfer tasks to detect and copy changes from the source storage system to the target storage system.
Ensure data is intact during transmission. For each transfer, the service performs integrity checks both in transit and at rest. These checks ensure that writing
The underlying data matches the data read from the source, thus verifying consistency.
Pricing per GB of data moved makes it cost-effective and predictable
Deploy in minutes

Storage Gateway file gateway enables continuous transfer of data to and from Amazon S3. The file gateway acts as a file system mount on the S3 bucket. The File Gateway NFS file share replaces the existing local NFS file server, allowing you to decommission the server. Doing so will free up local resources and reduce maintenance time.

Target

After completing this lab, you will be able to:

Deploy and activate the DataSync agent as an Amazon Elastic Compute Cloud (Amazon EC2) instance
Create a DataSync task to copy data from a Linux-based NFS server to an S3 bucket
Deploy and activate the Storage Gateway file gateway appliance as an EC2 instance
Create an NFS file share on the file gateway
Configure the Linux host to connect to the NFS share on the file gateway

lab environment

This lab starts with two Linux instances in different subnets: one instance acts 本地客户端主机and the other acts 本地 Linux NFS 文件服务器. You now need to DataSync 代理实例deploy to the same subnet as your local NFS file server. Then, configure the instance to copy the sample data to an S3 bucket. Finally, you need to do the following:

Deploy the file gateway appliance into the same subnet as the local NFS file server
Create an NFS file share on the file gateway device
Reconfigure the Linux client host to connect to the new share

The following image shows the resources provisioned for this lab and how they will be connected at the end of the lab:

Task 1: Connect to local NFS server

A test environment needed to be created to perform the proof of concept, so a Linux NFS file server and a Linux client host were deployed.

In this task, you complete the configuration of your environment by using a Linux client to mount an NFS file share hosted by a Linux NFS server instance. You then need to copy the data from the client instance to the NFS file share and confirm that the copy was successful.

The following figure shows the architecture of this task:

Task 1.1: Mount the local NFS share

Connect to the Client Linux instance through the session manager.

Command : To mount the local NFS share to the client instance, run the following command. WillReplace with NfsServerPrivateIp** value:

sudo mount <NfsServerPrivateIp>:/var/nfs /mnt/nfs

sudo mount 10.10.2.180:/var/nfs /mnt/nfs

Command : To verify that the NFS file share was mounted successfully, run the following command:

df -h

Note : The df command displays all currently mounted file systems and the available disk space on each system. The -h tag displays the size value in a more readable format, such as "1K" instead of "1000".

Expected output :

[root@ip-10-10-1-188 ~]# df -Th
Filesystem           Type      Size  Used Avail Use% Mounted on
devtmpfs             devtmpfs  465M     0  465M   0% /dev
tmpfs                tmpfs     473M     0  473M   0% /dev/shm
tmpfs                tmpfs     473M  416K  472M   1% /run
tmpfs                tmpfs     473M     0  473M   0% /sys/fs/cgroup
/dev/nvme0n1p1       xfs        10G  1.7G  8.4G  17% /
tmpfs                tmpfs      95M     0   95M   0% /run/user/0
10.10.2.180:/var/nfs nfs4       10G  1.7G  8.4G  17% /mnt/nfs

The client instance has two directories ( /data/DataSync and /data/FileGateway ) that contain the sample data for this lab. Each directory contains 10 .png files.

ll /data/DataSync

ll /data/FileGateway

Command : To copy the sample data from the /data/DataSync directory to an NFS file share, run the following command:

sudo cp /data/DataSync/*.* /mnt/nfs

Stay connected to this session. You will use this in a later task.

Open the NFS server instance terminal.

Command : In a terminal, run the following command to verify that the data in the client instance was copied successfully:

ls /var/nfs

Expected Output : The output should return the 10 .png files that you copied from the client instance.

Stay connected to this session. You will use this in a later task.

You have successfully mounted an NFS file share hosted by a Linux NFS server instance using the Linux client.

Task 2: Deploy and activate the DataSync agent instance

The foundation of the proof-of-concept environment is now complete and the DataSync agent can be deployed.

In this task, you deploy the DataSync agent as an EC2 instance and then activate it, as shown in the following image:

In a physical environment, DataSync can also be deployed as a VMware-based virtual machine.

In the Resources section of the EC2 control panel , select Instances .

Select ****Launch instances****.

Find the Name and tags section and make an entry for the Name field DataSync agent.

Find the Application and OS Images (Amazon Machine Image ) section.

Select the Browse more AMIs button. This will open the Choose an Amazon Machine Image (AMI) wizard.

In the navigation pane, select Community AMIs .

To search for the latest DataSync agent AMI, enter in the search bar aws-datasyncand press Enter.

To the right of the row with the highest version number, select Select .

The console returns to the launch instance screen.

Find the Instance type section.

Select t2.xlarge from the Instance type drop-down menu .

Note : The t2.xlarge instance type is the only instance type that can be successfully deployed in this lab. If you select any other type, a failure message appears at the end of the wizard.

Additional information : In this lab, the t2.xlarge instance type is used as an example only. When deploying the DataSync agent, refer to the documentation for the correct host size.

Find the Key pair ( login) section.

Select Proceed without a key pair (Not recommended) from the Key pair name - required drop-down menu .

Find the Network settings section.

Select the Edit button.

Configure the following settings from the drop-down menu:

For VPC, choose Lab VPC (Experimental VPC)
For Subnet, select Server Subnet
For Auto-assign public IP, select Enable

Select the Select existing security group button.

Select a security group with a name like xxxx-DataSyncAccess-xxxx from the Common security groups drop-down menu .

The DataSyncAccess security group will be configured to allow the following traffic:

Inbound: Port 80 (HTTP) for proxy activation
Outbound: Port 443 used for communication with the DataSync service
Outbound: Port 2049 for NFS v4.1 communication with the server subnet

In the Configure storage section, no changes are required.

Note : DataSync agent instances do not require additional storage beyond the 80GiB root volume.

Find the Summary section.

Select the Launch instance button.

The Launch Instance page appears .

Select the View all instances button.

The Amazon EC2 console appears .

Deployment of the DataSync agent instance takes several minutes. Monitor the deployment status and wait for the status check column to display 2/2 checks passed .

Select the DataSync agent instance from the list . In the lower pane , on the Details tab, locate the Public IPv4 address and Private IPv4 address values. Copy these two values into a text editor for easy use in subsequent steps. (This experiment: Public IPv4 address : 35.91.86.97; Private IPv4 address : 10.10.2.225)

In the AWS Management Console , search using the AWS search barDataSync , and then select the service from the list of results.

The Create Agent page appears .

Select Transfer data > Agents > Create Agent .

If the Deploy agent section is available, select Amazon EC2 from the drop-down menu .

In the Service endpoint section, leave the default Public service endpoints option.

In the Activation key section, for Agent address , enter the Public IPv4 address value for the DataSync agent instance that you copied earlier .

Select Get key.

Note : In the event that the activation page times out, the DataSync agent instance may still be online. Wait another minute or two, then refresh the page and try again.

There will be a green check mark on the next page, which lists the activation keys.

For Agent name, enter NFS DataSync agent.

Scroll down to the bottom of the page and select Create agent .

The DataSync agent page appears with the Created agent message at the top of the page.

The DataSync agent is now successfully deployed as an EC2 instance and activated.

激活密钥
K09GH-03C3J-8BUR8-9K5UA-816NP
代理地址
35.91.86.97

Task 3: Create and run the DataSync task

Data currently existing on an NFS file share can now be copied to an S3 bucket using DataSync.

In this task, you modify the local NFS server configuration to allow connections from the DataSync agent instance. Then, you need to create a new DataSync task to copy the data from the local NFS server to the S3 bucket.

Return to the NFS file server session you opened in Task 1 .

Command : To modify the /etc/exports file to allow connections to the DataSync agent instance, run the following command:

WillReplace with the Private IPv4 address of the DataSync agent instance that you copied in the previous task .10.10.2.225

sudo sh -c 'echo "/var/nfs <DataSyncAgentPrivateIp>(rw,fsid=2,sync,no_subtree_check)" >> /etc/exports'

sudo sh -c 'echo "/var/nfs 10.10.2.225(rw,fsid=2,sync,no_subtree_check)" >> /etc/exports'

Command : To activate changes to the /etc/exports configuration, run the following command:

sudo exportfs -a

Return to the DataSync agent page in the console.

In the Tasks section, select Create task.

Note : Tasks can also be created from the Tasks page (accessed from the left navigation pane).

From the Source location options section, select Create a new location .

From the Location type drop-down menu in the Configuration section , select Network File System (NFS ).

The system displays additional fields.

Make the following configuration:

For Agents, select NFS DataSync agent
For NFS Server, paste the NfsServerPrivateIp value
For Mount path, enter /var/nfs

At the bottom of the page, select Next .

From the Destination location options ( 目的地位置选项) section, select Create a new location .

From the Location type drop-down menu in the Configuration section , select Amazon S3 .

Configure the following:

For S3 bucket , choose the bucket whose name contains nfs-bucket .
For IAM role , choose the role whose name contains NfsS3BucketAccessRole . You can enter in this field to assist with the search.

Select Next .

Find the Task logging section.

Select Do not send logs to CloudWatch from the Log level drop-down menu .

Leave all other fields at their default options and select Next .

Review the settings you selected and select Create task.

A new page loads with the task ID and a Created task banner at the top.

Wait until Task status changes to Available , this should take about 1–2 minutes.

Select Start and Start with defaults in the upper right corner of the page . This operation will run the DataSync task.

Note : If Start with defaults is not available, select Start . A banner that reads Started execution appears at the top of the page .

To the right of the Started execution banner, select See execution details .

On the Execution details page, wait for the Execution status to change to Success , which should take approximately 3–4 minutes.

In the AWS Management Console , search using the AWS search barS3 , and then select the service from the list of results.

Select the bucket whose name contains nfsbucket to view its contents.

In addition to the aws-datasync-metadata file, it lists the 10 .png files that you copied to the NFS file share in Task 1.

Note : When you copy a file or folder to Amazon S3, there is a one-to-one correspondence between the file or folder and the object. File and folder metadata timestamps and POSIX permissions (including user IDs, group IDs, and permissions) are stored in Amazon S3 user metadata. File metadata stored in Amazon S3 user metadata interoperates with File Gateway, providing local file-based access to DataSync data stored in Amazon S3.

When DataSync copies from an NFS server, POSIX permissions for files and folders on the source are stored in Amazon S3 user metadata. When copying from an SMB file share, default POSIX permissions are stored in Amazon S3 user metadata.

When DataSync copies the object containing this user metadata back to the NFS server, the file metadata is restored. When copying back to an SMB file share, ownership is set according to the user configured in DataSync to access the file share, and default permissions are assigned.

The local NFS server configuration has been modified to allow connections to the DataSync agent instance. You also created a new DataSync task to copy data from the local NFS server to the S3 bucket.

Task 4: Deploy and activate Storage Gateway file gateway

Now that the existing data has been migrated from the NFS file share to Amazon S3, it is time to deploy a file gateway to host the new NFS file share.

In this task, you need to deploy the file gateway appliance as an EC2 instance and then activate it, as shown in the following figure:

In the AWS Management Console , search using the AWS search barStorage Gateway , and then select the service from the list of results.

The start page should appear. This page appears because you have not configured the Storage Gateway service in this zone.

Select Create gateway .

In the upper right corner of the page, verify that the AWS Region is the same as the Region value on the left side of the experiment page.

The Set Gateway page appears .

In the Gateway settings section, for Gateway name, enter File Gateway.

In the Gateway options section, select Amazon S3 File Gateway .

In the Platform options section, select Amazon EC2 , and then select the Customize your settings option.

In the Set up gateway on Amazon EC2 section, choose Launch instance .

A new tab opens with the EC2 instance launch wizard. This link automatically selects the correct Amazon Cloud Machine Image (AMI) that you must use with your File Gateway appliance.

Find the Name and tags section and make an entry for the Name field File Gateway appliance.

Find the Instance type section.

Select t2.xlarge from the Instance type drop-down menu .

Note :

The t2.xlarge instance type was the only instance type that could be successfully deployed in this lab. If you select any other type, a failure message appears at the end of the wizard.
In this lab, the t2.xlarge instance type is used as an example only. When deploying a Storage Gateway appliance, be sure to refer to the documentation for the correct device size.

Find the Key pair ( login) section.

Select Proceed without a key pair (Not recommended) from the Key pair name - required drop-down menu .

Find the Network settings section.

Select the Edit button.

Configure the following settings from the drop-down menu:

For VPC, choose Lab VPC (Experimental VPC)
For Subnet, select Server Subnet
For Auto-assign public IP, select Enable

Select the Select existing security group button.

Select a security group with a name like xxxx-FileGatewayAccess-xxxx from the Common security groups drop-down menu .

The FileGatewayAccess security group will be configured to allow the following traffic:

Inbound: Port 80 (HTTP) for gateway activation
Inbound: Port 2049 for NFS v4.1 traffic from the client subnet
Outbound: Port 443 for communication with Storage Gateway
Outbound: Port 2049 for NFS v4.1 communication with the client subnet

Additional information : For more information about the ports used by Storage Gateway, see Network and Firewall Requirements .

Go to the Configure storage section.

Note : There is already a volume named Root Volume here. Do not update this volume.

Select the Add new volume button and configure the following:

Enter 150 in the GiB text box.
From the EBS volume drop-down menu, select General Purpose SSD (gp2)

Find the Summary section.

Select the Launch instance button.

The Launch Instance page appears .

Select the View all instances button.

The Amazon EC2 console appears .

Deployment of the File Gateway appliance instance takes several minutes. Monitor the deployment status and wait for the status check column to display 2/2 checks passed .

Select your File Gateway instance from the list. In the lower pane , on the Details tab, locate Public IPv4 address . Copy the IP address listed. You will use this value when completing your File Gateway deployment.

Return to the AWS Storage Gateway tab in your browser . It should still be on the Setup Gateway page.

In Platform options , verify that Amazon EC2 is selected , select I completed all the steps… , and then select Next .

In the Endpoint options section, for Service endpoint , select Publicly accessible .

In the Gateway connection options section, select IP address , for IP address, paste the Public IPv4 address that you copied earlier for the File Gateway appliance instance , and then select Next (Next step).

Public IPv4 address 34.217.32.68
Private IPv4 address 10.10.2.190

Note : In the event that the activation page times out, the File Gateway instance may still be online. Wait another minute or two, then refresh the page and try again.

The Activate Gateway page appears .

View this page and select Activate gateway .

The Configure Gateway page appears .

On the Configure Gateway page, configure the following:

In the Configure cache storage section, wait for the preparing local disk to finish processing (approximately 1–2 minutes).

Select Cache from the Allocated to drop-down menu .

In the CloudWatch log group section, select Deactivate logging .

Select Configure .

Task 5: Create an NFS share on the file gateway and reconfigure the Linux client

Proof of concept is almost complete. The file gateway has been deployed and it is time to create an NFS file share to attach to Linux clients.

In this task, you will perform the following actions:

Create an NFS file share on the file gateway.
Reconfigure the Linux client to mount the new NFS file share.
Copy the second set of sample data to the new NFS file share.

If you have not already gone to the Gateway section of the Storage Gateway console that appeared in the previous task , search using the AWS search barStorage Gateway and select the service from the list of results.

In the navigation pane, make sure Gateways is selected .

Select Create file share .

Note : You can also create new file shares from the File shares section of the Storage Gateway console .

The Profile Sharing Settings page appears .

On the File Sharing Settings configuration page, configure the following:

For Gateway, select File Gateway (sgw-xxxx).
For File share protocol, select NFS.
For S3 bucket, select the bucket whose name contains nfsbucket from the drop-down menu.

Select Customize configuration to set additional properties instead of the default configuration.

Scroll to the bottom of the page and select Next .

The Amazon S3 storage settings page appears .

Configure the following:

For Storage class for new objects, select S3 Standard
For Object metadata, check:
- Guess MIME type
- Gateway accessible files to cross-account S3 bucket owner (cross-account S3 bucket owner accessible gateway files)
For Access to your S3 bucket, check Use an existing IAM role

arn:aws:iam::454566530271:role/LabStack-2848353a-a5d2-4c70-NfsS3BucketAccessRole-1ASGTSPZRM8F4

For IAM role, paste the NfsS3AccessPolicyARN value from the left side of the experiment page

Select Next .

Note : A warning message will appear indicating that the file share can be accessed from any location. AWS recommends that you be sure to grant access only to the clients that are required in your environment.

Select ****Add a client****.

To grant hosts in the client subnet access to the share, add an entry in the Allowed clients text field 10.10.1.0/24.

Select Next .

Verify that the Allowed clients list is updated with the value you added.

Check Create .

The file sharing overview page appears .

Wait for the Status column to show Available , which should take less than a minute.

Note : You may need to click the refresh icon at the top of the page from time to time.

Check the File share ID link.

In the lower pane, find the command to mount the file share on Linux and copy the command into a text editor.

In a text editor, replace [MountPath] in the command with /mnt/nfs

mount -t nfs -o nolock,hard 10.10.2.190:/labstack-2848353a-a5d2-4c70-b586-997e2b-nfsbucket-atrm8otxva0j [MountPath]

mount -t nfs -o nolock,hard 10.10.2.190:/labstack-2848353a-a5d2-4c70-b586-997e2b-nfsbucket-atrm8otxva0j /mnt/nfs

Return to the NFS client instance session that you opened in task 1.

Command : In the terminal, to disconnect the existing connection to the local NFS file server, run the following command:

sudo umount -f /mnt/nfs

Command : Enter sudoand paste the command in a text editor to mount the File Gateway NFS file share. The output of this command should be similar to the following:

sudo mount -t nfs -o nolock,hard 10.10.2.33:/nfs-bucket-ql-23453634245 /mnt/nfs

sudo mount -t nfs -o nolock,hard 10.10.2.190:/labstack-2848353a-a5d2-4c70-b586-997e2b-nfsbucket-atrm8otxva0j /mnt/nfs

Run this command.

Command : To verify that the NFS file share was mounted successfully, run the following command:

df -h

Expected output :

[root@ip-10-10-1-188 ~]# df -h
Filesystem                                                                   Size  Used Avail Use% Mounted on
devtmpfs                                                                     465M     0  465M   0% /dev
tmpfs                                                                        473M     0  473M   0% /dev/shm
tmpfs                                                                        473M  416K  472M   1% /run
tmpfs                                                                        473M     0  473M   0% /sys/fs/cgroup
/dev/nvme0n1p1                                                                10G  1.7G  8.4G  17% /
10.10.2.180:/var/nfs                                                         8.0E     0  8.0E   0% /mnt/nfs
tmpfs                                                                         95M     0   95M   0% /run/user/0
10.10.2.190:/labstack-2848353a-a5d2-4c70-b586-997e2b-nfsbucket-atrm8otxva0j  8.0E     0  8.0E   0% /mnt/nfs

Command : To verify that the 10 .png files previously copied to the NFS file share exist, run the following command:

ls /mnt/nfs

Expected output :

There are 10 .png files listed in the output.

Command : To copy the second set of data to the File Gateway NFS file share, run the following command:

sudo cp /data/FileGateway/*.* /mnt/nfs

Return to the AWS Management Console. Search using the AWS search barS3 and select the service from the list of results.

Select the bucket whose name starts with nfs-bucket and view its contents.

A total of 20 .png files are listed in the returned results: the 10 files copied to the NFS file share in Task 1 (1-10) and the 10 files just copied (11-20).

The NFS file share has been created on the file gateway, the Linux client has been reconfigured to mount the new NFS file share, and the second set of sample data has been copied to the new NFS file share.

Summarize

Successfully completed the following tasks:

Deploy and activate the DataSync agent as an EC2 instance
Create a DataSync task to copy data from a Linux-based NFS server to an S3 bucket
Deploy and activate the Storage Gateway file gateway appliance as an EC2 instance
Create an NFS file share on the file gateway
Configure the Linux host to connect to the NFS share on the file gateway