Introduction
Amazon S3 Files transforms your S3 buckets into fully-featured, high-performance file systems accessible directly from AWS compute resources like EC2, ECS, EKS, and Lambda. This eliminates the traditional trade-off between object storage and file systems, allowing you to enjoy the durability and cost-effectiveness of S3 while gaining interactive file system capabilities. With S3 Files, any change made on the file system is automatically synced back to the S3 bucket, and you have fine-grained control over data synchronization. This guide walks you through setting up and optimizing S3 Files for your workloads.
What You Need
- An active AWS account with appropriate IAM permissions to create and attach S3 file systems.
- An existing S3 bucket (general purpose) that you want to expose as a file system.
- A running AWS compute resource: either an EC2 instance, a container running on ECS or EKS, or a Lambda function.
- Network connectivity between your compute resource and S3 (e.g., VPC endpoints or private links).
- The Amazon S3 File System driver installed on your compute resource (for container environments) or the built-in support for Lambda and EC2.
- Familiarity with NFS v4.1+ operations (optional but helpful).
Step-by-Step Guide
Step 1: Create or Identify Your S3 Bucket
Before attaching S3 Files, ensure you have an S3 bucket ready. If you don’t have one, create a new general purpose bucket using the AWS Management Console, CLI, or SDK. Remember that S3 Files works with any general purpose bucket, so you can use existing buckets without modification. Keep the bucket name and AWS region handy.
Step 2: Prepare Your Compute Resource
Your compute resource (EC2 instance, ECS task, EKS pod, or Lambda function) must have network access to S3. If the resource is in a VPC, configure VPC endpoint for S3 or use a NAT gateway. For EC2, launch an instance with a security group that allows NFS traffic (port 2049) from the file system’s mount target. For containers, ensure the container runtime supports volume mounting with the S3 Files driver. For Lambda, check that your function has the necessary IAM role with s3:ListBucket and s3:GetObject permissions.
Step 3: Attach S3 Files to Your Compute Resource
S3 Files is attached automatically when you mount the file system. The attachment process differs based on the compute type:
- For EC2 instances: Use the standard
mountcommand with the NFS mount point provided by S3 Files. The mount command syntax ismount -t nfs -o nfsvers=4.1.:/ /local/mount/point - For ECS and EKS: Define a volume in your task definition or pod spec using the
s3volume driver. For ECS, specify thes3driver in the task definition’svolumessection. For EKS, use a PersistentVolume with the CSI driver for S3. - For Lambda: Mount the file system via the
efsconfiguration in the function’s settings (Lambda uses EFS access points, but S3 Files uses the same underlying NFS interface). You may need to configure the function’s VPC and mount target.
After mounting, the S3 bucket appears as a local directory. You can verify by listing files with ls.
Step 4: Work with Files and Directories
Once mounted, you can perform all standard NFS v4.1+ operations: create, read, update, and delete files and directories. S3 objects are automatically mapped to files (with key names as paths). For example, an object with key data/report.pdf appears as /mnt/s3/data/report.pdf. Any change you make on the file system is immediately reflected in the S3 bucket, and vice versa. You can also manage permissions using standard POSIX file permissions (if your compute resource supports them).
By default, S3 Files uses high-performance local storage for files that benefit from low-latency access. For large sequential reads, it automatically serves data directly from Amazon S3 to maximize throughput. Byte-range reads transfer only the requested bytes, minimizing data movement and costs.
Step 5: Optimize Performance with Caching and Pre-fetching
S3 Files includes intelligent caching mechanisms:
- High-performance storage: Frequently accessed files and metadata are stored on local storage for low latency.
- Intelligent pre-fetching: The system predicts your next data accesses and loads them in advance.
- Fine-grained control: You can configure what gets cached—whether to load full file data or only metadata. This is especially useful for workloads with predictable access patterns.
To adjust caching behavior, modify the mount options. For example, you can specify cache_mode=metadata to cache only metadata, reducing storage usage on the compute side.
Step 6: Share Data Across Compute Resources
One of the key benefits of S3 Files is the ability to attach the same file system to multiple compute resources simultaneously. This enables data sharing without duplication. Simply mount the same S3 bucket from different instances, containers, or functions. All changes made by one resource are visible to others in real time. This is ideal for collaborative ML training, multi-instance data processing, or shared configuration.
Tips
- Start with a test bucket: Before migrating production workloads, experiment with a small, non-critical bucket to understand behavior and performance.
- Monitor IAM permissions: Ensure the IAM role attached to your compute resource has the minimum required S3 permissions (
s3:GetObject,s3:PutObject,s3:DeleteObject,s3:ListBucket). Overly broad permissions can lead to security risks. - Use VPC endpoints: To reduce data transfer costs and improve latency, set up an S3 VPC endpoint (Gateway or Interface type) in the same VPC as your compute resources.
- Leverage caching wisely: If your workload involves random small reads, cache full file data. For large sequential reads, let S3 Files stream from S3 directly to avoid filling local storage.
- Test with your specific access patterns: Run benchmarks on your own data to fine-tune caching options and mount parameters.
- Keep the S3 Files driver updated: AWS releases improvements regularly—check documentation for the latest version.
- Remember the NFS v4.1+ requirement: Ensure your compute resource’s OS or container image includes NFS client support.