Amazon Elastic File System (EFS) Brain dump » –

Amazon Elastic File System (EFS) Brain dump » –

Amazon EFS

  • Scalable, elastic, cloud-native NFS file system
  • Provide simple, scalable, fully managed NFS file system for use with AWS Services and on-premises resources.
  • Build to scale on-demand to petabytes without interrupting applications. Eliminate to manage provision the storage, it is automatically scale your storage as needed.
  • EFS has 2 types: EFS standard and EFS Infrequent Access (IA).
  • EFS has lifecycle management (like S3 lifecycle manage) to help move files into EFS IA automatically.
  • EFS IA is cheaper file system.
  • Shared access to thousands of Amazon EC2 instances, enabling high level of aggregate throughput and IOPS with consistent low latencies.
  • Common use cases: Big data analytics, web serving and content management, application development & testing, database backups, containers storage…
  • EFS is Regional service storing data within and cross Available Zone for high availability and durability.
  • Amazon EC2 instances can access cross Available Zone, On-premises resources can access EFS via AWS DX and AWS VPN.
  • EFS can support over 10GB/s, more than 500,000 IOPS.
  • Using EFS Lifecycle management can reduce cost up to 92%.
  • Amazon EFS is compatible with all Linux-based AMIs for Amazon EC2.
  • You do not need to manage storage procurement and provisioning. EFS will grow and shrink automatically as you add or remove files.
  • AWS DataSync provides fast and secure way to sync existing file system to Amazon EFS, even from on-premise over any network connection, including AWS Direct Connect or AWS VPN.
  • Moving files to EFS IA by enabling Lifecycle management and choose age-off policy.
  • Files smaller than 128KB will remain on EFS standard storage, will not move to EFS IA even it is enabled.
  • Speed: EFS Standard storage is single-digit latencies, EFS IA storage is double-digit latencies.
  • Throughput:
    • 50MB/s baseline performance per TB of storage.
    • 100MB/s burst for 1TB
    • More than 1TB stored, storage can burst 100MB/s per TB.
    • Can have Amazon EFS Provisioned Throughput to provide higher throughput.
  • EFS’s objects are redundantly across Available Zone.
  • You can use AWS Backup to incremental backup your EFS.
  • Access to EFS:
    • Amazon EC2 instances inside VPC: access directly
    • Amazon EC2 Classic instance: via ClassicLink
    • Amazon EC2 instances in other VPCs: using VPC Peering Connection or VPC Transit Gateway.
  • EFS can store petabytes of storage. With EFS, you don’t need to provision in advance, EFS will automatically grow and shrink as files added or removed from the storage.
  • Mount EFS via NFS v4

Access Points

  • EFS Access points to simplify application access to shared datasets on EFS. EFS Access points can work with AWS IAM to enforce an user or group, and a directory for every file system request made through the access point.
  • You can create multiple access points and provide to some specific applications.

Encryption

  • EFS support encryption in transit and at rest.
  • You can configure the encryption at rest when creating EFS via console, api or CLI.
  • Encrypting your data is minimal effect on I/O latency and throughput.

EFS and On-premise access

  • To access EFS from on-premise, you have to have AWS DX or AWS VPN.
  • Standard tools like GNU to allow you to copy data from on-premise parallel. It can help faster copy. https://www.gnu.org/software/parallel/

Amazon FSx Windows workload

  • Window file server for Windows based application such as: CRM, ERP, .NET…
  • Backed by Native Windows file system.
  • Build on SSD storage.
  • Can access by thousands of Amazon EC2 at the same time, also provide connectivity to on-premise data center via AWS VPN or AWS DX.
  • Support multiple access from VPC, Available Zone, regions using VPC Peering and AWS Transit gateway.
  • High level throughput & sub-millisecond latency.
  • Amazon FSx for Windows File Server support: SMB, Windows NFS, Active Directory (AD) Integration, Distributed File System (DFS)
  • Amazon FSx can also mount to Amazon EC2 Linux based instances.

Amazon FSx for Luster

  • Fully managed file system that is optimized for HPC (high performance computing), machine learning, and media processing workloads.
  • Hundreds of GB per second of throughput at sub-millisecond latencies.
  • Can be integrated with Amazon S3, so you can join long-term datasets with a high performance system. Data can be automatically copied to and from Amazon S3 to Amazon FSx for Luster.
  • Amazon FSx for Lustra is POSIX-compliant, you can use your current Linux-based applications without having to make any changes.
  • Support read-after-write consistency and support File locking.
  • Amazon Luster can also be mounted to an Amazon EC2 instance.
  • Connect to onpremise via AWS DX, or AWS VPN.

Data Transfer

  • EFS Data transfers between Region using AWS DataSync
  • EFS Data transfer within Region using AWS Transfer Family endpoint

Limitations

Pricing

  • Pay for storage
  • Pay for read and write to files (EFS IA)

Leave a Reply