When talking to customers and users, this question is frequently left open with the note that they’ll check and tell us later. With storage being one of the main cost factors of cloud services such as Amazon’s AWS, this is not what it should be.

But who could blame them? It’s not like AWS is making it obvious to you how much of your storage resources (not only capacity but especially IOPS and throughput) you really use. It might be bad for AWS’ revenue.

Why We Built This

We believe that there is no reason to pay more than necessary. However, since it’s so hard to get hard facts on storage use, we tend to overprovision—by a lot.

Hence, we decided to do something about it. Today, we’re excited to share our new open-source tool – the AWS EBS Volume Usage Exporter!

What makes this particularly interesting is that, based on our experience, organizations typically utilize only 20-30% of their provisioned AWS EBS volumes. That means 70-80% of provisioned storage is sitting idle, quietly adding to your monthly AWS bill. Making someone else happy but you.

What Our Tool Does

The EBS Volume Usage Exporter runs in your EKS cluster and collects detailed metrics about your EBS volumes, including:

• Actual usage patterns
• IOPS consumption
• Throughput utilization
• Available disk space
• Snapshot information

All this data gets exported into a simple CSV file that you can analyze however you want.

If you like convenience, we’ve also built a handy calculator (that runs entirely in your browser – no data leaves your machine!) to help you quickly understand potential cost savings. Here’s the link to our EBS Volume Usage Calculator. No need to use it, though. The data is easy enough to get basic insight. Our calculator just automates the pricing and potential saving calculation based on current AWS price lists.

Super Simple to Get Started

To get you started quickly, we packaged everything as a Helm chart to make deployment as smooth as possible. You’ll need:

• An EKS cluster with cluster-admin privileges
• An S3 bucket
• Basic AWS permissions

The setup takes just a few minutes – we’ve included all the commands you need in our GitHub repository.

After a successful run, you can simply delete the helm chart deployment and be done with it. The exported data are available in the provided S3 bucket for download.

We Want Your Feedback!

This is just the beginning, and we’d love to hear from you!

Do the calculated numbers match your actual costs?
What other features would you find useful?

We already heard people asking for a tool that can run outside of EKS, and we’re looking into it. We would also love to extend support to utilize existing orchestration infrastructure such as DataDog, Dynatrace, or others. Most of the data is already available and should be easy to extract.

For those storage pros out there who can answer the EBS utilization question off the top of your head – we’d love to hear your stories, too!

Share your experiences and help us make this tool even better.

Try It Out!

The AWS EBS Volume Usage Exporter is open source and available now on GitHub. Give it a spin, and let us know what you think!

And hey – if you’re wondering whether you really need this tool, ask yourself: “Do I know exactly how much of my provisioned EBS storage is actually being used right now?”

If there’s even a moment of hesitation, you should check this out!

At simplyblock, we’re passionate about helping organizations optimize their cloud storage. This tool represents our commitment to the open-source community and our mission to eliminate cloud storage waste.

The post We Built a Tool to Help You Understand Your Real EBS Usage! appeared first on simplyblock.

For companies that use Amazon Web Services (AWS) for storage, having firm control over costs is key. Mismanaging storage can directly lead to soaring expenses and unexpected losses in a fast-paced world with fluctuating workloads. A reliable and effective way to keep costs in check for a business is to implement efficient storage solutions and in the right size. This can ensure solid growth and performance, making it a plan that works for AWS-reliant businesses.

This article will look at several good ways to manage storage costs on AWS and give you practical tips to size storage in changing environments.

What Does Right-Sizing Mean for AWS Storage?

Choosing the right size in AWS means going with suitable storage solution types and scales based on the particular needs of a business. It’s a major strategic factor that can help you achieve and maintain a competitive lead in the market by avoiding unnecessary costs and putting a stop to overspending. All it takes is actively monitoring storage policies, checking how much storage usually goes unused, and proactively making prompt changes accordingly.

AWS offers various storage types, including Amazon S3 for object storage, Amazon EBS for block storage, and Amazon EFS for file storage, each suitable for different applications. By right-sizing, businesses can avoid paying for idle storage resources and only use what’s necessary.

AWS Storage Services and The Cost-Savings They Offer

With AWS, you get a few storage options at different price points that you can choose from based on your business needs

• Amazon S3 (Simple Storage Service) offers an incredible amount of scalability, allowing growing businesses to adapt well. It works well for unorganized data and uses a pay-as-you-go system, which keeps costs down when storage needs change.
• Amazon EBS (Elastic Block Store) gives lasting block storage for EC2 instances. EBS prices change based on the type of volume, its size, and input/output actions, so you need to watch it to keep expenses in check.
• Amazon EFS (Elastic File System) is a managed file storage service that grows on its own, which helps applications that need shared storage. While it’s convenient, data costs can rise as volume grows.

To reduce overall cloud spending, it’s essential to understand which storage type suits your workloads and to handle these services.

Editor’s note: If you’re looking for ways to consolidate your Amazon EBS volumes, simplyblock got you covered.

Ways to Optimize Storage Size on AWS

1. Use Storage Class Levels

Amazon S3 offers various storage classes with different costs and speeds. You can save money by placing data in the appropriate class based on access frequency and retrieval speed needs. Here’s a breakdown:

• S3 Standard is best for accessed data, but it’s more expensive.
• S3 Infrequent Access (IA) is cheaper for less-used data that still needs retrieval.
• S3 Glacier and Glacier Deep Archive are the least expensive options for long-term accessed data.

You can cut costs without losing access by reviewing and moving data to suitable storage classes based on usage patterns.

2. Set Up Data Lifecycle Rules

Managing data lifecycles helps companies with changing storage needs to save money. AWS lets you make rules to move, store, or delete data based on certain conditions. With S3 lifecycle policies, you can set up your data to move from S3 Standard to S3 IA and then to Glacier or be removed after a set time.

3. Use Automatic Tracking and Warnings

AWS offers tools to keep an eye on storage use and costs, like AWS CloudWatch and AWS Budgets. These tools help spot wasted resources, odd spikes in use, or costs that go over your set budget. Setting up warnings through AWS Budgets can tell you when you’re close to your budget limit and stop extra costs before they pile up.

4. Make EBS Volumes the Right Size

Elastic Block Store (EBS) volumes often waste resources when they’re bigger than needed. Checking EBS use often can show volumes that aren’t used much or at all. AWS has a tool called EBS Right Sizing Recommendation that helps find volumes you can make smaller without slowing things down.

EBS provides Volume Types such as General Purpose (gp3), Provisioned IOPS (io2), and Throughput Optimized (st1). Picking the right volume type for each workload, plus sizing, cuts costs. You pay for the storage performance and capacity you need.

Editor’s note: Save up to 80% on your high-performance AWS storage costs.

Smart Ways to Cut AWS Storage Costs Further

1. Use Reserved Instances and Savings Plans

For workloads you can predict, think about AWS Reserved Instances (RIs) and Savings Plans. People often link these to EC2 instances, but they also offer cheap options for related EBS storage. RIs and Savings Plans let you promise to use a certain amount, giving you lower rates if you commit for one or three years. This works best for steady workloads that need the same amount of storage over time, where you’re less likely to buy too much.

Savings plans shouldn’t just be limited to storage solutions. You can also reconsider other costs, such as switching to a cheap web hosting service that still meets your business needs.

2. Make Multi-Region Storage More Efficient

AWS gives you ways to copy your data across different regions. This makes your data more secure and helps you recover if something goes wrong. But storing data in multiple regions can cost a lot because of copying and moving data between regions. To cut these costs, you can look at how people use your data and put it in regions close to most of your users.

3. Consider Spot Instances for Short-Term Storage Needs

Spot Instances offer a more affordable option to handle tasks that can cope with interruptions. You can use short-term storage on Spot Instances for less crucial brief projects where storage requirements fluctuate. When you combine Spot Instances with Amazon EBS or S3, you gain flexibility and cut costs. However, remember that AWS has the right to reclaim Spot Instances at any moment. This makes them unsuitable for critical or high-availability tasks.

Summing Up: Managing AWS Storage Costs

Smart AWS cost control begins with a hands-on strategy to size storage. This includes picking the right S3 storage types, setting up lifecycle rules, keeping an eye on EBS use, or taking advantage of reserved options. These methods can help you keep a lid on your storage bills.

When you check usage and put these tried-and-true tips into action, you’ll be in a better position to handle your AWS expenses. At the same time, you’ll keep the ability to scale and the reliability your workloads need. In a cloud world where storage costs can get out of hand, clever management will pay off. It’ll help your company stay nimble and budget-friendly.

The post AWS Cost Management: Strategies for Right-Sizing Storage in Dynamic Environments appeared first on simplyblock.

Amazon Elastic Block Store (EBS) provides scalable, high-performance block storage for use with Amazon EC2 instances. It is an essential component for businesses relying on AWS for their infrastructure, offering persistent, reliable storage that can be tailored to suit a wide variety of workloads. However, to maximize the effectiveness of your Amazon EBS usage, a range of open-source tools can streamline operations, optimize costs, and improve performance. In this post, we’ll explore nine must-know open-source tools that can help you get the most out of Amazon EBS.

1. EBS Snapper

EBS Snapper is an open-source tool that automates the creation and management of EBS snapshots. It helps manage snapshot policies by allowing users to schedule regular backups, ensuring data protection and compliance with backup policies. With EBS Snapper, you can automate the cleanup of old snapshots to save costs.

2. Cloud Custodian

Cloud Custodian is a rules engine for managing your AWS resources, including EBS. It helps ensure cost-effective use of Amazon EBS by automating actions such as snapshot creation, deletion of unused volumes, and enforcing lifecycle policies. Cloud Custodian is widely used for enforcing governance and operational best practices in AWS environments.

3. AWS Tools for PowerShell

AWS Tools for PowerShell allows you to manage your Amazon EBS volumes and snapshots using PowerShell scripting. With this tool, you can automate tasks such as provisioning new EBS volumes, resizing them, and managing backups, all through familiar PowerShell commands.

4. Boto3 (AWS SDK for Python)

Boto3 is the official AWS SDK for Python, enabling developers to interact programmatically with Amazon EBS. You can use Boto3 to create and manage EBS volumes, automate snapshot creation, and handle failover scenarios. It’s a great tool for developers who want to script complex tasks in their AWS environment.

5. Elastic Volumes CLI

The Elastic Volumes CLI is an open-source command-line tool that helps manage the resizing of Amazon EBS volumes. This tool allows you to automate the process of resizing volumes to meet changing storage needs, minimizing downtime and optimizing performance. It’s especially useful for dynamically scaling storage in response to workload changes.

6. EBS Optimizer

EBS Optimizer is a performance tuning tool that analyzes the usage of your EBS volumes and provides recommendations to optimize performance and cost. By monitoring IOPS and throughput, it helps you adjust volume types, resize volumes, or consolidate underutilized volumes to save on costs while maintaining performance.

7. ec2-snapper

ec2-snapper is an open-source tool for automating the snapshot process for EBS volumes attached to EC2 instances. It allows for easy configuration of snapshot schedules, retention policies, and email notifications, making it a simple solution for managing backups and disaster recovery in an AWS environment.

8. EBS Volume Cleaner

EBS Volume Cleaner is a small but effective tool that scans your AWS environment for orphaned or unused EBS volumes and helps you delete them to reduce costs. This tool is particularly useful in large-scale environments where it’s easy to lose track of unused resources that continue to incur charges.

9. Terraform

Terraform is an infrastructure-as-code tool that can be used to provision and manage Amazon EBS volumes in a scalable and automated way. With Terraform, you can define EBS resources in code and version them, allowing for efficient deployment and management of your storage infrastructure. Terraform’s flexibility and community support make it a popular choice for automating AWS resources.

Why Choose simplyblock for Amazon EBS?

While EBS provides flexible block storage, organizations often struggle with storage sprawl and rising costs. This is where SimplyBlock’s specialized EBS management approach creates unique value:

Intelligent Storage Consolidation

Simplyblock enables efficient consolidation of Amazon EBS volumes without compromising performance. By implementing intelligent volume pooling, organizations can reduce their EBS footprint significantly while maintaining the same performance levels. This approach helps eliminate storage silos and reduces costs by optimizing volume utilization.

Dynamic Resource Optimization

Simplyblock automatically manages EBS resources based on actual usage patterns. Instead of maintaining separate volumes with individual IOPS allocations, simplyblock’s pooling technology allows for dynamic resource sharing, ensuring applications get the performance they need while minimizing unused capacity and cost.

Simplified Storage Management

Simplyblock streamlines EBS management by providing a unified approach to volume provisioning and allocation. Rather than managing individual volumes, organizations can leverage simplyblock’s pooling capabilities to simplify storage operations and reduce administrative overhead, all while maintaining native AWS integration.

How to Optimize Amazon EBS with Open-source Tools

This guide explored nine essential open-source tools for Amazon EBS management, from EBS Snapper’s automated backups to Terraform’s infrastructure-as-code capabilities. While these tools excel at different aspects – Cloud Custodian for governance, Boto3 for programmatic control, and EBS Optimizer for performance tuning – proper implementation is crucial. Tools like ec2-snapper enable automated snapshots, while EBS Volume Cleaner helps optimize costs. Each tool offers unique capabilities for managing and optimizing EBS resources.

If you’re looking to further streamline your Amazon EBS operations, simplyblock offers comprehensive solutions that integrate seamlessly with these tools, helping you get the most out of your Amazon EBS environment.

Ready to optimize your Amazon EBS environment? Contact simplyblock today to learn how we can help you enhance performance, reduce costs, and streamline your AWS operations.

The post Best Open Source Tools for Amazon EBS appeared first on simplyblock.

The debate over the optimal storage solution has been ongoing. Local instance storage on AWS (i.e. ephemeral NVMe disk attached to EC2 instance) brings remarkable cost-performance ratios. It offers 20 times better performance and 10 times lower access latency than EBS. It’s a powerhouse for quick, ephemeral storage needs. In simple words, local NVME disk is very fast and relatively cheap, but not scalable and not persistent.

Recently, Vantage posted an article titled “Don’t use EBS for Cloud Native Services“. We agree with this problem statement, however we also strongly believe that there is a better solution that using Local NVMe SSD Storage on AWS as an alternative to EBS. Local NVMe to EBS is not like comparing apples to apples, but more like apples to oranges.

The Local Instance NVMe Storage Advantage

Local storage on AWS excels in speed and cost-efficiency, delivering performance that’s 20 times better and latency that’s 10 times lower compared to EBS. For certain workloads with temporary storage needs, it’s a clear winner. But, let’s acknowledge the reasons why data centers have traditionally separated storage and compute.

Overcoming Traditional Challenges of Local Storage

Simplyblock provides an innovatrive approach that marries the cost and performance advantages of local instance storage with the benefits of pooled cloud storage. It offers the best of both worlds—high-speed, low-latency performance near to local storage, coupled with the robustness and flexibility of pooled cloud storage.

Why Choose simplyblock on AWS?

1. Performance and Cost Efficiency: Enjoy the benefits of local storage without compromising on scalability, reliability, and high availability.
2. Data Protection: simplyblock employs advanced data protection mechanisms,
   ensuring that your data survives any instance outage.
3. Seamless Operations: Upgrade and maintain compute instances without impacting storage, ensuring continuous operations.
4. Data Services Galore: Unlock the potential of various data services without affecting local CPU performance.

While local instance storage has its merits, the future lies in a harmonious blend of the speed of local storage and the resilience of cloud-pooled storage. With simplyblock, we transcend the limitations of local NVMe disk, providing you with a storage solution that’s not just powerful but also versatile, scalable, and intelligently designed for the complexities of the cloud era.

The post Local NVMe Storage on AWS – Pros and Cons appeared first on simplyblock.

For environments with high-performance requirements such as transactional databases, where low-latency access and optimized storage costs are key, alternative solutions are essential. This is where simplyblock steps in, offering a new way to manage storage that addresses common pain points in traditional EBS or local NVMe disk usage—such as limited scalability, complex resizing processes, and the cost of underutilized storage capacity.

What is Simplyblock?

Simplyblock is known for providing top performance based on distributed (clustered) NVMe instance storage at low cost with great data availability and durability. Simplyblock provides storage to Linux instances and Kubernetes environments via the NVMe block storage and NVMe over Fabrics (using TCP/IP as the underlying transport layer) protocols and the simplyblock CSI Driver.

Simplyblock’s storage orchestration technology is fast. The service provides access latency between 100 us and 500 us, depending on the IO access pattern and deployment topology. That means that simplyblock’s access latency is comparable to, or even lower than on Amazon EBS io2 volumes, which typically provide between 200 us to 300 us.

To make sure we only provide storage which will keep up, we test simplyblock extensively. With simplyblock you can easily achieve more than 1 million IOPS at a 4KiB block size on single EC2 compute instances. This is several times higher than the most scalable Amazon EBS volumes, io2 Block Express. On the other hand, simplyblock’s cost of capacity is comparable to io2. However, with simplyblock IOPS come for free – at absolutely no extra charge. Therefore, depending on the capacity to IOPS ratio of io2 volumes, it is possible to achieve cost advantages up to 10x .

For customers requiring very low storage access latency and high IOPS per TiB, simplyblock provides the best cost efficiency available today.

Why Simplyblock over Simple Amazon EBS?

Many customers are generally satisfied with the performance of their gp3 EBS volumes. Access latency of 6 to 10 ms is fine, and they never have to go beyond the included 3,000 IOPS (on gp2 and gp3). They should still care for simplyblock, because there is more. Much more.

Benefits of Thin Provisioning

With gp3, customers have to pay for provisioned rather than utilized capacity (~USD 80 per TiB provisioned). According to our research, the average utilization of Amazon EBS gp3 volumes is only at ~30%. This means that customers are actually paying more than three times the price per TiB of utilized storage. That said, due to the low utilization below one-third, the actual price comes down to about USD 250 per TiB. The higher the utilization, the closer a customer would be to the projected USD 80 per TiB.

In addition to the price inefficiency, customers also have to manage the resizing of gp3 volumes when utilization reaches the current capacity limit. However, resizing has its own number of limitations in EBS it is only possible once every six hours. To mitigate potential issues during that time, volumes are commonly doubled in size.

On the other hand, simplyblock provides thin provisioned logical volumes. This means that you can provision your volumes nearly without any restriction in size. Think of growable partitions that are sliced out of the storage pool. Logical volumes can also be over-provisioned, meaning, you can set the requested storage capacity to exceed the storage pool’s current size. There is no charge for the over-provisioned capacity as long as you do not use it.

That said, simplyblock thinly provisions NVMe volumes from a storage pool which is either made up of distributed local instance storage or gp3 volumes. The underlying pool is resized before it runs out of storage capacity.

These means enable you to save massively on storage, while also simplifying your operations. No more manual or script-based resizing! No more custom alerts before running out of storage.

Benefits of Storage Tiering

But if you feel there should be even more potential to save on storage, you are absolutely right!

The total data stored on a single EBS volume has very different access patterns. Let’s explore together what the average database setup looks like. The typical corporate’s transactional database will easily qualify as a “hot” storage. It is commonly stored on SSD-based EBS volumes. Nobody would think of putting this database to slow file storage stored on HDD or Amazon S3.

In reality, however, data that belongs to a database is never homogeneous when it comes to performance requirements. There is, for example, the so-called database transaction log, often referred to as write-ahead log (WAL) or simply a database journal. The WAL is quite sensitive to access latency and requires a high IOPS rate for writes. On the other hand, the log is relatively small compared to the entire dataset in the database.

Furthermore, some other data files store tablespaces and index spaces. Many of them are read so frequently that they are always kept in memory. They do not depend on storage performance. Others are accessed less frequently, meaning they have to be loaded from storage every time they’re accessed. They require solid storage performance on read.

Last but not least, there are large tables which are commonly used for archiving or document storage. They are written or read infrequently and typically in large IO sizes (batches). While throughput speed is relevant for accessing this data, access latency is not.

To support all of the above use cases, simplyblock supports automatic tiering. Our tiering will place less frequently accessed data to either Amazon EBS (st2) or Amazon S3, called warm storage. The tiering implementation is optimized for throughput, hence large amounts of data can be written or read in parallel. Simplyblock automatically identifies individual segments of data, which qualify for tiering, and moves them automatically to secondary storage, and only after tiering was successful, cleaning them up on the “hot” tier. This reduces the storage demand in the hot pool.

The AWS cost ratio between hot and warm storage is about 5:1, cutting cost to about 20% for tiered data. Tiering is completely transparent to you and data is automatically read from tiered storage when requested.

Based on our observations, we often see that up to 75% of all stored data can be tiered to warm storage. This creates another massive potential in storage costs savings.

How to Prevent Data Duplication

But there is yet more to come.

The AWS’ gp3 volumes do not allow multi-attach, meaning the same volume cannot be attached to multiple virtual machines or containers at the same time. Furthermore, its reliability is also relatively low (indicated at 99.8% – 99.9%) compared to Amazon S3.

That means neither a loss of availability nor a loss of data can be ruled out in case of an incident.

Therefore, additional steps need to be taken to increase availability of the storage consuming service, as well as the reliability of the storage itself. The common measure is to employ storage replication (RAID-1, or application-level replication). However, this leads to additional operational complexity, utilization of network bandwidth, and to a duplication of storage demand (which doubles the storage capacity and cost).

Simplyblock mitigates the requirement to replicate storage. First, the same thinly provisioned volume can be attached to more than one Amazon EC2 instance (or container) and, second, the reliability of each individual volume is higher (99.9999%) due to the internal use of erasure coding (parity data) to protect the data.

Multi-attach helps to cut the storage cost by half.

The Cost of Backup

Last but not least, backups. Yes there is even more.

A snapshot taken from an Amazon EBS volume is stored in an S3-like storage. However, AWS charges significantly more per TiB than for the same storage directly on S3. Actually about 3.5 times.

Snapshots taken from simplyblock logical volumes, however, are stored into a standard Amazon S3 bucket and based on the standard S3 pricing, giving you yet another nice cost reduction.

Near-Zero RPO Disaster Recovery

Anyhow, there is one more feature that we really want to talk about. Disaster recovery is an optional feature. Our DR comes at a minimum RPO and can be deployed without any redundancy on either the block storage or the compute layer between zones. Additionally, no data transfers between zones are needed.

Simplyblock employs asynchronous replication to store any change on the storage pool to an S3 bucket. This enables a fully crash-consistent and near-real-time option for disaster recovery. You can bootstrap and restart your entire environment after a disaster. This works in the same or a different availability zone and without having to take care of backup management yourself.

And if something happened, accidental deletion or even a successful ransomware attack which encrypted your data. Simplyblock is here to help. Our asynchronous replication journal provides full Point-in-Time-Recovery functionality on the block storage layer. No need for your service or database to support it. Just rewind the storage to whatever point in time in the past.

It also utilizes write- and deletion-protected on its S3 bucket making the journal itself resilient to ransomware attacks. That said, simplyblock provides a sophisticated solution to disaster recovery and cybersecurity breaches without the need for manual backup management.

Simplyblock is Storage Optimization – just for you

Simplyblock provides a number of advantages for environments that utilize a large number of Amazon EBS gp2 or gp3 volumes. Thin provisioning enables you to consolidate unused storage capacity and minimize the spent. Due to the automatic pool enlargement (increasing the pool with additional EBS volumes or storage nodes), you’ll never run out of storage space but also only require the least amount.

Together with automatic tiering, you can move infrequently used data blocks to warm or even cold storage. Fully transparent to the application. The same is true for our disaster recovery. Built into the storage layer, every application can benefit from point in time recovery, removing almost all RPO (Recovery Point Objective) from your whole infrastructure. And with consistent snapshots across volumes, you can enable a full-blown infrastructure recovery in case of an availability zone outage, right from ground up.

With simplyblock you get more features than mentioned here. Get started right away and learn about our other features and benefits.

The post Simplyblock for AWS: Environments with many gp2 or gp3 Volumes appeared first on simplyblock.

Why Migrate to AWS?

Migrating to AWS offers numerous benefits, including scalability, cost savings, improved performance, and enhanced security. AWS’s extensive range of storage solutions caters to diverse needs, from simple object storage to high-performance block storage. By leveraging AWS’s robust infrastructure, businesses can focus on innovation and growth without worrying about underlying IT challenges.

Understanding AWS Storage Options

Before diving into the migration process, it’s crucial to understand the various storage options AWS offers:

• Amazon S3 (Simple Storage Service) Amazon S3 is an object storage service that provides scalability, data availability, security, and performance. It’s ideal for storing and retrieving data at any time.
• Amazon EBS (Elastic Block Store) Amazon EBS provides block storage for EC2 instances. It’s suitable for applications requiring low-latency data access and offers different volume types optimized for performance and cost.
• Amazon EFS (Elastic File System) Amazon EFS is designed to be highly scalable and elastic. It provides scalable file storage for use with AWS Cloud services and on-premises resources.
• Amazon Glacier Amazon Glacier is a secure, durable, and extremely low-cost cloud storage service for data archiving and long-term backup. It’s ideal for data that is infrequently accessed

AWS provides several migration tools, such as AWS DataSync and AWS Snowball, to ensure a smooth and efficient data migration process. Based on your data volume and migration requirements, choose the right tool.

How is data stored in AWS? AWS stores the data of each storage service separately. That means that AWS storage services are not synchronized and your data might be frequently duplicated multiple times. Coordination between AWS storage services might be resolved using orchestration tools such as simplyblock.

Steps for Migrating to AWS

1. Assess your Current Environment

Begin by evaluating your current storage infrastructure. Identify the types of data you store, how often it’s accessed, and any compliance requirements. This assessment will help you choose the right AWS storage services for your needs.

2. Plan your Migration Strategy

Develop a comprehensive migration plan that outlines the steps, timelines, and resources required. Decide whether you’ll use a lift-and-shift approach, re-architecting, or a hybrid strategy.

3. Choose the right AWS Storage Services

Based on your assessment, select the appropriate AWS storage services. For instance, Amazon S3 can be used for object storage, EBS for block storage, and EFS for scalable file storage.

4. Set up the AWS Environment

Set up your AWS environment, including creating an AWS account, configuring Identity and Access Management (IAM) roles, and setting up Virtual Private Clouds (VPCs).

5. Use AWS Migration Tools

AWS offers several tools to assist with migration, such as

• AWS Storage Gateway, which bridges your on-premises data and AWS Cloud storage
• AWS DataSync automates moving data between on-premises storage and AWS
• AWS Snowball physically transports large amounts of data to AWS.

6. Migrate Data

Start migrating your data using the chosen AWS tools and services. Ensure data integrity and security during the transfer process. Test the migrated data to verify its accuracy and completeness.

7. Optimize Storage Performance

After migration, monitor and optimize your storage performance. Use AWS CloudWatch to track performance metrics and make necessary adjustments to enhance efficiency.

8. Ensure Data Security and Compliance

AWS provides various security features to protect your data, including encryption, access controls, and monitoring. Ensure your data meets regulatory compliance requirements.

9. Validate and Test

Conduct thorough testing to validate that your applications function correctly in the new environment. Ensure that data access and performance meet your expectations.

10. Decommission Legacy Systems

Once you’ve confirmed your data’s successful migration and testing, you can decommission your legacy storage systems. Ensure all data has been securely transferred and backed up before decommissioning.

Common Challenges in AWS Migration

1. Data Transfer Speed

Large data transfers can take time. Use tools like AWS Snowball for faster data transfer.

2. Data Compatibility

Ensure your data formats are compatible with AWS storage services. Consider data transformation if necessary.

3. Security Concerns

Data security is paramount. Utilize AWS security features such as encryption and IAM roles.

4. Cost Management

Monitor and manage your AWS storage costs. Use AWS Cost Explorer and set up budget alerts.

Benefits of AWS Storage Solutions

1. Scalability: AWS storage solutions scale according to your needs, ensuring you never run out of space.
2. Cost-Effectiveness: Pay only for the storage you actually use and leverage different storage tiers to optimize costs.
3. Reliability: AWS guarantees high availability and durability for your data.
4. Security: Robust security features protect your data against unauthorized access and threats.
5. Flexibility: Choose from various storage options for different workloads and applications.

Conclusion

Migrating to AWS from a storage perspective involves careful planning, execution, and optimization. By understanding the various AWS storage options and following a structured migration process, you can ensure a smooth transition to the cloud. AWS’s comprehensive suite of tools and services simplifies the migration journey, allowing you to focus on leveraging the cloud’s benefits for your business.

FAQs

How Simplyblock can be used with AWS Migration

Migrating to AWS can be a complex process, but using simplyblock can significantly simplify this journey while optimizing your costs, too.

Simplyblock software provides a seamless bridge between local NVMe disk, Amazon EBS, and Amazon S3, integrating these storage options into a cohesive system designed for the ultimate scale and performance of IO-intensive stateful workloads. By combining the high performance of local NVMe storage with the reliability and cost-efficiency of EBS (gp2 and gp3 volumes) and S3, respectively, simplyblock enables enterprises to optimize their storage infrastructure for stateful applications, ensuring scalability, cost savings, and enhanced performance. With simplyblock, you can save up to 80% of your AWS database storage costs.

Our technology uses NVMe over TCP for minimal access latency, high IOPS/GB, and efficient CPU core utilization, outperforming local NVMe disks and Amazon EBS in cost/performance ratio at scale. Ideal for high-performance Kubernetes environments, simplyblock combines the benefits of local-like latency with the scalability and flexibility necessary for dynamic AWS EKS deployments, ensuring optimal performance for I/O-sensitive workloads like databases. Using erasure coding (a better RAID) instead of replicas, simplyblock minimizes storage overhead while maintaining data safety and fault tolerance. This approach reduces storage costs without compromising reliability.

Simplyblock also includes additional features such as instant snapshots (full and incremental), copy-on-write clones, thin provisioning, compression, encryption, and many more – in short, there are many ways in which simplyblock can help you optimize your cloud costs. Get started using simplyblock right now and see how simplyblock can simplify and optimize your AWS migration. Simplyblock is available on AWS Marketplace.

The post AWS Migration: How to Migrate into the Cloud? Data Storage Perspective. appeared first on simplyblock.

Storage optimization in AWS presents unique challenges. Organizations frequently overprovision Amazon EBS volumes, leading to poor utilization rates averaging just 30%. Meanwhile, the complexity of managing multiple storage types – from high-performance io2 volumes to cost-effective S3 buckets – can make it difficult to implement effective tiering strategies. For companies running databases and data-intensive applications, these challenges are particularly acute.

This guide explores proven best practices for AWS storage optimization, focusing on key areas including:

• Strategic storage provisioning and capacity planning
• Effective use of storage tiers and volume types
• Performance optimization techniques
• Cost reduction strategies through improved resource utilization
• Automated storage management and monitoring

Whether you’re running managed database services, operating observability platforms, or managing enterprise applications, these optimization strategies can help you achieve the ideal balance of performance, cost, and operational efficiency in your AWS environment.

Introduction to AWS Storage

What is AWS Storage?

AWS Storage refers to the various cloud storage solutions provided by Amazon Web Services (AWS). Core AWS Storage services are S3, EBS and EFS. These services enable users to store, manage, and retrieve data over the internet, offering scalable and secure storage options tailored to different needs. AWS Storage solutions are integral for businesses and developers who require reliable, high-performance storage that can grow with their demands.

Why is AWS Storage Important?

AWS Storage services are crucial for managing vast amounts of data efficiently. It provides flexibility, scalability, and cost-effectiveness, making it suitable for a wide range of applications—from simple data backup to complex data analytics and high-performance computing. Understanding AWS Storage types and costs helps businesses optimize their data management strategies and budgets.

AWS Storage offers scalable, secure, and cost-effective solutions for all your data management needs.

AWS Storage Types: Overview and Use Cases

Amazon EBS (Elastic Block Store)

What it is: Block-level storage volumes attached to EC2 instances, behaving like physical hard drives.

Pros:

• High performance with low latency (especially io2)
• Consistent I/O performance
• Supports live configuration changes
• Automatic replication within AZ
• Supports snapshots and encryption

Cons:

• Limited to single AZ
• Can only attach to one instance (except multi-attach io2)
• Relatively expensive, especially for high IOPS
• Pay for provisioned capacity, not used capacity
• Volume size changes limited to every 6 hours

Best for:

• Database storage
• Boot volumes
• Enterprise applications requiring consistent I/O
• Development and test environments

Amazon S3 (Simple Storage Service)

What it is: Highly durable object storage service accessible via HTTP/HTTPS.

Pros:

• Unlimited storage capacity
• 99.999999999% durability
• Cross-region availability
• Multiple storage tiers
• Pay only for what you use
• Highly scalable and cost-effective

Cons:

• Higher latency than block storage
• Not suitable for operating systems or databases
• Can be expensive for frequent data access
• Object size limitations
• No file system interface

Best for:

• Static website hosting
• Backup and archive
• Data lakes
• Content distribution
• Application assets

Local Instance Storage

What it is: Physical storage attached to EC2 instance hardware.

Pros:

• Extremely low latency
• Very high IOPS
• No additional cost beyond instance price
• Ideal for temporary storage
• Highest performance option

Cons:

• Data lost when instance stops
• Cannot be detached/reattached
• Size limited by instance type
• No replication or backup
• No data persistence

Best for:

• Cache storage
• Temporary processing
• High-performance scratch space
• Buffer/queue storage
• Instance-specific workloads

Amazon EFS (Elastic File System)

What it is: Fully managed NFS file system for EC2 instances.

Pros:

• Shared access across multiple instances
• Automatic scaling
• Cross-AZ replication
• Pay for used storage only
• Linux-compatible file system

Cons:

• Higher latency than EBS
• More expensive than S3
• Limited to Linux workloads
• Performance scales with size
• Regional service only

Best for:

• Shared file storage
• Content management systems
• Development environments
• Web serving
• Analytics applications

Comparing AWS Storage Types: A Deep Dive

Amazon EBS vs Local Instance Storage (NVMe)

Performance Characteristics:

• EBS:
  • 200-500μs latency for io2
  • Limited IOPS (64,000 max per volume)
  • Network-attached storage with consistent performance
  • Bandwidth limited by network
• Local NVMe:
  • Ultra-low latency (100μs or less)
  • Very high IOPS (millions possible)
  • Direct-attached storage
  • No network bandwidth limitations

Durability & Availability:

• EBS:
  • 99.8-99.9% durability
  • Persists independently of instance
  • Survives instance stops/starts
  • Supports snapshots
• Local NVMe:
  • Ephemeral storage
  • Data lost on instance stop
  • No built-in replication
  • No snapshot support

Amazon EBS vs S3

Performance Characteristics:

• EBS:
  • Low latency (milliseconds)
  • Block-level access
  • Consistent I/O performance
  • Limited to single AZ
• S3:
  • Higher latency (tens of milliseconds)
  • Object-level access
  • Unlimited scale
  • Global access

Cost Structure:

• EBS:
  • Pay for provisioned capacity
  • Additional IOPS costs for io2
  • Snapshot storage costs
  • Cross-AZ data transfer fees
• S3:
  • Pay for used storage only
  • Tiered pricing based on volume
  • Access frequency pricing options
  • Cheaper for large datasets

Local NVMe vs S3

Use Case Optimization:

• Local NVMe:
  • High-performance databases
  • Real-time analytics
  • Cache layers
  • Temporary processing
• S3:
  • Long-term storage
  • Data lakes
  • Static content
  • Backup/archive

Management Overhead:

• Local NVMe:
  • Requires careful capacity planning
  • No built-in data protection
  • Instance type dependent
  • Complex redundancy needs
• S3:
  • Fully managed service
  • Built-in redundancy
  • Automatic scaling
  • Lifecycle management

Choosing the Right Storage Type

For Database Workloads:

1. High-Performance Requirements:
   • Primary: Local NVMe
   • Secondary: EBS io2
   • Archive: S3
2. Cost-Sensitive Applications:
   • Primary: EBS gp3
   • Secondary: S3
   • Archive: S3 Glacier

For Analytics Workloads:

1. Real-time Analytics:
   • Hot data: Local NVMe
   • Warm data: EBS
   • Cold data: S3
2. Batch Processing:
   • Processing: Local NVMe
   • Source data: S3
   • Results: EBS/S3

Implementing AWS storage solutions tailored to your specific needs ensures you get the most out of your cloud investment.

Understanding AWS Storage Challenges

Storage optimization in AWS presents several key challenges that organizations must address:

• Underutilization: Organizations frequently overprovision Amazon EBS volumes, leading to poor utilization rates averaging just 30%
• Complex Management: Balancing multiple storage types, from high-performance io2 volumes to cost-effective S3 buckets
• Performance Tradeoffs: Finding the right balance between cost and performance for different workload types
• Scaling Costs: Managing growing storage expenses as data volumes expand

AWS Storage Pricing and Cost Optimization

Understanding AWS Storage Costs

AWS storage pricing encompasses multiple components that organizations must carefully consider. At its core, storage costs are based on the volume of data stored, with pricing varying significantly across different storage types. While S3 charges for actual usage, EBS volumes bill for provisioned capacity regardless of utilization. This distinction becomes crucial for cost optimization strategies.

Data transfer costs represent another significant component of storage expenses. AWS charges for data movement between regions and from AWS to the internet, though transfers into AWS and within the same region are typically free or lower cost. Organizations should carefully architect their applications to minimize costly cross-region data transfers.

Request and retrieval costs, while often overlooked, can substantially impact the total storage bill. Services like S3 charge for both PUT/GET operations, while Glacier adds retrieval fees based on speed requirements. Understanding these operational costs is crucial for accurately forecasting storage expenses.

Cost Estimation and Management

The AWS Pricing Calculator serves as an invaluable tool for projecting storage costs before deployment. Organizations can model different scenarios, comparing costs across storage types and usage patterns. This proactive approach helps avoid unexpected expenses and enables better budgeting decisions.

Regular bill analysis provides insights into actual storage usage patterns and costs. AWS Cost Explorer and detailed billing reports help identify cost drivers, unused resources, and opportunities for optimization. Monthly reviews of these reports should be standard practice for effective cost management.

AWS Storage Optimization Strategies

Selecting the appropriate storage type for each workload represents one of the most effective cost optimization strategies. For instance, frequently accessed data might justify the higher costs of EBS io2 volumes, while rarely accessed data could be more cost-effectively stored in S3 Glacier. Understanding access patterns and performance requirements enables informed decision-making.

Lifecycle policies automate the movement of data between storage tiers based on age or access patterns. For example, moving infrequently accessed data from S3 Standard to S3 Glacier after 90 days can significantly reduce storage costs while maintaining data accessibility when needed.

Data compression and deduplication technologies can substantially reduce storage requirements and costs. Modern compression algorithms offer excellent compression ratios with minimal performance impact, making them particularly valuable for large datasets or backup storage.

Ongoing Management

Regular storage audits should be a cornerstone of any cost optimization strategy. These reviews help identify orphaned snapshots, unused volumes, and opportunities for storage consolidation. Organizations should establish processes for regular cleanup and right-sizing of storage resources.

Implementing proper tagging and monitoring strategies enables better cost allocation and usage tracking. Tags help attribute costs to specific projects or departments, while monitoring helps identify usage patterns and potential cost optimization opportunities. This data-driven approach ensures storage resources are used efficiently and cost-effectively.

Frequently Asked Questions (FAQs)

What is the Cheapest AWS Storage Option?

Amazon Glacier is the most cost-effective storage option for long-term archival needs, though it has higher retrieval costs compared to other services.

How can i Reduce my AWS Storage Costs?

To reduce costs, choose the appropriate storage type, use lifecycle policies to transition data to lower-cost storage, compress data, and regularly review your storage usage.

What is the Difference between S3 and EBS?

S3 is an object storage service suitable for storing and retrieving any amount of data, while EBS provides block storage for use with EC2 instances, offering high performance and low latency.

How do i Estimate AWS Storage Costs?

Estimate costs using the AWS Pricing Calculator, which factors in storage type, amount of data, data transfer, and retrieval requests. Review your AWS bill for accurate cost management.

Can i use Multiple AWS Storage Types Together?

No, you can’t use multiple AWS storage types together for a single workload. You can however use simplyblock orchestration to combine NVMe disk, EBS and S3 in a single solution.

Simplyblock integrates seamlessly with AWS storage services, offering cost-efficient yet high-performance cloud storage at scale in a single solution.

How Simplyblock Can Be Used To Optimize AWS Storage Cost?

Simplyblock can help you optimize AWS storage costs and utilize various AWS storage types effectively by providing a seamless bridge between local NVMe disk, Amazon EBS, and Amazon S3, integrating these storage options into a single, cohesive system designed for ultimate scale and performance of IO-intensive stateful workloads. By combining the high performance of local NVMe storage with the reliability and cost-efficiency of EBS and S3 respectively, simplyblock enables enterprises to optimize their storage infrastructure for stateful applications, ensuring scalability, cost savings, and enhanced performance. With simplyblock, you can save up to 80% on your EBS costs on AWS.

Ideal for high-performance Kubernetes environments, simplyblock combines the benefits of local-like latency with the scalability and flexibility necessary for dynamic AWS EKS deployments, ensuring optimal performance for I/O-sensitive workloads like databases. Using erasure coding (a better RAID) instead of replicas helps to minimize storage overhead without sacrificing data safety and fault tolerance. Simplyblock uses NVMe over TCP for minimal access latency, high IOPS/GB, and efficient CPU core utilization, surpassing local NVMe disks and Amazon EBS in cost/performance ratio at scale. Moreover, simplyblock can be used alongside various AWS storage types, ensuring a versatile storage solution.

Additional features such as instant snapshots (full and incremental), copy-on-write clones, thin provisioning, compression, encryption, and many more, simplyblock meets your requirements before you set them. Get started using simplyblock right now or learn more about our feature set. Simplyblock is available on AWS Marketplace.

The post AWS Storage Optimization: Best Practices for Cost and Performance appeared first on simplyblock.

Amazon EBS: the Limitations

When building out a new system, quick iterations are generally key. That includes fast turnovers to test ideas, or validate approaches. Using out of the box services, like Amazon EBS helps with these requirements. Cloud providers like AWS offer these services to get customers started quickly.

That said, cloud block storage volumes, such as Amazon EBS (gp3, io2, io2 Block Express), provide fast storage with high IOPS, and low latency for your compute instances (Amazon EC2) or Kubernetes environments (Amazon EKS or self-hosted) in the same availability zones.

While quick to get started, eventually you may run into the following shortcomings, which will make scaling either complicated or expensive.

Limited Free IOPS: The number of free IOPS is limited on a per volume basis, meaning that if you need high IOPS numbers, you have to pay extra. Sometimes you even have to change the volume type (gp3 only supports up to 16k IOPS, whereas io2 Block Express supports up to 256k IOPS).

Limited Durability: Depending on the selected volume type you’ll have to deal with limited data durability and availability (e.g. gp3 offers only 99.9% availability). There is no way to buy your way out of it, except using two volumes with some RAID1 like configuration.

No Thin Provisioning: Storage cost is paid per provisioned capacity per time unit and not by actual usage (which is around 50% less). When a volume is created, the given capacity is used for calculating the price, meaning if you create a 1TB volume but only use 100GB, you’ll still pay for the 1TB.

Limited Capacity Scalability: While volumes can be grown in size, you can only increase the capacity once every 6h (at least on Amazon EBS). Therefore, you need to estimate upfront how large the volume will grow in that time frame. If you miscalculated, you’ll run out of memory.

No Replication Across Availability Zones: Volumes cannot be replicated across availability zones, limiting the high availability if there are issues with one of the availability zones. This can be mitigated using additional tools, but they invoke additional cost.

Missing Multi-Attach: Attaching a volume to multiple compute instances or containers offers shared access to a dataset. Depending on volume-type, there is no option to multi-attach a volume to multiple instances.

Limited Latency: Depending on volume type, the access latency of a volume is located in the single or double-digit milliseconds range. Latency may also fluctuate. With low and predictable latency requirements, you may be limited here.

Simplyblock, Elastic Block Storage for the Cloud-Age

Simplyblock is built from the ground up to break-free of the typical cloud limitations and provide sub-millisecond predictable latency, virtually unlimited IOPS, while empowering you with scalability and a minimum five9s (99.999%) availability.

To overcome your typical cloud block storage service limitations, simplyblock implements a transparent and dynamic pooling of cloud-based block storage volumes, combining the individual drives into one large storage pool.

In its simplest form, block storage is pooled in a single, but separated storage node (typically a virtual machine). From this pooled storage, which can be considered a single, large virtual disk (or stripe), we carve out logical volumes. These logical volumes can differ in capacity and their particular performance characteristics (IOPS, throughput). All logical volumes are thin-provisioned, thus making more efficient use of raw disk space available to you.

On the client-side, meaning Linux and Windows, no additional drivers are required. Simplyblock’s logical volumes are exported as NVMe devices and use the NVMe over Fabrics industry standard for fast SSD storage, hence the NVMe initiator protocol for NVMe over TCP is already part of the operating system kernels on Linux and the latest versions of Windows Server. Simplyblock logical volumes are designed for ease of use and the out of the box experience, simply by partitioning them and formatting them with any operating-system specific file system.

Additional data services include instant snapshotting of volumes (for fast backup) and instant cloning (for speed and storage efficiency) as well as storage replication across availability zones (for disaster recovery purposes). All of this is powered by the copy-on-write nature of the simplyblock storage engine.

Last but not least, logical volumes can also be multi-attached to multiple compute instances.

More importantly though, simplyblock has its own Kubernetes CSI driver for automated container storage lifecycle management under the Kubernetes ecosystem.

Scaling out the Simplyblock Storage Pool

If the processing power of a single storage node isn’t sufficient anymore, or high-availability is required, you will use a cluster. When operating as a cluster, multiple storage nodes are combined as a single virtual storage pool and compute instances are connected to all of them.

In this scenario, a transparent online fail-over mechanism takes care of switching the connection of logical volumes from one node to another in case of connection issues. This mechanism (NVMe multi-pathing with ANA) is already built into the operating system kernels of Linux and Windows Server, therefore, no additional software is required on the clients.

It is important to note that clusters can be expanded by either increasing the attached block storage pools (on the storage nodes) or by adding additional storage nodes to the cluster. This expansion can happen online, doesn’t require any downtime, and eventually results in an automatically re-balanced storage cluster (background operation).

Simplyblock and Microsecond Latency

When double-digit microsecond latency is required, simplyblock can utilize client-local NVMe disks as caches.

In this case simplyblock can boost read IOPS and decrease read access latency to below 100 microseconds. To achieve this you configure the local NVMe devices as a write-through (read) cache. Simplyblock client-local caches are deployed as containers. In a typical Kubernetes environment, this is done as part of CSI driver deployment via the helm chart. Caches are transparent to the compute instances and containers and look like any access to a local NVMe storage. They are, however, managed as part of the simplyblock cluster.

Simplyblock as Hyper-converged instead of Disaggregated

If you have sufficient spare capacity (CPU, RAM resources) on your compute instances and don’t want to deploy additional, separated storage nodes, a hyper-converged setup can be chosen as well. A hyper-converged setup is more cost-efficient as no additional virtual server is required.

On the other hand, resources are shared between services consuming storage and the simplyblock storage engine. While this is isn’t necessarily a problem, it requires some additional capacity planning on your end. Simplyblock generally recommends a disaggregated setup where storage and compute resources are strictly separated.

Simplyblock: Scalable Elastic Block Storage Made Easy

No matter what deployment strategy you choose, storage nodes are connected to simplyblock’s the hosted and managed control plane. The control plane is highly scalable and can serve thousands of storage clusters. That said, if you deploy multiple clusters, the management infrastructure is shared between all of them.

Likewise, all of the shown deployment options are realized using different deployment configurations of the same core components. Additionally, none of the configurations requires special software on the client side.

Anyhow, simplyblock enables you to build your own elastic block storage, overcoming the shortcomings of typical cloud provider offered services, such as Amazon EBS. Simplyblock provides advantages as overcome the (free) IOPS limit with a single volumes benefitting from 100k or more free IOPS (compared to e.g. 3,000 for aws gp3) reach read access latency lower than 100 microseconds multi-attach a volume to many instances replicate a volume across availability zones (synchronous and asynchronous replication) and bring down the cost of capacity / increase storage efficiency by multiple times via thin provisioning and copy-on-write clones scale the cluster according to your needs with zero downtime cluster extension

If you want to learn more about simplyblock see why simplyblock, or do you want to get started right away ?

The post Block Storage Volume Pooling for the Cloud-Age appeared first on simplyblock.

Editorial: This blog post is written by a guest author, Steven Sklar from QuestDB. It appeared first on his private blog at sklar.rocks. We appreciate his contributions to the Kubernetes ecosystem and wanted to thank him for letting us repost his article. Steven, you rock! 🔥

The Container Storage Interface is an API specification that enables developers to build custom drivers which handle the provisioning, attaching, and mounting of volumes in containerized workloads. As long as a driver correctly implements the CSI API spec, it can be used in any supported Container Orchestration system, like Kubernetes. This decouples persistent storage development efforts from core cluster management tooling, allowing for the rapid development and iteration of storage drivers across the cloud native ecosystem.

In Kubernetes, the CSI has replaced legacy in-tree volumes with a more flexible means of managing storage mediums. Previously, in order to take advantage of new storage types, one would have had to upgrade an entire cluster’s Kubernetes version to access new PersistentVolume API fields for a new storage type. But now, with the plethora of independent CSI drivers available, you can add any type of underlying storage to your cluster instantly, as long as there’s a driver for it.

But what if existing drivers don’t provide the features that you require and you want to build a new custom driver? Maybe you’re concerned about the ramifications of migrating from in-tree to CSI volumes? Or, you simply want to learn more about how persistent storage works in Kubernetes? Well, you’re in the right place! This article will describe what the CSI is and detail how it’s implemented in Kubernetes.

It’s APIs all the way down

Like many things in the Kubernetes ecosystem, the Container Storage Interface is actually just an API specification. In the container-storage-interface/spec GitHub repo, you can find this spec in 2 different versions:

1. A protobuf file that defines the API schema in gRPC terms
2. A markdown file that describes the overall system architecture and goes into detail about each API call

What I’m going to discuss in this section is an abridged version of that markdown file, while borrowing some nice ASCII diagrams from the repo itself!

Architecture

A CSI Driver has 2 components, a Node Plugin and a Controller Plugin. The Controller Plugin is responsible for high-level volume management; creating, deleting, attaching, detatching, snapshotting, and restoring physical (or virtualized) volumes. If you’re using a driver built for a cloud provider, like EBS on AWS, the driver’s Controller Plugin communicates with AWS HTTPS APIs to perform these operations. For other storage types like NFS, EXSI, ZFS, and more, the driver sends these requests to the underlying storage’s API endpoint, in whatever format that API accepts.

Editorial: The same is true for simplyblock. Simplyblock’s CSI driver implements all necessary, and following described calls, making it a perfect drop-in replacement for Amazon EBS. If you want to learn more read: Why simplyblock.

On the other hand, the Node Plugin is responsible for mounting and provisioning a volume once it’s been attached to a node. These low-level operations usually require privileged access, so the Node Plugin is installed on every node in your cluster’s data plane, wherever a volume could be mounted.

The Node Plugin is also responsible for reporting metrics like disk usage back to the Container Orchestration system (referred to as the “CO” in the spec). As you might have guessed already, I’ll be using Kubernetes as the CO in this post! But what makes the spec so powerful is that it can be used by any container orchestration system, like Nomad for example, as long as it abides by the contract set by the API guidelines.

The specification doc provides a few possible deployment patterns, so let’s start with the most common one.

CO "Master" Host
+-------------------------------------------+
|                                           |
|  +------------+           +------------+  |
|  |     CO     |   gRPC    | Controller |  |
|  |            +----------->   Plugin   |  |
|  +------------+           +------------+  |
|                                           |
+-------------------------------------------+

CO "Node" Host(s)
+-------------------------------------------+
|                                           |
|  +------------+           +------------+  |
|  |     CO     |   gRPC    |    Node    |  |
|  |            +----------->   Plugin   |  |
|  +------------+           +------------+  |
|                                           |
+-------------------------------------------+ 

Since the Controller Plugin is concerned with higher-level volume operations, it does not need to run on a host in your cluster’s data plane. For example, in AWS, the Controller makes AWS API calls like ec2:CreateVolume, ec2:AttachVolume, or ec2:CreateSnapshot to manage EBS volumes. These functions can be run anywhere, as long as the caller is authenticated with AWS. All the CO needs is to be able to send messages to the plugin over gRPC. So in this architecture, the Controller Plugin is running on a “master” host in the cluster’s control plane.

Volume Lifecycle

The spec provides a flowchart of basic volume operations, also in the form of a cool ASCII diagram:

   CreateVolume +------------+ DeleteVolume
 +------------->|  CREATED   +--------------+
 |              +---+----^---+              |
 |       Controller |    | Controller       v
+++         Publish |    | Unpublish       +++
|X|          Volume |    | Volume          | |
+-+             +---v----+---+             +-+
                | NODE_READY |
                +---+----^---+
               Node |    | Node
            Publish |    | Unpublish
             Volume |    | Volume
                +---v----+---+
                | PUBLISHED  |
                +------------+

Mounting a volume is a synchronous process: each step requires the previous one to have run successfully. For example, if a volume does not exist, how could we possibly attach it to a node?

When publishing (mounting) a volume for use by a workload, the Node Plugin first requires that the Controller Plugin has successfully published a volume at a directory that it can access. In practice, this usually means that the Controller Plugin has created the volume and attached it to a node. Now that the volume is attached, it’s time for the Node Plugin to do its job. At this point, the Node Plugin can access the volume at its device path to create a filesystem and mount it to a directory. Once it’s mounted, the volume is considered to be published and it is ready for a containerized process to use. This ends the CSI mounting workflow.

Continuing the AWS example, when the Controller Plugin publishes a volume, it calls ec2:CreateVolume followed by ec2:AttachVolume. These two API calls allocate the underlying storage by creating an EBS volume and attaching it to a particular instance. Once the volume is attached to the EC2 instance, the Node Plugin is free to format it and create a mount point on its host’s filesystem.

Here is an annotated version of the above volume lifecycle diagram, this time with the AWS calls included in the flow chart.

   CreateVolume +------------+ DeleteVolume
 +------------->|  CREATED   +--------------+
 |              +---+----^---+              |
 |       Controller |    | Controller       v
+++         Publish |    | Unpublish       +++
|X|          Volume |    | Volume          | |
+-+                 |    |                 +-+
                    |    |
  |    | 
                    |    |
  |    | 
                    |    |
                +---v----+---+
                | NODE_READY |
                +---+----^---+
               Node |    | Node
            Publish |    | Unpublish
             Volume |    | Volume
                +---v----+---+
                | PUBLISHED  |
                +------------+

If a Controller wants to delete a volume, it must first wait for the Node Plugin to safely unmount the volume to preserve data and system integrity. Otherwise, if a volume is forcibly detached from a node before unmounting it, we could experience bad things like data corruption. Once the volume is safely unpublished (unmounted) by the Node Plugin, the Controller Plugin would then call ec2:DetachVolume to detach it from the node and finally ec2:DeleteVolume to delete it, assuming that the you don’t want to reuse the volume elsewhere.

What makes the CSI so powerful is that it does not prescribe how to publish a volume. As long as your driver correctly implements the required API methods defined in the CSI spec, it will be compatible with the CSI and by extension, be usable in COs like Kubernetes and Nomad.

Running CSI Drivers in Kubernetes

What I haven’t entirely make clear yet is why the Controller and Node Plugins are plugins themselves! How does the Container Orchestrator call them, and where do they plug into?

Well, the answer depends on which Container Orchestrator you are using. Since I’m most familiar with Kubernetes, I’ll be using it to demonstrate how a CSI driver interacts with a CO.

Deployment Model

Since the Node Plugin, responsible for low-level volume operations, must be running on every node in your data plane, it is typically installed using a DaemonSet. If you have heterogeneous nodes and only want to deploy the plugin to a subset of them, you can use node selectors, affinities, or anti-affinities to control which nodes receive a Node Plugin Pod. Since the Node Plugin requires root access to modify host volumes and mounts, these Pods will be running in privileged mode. In this mode, the Node Plugin can escape its container’s security context to access the underlying node’s filesystem when performing mounting and provisioning operations. Without these elevated permissions, the Node Plugin could only operate inside of its own containerized namespace without the system-level access that it requires to provision volumes on the node.

The Controller Plugin is usually run in a Deployment because it deals with higher-level primitives like volumes and snapshots, which don’t require filesystem access to every single node in the cluster. Again, lets think about the AWS example I used earlier. If the Controller Plugin is just making AWS API calls to manage volumes and snapshots, why would it need access to a node’s root filesystem? Most Controller Plugins are stateless and highly-available, both of which lend themselves to the Deployment model. The Controller also does not need to be run in a privileged context.

Event-Driven Sidecar Pattern

Now that we know how CSI plugins are deployed in a typical cluster, it’s time to focus on how Kubernetes calls each plugin to perform CSI-related operations. A series of sidecar containers, that are registered with the Kubernetes API server to react to different events across the cluster, are deployed alongside each Controller and Node Plugin. In a way, this is similar to the typical Kubernetes controller pattern, where controllers react to changes in cluster state and attempt to reconcile the current cluster state with the desired one.

There are currently 6 different sidecars that work alongside each CSI driver to perform specific volume-related operations. Each sidecar registers itself with the Kubernetes API server and watches for changes in a specific resource type. Once the sidecar has detected a change that it must act upon, it calls the relevant plugin with one or more API calls from the CSI specification to perform the desired operations.

Controller Plugin Sidecars

Here is a table of the sidecars that run alongside a Controller Plugin:

How do these sidecars work together? Let’s use an example of a StatefulSet to demonstrate. In this example, we’re dynamically provisioning our PersistentVolumes (PVs) instead of mapping PersistentVolumeClaims (PVCs) to existing PVs. We start at the creation of a new StatefulSet with a VolumeClaimTemplate.

---
apiVersion: apps/v1
kind: StatefulSet
spec:
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-storage-class"
      resources:
        requests:
         storage: 1Gi

Creating this StatefulSet will trigger the creation of a new PVC based on the above template. Once the PVC has been created, the Kubernetes API will notify the external-provisioner sidecar that this new resource was created. The external-provisioner will then send a CreateVolume message to its neighbor Controller Plugin over gRPC. From here, the CSI driver’s Controller Plugin takes over by processing the incoming gRPC message and will create a new volume based on its custom logic. In the AWS EBS driver, this would be an ec2:CreateVolume call.

At this point, the control flow moves to the built-in PersistentVolume controller, which will create a matching PV and bind it to the PVC. This allows the StatefulSet’s underlying Pod to be scheduled and assigned to a Node.

Here, the external-attacher sidecar takes over. It will be notified of the new PV and call the Controller Plugin’s ControllerPublishVolume endpoint, mounting the volume to the StatefulSet’s assigned node. This would be the equivalent to ec2:AttachVolume in AWS.

At this point, we have an EBS volume that is mounted to an EC2 instance, all based on the creation of a StatefulSet, PersistentVolumeClaim, and the work of the AWS EBS CSI Controller Plugin.

Node Plugin Sidecars

There is only one unique sidecar that is deployed alongside the Node Plugin; the node-driver-registrar. This sidecar, running as part of a DaemonSet, registers the Node Plugin with a Node’s kubelet. During the registration process, the Node Plugin will inform the kubelet that it is able to mount volumes using the CSI driver that it is part of. The kubelet itself will then wait until a Pod is scheduled to its corresponding Node, at which point it is then responsible for making the relevant CSI calls ( PublishVolume ) to the Node Plugin over gRPC.

Common Sidecars

There is also a livenessprobe sidecar that runs in both the Container and Node Plugin Pods that monitors the health of the CSI driver and reports back to the Kubernetes Liveness Probe mechanism.

Communication over Sockets

How do these sidecars communicate with the Controller and Node Plugins? Over gRPC through a shared socket! So each sidecar and plugin contains a volume mount pointing to a single unix socket.

This diagram highlights the pluggable nature of CSI Drivers. To replace one driver with another, all you have to do is simply swap the CSI Driver container with another and ensure that it’s listening to the unix socket that the sidecars are sending gRPC messages to. Becase all drivers advertise their own different capabilities and communicate over the shared CSI API contract, it’s literally a plug-and-play solution.

Conclusion

In this article, I only covered the high-level concepts of the Container Storage Interface spec and implementation in Kubernetes. While hopefully it has provided a clearer understanding of what happens once you install a CSI driver, writing one requires significant low-level knowledge of both your nodes’ operating system(s) and the underlying storage mechanism that your driver is implementing. Luckily, CSI drivers exist for a variety of cloud providers and distributed storage solutions, so it’s likely that you can find a CSI driver that already fulfills your requirements. But it always helps to know what’s happening under the hood in case your particular driver is misbehaving.

If this article interests you and you want to learn more about the topic, please let me know! I’m always happy to answer questions about CSI Drivers, Kubernetes Operators, and a myriad of other DevOps-related topics.

The post How the CSI (Container Storage Interface) Works appeared first on simplyblock.

As businesses increasingly migrate to the cloud, or build newer applications as cloud-native services, understanding the cloud cost becomes essential for cost-effective operations. With Amazon EBS often making up 50% or more of the cloud cost, it is important to grasp the intricacies of Amazon EBS pricing, explore the key concepts, and find the main factors that influence cost, as well as strategies to optimize expenses.

Understanding Amazon EBS

Amazon EBS provides scalable block-level storage volumes for use with Amazon EKS Persistent Volumes, EC2 instances, and other Amazon services. It offers various volume types, each designed for specific use cases, such as General Purpose (SSD), Provisioned IOPS (SSD), and HDD based. The choice of volume type significantly impacts performance and cost, making it vital to align storage configurations with application requirements.

Amazon EBS Pricing Breakdown

AWS pricing is complicated and requires a lot of studying the different regions, available options, as well as some good estimations of a service’s own behavior in terms of speed and capacity requirements.

Amazon EBS provides a set of different factors that influence availability, performance, capacity, and most prominently the cost.

Volume Type and Performance

Different workloads demand different levels of performance. Understanding the nature of your applications and selecting the appropriate volume type is crucial to balance cost and performance. The available volume types will be discussed further down in the blog post.

Volume Size

Amazon EBS volumes come in various sizes, and costs scale with the amount of provisioned storage per volume. Assessing the storage storage requirements and adjusting volume sizes accordingly to avoid over-provisioning can influence quite significantly.

Snapshot Costs

Creating snapshots for backup and disaster recovery is a common practice. However, snapshot costs can accumulate, especially as the frequency and volume of snapshots increase, the cost scales with the number and types of snapshots created. Additionally, there are two types of snapshots, standard, which is the default, and archive, which is cheaper on the storage side, but incurs cost when being restored. Implementing a snapshot management strategy to control expenses is crucial.

Throughput and I/O Operations

Throughput and I/O operations may or may not incur additional costs, depending on the selected volume type.

While data transfer is often easy to estimate, the necessary values for throughput and I/O operations per second (also known as IOPS ) are much harder. Especially IOPS can be a fair amount of the spending when running io-intensive workloads, such as databases, data warehouses, high-load webservers, or similar.

Be mindful of the amount of data transferred in and out of your EBS volumes, as well as the number of I/O operations performed.

Amazon EBS Volume Types

As mentioned above, Amazon EBS has quite the set of different volume types. Some are designed for specific use cases or to provide a cost-effective alternative, while others are older or newer generations for the same usage scenario.

An in-depth technical description of the different volume types can be found on AWS’ documentation .

Cheap Storage Volumes (st1 / Sc1)

The first category is designed for storage volumes that require large amounts of data storage which, at the same time, doesn’t need to provide the highest performance characteristics.

Being based upon HDD disks, the access latency is high and transfer speed is fairly low. The volume can be scaled up to 16TiB each though, reaching a high capacity at a cheap price.

Durability is typically given as 99.8% – 99.9%, meaning that the volume can be offline for roughly 9h per year. Warm ( throughput optimized) and cold volumes are available, relating to the types st1 and sc1 respectively.

General Purpose Volumes (gp2 / Gp3)

The second category is, what AWS calls, general purpose. It has the widest applicability and is the default option when looking for an Amazon EBS volume.

When creating volumes, gp2 should be avoided, being the old generation at the same price but with less features. That said, gp3 provides higher throughput and IOPS over st1 and sc1 volumes due to being SSD-based storage. Like the HDD-based services, durability is in the same range of 99.8% – 99.9%, leading to up to 9h per year unavailability. Likewise with capacity. Volumes can be scaled up to 16TiB each and therefore are perfect for a variety of use cases, such as boot volumes, simple transactional workloads, smaller databases, and similar.

Provisioned IOPS Volumes (io1 / Io2)

The third option are high-performance SSD (and NVMe) based volumes.

Amazon EBS Pricing

Prices for Amazon EBS volumes and additional upgrades depend on the region they are created in. For that reason, it is not possible to give an exact explanation of the pricing. There is, however, the chance to give an overview of what features have separate prices, and an example for one specific region.

The base Amazon EBS volume types normal price from cheapest to most expensive (GB-month):

1. HDD-based sc1 2. HDD-based st1 3. SSD-based gp2 4. SSD-based gp3 5. SSD-based io1 and io2

In addition to the base pricing, there are certain capabilities or aspects which can be increased for an additional cost, such as I/O Operations per Second (IOPS) Throughput

Amazon EBS Pricing example

And this is where it gets a bit more complicated. Every type of volume has its own set of base, and maximum available capabilities. Not all capabilities are available on all volume types though.

In our example, we want to create an Amazon EBS volume of type io2 in the US-EAST with 10 TB storage capacity. In addition we want to increase the available IOPS to 80,000 – just to make it complicated. For newer io2 volumes, the throughput scales proportionally with provisioned IOPS up to 4,000 MiB/s, meaning we don’t have to pay extra.

Base price for the io2 volume: The volume’s base cost is 0.125 USD/GB-month. That said, our 10 TB volume comes up to 1,250 USD per month.

Throughput capability pricing: The throughput of up to 4,000 MiB/s is automatically scaled proportionally to the provisioned IOPS, so all is good here. For other volume types, additional throughput (over the base amount) can be bought.

IOPS capability pricing: The pricing for IOPS can be as complicated as with io2 volumes. These have multiple “discount stages”. The prices are split at 32,000 and 64,000 IOPS.

With that in mind, the IOPS pricing can be broken down into: 0-32,000 IOPS * 0.065 USD/IOPS-month = 2,080 USD/month 32,001 – 64,000 IOPS * 0.046 USD/IOPS-month = 1,417.95 USD/month 64,001 – 80,000 IOPS * 0.032 USD/IOPS-month = 511.97 USD/month

Cost of the io2 volume: That means, including all cost factors (USD 1,250.00 + USD 2,080.00, USD 1,417.95, USD 511.97), the cost builds up to a monthly fee of USD 5,259.92 – for a single volume.

Strategies to Optimize Amazon EBS Spending

Amazon EBS volumes can be expensive as just shown. Therefore, it is important to keep the following strategies for cost reduction and optimization in mind.

Rightsize your Volumes

Regularly assess your storage requirements and resize volumes accordingly. Downsizing or upsizing volumes based on actual needs can result in significant cost savings. If auto-growing of volumes is enabled, keep the disk growth in check. Log files, or similar, running amok can blow your spend limit in hours.

Utilize Provisioned IOPS Wisely

Provisioned IOPS volumes offer high-performance storage but come at a high cost. Use them judiciously (and not ludicrously) for applications that require consistent and low-latency performance, and consider alternatives for less demanding workloads.

Implement Snapshot Lifecycle Policies

Set up lifecycle policies for snapshots to manage retention periods and reduce unnecessary storage costs. Periodically review and clean up outdated snapshots to optimize storage usage.

Leverage EBS-Optimized Instances

Use EC2 instances that are EBS-optimized for better performance. This ensures that the network traffic between EC2 instances and EBS volumes does not negatively impact overall system performance.

Conclusive Thoughts

As businesses continue to leverage AWS services, understanding and optimizing Amazon EBS spending is a key aspect of efficient cloud management. By carefully selecting the right volume types, managing sizes, and implementing cost-saving strategies, organizations can strike a balance between performance and cost-effectiveness in their cloud storage infrastructure. Regular monitoring and adjustment of storage configurations will contribute to a well-optimized and cost-efficient AWS environment.

If this feels too complicated or the requirements are hard to predict, simplyblock offers an easier, more scalable, and future-proof solution. Running right in your AWS account, providing you with the fastest and easiest way to build your own Amazon EBS alternative for Kubernetes, and save 60% and more on storage cost at the same time. Learn here how simplyblock works.

The post AWS EBS Pricing: A Comprehensive Guide appeared first on simplyblock.