In 2024, data-driven organizations increasingly recognize the crucial importance of adopting NVMe storage solutions to stay competitive. With NVMe adoption still below 30%, there’s significant room for growth as companies seek to optimize their database performance and storage efficiency. We’ve looked at how major tech companies have tackled database optimization and scalability challenges, often turning to self-hosted database solutions and NVMe storage.

While it’s interesting to see what Netflix or Pinterest engineers are investing their efforts into, it is also essential to ask yourself how your organization is adopting new technologies. As companies grow and their data needs expand, traditional database setups often struggle to keep up. Let’s look at some examples of how some of the major tech players have addressed these challenges.

Pinterest’s Journey to Horizontal Database Scalability with TiDB

Pinterest, which handles billions of pins and user interactions, faced significant challenges with its HBase setup as it scaled. As their business grew, HBase struggled to keep up with evolving needs, prompting a search for a more scalable database solution. They eventually decided to go with TiDB as it provided the best performance under load.

Selection Process:

• Evaluated multiple options, including RocksDB, ShardDB, Vitess, VoltDB, Phoenix, Spanner, CosmosDB, Aurora, TiDB, YugabyteDB, and DB-X.
• Narrowed down to TiDB, YugabyteDB, and DB-X for final testing.

Evaluation:

• Conducted shadow traffic testing with production workloads.
• TiDB performed well after tuning, providing sustained performance under load.

TiDB Adoption:

• Deployed 20+ TiDB clusters in production.
• Stores over 200+ TB of data across 400+ nodes.
• Primarily uses TiDB 2.1 in production, with plans to migrate to 3.0.

Key Benefits:

• Improved query performance, with 2-10x improvements in p99 latency.
• More predictable performance with fewer spikes.
• Reduced infrastructure costs by about 50%.
• Enabled new product use cases due to improved database performance.

Challenges and Learnings:

• Encountered issues like TiCDC throughput limitations and slow data movement during backups.
• Worked closely with PingCAP to address these issues and improve the product.

Future Plans:

• Exploring multi-region setups.
• Considering removing Envoy as a proxy to the SQL layer for better connection control.
• Exploring migrating to Graviton instance types for a better price-performance ratio and EBS for faster data movement (and, in turn, shorter MTTR on node failures).

Uber’s Approach to Scaling Datastores with NVMe

Uber, facing exponential growth in active users and ride volumes, needed a robust solution for their datastore “Docstore” challenges.

Hosting Environment and Limitations:

• Initially on AWS, later migrated to hybrid cloud and on-premises infrastructure
• Uber’s massive scale and need for customization exceeded the capabilities of managed database services

Uber’s Solution: Schemaless and MySQL with NVMe

• Schemaless: A custom solution built on top of MySQL
• Sharding: Implemented application-level sharding for horizontal scalability
• Replication: Used MySQL replication for high availability
• NVMe storage: Leveraged NVMe disks for improved I/O performance

Results:

• Able to handle over 100 billion queries per day
• Significantly reduced latency for read and write operations
• Improved operational simplicity compared to Cassandra

Discord’s Storage Evolution and NVMe Adoption

Discord, facing rapid growth in user base and message volume, needed a scalable and performant storage solution.

Hosting Environment and Limitations:

• Google Cloud Platform (GCP)
• Discord’s specific performance requirements and need for customization led them to self-manage their database infrastructure

Discord’s storage evolution:

1. MongoDB: Initially used for its flexibility, but faced scalability issues
2. Cassandra: Adopted for better scalability but encountered performance and maintenance challenges
3. ScyllaDB: Finally settled on ScyllaDB for its performance and compatibility with Cassandra

Discord also created a solution, “superdisk” with a RAID0 on top of the Local SSDs, and a RAID1 between the Persistent Disk and RAID0 array. They could configure the database with a disk drive that would offer low-latency reads while still allowing us to benefit from the best properties of Persistent Disks. One can think of it as a “simplyblock v0.1”.

Key improvements with ScyllaDB:

• Reduced P99 latencies from 40-125ms to 15ms for read operations
• Improved write performance, with P99 latencies dropping from 5-70ms to a consistent 5ms
• Better resource utilization, allowing Discord to reduce their cluster size from 177 Cassandra nodes to just 72 ScyllaDB nodes

Summary of Case Studies

In the table below, we can see a summary of the key initiatives taken by tech giants and their respective outcomes. What is notable, all of the companies were self-hosting their databases (on Kubernetes or on bare-metal servers) and have leveraged local SSD (NVMe) for improved read/write performance and lower latency. However, at the same time, they all had to work around data protection and scalability of the local disk. Discord, for example, uses RAID to mirror the disk, which causes significant storage overhead. Such an approach doesn’t also offer a logical management layer (i.e. “storage/disk virtualization”). In the next paragraphs, let’s explore how simplyblock adds even more performance, scalability, and resource efficiency to such setups.

The Role of Intelligent Storage Optimization in NVMe-Based Systems

While these case studies demonstrate the power of NVMe and optimized database solutions, there’s still room for improvement. This is where intelligent storage optimization solutions like simplyblock are spearheading market changes.

Simplyblock vs. Local NVMe SSD: Enhancing Database Scalability

While local NVMe disks offer impressive performance, simplyblock provides several critical advantages for database scalability. Simplyblock builds a persistent layer out of local NVMe disks, which means that is not just a cache and it’s not just ephemeral storage. Let’s explore the benefits of simplyblock over local NVMe disk:

1. Scalability: Unlike local NVMe storage, simplyblock offers dynamic scalability, allowing storage to grow or shrink as needed. Simplyblock can scale performance and capacity beyond the local node’s disk size, significantly improving tail latency.
2. Reliability: Data on local NVMe is lost if an instance is stopped or terminated. Simplyblock provides advanced data protection that survives instance outages.
3. High Availability: Local NVMe loses data availability during the node outage. Simplyblock ensures storage remains fully available even if a compute instance fails.
4. Data Protection Efficiency: Simplyblock uses erasure coding (parity information) instead of triple replication, reducing network load and improving effective-to-raw storage ratios by about 150% (for a given amount of NVMe disk, there is 150% more usable storage with simplyblock).
5. Predictable Performance: As IOPS demand increases, local NVMe access latency rises, often causing a significant increase in tail latencies (p99 latency). Simplyblock maintains constant access latencies at scale, improving both median and p99 access latency. Simplyblock also allows for much faster write at high IOPS as it’s not using NVMe layer as write-through cache, hence its performance isn’t dependent on a backing persistent storage layer (e.g. S3)
6. Maintainability: Upgrading compute instances impacts local NVMe storage. With simplyblock, compute instances can be maintained without affecting storage.
7. Data Services: Simplyblock provides advanced data services like snapshots, cloning, resizing, and compression without significant overhead on CPU performance or access latency.
8. Intelligent Tiering: Simplyblock automatically moves infrequently accessed data to cheaper S3 storage, a feature unavailable with local NVMe.
9. Thin Provisioning: This allows for more efficient use of storage resources, reducing overprovisioning common in cloud environments.
10. Multi-attach Capability: Simplyblock enables multiple nodes to access the same volume, which is useful for high-availability setups without data duplication. Additionally, multi-attach can decrease the complexity of volume management and data synchronization.

Technical Deep Dive: Simplyblock’s Architecture

Simplyblock’s architecture is designed to maximize the benefits of NVMe while addressing common cloud storage challenges:

1. NVMe-oF (NVMe over Fabrics) Interface: Exposes storage as NVMe volumes, allowing for seamless integration with existing systems while providing the low-latency benefits of NVMe.
2. Distributed Data Plane: Uses a statistical placement algorithm to distribute data across nodes, balancing performance and reliability.
3. Logical Volume Management: Supports thin provisioning, instant resizing, and copy-on-write clones, providing flexibility for database operations.
4. Asynchronous Replication: Utilizes a block-storage-level write-ahead log (WAL) that’s asynchronously replicated to object storage, enabling disaster recovery with near-zero RPO (Recovery Point Objective).
5. CSI Driver: Provides seamless integration with Kubernetes, allowing for dynamic provisioning and lifecycle management of volumes.

Below is a short overview of simplyblock’s high-level architecture in the context of PostgreSQL, MySQL, or Redis instances hosted in Kubernetes. Simplyblock creates a clustered shared pool out of local NVMe storage attached to Kubernetes compute worker nodes (storage is persistent, protected by erasure coding), serving database instances with the performance of local disk but with an option to scale out into other nodes (which can be either other compute nodes or separate, disaggregated, storage nodes). Further, the “colder” data is tiered into cheaper storage pools, such as HDD pools or object storage.

Applying Simplyblock to Real-World Scenarios

Let’s explore how simplyblock could enhance the setups of the companies we’ve discussed:

Pinterest and TiDB with simplyblock

While TiDB solved Pinterest’s scalability issues, and they are exploring Graviton instances and EBS for a better price-performance ratio and faster data movement, simplyblock could potentially offer additional benefits:

1. Price/Performance Enhancement: Simplyblock’s storage orchestration could complement Pinterest’s move to Graviton instances, potentially amplifying the price-performance benefits. By intelligently managing storage across different tiers (including EBS and local NVMe), simplyblock could help optimize storage costs while maintaining or even improving performance.
2. MTTR Improvement & Faster Data Movements: In line with Pinterest’s goal of faster data movement and reduced Mean Time To Recovery (MTTR), simplyblock’s advanced data management capabilities could further accelerate these processes. Its efficient data protection with erasure coding and multi-attach capabilities helps with smooth failovers or node failures without performance degradation. If a node fails, simplyblock can quickly and autonomously rebuild the data on another node using parity information provided by erasure coding, eliminating downtime.
3. Better Scalability through Disaggregation: Simplyblock’s architecture allows for the disaggregation of storage and compute, which aligns well with Pinterest’s exploration of different instance types and storage options. This separation would provide Pinterest with greater flexibility in scaling their storage and compute resources independently, potentially leading to more efficient resource utilization and easier capacity planning.

Uber’s Schemaless

While Uber’s custom Schemaless solution on MySQL with NVMe storage is highly optimized, simplyblock could still offer benefits:

1. Unified Storage Interface: Simplyblock could provide a consistent interface across Uber’s diverse storage needs, simplifying operations.
2. Intelligent Data Placement: For Uber’s time-series data (like ride information), simplyblock’s tiering could automatically optimize data placement based on age and access patterns.
3. Enhanced Disaster Recovery: Simplyblock’s asynchronous replication to S3 could complement Uber’s existing replication strategies, potentially improving RPO.

Discord and ScyllaDB

Discord’s move to ScyllaDB already provided significant performance improvements, but simplyblock could further enhance their setup:

1. NVMe Resource Pooling: By pooling NVMe resources across nodes, simplyblock would allow Discord to further reduce their node count while maintaining performance.
2. Cost-Efficient Scaling: For Discord’s rapidly growing data needs, simplyblock’s intelligent tiering could help manage costs as data volumes expand.
3. Simplified Cloning for Testing: Simplyblock’s instant cloning feature could be valuable for Discord’s development and testing processes.It allows for quick replication of production data without additional storage overhead.

What’s next in the NVMe Storage Landscape?

The case studies from Pinterest, Uber, and Discord highlight the importance of continuous innovation in database and storage technologies. These companies have pushed beyond the limitations of managed services like Amazon RDS to create custom, high-performance solutions often built on NVMe storage.

However, the introduction of intelligent storage optimization solutions like simplyblock represents the next frontier in this evolution. By providing an innovative layer of abstraction over diverse storage types, implementing smart data placement strategies, and offering features like thin provisioning and instant cloning alongside tight integration with Kubernetes, simplyblock spearheads market changes in how companies approach storage optimization.

As data continues to grow exponentially and performance demands increase, the ability to intelligently manage and optimize NVMe storage will become ever more critical. Solutions that can seamlessly integrate with existing infrastructure while providing advanced features for performance, cost optimization, and disaster recovery will be key to helping companies navigate the challenges of the data-driven future.

The trend towards NVMe adoption, coupled with intelligent storage solutions like simplyblock is set to reshape the database infrastructure landscape. Companies that embrace these technologies early will be well-positioned to handle the data challenges of tomorrow, gaining a significant competitive advantage in their respective markets.

The post NVMe Storage for Database Optimization: Lessons from Tech Giants appeared first on simplyblock.

In preparation for KubeCon + CloudNativeCon, we at simplyblock figured it was essential to create a guide beyond the conference halls, highlighting the best spots to truly experience Salt Lake City. As a remote only company, we believe this short escapade, with the chance to get the team together, shouldn’t just be about work but also enjoying the local culture, food, and fun. But we’re not the gatekeeping type over here, so we want to share this guide with you in the hope to inspire you to make the most of your time there. After all, work hard, play hard—that’s our motto, and we hope you’ll join us in embracing both during the conference.

Salt Lake City’s TRAX system is your go-to for easily navigating the city. The straightforward routes and convenient stops make it a breeze to get around, much like finding the right Kubernetes pod. Grab a day pass for unlimited travel, and you’ll be ready to explore the city without any detours. To make your journey even smoother, check out this link for all the details on how to ride TRAX, including tips on routes, stops, and purchasing passes.

Here’s a link to the Google Maps location for all the places to make it more convenient.

By the way, you might see us at one of these places, so don’t be shy about greeting us. The worst we can do is improve your storage efficiency. Alright, I’ll stop pitching, read on and make your plans!

🏛️ Landmark Highlights

Salt Lake City is rich in history and architecture, with several landmarks to explore. Don’t miss the Cathedral of the Madeleine (1) ( maps ), a beautiful example of Romanesque and Gothic Revival design. Inside, admire vibrant stained glass, intricate woodwork, and serene murals. Whether attending a service or simply enjoying the peaceful atmosphere, it’s a must-visit spot in the city.

Secondly, don’t miss the Utah State Capitol (2) ( maps ), a stunning architectural landmark perched on Capitol Hill. This neoclassical building offers free guided tours and breathtaking views of Salt Lake City and the surrounding mountains. Inside, explore the rich history of Utah through various exhibits, and enjoy the peaceful ambiance of the beautifully landscaped grounds, perfect for a leisurely stroll.

❄️ Winter Thrills

Salt Lake City isn’t just a hub for tech enthusiasts during KubeCon + CloudNativeCon—it’s also a vibrant winter wonderland that offers something for everyone. After you’ve tackled all the conference sessions, why not unwind by ice skating at the Gallivan Center ? (3) ( Maps ) This charming downtown rink is a perfect way to enjoy the festive atmosphere—just remember to bundle up, because even tech pros feel the chill!

For a more serene experience, visit Temple Square (4) ( Maps ), which turns into a dazzling holiday light display in the winter—think of it as a peaceful stroll through a winter wonderland that’ll definitely brighten your day.

🍽️ Food Adventures Await

Heading to KubeCon + CloudNativeCon in Salt Lake City and eager to dive into the local food scene? We’ve got you covered with some of the city’s top dining spots, each offering a unique taste of Salt Lake’s diverse culinary landscape. For authentic Mexican cuisine, head to Red Iguana **** (5) **** ( Maps ), famous for its rich and flavorful moles that will keep you coming back for more. If you’re in the mood for a contemporary twist on American classics, The Copper Onion (6) ( Maps ) serves up bold, locally sourced dishes like succulent pork belly and house-made pastas. For sushi lovers, Takashi (7) ( Maps ) is a must-visit, offering fresh, innovative rolls and sashimi that rival coastal cities.

HSL (8) ( Maps ) provides a chic yet cozy atmosphere with modern American dishes like roasted chicken with seasonal veggies, perfect for unwinding after a day of sessions. And when you need a coffee break, The Rose Establishment (9) ( Maps ) is your go-to spot for quality coffee and artisanal pastries—a cozy hideaway to recharge between conference activities.

🍻 Don’t Forget to Enjoy the Night

Looking for the perfect spot to unwind after a day at KubeCon + CloudNativeCon in Salt Lake City? Here are some top picks for a fun and relaxing evening. Start with The Red Door (10) ( Maps ), a cozy lounge with a speakeasy vibe, where you can sip on craft cocktails and decompress after all the tech talk.

If you’re in the mood for some nostalgia, head to Quarters Arcade Bar (11) ( Maps ), where you can relive your childhood with classic arcade games while enjoying a drink. For craft beer enthusiasts, Beer Bar (12) ( Maps ) offers a wide selection of local brews in a laid-back atmosphere, making it an ideal spot to kick back with colleagues.

Whiskey Street (13) ( Maps ) is the place for those who appreciate a good whiskey. Its extensive selection, lively atmosphere, and perfect blend of elegance and comfort make it a great spot to enjoy expertly crafted cocktails and delicious food.

⛷ Salt Lake Ski Resorts

Salt Lake City is the ultimate destination for skiing in November, thanks to its unique combination of weather, terrain, and resort accessibility. Nestled at the base of the Wasatch Mountains (worth visiting the area in and of itself), Salt Lake benefits from early-season snowfalls that blanket the region’s world-renowned ski resorts, often making them operational as early as mid-November. The area’s terrain is diverse – ranging from steep and challenging slopes for experts to wide and groomed runs perfect for beginners, ensuring a tailored experience for every skier.

Moreover, Salt Lake City is just a short drive from over 10 major ski resorts , making it incredibly convenient for visitors to access top-tier slopes without long travel times. The efficient transport system, in the form of the ski bus can help you travel to and enjoy all the different ski slopes. If you don’t want to travel by bus, there are multiple options here to pick from. In addition to this, you might also want to check out the bundled pricing (with additional perks) for ski resorts.

This may be subjective, but according to our highly adept (world-class) skiing experts at simplyblock, we have come up with the list of the top 5 resorts that you may want to visit:

These are more for the experienced, and you can get a bundled pass for both here (also known to be in an area called Little Cottonwood):

Alta Ski Area ( Maps ): Alta Ski Area, one of the first ski areas in the U.S., is renowned for its steep terrain and deep powder, averaging 546 inches of snow annually. Covering 2,614 acres with a 2,538-foot vertical drop, 45% of the terrain caters to beginner and intermediate skiers. Six lifts provide access to its varied slopes, making it a top destination for experienced skiers. However, you can find multiple resorts in the Alta ski area, our favorite pick is Snowpine lodge ( Maps ). You can view the prices here . Snowbird Resort ( Maps ): Snowbird’s top elevation is 11,000 feet at Hidden Peak, with a 7,760-foot base at Baby Thunder. The resort’s tram ascends 2,900 vertical feet in just 7 minutes. It gets 500 inches of dry Utah powder annually. SKIING Magazine has ranked Alta and Snowbird the No. 1 resort in the United States. You can view the prices here .

These are more for fun and the beginner (also known to be in an area called Big Cottonwood):

Brighton Resort (Maps): Brighton Resort offers 1,050 acres of skiable terrain with a 1,745-foot vertical drop. Receiving 500 inches of snow annually, Brighton is family-friendly and accessible, with 66 runs and lifts that include five quads, a triple, and a magic carpet. Night skiing is available on 200 acres, making it a well-rounded destination just 35 miles from Salt Lake City. You can view the prices here . Solitude Resort ( Maps ): Solitude Mountain Resort spans 1,200 acres with a 2,494-foot vertical drop and receives 500 inches of snow each year. With 87 runs catering to all skill levels, the resort offers a peaceful skiing experience. Its eight lifts, including four high-speed quads, provide quick access to varied terrain, from groomed runs to powder-filled glades, making Solitude an ideal escape for skiers of all levels. You can view the prices here .

And a bonus tubing and skiing location, the home of the 2002 Winter Olympics:

Soldier hollow Nordic Centre (Maps): Nestled in Wasatch Mountain State Park, Soldier Hollow is famous for its Olympic legacy and offers over 20 miles of cross-country skiing trails through scenic landscapes. It’s a winter paradise for both athletes and the public, with activities like cross-country skiing, snowshoeing, and even public biathlon courses. A highlight is the snow tubing lanes, the longest in Utah, stretching over 1,200 feet. The day lodge provides ski rentals and food, making it an ideal spot for a winter retreat and family fun. We think Zermatt Utah Resort ( Maps ) is relatively close and reasonable. You can find the locations of all of the skiing resorts below and on this map list:

📜 Closing Note

If you’re attending KubeCon + CloudNativeCon NA 2024 in Salt Lake City, we’re excited to let you know that simplyblock will be there too!

Simplyblock offers cutting-edge storage orchestration solutions tailored for IO-intensive stateful workloads in Kubernetes, including databases and analytics. A single system seamlessly connects local NVMe disks, GP3 volumes, and S3, making it easier to manage storage capacity and performance. With smart NVMe caching, thin provisioning, storage tiering, and volume pooling, we enhance database performance while reducing costs—all without requiring changes to your existing AWS infrastructure.

We invite you to visit the simplyblock booth at the event! Swing by to learn more about how we can optimize your storage solutions and pick up some exclusive freebies. We can’t wait to meet you and discuss how we can help improve your Kubernetes experience. See you there!

The post KubeCon + CloudNativeCon NA 2024: Your Salt Lake City Guide appeared first on simplyblock.