The Kubernetes ecosystem is vibrant and ever-expanding, driven by a community of developers who are committed to enhancing the way we manage and deploy applications. Open-source tools have become an essential part of this ecosystem, offering a wide range of functionalities that streamline Kubernetes operations. These tools are crucial for automating tasks, improving efficiency, and ensuring that your Kubernetes clusters run smoothly. As Kubernetes continues to gain popularity, the demand for robust and reliable open-source tools has also increased. Developers and operators are constantly on the lookout for tools that can help them manage their Kubernetes environments more effectively. In this post, we will explore nine must-know open-source tools that can help you optimize your Kubernetes environment.

1. Helm

Helm is often referred to as the package manager for Kubernetes. It simplifies the process of deploying, managing, and versioning Kubernetes applications. With Helm, you can define, install, and upgrade even the most complex Kubernetes applications. By using Helm Charts, you can easily share your application configurations and manage dependencies between them.

2. Prometheus

Prometheus is a leading monitoring and alerting toolkit that’s widely adopted within the Kubernetes community. It collects metrics from your Kubernetes clusters, stores them, and allows you to query and visualize the data. Prometheus is essential for keeping an eye on your infrastructure’s performance and spotting issues before they become critical.

3. Kubectl

Kubectl is the command-line tool that allows you to interact with your Kubernetes clusters. It is indispensable for managing cluster resources, deploying applications, and troubleshooting issues. Whether you’re scaling your applications or inspecting logs, Kubectl provides the commands you need to get the job done.

4. Kustomize

Kustomize is a configuration management tool that helps you customize Kubernetes objects through a file-based approach. It allows you to manage multiple configurations without duplicating YAML manifests. Kustomize’s native support in Kubectl makes it easy to integrate into your existing workflows.

5. Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It enables you to manage your application deployments through Git repositories, ensuring that your applications are always in sync with your Git-based source of truth. Argo CD offers features like automated sync, rollback, and health status monitoring, making it a powerful tool for CI/CD pipelines.

6. Istio

Istio is an open-source service mesh that provides traffic management, security, and observability for microservices. It simplifies the complexity of managing network traffic between services in a Kubernetes cluster. Istio helps ensure that your applications are secure, reliable, and easy to monitor.

7. Fluentd

Fluentd is a versatile log management tool that helps you collect, process, and analyze logs from various sources within your Kubernetes cluster. With Fluentd, you can unify your log data into a single source and easily route it to multiple destinations, making it easier to monitor and troubleshoot your applications.

8. Velero

Velero is a backup and recovery solution for Kubernetes clusters. It allows you to back up your cluster resources and persistent volumes, restore them when needed, and even migrate resources between clusters. Velero is an essential tool for disaster recovery planning in Kubernetes environments.

9. Kubeapps

Kubeapps is a web-based UI for deploying and managing applications on Kubernetes. It provides a simple and intuitive interface for browsing Helm charts, managing your applications, and even configuring role-based access control (RBAC). Kubeapps makes it easier for developers and operators to work with Kubernetes applications.

Conclusion

These nine open-source tools are integral to optimizing and managing your Kubernetes environment. Each of them addresses a specific aspect of Kubernetes operations, from monitoring and logging to configuration management and deployment. By integrating these tools into your Kubernetes workflow, you can enhance your cluster’s efficiency, reliability, and security.

However, there is more. Simplyblock offers a wide range of benefits to many of the above tools, either by enhancing their capability with high performance and low latency storage options, or by directly integrating with them.

Simplyblock is the intelligent storage orchestrator for Kubernetes . We provide the Kubernetes community with easy to use virtual NVMe block devices by combining the power of Amazon EBS and Amazon S3, as well as local instance storage. Seamlessly integrated as a StorageClass (CSI) into Kubernetes, simplyblock enables Kubernetes workloads with a requirement for high IOPS and ultra low latency. Deployed directly into your AWS account, simplyblock takes full responsibility of your data and storage infrastructure, scaling and growing dynamically to meet your storage demands at any point in time.

Why Choose Simplyblock for Kubernetes?

Choosing simplyblock for your Kubernetes workloads comes with several compelling benefits to optimize your workload performance, scalability, and cost-efficiency. Elastic block storage powered by simplyblock is designed for IO-intensive and predictable low latency workloads.

• Increase Cost-Efficiency: Optimize resource scaling to exactly meet your current requirements and reduce the overall cloud spend. Grow as needed, not upfront.
• Maximize Reliability and Speed: Get the best of both worlds with ultra low latency of local instance storage combined with the reliability of Amazon EBS and Amazon S3.
• Enhance Security: Get an immediate mitigation strategy for availability zone, and even region, outages using simplyblock’s S3 journaling and Point in Time Recovery (PITR) for any application.

If you’re looking to further streamline your Kubernetes operations, simplyblock offers comprehensive solutions that integrate seamlessly with these tools, helping you get the most out of your Kubernetes environment.

Ready to take your Kubernetes management to the next level? Contact simplyblock today to learn how we can help you simplify and enhance your Kubernetes journey.

The post Best Open Source Tools For Kubernetes appeared first on simplyblock.

In this installment of podcast, we’re joined by Alessandro Vozza ( Twitter/X , Github ) , a prominent figure in the Kubernetes and cloud-native community , who talks about his new project, Kubespaces, which aims to simplify Kubernetes deployment by offering a namespace-as-a-service. He highlights the importance of maintaining the full feature set of Kubernetes while ensuring security and isolation for multi-tenant environments. Alessandro’s vision includes leveraging the Kubernetes API to create a seamless, cloud-agnostic deployment experience, ultimately aiming to fulfill the promises of platform engineering and serverless computing. He also discusses the future trends in Kubernetes and the significance of environmental sustainability in technology.

Chris Engelbert: Hello, everyone. Welcome back to the next episode of simplyblock’s Cloud Commute podcast. Today, I have another incredible guest. I know I say that every time, but he’s really incredible. He’s been around in the Kubernetes space for quite a while. And I think, Alessandro, the best way is just to introduce yourself. Who are you? What have you done in the past, and what are you doing right now?

Alessandro Vozza: Thank you for having me. Well, I’m Alessandro, yes, indeed. I’ve been around for some time in the cloud-native community. I’m Italian, from the south of Italy, and I moved to Amsterdam, where I live currently, about 20 years ago, to get my PhD in chemistry. And then after I finished my PhD, that’s my career. So I went through different phases, always around open source, of course. I’ve been an advocate for open source, and a user of open source since the beginning, since I could lay my hands on a keyboard.

That led me to various places, of course, and various projects. So I started running the DevOps meetup in Amsterdam back in the day, 10, 11 years ago. Then from there, I moved to the OpenStack project and running the OpenStack community. But when I discovered Kubernetes, and what would become the Cloud Native Computing Foundation, I started running the local meetup. And that was kind of a turning point for me. I really embraced the community and embraced the project and started working on the things. So basically what I do is organize the meetup and organize the KCDs, the Kubernetes Community Days in Amsterdam, in Utrecht, around the country. That kind of led me through a natural process to be a CNCF Ambassador, which are people that represent or are so enthusiastic about the way the Cloud Native Computing Foundation works and the community, that are naturally elected to be the face or the ambassadors for the project, for the mission.

At this moment, I still do that. It’s my honor and pleasure to serve the community, to create, to run monthly meetups and KCDs and help other communities thrive as well. So the lessons learned in the Netherlands, in the meetups and in the conferences, we try to spread them as much as possible. We are always available for other communities to help them thrive as well. So that’s been me in a nutshell. So all about community. I always say I’m an average programmer, I’m an average engineer, but where I really shine is to organize these events and to get the people together. I get a kick out of a successful event where people form connections and grow together. So that’s what drives me in my very core.

Chris Engelbert: I like how you put this. You really shine in bringing engagement to the community, helping people to shine themselves, to grow themselves. I think that is a big part of being a developer advocate or in the developer relations space in general. You love this sharing of information, helping other people to get the most out of it.

Alessandro Vozza: Actually, I used to be, or I still do play the bass, electric bass and double bass. And the bass player stays in the back next to the drummer and he creates the conditions so the other members of the band shine. So the guitar player usually stays in front, the bass player is the guy that stays back and is happy to create the foundations and cover the music to really shine. And that’s maybe my nature. So maybe it reflects from the fact that I always love playing the bass and being that guy in a band.

Chris Engelbert: I love that. That’s a great analogy. I never thought about that, but that is just brilliant. And I actually did the same thing in the past, so there may be some truth to that. So we met a few weeks ago in Amsterdam, actually at AWS Summit Amsterdam.

And I invited you because I thought you were still with the previous company, but you’re doing something new right now. So before that, you were with Solo.io , an API gateway, networking, whatever kind of thing. But you’re doing your own thing. So tell us about it.

Alessandro Vozza: Yeah. So it was a great year doing DevRel and so much fun going and speaking about service mesh, which is something that I really believe it’s going to, it’s something that everybody needs, but I know it’s a controversial, but it’s something that I really, you got to believe in it. You know, when you are a developer advocate, when you represent a company or community, the passion is important. You cannot have passion for something you don’t believe in, for something that you don’t completely embrace. And that was great. And we had so much fun for about a year or a bit more. But then I decided that I’m too young to settle, as always, like I’m only 48, come on, I have a good 10 years of engineering work to do. So I decided that I wanted to work on something else, on something mine, more, more mine, more an idea that I had, and I want to see it develop.

Filling a gap in the market and a real need for developers to have a flexible environment, environments to deploy their applications. So fulfilling the promises of platform engineering as a self-service platform to deploy applications. So the idea goes around the namespace. What is a namespace? Of course, it’s what the unit of deployment in Kubernetes really, it’s this magical place where developers can be free and can deploy their application without the control within the guard rails of whatever the system means, the cluster administrator sets.

But developers really love freedom. So developers don’t want to have to interact even with the sysops or sysadmins. In fact, developers love Heroku. So Heroku, I think, is the hallmark of developer experience where you just can deploy whatever you want, all your code, all your applications in a place and it’s automatically exposed and you can manage by yourself everything about your application.

I want to reproduce that. I want to get inspired by that particular developer experience. But because I love Kubernetes, of course, and because I really believe that the Kubernetes APIs are the cornerstone, the golden standards of cloud-native application deployment. So I want to offer the same experience but through the Kubernetes API. So how you do that, and that’s, of course, like this evolving product, me and a bunch of people are still working on, define exactly what does it mean and how it’s going to work. But the idea is that we offer namespace-as-a-service. What really matters to developers is not the clusters, is not the VMs or the networks or all the necessary evil that you need to run namespaces. But what really matters is the namespace, is a place where they can deploy their application. So what if we could offer the best of both worlds, kind of like the promises of serverless computing, right? So you are unburdened by infrastructure. Of course, there is infrastructure somewhere, the cloud is just somebody else’s computer, right? So it’s not magic, but it feels like magic because of the clever arrangement of servers in a way that you don’t see them, but they are still there.

So imagine a clusterless Kubernetes. The experience of Kubernetes, the API really, so all the APIs that you learn to love and embrace without the burden of infrastructure. That’s the core idea.

Chris Engelbert: So that means it’s slightly different from those app platforms like Fargate or what’s the Azure and GCP ones, Cloud Run and whatever. So it’s slightly different, right? Because you’re still having everything Kubernetes offers you. You still have your CRDs or your resource definitions, but you don’t have to manage Kubernetes on its own because it’s basically a hosted platform. Is that correct?

Alessandro Vozza: Yeah. So those platforms, of course, they are meant to run single individual application pods, but they don’t feel like Kubernetes. I don’t understand. For me, because I love it so much, I think developers love to learn also new things. So developers will love to have a Kubernetes cluster where they can do what they like, but without the burden of managing it. But this CloudRun and ACI and Fargate, they are great tools, of course, and you can use them to put together some infrastructure, but they’re still limiting in what you can deploy. So you can deploy this single container, but it’s not a full-fledged Kubernetes cluster. And I think it’s still tripling in a way that you don’t have the full API at your disposal, but you have to go through this extra API layer. It’s a bespoke API, so you got to learn Cloud Run, you got to learn ACI, you got to learn Fargate, but they are not compatible with each other. They are very cloud specific, but a Kubernetes API is cloud agnostic, and that’s what I want to build.

What we seek to build is to have a single place where you can deploy in every cloud, in every region, in some multi-region, multi-cloud, but through the same API layer, which is the pure and simple Kubernetes API.

Chris Engelbert: I can see there’s two groups of people, the ones that say, just hide all the complexity from Kubernetes. And you’re kind of on the other side, I wouldn’t say going all the way, like you want the complexity, but you want the feature set, the possibilities that Kubernetes still offers you without the complexity of operating it. That’s my feeling.

Alessandro Vozza: Yeah, the complexity lies in the operation, in the upgrades, the security, to properly secure a Kubernetes cluster, it takes a PhD almost, so there’s a whole sort of ecosystem dedicated to secure a cluster. But in Kubespaces, we can take care of it, we can make sure that the clusters are secure and compliant, while still offering the freedom to the developers to deploy what they need and they like. I think we underestimate the developers, so they love to tinker with the platform, so they love freedom, they don’t want the burden, even to interact with the operation team.

And so the very proposal here is that you don’t need an operation team, you don’t need a platform engineering team, it’s all part of the platform that we offer. And you don’t even need an account in Azure or AWS, you can select which cloud and which region to deploy to completely seamlessly and without limits.

Chris Engelbert: Okay, so that means you can select, okay, I need a Kubernetes cluster namespace, whatever you want to call it, in Azure, in Frankfurt or in Western Europe, whatever they call it.

Alessandro Vozza: Yeah. Okay, so yeah, it is still a thing, so people don’t want to be in clouds that don’t trust, so if you don’t want to be in Azure, you should not be forced to. So we offer several infrastructure pieces, clusters, even if the word cluster doesn’t even appear anywhere, because it’s by design, we don’t want people to think in terms of clusters, we want people to think in terms of namespaces and specifically tenants, which are just a collection of namespaces, right? So it’s a one namespace is not going to cut it, of course, you want to have multiple to assign to your teams, to group them in environments like that, prod or test, and then assign them to your team, to your teams, so they can deploy and they’re fun with their namespaces and tenants.

Chris Engelbert: Yeah, I think there’s one other thing which is also important when you select a cloud and stuff, you may have other applications or other services already in place, and you just want to make sure that you have the lowest latency, you don’t have to pay for throughput, and stuff like that. Something that I always find complicated with hosted database platforms, to be honest, because you have to have them in the same region somehow.

Alessandro Vozza: Yeah, that’s also a political reason, right? Or commercial reason that prevents you from that.

Chris Engelbert: Fair, fair. There’s supposed to be people that love Microsoft for everything.

Alessandro Vozza: I love Microsoft, of course, been there for seven years. I’m not a fanboy, maybe I am a little, but that’s all right. Everybody, that’s why the world is a beautiful place. Everybody is entitled to his or her opinion, and that’s all right.

Chris Engelbert: I think Microsoft did a great job with the cloud, and in general, a lot of the changes they did over the last couple of decades, like the last two decades, I think there are still the teams like the Office and the Windows team, which are probably very enterprise-y still, but all of the other ones. For me specifically, the Java team at Microsoft, they’re all doing a great job, and they seem to be much easier and much more community driven than the others.

Alessandro Vozza: I was so lucky because I was there, so I saw it with my own eyes, the unfolding of this war machine of Microsoft. There was this tension of beating Amazon at their own game. Seven years ago, we had this mission of really, really demonstrating that Microsoft was serious about open source, about cloud, and it paid off, and they definitely put Microsoft back on the map. I’m proud and very, very grateful to be here. You have been there, Microsoft joining the Linux Foundation, the Cloud Native Computing Foundation really being serious about Cloud Native, and now it works.

Chris Engelbert: I agree. The Post-Balmer era is definitely a different world for Microsoft. All right, let’s get back to Kubespaces, because looking at the time, we’re at 17. You said it’s, I think it’s a shared resource. You see the Kubernetes as a multi-tenant application, so how does isolation work between customers? Because I think that is probably a good question for a lot of security-concerned people.

Alessandro Vozza: Yeah, so of course, in the first incarnation would be a pure play SaaS where you have shared tenants. I mean, it’s an infrastructure share among customers. That’s by design the first iteration. There will be more, probably where we can offer dedicated clusters to specific customers. But in the beginning, it will be based on a mix of technologies between big cluster and Firecracker, which ensure better isolation of your workload. So it is indeed one piece of infrastructure where multiple customers will throw their application, but you won’t be able to see each other. Everybody gets his own API endpoint for Kubernetes API, so you will not be able. RBAC is great, and it works, of course, and it’s an arcane magic thing and it’s arcane knowledge. Of course, to properly do RBAC is quite difficult. So instead of risking to make a mistake in some cluster role or role, and then everybody can see everything, you better have isolation between tenants. And that comes with a popular project like big cluster, which has been already around for five years. So that’s some knowledge there already.

And even an other layer of isolation, things like Kata Container and Firecracker, they provide much better isolation at the container runtime level. So even if you escape from the container, from the jail of the container, you only can see very limited view of the world and you cannot see the rest of the infrastructure. So that’s the idea of isolating workloads between customers. You could find, of course, flaws in it, but we will take care of it and we will have all the monitoring in place to prevent it, it’s a learning experience. We want to prove to ourselves first and to customers that we can do this.

Chris Engelbert: Right. Okay. For the sake of time, a very, very… well, I think because you’re still building this thing out, it may be very interesting for you to talk about that. I think right now it’s most like a one person thing. So if you’re looking for somebody to help with that, now is your time to ask for people.

Alessandro Vozza: Yeah. If the ideas resonate and you want to build a product together, I do need backend engineers, front-end engineers, or just enthusiastic people that believe in the idea. It’s my first shot at building a product or building a startup. Of course, I’ve been building other businesses before, consulting and even a coworking space called Cloud Pirates. But now I want to take a shot at building a product and see how it goes. The idea is sound. There’s some real need in the market. So it’s just a matter of building it, build something that people want. So don’t start from your ideas, but just listen to what people tell you to build and see how it goes. So yeah, I’ll be very happy to talk about it and to accept other people’s ideas.

Chris Engelbert: Perfect. Last question, something I always have to ask people. What do you think will be the next big thing in Kubernetes? Is it the namespace-as-a-service or do you see anything else as well?

Alessandro Vozza: If I knew, of course, in the last KubeCon in Paris, of course, the trends are clear, this AI, this feeding into AI, but also helping AI thrive from Cloud Native. So this dual relationship with the Gen AI and the new trends in computing, which is very important. But of course, if you ask people, there will be WebAssembly on the horizon, not replacing containers, but definitely becoming a thing. So there are trends. And that’s great about this community and this technologies that it’s never boring. So there’s always something new to learn. And I’m personally trying to learn every day. And if it’s not WebAssembly, it’s something else, but trying to stay updated. This is fun. And challenges your convention, your knowledge every day. So this idea from Microsoft that I learned about growth mindset, what you should know now is never enough if you think ahead. And it’s a beautiful thing to see. So it’s something that keeps me every day.

Now I’m learning a lot of on-premise as well. These are also trying to move workloads back to the data centers. There are reasons for it. And one trend is actually one very important one. And I want to shout out to the people in the Netherlands also working on it is green computing or environmental sustainability of software and infrastructure. So within the CNCF, there is the Technical Advisory Group environmental sustainability, which we’re collaborating with. We are running the environmental sustainability week in October. So worldwide events all around getting the software we all love and care to run greener and leaner and less carbon intense. And this is not just our community, but it’s the whole planet involved. Or at least should be concerned for everybody concerned about the future of us. And I mean, I have a few kids, so I have five kids. So it’s something that concerns me a lot to leave a better place than I found it.

Chris Engelbert: I think that is a beautiful last statement, because we’re running out of time. But in case you haven’t seen the first episode of a podcast, that may be something for you because we actually talked to Rich Kenny from Interact and they work on data center sustainability, kind of doing the same thing on a hardware level. Really, really interesting stuff. Thank you very much. It was a pleasure having you. And for the audience, next week, same time, same place. I hope you’re listening again. Thank you.

Alessandro Vozza: Thank you so much for having me. You’re welcome.

The post Easy Developer Namespaces with Multi-tenant Kubernetes with Alessandro Vozza from Kubespaces appeared first on simplyblock.

Therefore, a Postgres Operator is a critical component of the database setup. In addition to the “simply” management of the Postgres cluster, it often also provides additional integration with external tools like pgpool II or pgbouncer (for connection pooling), pgbackrest or Barman (for backup and restore), as well as Postgres extension management.

Critical components should be chosen wisely. Oftentimes, it’s hard to exchange one tool for another later on. Additionally, depending on your requirements, you may need a tool with commercial support available, while others prefer the vast open source ecosystem. If both, even better.

But how do you choose your Postgres Operator of choice? Let’s start with a quick introduction of the most common options.

Stolon

Stolon is one of the oldest operators available. Originally released in November 2015 it even predates the term Kubernetes Operator .

The project is well known and has over 4.5k stars on GitHub. However the age shows. Many features aren’t cloud-native, it doesn’t support CRDs (custom resource definition), hence configuration isn’t according to Kubernetes. All changes are handled through a command line tool. Apart from that, the last release 0.17.0 is from September 2021, and there isn’t really any new activity on the repository. You can read that as “extremely stable” or “almost abandoned”. I guess both views on the matter are correct. Anyhow, from my point of view, the lack of activity is concerning for a tool that’s potentially being used for 5-10 years, especially in a fast-paced world like the cloud-native one.

Personally, I don’t recommend using it for new setups. I still wanted to mention it though since it deserves it. It did and does a great job for many people. Still, I left it out of the comparison table further below.

CloudNativePG

CloudNativePG is the new kid on the block, or so you might think. Its official first commit happened in March 2022. However, CloudNativePG was originally developed and sponsored by EDB (EnterpriseDB), one of the oldest companies built around PostgreSQL.

As the name suggests, CloudNativePG is designed to bring the full cloud-native feeling across. Everything is defined and configured using CRDs. No matter if you need to create a new Postgres cluster, want to define backup schedules, configure automatic failover, or scale up and down. It’s all there. Fully integrated into your typical Kubernetes way of doing things. Including a plugin for kubectl.

On GitHub , CloudNativePG skyrocketed in terms of stargazers and collected over 3.6k stars over the course of its lifetime. And while it’s not officially a CNCF (Cloud Native Computing Foundation) project, the CNCF stands pretty strongly behind it. And not to forget, the feature set is on par with older projects. No need to hide here.

All in all, CloudNativePG is a strong choice for a new setup. Fully in line with your Kubernetes workflows, feature rich, and a strong and active community.

Editor’s note: Btw, we had Jimmy Angelakos as a guest on the Cloud Commute podcast , and he talks about CloudNativePG. You should listen in.

Crunchy Postgres Operator (PGO)

PGO, or the Postgres Operator from Crunchy Data , has been around since March 2017 and is a very beloved choice by a good chunk of people. On GitHub, PGO has over 3.7k stars, an active community, quick response times to issues, and overall a pretty stable timeline activity.

Postgres resources are defined and managed through CRDs, as are Postgres users. Likewise, PGO provides integration with a vast tooling ecosystem, such as patroni (for automatic failover), pgbackrest (for backup management), pgbouncer (connection pooling), and more.

A good number of common extensions are provided out of the box, however, adding additional extensions is a little bit more complicated.

Overall, Crunchy PGO is a solid, production-proven option, also into the future. While not as fresh and hip as CloudNativePG anymore, it checks all the necessary marks, and it does so for many people for many years.

OnGres StackGres

StackGres by OnGres is also fairly new and tries to do a few things differently. While all resources can be managed through CRDs, they can also be managed through a CLI, and even a web interface. Up to the point that you can create a resource through a CRD, change a small property through the CLI, and scale the Postgres cluster using the webui. All management interfaces are completely interchangeable.

Same goes for extension management. StackGres has the biggest amount of supported Postgres extensions available, and it’s hard to find an extension which isn’t supported out of the box.

In terms of tooling integration, StackGres supports all the necessary tools to build a highly available, fault tolerant, automatically backed up and restored, scalable cluster.

In comparison to some of the other operators I really like the independent CRD types, giving a better overview of a specific resource instead of bundling all of the complexity of the Postgres and tooling ecosystem configuration in one major CRD with hundreds and thousands of lines of code.

StackGres is my personal favorite, even though it only accumulated around 900 starts on GitHub so far.

While still young, the team behind it is a group of veteran PostgreSQL folks. They just know what they do. If you prefer a bit of a bigger community, and less of a company driven project, you’ll be better off with CloudNativePG, but apart from that, StackGres is the way to go.

Editor’s note: We had Álvaro Hernández, the founder and CEO of OnGres , in our Cloud Commute podcast, talking about StackGres and why you should use it. Don’t miss the first hand information.

AppsCode KubeDB

KubeDB by AppsCode is different. In development since 2017, it’s well known in the community, but an all-in commercial product. That said, commercial support is great and loved.

Additionally, KubeDB isn’t just PostgreSQL. It supports Elasticsearch, MongoDB, Redis, Memcache, MySQL, and more. It’s a real database-in-a-box deployment solution.

All boxes are checked for KubeDB and there isn’t much to say about it other than, if you need a commercially supported operator for Postgres, look no further than KubeDB.

Zalando Postgres Operator

Last but not least, the Postgres Operator from Zalando . Yes, that Zalando, the one that sells shoes and clothes.

Zalando is a big Postgres user and started early with the cloud-native journey. Their operator has been around since 2017 and has a sizable fanbase. On GitHub the operator managed to collect over 4k stars, has a very active community, a stable release cadence, and is a great choice.

In terms of integrations with the tooling ecosystem, it provides less flexibility and is slightly opinionated towards how things are done. It was developed for Zalando’s own infrastructure first and foremost though.

Anyhow, the Zalando operator has been and still is a great choice. I actually used it myself for my previous startup and it just worked.

Which Postgres Kubernetes Operator should I Use?

You already know the answer, it depends. I know we all hate that answer, but it is true. As hinted at in the beginning, if you need commercial support certain options are already out of the scope.

It also depends if you already have another Postgres cluster with an operator running. If it works, is there really a reason to change it or introduce another one for the new cluster?

Anyhow, below is a quick comparison table of features and supported versions that I think are important.

Supported PostgreSQL Features

High Availability and Backup Features

Kubernetes Specific Features

How to choose your Postgres Operator?

The graph below shows the GitHub stars of the above Postgres Operators. What we see is a clear domination of Stolon, PGO, and Zalando’s PG Operator, with CloudNativePG rushing in from behind. StackGres, while around longer than CloudNativePG doesn’t have the community backing behind it, yet. But GitHub stars aren’t everything.

All of the above tools are great options, with the exception of Stolon, which isn’t a bad tool, but I’m concerned about the lack of activity. Make of it what you like.

Before closing, I want to quickly give some honorable mentions to two further tools.

Percona has an operator for PostgreSQL, but the community is very small right now. Let’s see if they manage to bring it on par with the other tools. If you use other Percona tools, it’s certainly worth giving it a look: Percona Operator for PostgreSQL .

The other one is the External PostgreSQL Server Operator by MoveToKube . It didn’t really fit the topic of this blog post as it’s less of a Postgres Operator but more of a database (in Postgres’ relational entity sense) and users management tool. Meaning, it uses CRDs to add, update, remove databases in external PG servers, as it does for Postgres users. Anyhow, this tool also works with services like Timescale Cloud, Amazon RDS, and many more. Worth mentioning and maybe you can make use of it in the future.

The post How to choose your Kubernetes Postgres Operator? appeared first on simplyblock.

Amazon EBS: the Limitations

When building out a new system, quick iterations are generally key. That includes fast turnovers to test ideas, or validate approaches. Using out of the box services, like Amazon EBS helps with these requirements. Cloud providers like AWS offer these services to get customers started quickly.

That said, cloud block storage volumes, such as Amazon EBS (gp3, io2, io2 Block Express), provide fast storage with high IOPS, and low latency for your compute instances (Amazon EC2) or Kubernetes environments (Amazon EKS or self-hosted) in the same availability zones.

While quick to get started, eventually you may run into the following shortcomings, which will make scaling either complicated or expensive.

Limited Free IOPS: The number of free IOPS is limited on a per volume basis, meaning that if you need high IOPS numbers, you have to pay extra. Sometimes you even have to change the volume type (gp3 only supports up to 16k IOPS, whereas io2 Block Express supports up to 256k IOPS).

Limited Durability: Depending on the selected volume type you’ll have to deal with limited data durability and availability (e.g. gp3 offers only 99.9% availability). There is no way to buy your way out of it, except using two volumes with some RAID1 like configuration.

No Thin Provisioning: Storage cost is paid per provisioned capacity per time unit and not by actual usage (which is around 50% less). When a volume is created, the given capacity is used for calculating the price, meaning if you create a 1TB volume but only use 100GB, you’ll still pay for the 1TB.

Limited Capacity Scalability: While volumes can be grown in size, you can only increase the capacity once every 6h (at least on Amazon EBS). Therefore, you need to estimate upfront how large the volume will grow in that time frame. If you miscalculated, you’ll run out of memory.

No Replication Across Availability Zones: Volumes cannot be replicated across availability zones, limiting the high availability if there are issues with one of the availability zones. This can be mitigated using additional tools, but they invoke additional cost.

Missing Multi-Attach: Attaching a volume to multiple compute instances or containers offers shared access to a dataset. Depending on volume-type, there is no option to multi-attach a volume to multiple instances.

Limited Latency: Depending on volume type, the access latency of a volume is located in the single or double-digit milliseconds range. Latency may also fluctuate. With low and predictable latency requirements, you may be limited here.

Simplyblock, Elastic Block Storage for the Cloud-Age

Simplyblock is built from the ground up to break-free of the typical cloud limitations and provide sub-millisecond predictable latency, virtually unlimited IOPS, while empowering you with scalability and a minimum five9s (99.999%) availability.

To overcome your typical cloud block storage service limitations, simplyblock implements a transparent and dynamic pooling of cloud-based block storage volumes, combining the individual drives into one large storage pool.

In its simplest form, block storage is pooled in a single, but separated storage node (typically a virtual machine). From this pooled storage, which can be considered a single, large virtual disk (or stripe), we carve out logical volumes. These logical volumes can differ in capacity and their particular performance characteristics (IOPS, throughput). All logical volumes are thin-provisioned, thus making more efficient use of raw disk space available to you.

On the client-side, meaning Linux and Windows, no additional drivers are required. Simplyblock’s logical volumes are exported as NVMe devices and use the NVMe over Fabrics industry standard for fast SSD storage, hence the NVMe initiator protocol for NVMe over TCP is already part of the operating system kernels on Linux and the latest versions of Windows Server. Simplyblock logical volumes are designed for ease of use and the out of the box experience, simply by partitioning them and formatting them with any operating-system specific file system.

Additional data services include instant snapshotting of volumes (for fast backup) and instant cloning (for speed and storage efficiency) as well as storage replication across availability zones (for disaster recovery purposes). All of this is powered by the copy-on-write nature of the simplyblock storage engine.

Last but not least, logical volumes can also be multi-attached to multiple compute instances.

More importantly though, simplyblock has its own Kubernetes CSI driver for automated container storage lifecycle management under the Kubernetes ecosystem.

Scaling out the Simplyblock Storage Pool

If the processing power of a single storage node isn’t sufficient anymore, or high-availability is required, you will use a cluster. When operating as a cluster, multiple storage nodes are combined as a single virtual storage pool and compute instances are connected to all of them.

In this scenario, a transparent online fail-over mechanism takes care of switching the connection of logical volumes from one node to another in case of connection issues. This mechanism (NVMe multi-pathing with ANA) is already built into the operating system kernels of Linux and Windows Server, therefore, no additional software is required on the clients.

It is important to note that clusters can be expanded by either increasing the attached block storage pools (on the storage nodes) or by adding additional storage nodes to the cluster. This expansion can happen online, doesn’t require any downtime, and eventually results in an automatically re-balanced storage cluster (background operation).

Simplyblock and Microsecond Latency

When double-digit microsecond latency is required, simplyblock can utilize client-local NVMe disks as caches.

In this case simplyblock can boost read IOPS and decrease read access latency to below 100 microseconds. To achieve this you configure the local NVMe devices as a write-through (read) cache. Simplyblock client-local caches are deployed as containers. In a typical Kubernetes environment, this is done as part of CSI driver deployment via the helm chart. Caches are transparent to the compute instances and containers and look like any access to a local NVMe storage. They are, however, managed as part of the simplyblock cluster.

Simplyblock as Hyper-converged instead of Disaggregated

If you have sufficient spare capacity (CPU, RAM resources) on your compute instances and don’t want to deploy additional, separated storage nodes, a hyper-converged setup can be chosen as well. A hyper-converged setup is more cost-efficient as no additional virtual server is required.

On the other hand, resources are shared between services consuming storage and the simplyblock storage engine. While this is isn’t necessarily a problem, it requires some additional capacity planning on your end. Simplyblock generally recommends a disaggregated setup where storage and compute resources are strictly separated.

Simplyblock: Scalable Elastic Block Storage Made Easy

No matter what deployment strategy you choose, storage nodes are connected to simplyblock’s the hosted and managed control plane. The control plane is highly scalable and can serve thousands of storage clusters. That said, if you deploy multiple clusters, the management infrastructure is shared between all of them.

Likewise, all of the shown deployment options are realized using different deployment configurations of the same core components. Additionally, none of the configurations requires special software on the client side.

Anyhow, simplyblock enables you to build your own elastic block storage, overcoming the shortcomings of typical cloud provider offered services, such as Amazon EBS. Simplyblock provides advantages as overcome the (free) IOPS limit with a single volumes benefitting from 100k or more free IOPS (compared to e.g. 3,000 for aws gp3) reach read access latency lower than 100 microseconds multi-attach a volume to many instances replicate a volume across availability zones (synchronous and asynchronous replication) and bring down the cost of capacity / increase storage efficiency by multiple times via thin provisioning and copy-on-write clones scale the cluster according to your needs with zero downtime cluster extension

If you want to learn more about simplyblock see why simplyblock, or do you want to get started right away ?

The post Block Storage Volume Pooling for the Cloud-Age appeared first on simplyblock.

In this installment of the podcast, we talked to Steven Sklar ( his private blog , X/Twitter ) from QuestDB , a company producing a time series database for large IoT, metrics, observability and other time-component data sets, talks about how they implemented their database offering, from building their own operator to how storage is handled.

Chris Engelbert: Hello, everyone. Welcome back to another episode of simpleblock’s Cloud Commute Podcast. Today, I’m joined by Steven Sklar from QuestDB. He was recommended by a really good friend and an old coworker who’s also at QuestDB. So hello, Steven, and good to have you.

Steven Sklar: Thank you. It’s really a pleasure to be here, and I’m looking forward to our chat.

Chris Engelbert: All right, cool. So maybe just start with a quick introduction. I mean, we already know your name, and I hope I pronounced that correctly. But what else is to talk about you?

Steven Sklar: Sure. So I kind of have a nontraditional background. I started with a degree in economics and finance and worked on Wall Street for a little bit. I like to say in most of my conference talks on the first slide that my first programming language was actually Excel VBA, which I do still have a soft spot for. And I found myself on a bond trading desk and kind of reached the boundaries of Excel and started working in C# and SQL Server, realized I liked that a lot more than just kind of talking to people on the phone and negotiating over various mortgage bonds and things. So I moved into the IT realm and software development and have been developing software ever since. So I moved on from C# into the Python world, moved on from finance into the startup world, and I currently am QuestDB, as you mentioned earlier.

Chris Engelbert: Right. So maybe you can say a few words about QuestDB. What is it? What does it do? And why do people want to use it?

Steven Sklar: Sure. QuestDB is a time series database with a focus on high performance. And I like to think of ease of usability. So we can ingest up to like millions of rows per second on some benchmarks, which is just completely mind-blowing to me. It’s actually written primarily in Java, which doesn’t necessarily go hand in hand with high performance, but we’ve rewritten most of the standard library to avoid memory allocation. So I know it actually truly is high performance. We’ve actually been introducing Rust as well into the code base. You can query the database using plain old SQL. And it really fits into several use cases, like financial tick by tick data and sensor data. I have one going on in my house right now, collecting all of my smart home stuff from Home Assistant. And I mean, yes, I’ve been here for around a year and a half, I want to say. And it’s been a great ride.

Chris Engelbert: Right. So you mentioned time series. And I’m aware what time series are because I’ve been at a competitor before that. So Jaromir and I went slightly different directions, but we both ended up in the time series world. But for the audience, that may not be perfectly aware what time series are. You already mentioned tick data from the financial background. You also mentioned Home Assistant and IoT data, which is great because I’m doing the same thing for me. It’s most like energy consumption and stuff. But maybe you have some more examples.

Steven Sklar: Sure. Kind of a canonical one is monitoring and metrics. So any kind of data, I think, has a time component. Because it’s and I think you need a specialized database. A lot of people ask, well, why not just use Postgres or any of the common databases? And you could, but you’re probably not going to scale. And you’re going to hit a point where your queries are just not performing. And time series databases, in many cases, ours in particular as well, I can speak to, is a columnar database. So it stores data in a different format than you normally would see in a traditional database. And that makes querying and actually ingesting data from a wide range of sources much more efficient. And you kind of like to think of it as, I don’t want to put myself on the spot and do mental math. But imagine if you have 10,000 devices that are sending information to your database for more than a second. It’s not that big of a deal. But maybe, let’s say, you scale and you end up with a million devices. All of a sudden, you’re dealing with tremendous amounts of data going into your database that you need to manage. And that’s a different problem, I think, than your typical relational database.

Chris Engelbert: Right. And I think you brought up a good example. Most of the time when we talk about devices as I said, I’m coming from a kind of similar background. It’s not like a device just sends you a single data point. When we talk about connected cars, they actually send thousands to 100,000 of data, position information, all kinds of metrics about the car itself, the electronics, and all that kind of stuff. And that comes down to quite a massive amount of data. So yeah, I agree with you. An actual time series database is super important. You mentioned columnar storage. Maybe you can say a few words about how that is different from, I guess, your Excel sheet.

Steven Sklar: Sure. Well, I guess I don’t know if I can necessarily compare it to my Excel spreadsheet, since that’s its own weird XML format, of course. But columnar data, I guess, is different from, let’s say, tabular data in your typical database. Tabular data is generally stored in the table format, where all of your columns and rows are kind of stored together versus columnar in a data store, each column is its own separate file. And that kind of makes it more efficient when you’re working in a time component, because time is generally your index. You’re not really indexing on a lot of things like primary key type things. You’re really just mostly indexing on time, like what happened at this point in time or over this time period. Because of that, we’re able to optimize the storage model to allow faster querying and also ingestion as well. And just for clarity, I’m not a core developer. I’m more of a cloud guy, so I hope I got those details right.

Chris Engelbert: I think you get the gist of it. But for QuestDB, that means it still looks like a tabular kind of database. So you still have your typical tables, but the individual columns are stored separately. Is that correct?

Steven Sklar: Correct.

Chris Engelbert: Ok, cool. So you said you’re a cloud guy. But as far as I know, you can install QuestDB locally, on-prem. You can install it into your own private cloud. I think there is the QuestDB cloud, which is the hosted platform. Well, not I guess. I know that it is. So maybe what is special about that? Does it have special features? Or is that mostly about the convenience of getting the managed database and getting rid of all the work you have to do when you run your own database, which can be complicated.

Steven Sklar: Absolutely. So actually, both. Obviously, you don’t have to manage it, and that’s great. You can leave it to the experts. That’s already worth the price of admission, I think. Additionally, we have the enterprise, the QuestDB enterprise, which has additional features. And all of those features, like role-based authentication and replication that’s coming soon and compression of your data on disk, are all things that you get automatically through the cloud.

Chris Engelbert: Ok, so that means I have to buy QuestDB enterprise when I want to have those features on prem, but I get them on the cloud right away.

Steven Sklar: Correct.

Chris Engelbert: Ok, cool. And correct me if I’m wrong, but I think from a client perspective, it uses the Postgres protocol. So any Postgres client is a QuestDB client, basically.

Steven Sklar: Absolutely, 100%.

Chris Engelbert: All right, so that means as an application developer, it’s super simple. I’ll basically drop in QuestDB instead of Postgres or anything else. So yeah, let’s talk a little bit about the cloud then. Maybe you can elaborate a little bit on the stack you’re running on. I’m not sure how much you can actually say, but anything you can share will probably be beneficial to everyone.

Steven Sklar: Oh, yeah, no problem. So we run on AWS. We run on Kubernetes. And we also– I guess one thing that I’m particularly proud of is an operator that I wrote to orchestrate all these databases. So our model, which is not necessarily your bread and butter Kubernetes deployment, is actually a single-tenant model. So we have one database per instance. And when you’re running Kubernetes, you kind of think, why do you care about what nodes you’re running on? Shouldn’t all that be abstracted away? And I would agree. We primarily use Kubernetes for its orchestration. But we want to avoid the noisy neighbor problem. We want to make it easy for users to change instances and instance types quickly. We want users to be able to shut down their database. And we still have the volume. So all these things, we could orchestrate them directly through Kubernetes. But we decided to use single-tenant nodes for that.

Chris Engelbert: Right. So let me see. So that means you’re using Kubernetes, as you said, mostly for orchestration, which means it’s more like if the database for some reason goes down or you have to have maintenance or you want to upgrade. It’s more about the convenience of having something managing that instead of doing it manually, right?

Steven Sklar: Exactly. And so I think we really thought, ok and this is a little bit before my time, but you could always roll your own cluster. But there’s so many things that are baked into Kubernetes these days, like monitoring and logs and metrics and networking and DNS and all these things that I don’t necessarily want to spend all my time on. I want to build a product. And by using Kubernetes and leveraging those components, we were able to build the cloud incredibly quickly, get us up and running, and then now expand upon it in the future. And that’s why, again, I mentioned the operator earlier. That was not originally part of the cloud. The cloud still has in a more limited capacity what we call a provisioner. So basically, if you’re interacting with the cloud and you make a new database, basically send a message to a queue, and that message will be picked up by a provisioner. And previously, that provisioner would say, ok, you want a database. Let’s make a stateful set. Let’s make a persistent volume. Let’s make these networking policies. Let’s do all of these things. If there’s an error, we can roll back. And we have retries. So it’s fairly sophisticated. But we ended up moving towards this operator model, which instead of the provisioner managing each of these individual components, it just manages one QuestDB resource. And our operator now handles all of those other little things. So I think that’s much more flexible for us in terms of, A, simplifying the provisioner code, and also by adding new features instead of having to work in this ever-growing web of Python. Now, it’s really just adding a snippet here and there to our reconciliation inside of everything.

Chris Engelbert: Right. You mentioned that the database is mostly written in Java. Most operators are written in Go. So what about your operator? Is it Java?

Steven Sklar: It’s Go.

Chris Engelbert: That’s fair. To be honest, I think the vast majority is. So you mentioned AWS. But I think that you are mostly talking about QuestDB Cloud, right? I think from a user’s perspective, do I use a helm chart or do I also use the operator to install it myself?

Steven Sklar: Yes. So the operator is actually only limited to the cloud because it’s built specifically to manage our own infrastructure with our own assumptions. We do have a helm chart and an open source image on Docker Hub. So I’ve used that plenty of times more than I can count.

Chris Engelbert: Ok, fair enough. So you basically support all cloud environments, all on-premise. But when you go for QuestDB Cloud, that is AWS, which I think is a fair decision. It is the biggest environment by far. So from a storage engine perspective, how much can you share? Can you talk about some fancy details? Like what kind of storage do you use? Do you use the local NVMe storage attached to the virtual machine or EBS volumes?

Steven Sklar: Yeah. So in our cloud, we have both actually NVMe and EBS. Most customers end up using EBS. And honestly, EBS is simpler to provision. But I do want to actually talk about some cool stuff that we’ve done with compression. Because we actually never implemented our own compression algorithm. We’re running on top of ZFS and using their compression algorithm. And we’ve actually– there’s an issue about data corruption, potentially, using mmap on ZFS, or rather a combination of mmap and traditional sys calls, the pwrite and preads. And what we do is actually identify when we’re running on ZFS and then decide to only use mmap calls to avoid this issue. And I think what we’ve done is pretty cool also on the storage side of orchestrating this whole thing. Because ZFS has its own notion of snapshots, its own notion of replication, its own notion of ZPools. And to simplify things, again, because we’re running this kind of I don’t necessarily want to say antiquated, but we’re running a single-tenant model, which might not be in vogue these days. What we actually do is we create one ZPool per volume and throw our QuestDB on the ZPool, enabling compression. And we’ve written our own CSI storage driver that sits in the middle of Kubernetes and other cloud providers so that we’re able to pass calls onto the cloud providers if, let’s say, we need to create or delete a volume using the cloud provider API. But when it comes to mounting specific ZFS and running ZFS-related commands, we actually take control of that and perform that in our own driver. I don’t know when this is going to be released, but I’m actually talking about this in Atlanta next week.

Chris Engelbert: No. Next week is a little bit early. Currently, I’m doing a couple of recordings, building a little bit of a pipeline. Because of conferences, the same thing will be in Paris for KubeCon next week. So there is a little bit. No, I don’t know the exact date. I think it’s in three or four weeks. So it’s a little bit out. But I guess your talk may be recorded. And public by then. So if that is the case, I’m happy if you drop it over and I put it into the show notes, people will love that. So you said when you run on, or when you detect that you run on ZFS, you use mmap. So you basically map the file into memory. And you change the memory positions directly. And then you fsync it. Or how does it work? How do I have to think about that?

Steven Sklar: Oh, boy. Ok. This is getting a little out of my– So you always use mmap regardless. But the issue is when you combine mmap with traditional sys calls on ZFS. And so what we do is we basically turn off those other sys calls and only use mmap when we’re writing to our files. In terms of the specifics of when we sync and things like that, I wish I could answer it right off of the bat.

Chris Engelbert: That’s totally fine. So just to sneak in a little shameless plug here, we should totally look into getting QuestDB running on simplyblock. I think that could be a really interesting thing. Because you mentioned ZFS, it’s basically ZFS on steroids. ZFS from my perspective, I mean, I’m running a ZFS file server in the basement. It saved me a couple of times with a broken hard disk. It’s just an incredible piece of technology. I agree with that. And it’s interesting because I’ve seen a lot of people running database on ZFS. And ZFS is all about reliability. It’s not necessarily about the highest performance. So it’s interesting you choose ZFS and you say, that’s perfect and works great for us. So because we’re almost running out of time, as I said earlier, 20 minutes is super short. When you look at cloud and databases and the world as a whole, whatever you want to talk about, what do you think is the next big trend or the current big trend? What is coming? What do you think would be really cool?

Steven Sklar: Yeah. So I guess I’m not going to talk about the existential crisis I’m having with Devin and the AI bots because it’s just a little depressing for me right now. But I think one thing that I’ve been seeing over the past few years that I find very interesting is this move away from cloud and back into your own data center. I think having control over your data is something that’s incredibly important to basically everyone now. And I think it’s to find a happy medium as a DevOps engineer between all the wonderful cloud APIs that you can use and going in the server room and kind of hooking things up. There’s probably a happy medium there somewhere that I think is an area that is going to start growing in the future. You see a lot of on-prem Kubernetes type things, Kubernetes on edge maybe. And for me, it presents a lot of interesting challenges because I spent most of my career in startups working on the cloud and understanding the fundamentals of not just the cloud APIs but operating systems and hardware a little bit. And so kind of figuring out where to draw that line in terms of what knowledge is transferable to this new paradigm will be interesting. And I think that’s a new trend that I’ve been focused on at least over the past couple of months.

Chris Engelbert: That is interesting that you mentioned that because it is kind of that. When the cloud became big, everyone wanted to move to the cloud because it was like “cheaper” in air quotes. And I think– well, the next step was serverless because it is yet even cheaper, which we all know is not necessarily true. And I see kind of the same thing. Now people realize that not every workload actually works perfectly or is a great fit for the cloud and people slowly start moving back or at least going back to not necessarily cloud instance but co-located servers or virtual machines, like plain virtual machines and just taking those for the workloads that do not need to be super scalable or super elastic. Well, thank you very much. That was very delightful. It was a pleasure having you.

Steven Sklar: Thank you.

Chris Engelbert: Thank you for being here and for the audience. I hope to– well, not see you, but hear you next time, next week. Thank you very much.

Steven Sklar: Thank you. Take care.

The post Building a Time Series Database in the Cloud with Steven Sklar from QuestDB (video + interview) appeared first on simplyblock.

Years ago, somebody asked me in an interview what I think of Docker, and if I see a future for containerization. At the time my answer was quick and easy. First of all, containerization wasn’t a new concept. BSD, Solaris, as well as other systems had them for years. They were kind of new to Linux though (at least in a widespread fashion), so they were here to stay. It was the next logical evolutionary step to virtualization. Docker, however, at least in my mind, was different. Towards Docker I simply answered “it’s the best tool we have today, but I hope it won’t be the final solution we can come up with.” While Docker turned around and is just coming back, the tooling we use today is unanimously built upon the specs defined by the open container initiative (OCI) and its OCI image format.

So what will the future hold for Kubernetes? Is it going to stay or is it going to step into the abyss and will it “just be another platform that was replaced by something else”, as Michael Levan wrote in The Future of Kubernetes .

The Rise of Kubernetes

Jokingly, when looking up container orchestration in the dictionary, you’ll probably find the synonym Kubernetes, but it took Kubernetes about a decade to get to where it is today.

Initially built by Google on the key learnings of Borg, Google’s internal container orchestration platform, K8s was released in September 2014. By the release of Kubernetes, Borg itself was already over a decade old. By 2013 many of the original team members of the Borg started to look into the next step. Project 7 was born.

At its release, Kubernetes was still using Docker underneath. A combination that most probably helped elevate Kubernetes popularity. Docker was extremely popular at the time, but people started to find insufficiencies when trying to organize and run large numbers of containers. Kubernetes was about to fix it. With its concepts of building blocks, independently deployed and the actors (or agents) it was easy to extend but still understandable. In addition, resources are declarative by nature (written as JSON or YAML files), which enables version control of those definitions.

Ever since the release, K8s enabled more and more use cases, hence more companies started using it. I think a major step for the adoption was the release of Helm in 2016 which simplified the deployment process for more complex applications and enabled an “out of the box” experience. Now Kubernetes was “easy” (don’t quote me on easy though!).

Today every cloud provider, and their mothers offer a managed Kubernetes environment. Due to a set of standard interfaces, those services are mostly interchangeable. One of the big benefits of Kubernetes. Anyhow, it’s only mostly. The small inconsistencies and implementations or performance differences may give you quite the time of your life. Not the good one though. But let’s call it “run everywhere”, because it mostly is.

A great overview of the full history of Kubernetes, with all major milestones, can be found at The History of Kubernetes by Ferenc Hámori .

Kubernetes, Love or Hate

When we look into the community, opinions on Kubernetes diverge. Many people point out the internal (yet hidden) complexity of Kubernetes. A complexity which only increases with new features and additional functionality (also third-party) being added. This complexity is real, a reason why folks like Kelsey Hightower or Abdelfettah Sghiouar call Kubernetes a platform to build platforms (listen to our Cloud Commute podcast episode with Abdelfettah ), meaning it should be used by the cloud providers (or company internal private cloud teams) to build a platform for container deployment, but it shouldn’t be used by the developers or just everyone. However, Kelsey also claimed that Kubernetes is a good place to start, not the endgame.

On the other end of the spectrum you have people that refer to Kubernetes as the operating system of the cloud area. And due to the extensibility and feature richness, they’re probably not too far off. Modern operating systems have mostly one job, abstract away the underlying hardware and its features. That said, Kubernetes abstracts away many aspects of the cloud infrastructure and the operational processes necessary to run containers. In that sense, yes, Kubernetes is probably a Cloud OS. Especially since we started to see implementations of the Kubernetes running on operating systems other than Linux. Looking at you Microsoft.

If you’re interested in learning more about the idea of Kubernetes as a Cloud OS, Natan Yellin from Robusta Dev wrote a very insightful article named Why Kubernetes will sustain the next 50 years .

What are the next Steps?

The most pressing questions for Kubernetes as it stands today, what will be next? Where will it evolve? Are we close to the end of the line?

Looking back at Borg, a decade in, Google decided it was time to reiterate on orchestration, and build upon the lessons learned. Kubernetes is about to hit its 10 year anniversary soon. So does that mean it’s time for another iteration?

Many features in Kubernetes, such as secrets, were fine 10 years ago. Today we know that an encoded “secret” is certainly not enough. Temporary user accounts, OIDC, and similar technologies can and are integrated into K8s already, increasing the complexity of it.

Looking beyond Kubernetes, technology always runs in three phases, the beginning or adoption phase, the middle where everyone “has” to use it, and the end, where companies start to phase it out. Personally, I feel that Kubernetes is at its height right now, standing right in the middle.

That doesn’t give us any prediction about the time frame for hitting the end though. At the moment it looks like Kubernetes will keep going and growing for a while. It doesn’t show any signs of slowing down.

Other technologies, like micro virtual machines, using kata containers or Firecracker , are becoming more popular, offering higher isolation (hence security), but aren’t as efficient. The important element though, they offer a CRI compatible interface. Meaning, they can be used as an alternative runtime underneath Kubernetes.

In the not too distant future, I see K8s offering multiple runtime environments, just as it offers multiple storage solutions today. Enabling running simple services in normal containers, but moving services with higher isolation needs to micro VMs.

And there are other interesting developments, based on Kubernetes, too. Edgeless Systems implements a confidential computing solution, provided as a K8s distribution named Constellation. Confidential computing makes use of CPU and GPU features that help to hardware-encrypt memory, not only for the whole system memory space, but per virtual machine, or even per container. That enables a whole set of new use cases, with end to end encryption for highly confidential calculations and data processes. While it’s possible to use it outside Kubernetes, the orchestration and operation benefits of running those calculations inside containers, making them easy to deploy and update. If you want to learn more about Constellation, we had Moritz Eckert from Edgeless Systems in our podcast not too long ago.

Future or Fad?

So, does Kubernetes have a bright future and will stand for the next 50 years, or will we realize that it is not what we’re looking for very soon-ish.

If somebody would ask me today what I think about Kubernetes, I think I would answer similarly to my Docker answer. It is certainly the best tool we have today, making it the to-go container orchestration tool of today. Its ever increasing complexity makes it hard to see the same in the future though. I think there are a lot of new lessons learned again. It’s probably time for a new iteration. Not today, not tomorrow, but somewhere in the next few years.

Maybe this new iteration isn’t an all new tool, but Kubernetes 2.0, who knows – but something has to change. Technology doesn’t stand still, the (container) world is different from what it was 10 years ago.

If you asked somebody in the beginning of containerization, it was all about how containers have to be stateless, and what do we do today? We deploy databases into K8s, and we love it. Cloud-nativeness isn’t just stateless anymore, but I’d argue a good one-third of the container workloads may be stateful today (with ephemeral or persistent state), and it will keep increasing. The beauty of orchestration, automatic resource management, self-healing infrastructure, and everything in between is just too incredible to not use it for “everything”.

Anyhow, whatever happens to Kubernetes itself (maybe it will become an orchestration extension of the OCI?!), I think it will disappear from the eyes of the users. It (or its successor) will become the platform to build container runtime platforms. But to make that happen, debug features need to be made available. At the moment you have to look way too deep into Kubernetes or agent logs to find out and fix issues. The one who never had to find why a Let’s Encrypt certificate isn’t updating may raise hand now.

To bring it to a close, Kubernetes certainly isn’t a fad, but I strongly hope it’s not going to be our future either. At least not in its current incarnation.

The post Kubernetes: Future or Fad? appeared first on simplyblock.

This interview is part of the simplyblock Cloud Commute Podcast, available on Youtube, Spotify, iTunes/Apple Podcasts, Pandora, Samsung Podcasts, and our show site.

Key Takeaways

Q: Should you deploy PostgreSQL in Kubernetes?

Deploying PostgreSQL in Kubernetes is a strategic move for organizations aiming for flexibility and scalability. Álvaro emphasizes that Kubernetes abstracts the underlying infrastructure, allowing PostgreSQL to run consistently across various environments—whether on-premise or in the cloud. This approach not only simplifies deployments but also ensures that the database is resilient and highly available.

Q: What are the main challenges of running PostgreSQL on Kubernetes?

Running PostgreSQL on Kubernetes presents unique challenges, particularly around storage and network performance. Network disks, commonly used in cloud environments, often lag behind local disks in performance, impacting database operations. However, these challenges can be mitigated by carefully choosing storage solutions and configuring Kubernetes to optimize performance. Furthermore, managing PostgreSQL’s ecosystem—such as backups, monitoring, and high availability—requires robust tooling and expertise, which can be streamlined with solutions like StackGres.

Q: Why should you use Kubernetes for PostgreSQL?

Kubernetes offers a powerful platform for running PostgreSQL due to its ability to abstract infrastructure details, automate deployments, and provide built-in scaling capabilities. Kubernetes facilitates the management of complex PostgreSQL environments, making it easier to achieve high availability and resilience without being locked into a specific vendor’s ecosystem.

Q: Can I use PostgreSQL on Kubernetes with PGO?

Yes, you can. Tools like the PostgreSQL Operator (PGO) for Kubernetes simplify the management of PostgreSQL clusters by automating routine tasks such as backups, scaling, and updates. These operators are essential for ensuring that PostgreSQL runs efficiently on Kubernetes while reducing the operational burden on database administrators.

In addition to highlighting the key takeaways, it’s essential to provide deeper context and insights that enrich the listener’s understanding of the episode. By offering this added layer of information, we ensure that when you tune in, you’ll have a clearer grasp of the nuances behind the discussion. This approach enhances your engagement with the content and helps shed light on the reasoning and perspective behind the thoughtful questions posed by our host, Chris Engelbert. Ultimately, this allows for a more immersive and insightful listening experience.

Key Learnings

Q: How does Kubernetes scheduler work with PostgreSQL?

Kubernetes uses its scheduler to manage how and where PostgreSQL instances are deployed, ensuring optimal resource utilization. However, understanding the nuances of Kubernetes’ scheduling can help optimize PostgreSQL performance, especially in environments with fluctuating workloads.

simplyblock Insight: Leveraging simplyblock’s solution, users can integrate sophisticated monitoring and management tools with Kubernetes, allowing them to automate the scaling and scheduling of PostgreSQL workloads, thereby ensuring that database resources are efficiently utilized and downtime is minimized. Q: What is the best experience of running PostgreSQL in Kubernetes?

The best experience comes from utilizing a Kubernetes operator like StackGres, which simplifies the deployment and management of PostgreSQL clusters. StackGres handles critical functions such as backups, monitoring, and high availability out of the box, providing a seamless experience for both seasoned DBAs and those new to PostgreSQL on Kubernetes.

simplyblock Insight: By using simplyblock’s Kubernetes-based solutions, you can further enhance your PostgreSQL deployments with features like dynamic scaling and automated failover, ensuring that your database remains resilient and performs optimally under varying loads. Q: How does disk access latency impact PostgreSQL performance in Kubernetes?

Disk access latency is a significant factor in PostgreSQL performance, especially in Kubernetes environments where network storage is commonly used. While network storage offers flexibility, it typically has higher latency compared to local storage, which can slow down database operations. Optimizing storage configurations in Kubernetes is crucial to minimizing latency and maintaining high performance.

simplyblock Insight: simplyblock’s advanced storage solutions for Kubernetes can help mitigate these latency issues by providing optimized, low-latency storage options tailored specifically for PostgreSQL workloads, ensuring your database runs at peak efficiency. Q: What are the advantages of clustering in PostgreSQL on Kubernetes?

Clustering PostgreSQL in Kubernetes offers several advantages, including improved fault tolerance, load balancing, and easier scaling. Kubernetes operators like StackGres enable automated clustering, which simplifies the process of setting up and managing a highly available PostgreSQL cluster.

simplyblock Insight: With simplyblock, you can easily deploy clustered PostgreSQL environments that automatically adjust to your workload demands, ensuring continuous availability and optimal performance across all nodes in your cluster.

Additional Nugget of Information

Q: What are the advantages of clustering in Postgres? A: Clustering in PostgreSQL provides several benefits, including improved performance, high availability, and better fault tolerance. Clustering allows multiple database instances to work together, distributing the load and ensuring that if one node fails, others can take over without downtime. This setup is particularly advantageous for large-scale applications that require high availability and resilience. Clustering also enables better scalability, as you can add more nodes to handle increasing workloads, ensuring consistent performance as demand grows.

Conclusion

Deploying PostgreSQL on Kubernetes offers powerful capabilities but comes with challenges. Álvaro Hernández Tortosa highlights how StackGres simplifies this process, enhancing performance, ensuring high availability, and making PostgreSQL more accessible. With the right tools and insights, you can confidently manage PostgreSQL in a cloud-native environment.

Full Video Transcript

Chris Engelbert: Welcome to this week’s episode of Cloud Commute podcast by simplyblock. Today, I have another incredible guest, a really good friend, Álvaro Hernández from OnGres. He’s very big in the Postgres community. So hello, and welcome, Álvaro.

Álvaro Hernández Tortosa: Thank you very much, first of all, for having me here. It’s an honor.

Chris Engelbert: Maybe just start by introducing yourself, who you are, what you’ve done in the past, how you got here. Well, except me inviting you.

Álvaro Hernández Tortosa: OK, well, I don’t know how to describe myself, but I would say, first of all, I’m a big nerd, big fan of open source. And I’ve been working with Postgres, I don’t know, for more than 20 years, 24 years now. So I’m a big Postgres person. There’s someone out there in the community that says that if you say Postgres three times, I will pop up there. It’s kind of like Superman or Batman or these superheroes. No, I’m not a superhero. But anyway, professionally, I’m the founder and CEO of a company called OnGres. Let’s guess what it means, On Postgres. So it’s pretty obvious what we do. So everything revolves around Postgres, but in reality, I love all kinds of technology. I’ve been working a lot with many other technologies. I know you because of being a Java programmer, which is kind of my hobby. I love programming in my free time, which almost doesn’t exist. But I try to get some from time to time. And everything related to technology in general, I’m also a big fan and supporter of open source. I have contributed and keep contributing a lot to open source. I also founded some open source communities, like for example, I’m a Spaniard. I live in Spain. And I founded Debian Spain, an association like, I don’t know, 20 years ago. More recently, I also founded a foundation, a nonprofit foundation also in Spain called Fundación PostgreSQL. Again, guess what it does? And I try to engage a lot with the open source communities. We, by the way, organized a conference for those who are interested in Postgres in the magnificent island of Ibiza in the Mediterranean Sea in September this year, 9th to 11th September for those who want to join. So yeah, that’s probably a brief intro about myself.

Chris Engelbert: All right. So you are basically the Beetlejuice of Postgres. That’s what you’re saying.

Álvaro Hernández Tortosa: Beetlejuice, right. That’s more upper bid than superheroes. You’re absolutely right.

Chris Engelbert: I’m not sure if he is a superhero, but he’s different at least. Anyway, you mentioned OnGres. And I know OnGres isn’t really like the first company. There were quite a few before, I think, El Toro, a database company.

Álvaro Hernández Tortosa: Yes, Toro DB.

Chris Engelbert: Oh, Toro DB. Sorry, close, close, very close. So what is up with that? You’re trying to do a lot of different things and seem to love trying new things, right?

Álvaro Hernández Tortosa: Yes. So I sometimes define myself as a 0.x serial entrepreneur, meaning that I’ve tried several ventures and sold none of them. But I’m still trying. I like to try to be resilient, and I keep pushing the ideas that I have in the back of my head. So yes, I’ve done several ventures, all of them, around certain patterns. So for example, you’re asking about Toro DB. Toro DB is essentially an open source software that is meant to replace MongoDB with, you guessed it, Postgres, right? There’s a certain pattern in my professional life. And Toro DB was. I’m speaking in the past because it no longer unfortunately maintained open source projects. We moved on to something else, which is OnGres. But the idea of Toro DB was to essentially replicate from Mongo DB live these documents and in the process, real time, transform them into a set of relational tables that got stored inside of a Postgres database. So it enabled you to do SQL queries on your documents that were MongoDB. So think of a MongoDB replica. You can keep your MongoDB class if you want, and then you have all the data in SQL. This was great for analytics. You could have great speed ups by normalizing data automatically and then doing queries with the power of SQL, which obviously is much broader and richer than query language MongoDB, especially for analytics. We got like 100 times faster on most queries. So it was an interesting project.

Chris Engelbert: So that means you basically generated the schema on the fly and then generated the table for that schema specifically? Interesting.

Álvaro Hernández Tortosa: Yeah, it was generating tables and columns on the fly.

OnGres StackGres: Operator for Production-Grade PostgreSQL on Kubernetes

Chris Engelbert: Right. Ok, interesting. So now you’re doing the OnGres thing. And OnGres has, I think, the main product, StackGres, as far as I know. Can you tell a little bit about that?

Álvaro Hernández Tortosa: Yes. So OnGres, as I said, means On Postgres. And one of our goals in OnGres is that we believe that Postgres is a fantastic database. I don’t need to explain that to you, right? But it’s kind of the Linux kernel, if I may use this parallel. It’s a bit bare bones. You need something around it. You need a distribution, right? So Postgres is a little bit the same thing. The core is small, it’s fantastic, it’s very featureful, it’s reliable, it’s trustable. But it needs tools around it. So our vision in OnGres is to develop this ecosystem around this Postgres core, right? And one of the things that we experience during our professional lifetime is that Postgres requires a lot of tools around it. It needs monitoring, it needs backups, it needs high availability, it needs connection pooling.

By the way, do not use Postgres without connection pooling, right? So you need a lot of tools around. And none of these tools come from a core. You need to look into the ecosystem. And actually, this is good and bad. It’s good because there’s a lot of options. It’s bad because there’s a lot of options. Meaning which one to choose, which one is good, which one is bad, which one goes with a good backup solution or the good monitoring solution and how you configure them all. So this was a problem that we coined as a stack problem. So when you really want to run Postgres in production, you need the stack on top of Postgres, right? To orchestrate all these components.

Now, the problem is that we’ve been doing this a lot of time for our customers. Typically, we love infrastructure scores, right? And everything was done with Ansible and similar tools and Terraform for infrastructure and Ansible for orchestrating these components. But the reality is that every environment into which we looked was slightly different. And we can just take our Ansible code and run it. You’ve got this stack. But now the storage is different. Your networking is different. Your entry point. Here, one is using virtual IPs. That one is using DNS. That one is using proxies. And then the compute is also somehow different. And it was not reusable. We were doing a lot of copy, paste, modify, something that was not very sustainable. At some point, we started thinking, is there a way in which we can pack this stack into a single deployable unit that we can take essentially anywhere? And the answer was Kubernetes. Kubernetes provides us this abstraction where we can abstract away this compute, this storage, this bit working and code against a programmable API that we can indeed create this package. So that’s a StackGres.

StackGres is the stack of components you need to run production Postgres, packaging a way that is uniform across any environment where you want to run it, cloud, on-prem, it doesn’t matter. And is production ready! It’s packaged at a very, very high level. So basically you barely need, I would say, you don’t need Postgres knowledge to run a production ready enterprise quality Postgres cluster introduction. And that’s the main goal of StackGres.

Chris Engelbert: Right, right. And as far as I know, I think it’s implemented as a Kubernetes operator, right?

Álvaro Hernández Tortosa: Yes, exactly.

Chris Engelbert: And there’s quite a few other operators as well. But I know that StackGres has some things which are done slightly differently. Can you talk a little bit about that? I don’t know how much you wanna actually make this public right now.

Álvaro Hernández Tortosa: No, actually everything is open source. Our roadmap is open source, our issues are open source. I’m happy to share everything. Well, first of all, what I would say is that the operator pattern is essentially these controllers that take actions on your cluster and the CRDs. We gave a lot of thought to these CRDs. I would say that a lot of operators, CRDs are kind of a byproduct. A second thought, “I have my objects and then some script generates the CRDs.” No, we said CRDs are our user-facing API. The CRDs are our extended API. And the goal of operators is to abstract the way and package business logic, right? And expose it with a simple user interface.

So we designed our CRDs to be very, very high level, very amenable to the user, so that again, you don’t require any Postgres expertise. So if you look at the CRDs, in practical terms, the YAMLs, right? The YAMLs that you write to deploy something on StackGres, they should be able to deploy, right? You could explain to your five-year-old kid and your five-year-old kid should be able to deploy Postgres into a production-quality cluster, right? And that’s our goal. And if we didn’t fulfill this goal, please raise an issue on our public issue tracker on GitLab because we definitely have failed if that’s not true. So instead of focusing on the Postgres usual user, very knowledgeable, very high level, most operators focused on low level CRDs and they require Postgres expertise, probably a lot. We want to make Postgres more mainstream than ever, right? Postgres increases in popularity every year and it’s being adopted by more and more organizations, but not everybody’s a Postgres expert. We want to make Postgres universally accessible for everyone. So one of the things is that we put a lot of effort into this design. And we also have, instead of like a big one, gigantic CRD. We have multiple. They actually can be attached like in an ER diagram between them. So you understand relationships, you create one and then you reference many times, you don’t need to restart or reconfigure the configuration files. Another area where I would say we have tried to do something is extensions. Postgres extensions is one of the most loved, if not the most loved feature, right?

And StackGres is the operator that arguably supports the largest number of extensions, over 200 extensions of now and growing. And we did this because we developed a custom solution, which is also open source by StackGres, where we can load extensions dynamically into the cluster. So we don’t need to build you a fat container with 200 images and a lot of security issues, right? But rather we deploy you a container with no extensions. And then you say, “I want this, this, this and that.” And then they will appear in your cluster automatically. And this is done via simple YAML. So we have a very powerful extension mechanism. And the other thing is that we not only expose the usual CRD YAML interface for interacting with StackGres, it’s more than fine and I love it, but it comes with a fully fledged web console. Not everybody also likes the command line or GitOps approach. We do, but not everybody does. And it’s a fully fledged web console which also supports single sign-on, where you can integrate with your AD, with your OIDC provider, anything that you want. Has detailed fine-grained permissions based on Kubernetes RBAC. So you can say, “Who can create clusters, who can view configurations, who can do anything?” And last but not least, there’s a REST API. So if you prefer to automate and integrate with another kind of solution, you can also use the REST API and create clusters and manage clusters via the REST API. And these three mechanisms, the YAML files, CRDs, the REST API and the web console are fully interchangeable. You can use one for one operation, the other one for everything goes back to the same. So you can use any one that you want.

And lately we also have added sharding. So sharding scales out with solutions like Citus, but we also support foreign interoperability, Postgres with partitioning and Apache ShardingSphere. Our way is to create a cluster of multiple instances. Not only one primary and one replica, but a coordinator layer and then shards, and it shares a coordinator of the replica. So typically dozens of instances, and you can create them with a simple YAML file and very high-level description, requires some knowledge and wires everything for you. So it’s very, very convenient to make things simple.

Chris Engelbert: Right. So the plugin mechanism or the extension mechanism, that was exactly what I was hinting at. That was mind-blowing. I’ve never seen anything like that when you showed it last year in Ibiza. The other thing that is always a little bit of like a hat-scratcher, I think, for a lot of people when they hear that a Kubernetes operator is actually written in Java. I think RedHat built the original framework. So it kind of makes sense that RedHat is doing that, I think the original framework was a Go library. And Java would probably not be like the first choice to do that. So how did that happen?

Álvaro Hernández Tortosa: Well, at first you’re right. Like the operator framework is written in Go and there was nothing else than Go at the time. So we were looking at that, but our team, we had a team of very, very senior Java programmers and none of them were Go programmers, right? But I’ve seen the Postgres community and all the communities that people who are kind of more in the DevOps world, then switching to Go programmers is a bit more natural, but at the same time, they are not senior from a Go programming perspective, right? The same would have happened with our team, right? They would switch from Java to Go. They would have been senior in Go, obviously, right? So it would have taken some time to develop those skills. On the other hand, we looked at what is the technology behind, what is an operator? An operator is no more than essentially an HTTP server that receives callbacks from Kubernetes and a client because it makes calls to Kubernetes. And HTTP clients and servers can read written in any language. So we look at the core, how complicated this is and how much does this operator framework bring to you? How we saw that it was not that much.

And actually something, for example, just mentioned before, the CRDs are kind of generated from your structures and we really wanted to do the opposite way. This is like the database. You use an ORM to read your database existing schema that we develop with all your SQL capabilities or you just create an object and let that generate a database. I prefer the format. So we did the same thing with the CRDs, right? And we wanted to develop them. So Java was more than okay to develop a Kubernetes operator and our team was expert in Java. So by doing it in Java, we were able to be very efficient and deliver a lot of value, a lot of features very, very fast without having to retrain anyone, learn a new language, or learn new skills. On top of this, there’s sometimes a concern that Java requires a JVM, which is kind of a heavy environment, right? And consumes memory and resources, and disk. But by default, StackGres uses a compilation technology and will build a whole project around it called GraalVM. And this allows you to generate native images that are indistinguishable from any other binary, Linux binary you can have with your system. And we deploy StackGres with native images. You can also switch JVM images if you prefer. We over expose both, but by default, there are native images. So at the end of the day, StackGres is several megabytes file, Linux binary and the container and that’s it.

Chris Engelbert: That makes sense. And I like that you basically pointed out that the efficiency of the existing developers was much more important than like being cool and going from a new language just because everyone does. So we talked about the operator quite a bit. Like what are your general thoughts on databases in the cloud or specifically in Kubernetes? What are like the issues you see, the problems running a database in such an environment? Well, it’s a wide topic, right? And I think one of the most interesting topics that we’re seeing lately is a concern about cost and performance. So there’s kind of a trade off as usual, right?

Álvaro Hernández Tortosa: There’s a trade off between the convenience I want to run a database and almost forget about it. And that’s why you switched to a cloud managed service which is not always true by the way, because forgetting about it means that nobody’s gonna then back your database, repack your tables, right? Optimize your queries, analyze if you haven’t used indexes. So if you’re very small, that’s more than okay. You can assume that you don’t need to touch your database even if you grow over a certain level, you’re gonna need the same DBAs, the same, at least to operate not the basic operations of the database which are monitoring, high availability and backups. So those are the three main areas that a managed service provides to you.

But so there’s convenience, but then there’s an additional cost. And this additional cost sometimes is quite notable, right? So it’s typically around 80% premium on a N+1/N number of instances because sometimes we need an extra even instance for many cloud services, right? And that multiply by 1.8 ends up being two point something in the usual case. So you’re overpaying that. So you need to analyze whether this is good for you from this perspective of convenience or if you want to have something else. On the other hand, almost all cloud services use network disks. And these network disks are very good and have improved performance a lot in the last years, but still they are far from the performance of a local drive, right? And running databases with local drives has its own challenges, but they can be addressed. And you can really, really move the needle by kind of, I don’t know if that’s the right term to call it self-hosting, but this trend of self-hosting, and if we could marry the simplicity and the convenience of managed services, right?

With the ability of running on any environment and running on any environment at a much higher performance, I think that’s kind of an interesting trend right now and a good sweet spot. And Kubernetes, to try to marry all the terms that you mentioned in the question, actually is one driver towards this goal because it enables us infrastructure independence and it enables both network disks and local disks and equally the same. And it’s kind of an enabler for this pattern that I see more trends, more trends as of now, more important and one that definitely we are looking forward to.

Chris Engelbert: Right, I like that you pointed out that there’s ways to address the local storage issues, just shameless plug, we’re actually working on something.

Álvaro Hernández Tortosa: I heard something.

The Biggest Trend in Containers?

Chris Engelbert: Oh, you heard something. (laughing) All right, last question because we’re also running out of time. What do you see as the biggest trend right now in containers, cloud, whatever? What do you think is like the next big thing? And don’t say AI, everyone says that.

Álvaro Hernández Tortosa: Oh, no. Well, you know what? Let me do a shameless plug here, right?

Chris Engelbert: All right. I did one. (laughing)

Álvaro Hernández Tortosa: So there’s a technology we’re working on right now that works for our use case, but will work for many use cases also, which is what we’re calling dynamic containers. So containers are essential as something that is static, right? You build a container, you have a build with your Dockerfile, whatever you use, right? And then that image is static. It is what it is. Contains the layers that you specified and that’s all. But if you look at any repository in Docker Hub, right? There’s plenty of tags. You have what, for example, Postgres. There’s Postgres based on Debian. There’s Postgres based on Alpine. There’s Postgres with this option. Then you want this extension, then you want this other extension. And then there’s a whole variety of images, right? And each of those images needs to be built independently, maintained, updated independently, right? But they’re very orthogonal. Like upgrading the Debian base OS has nothing to do with the Postgres layer, has nothing to do with the timescale extension, has nothing to do with whether I want the debug symbols or not. So we’re working on technology with the goal of being able to, as a user, express any combination of items I want for my container and get that container image without having to rebuild and maintain the image with the specific parameters that I want.

Chris Engelbert: Right, and let me guess, that is how the Postgres extension stuff works.

Álvaro Hernández Tortosa: It is meant to be, and then as a solution for the Postgres extensions, but it’s actually quite broad and quite general, right? Like, for example, I was discussing recently with some folks of the OpenTelemetry community, and the OpenTelemetry collector, which is the router for signals in the OpenTelemetry world, right? Has the same architecture, has like around 200 plugins, right? And you don’t want a container image with those 200 plugins, which potentially, because many third parties may have some security vulnerabilities, or even if there’s an update, you don’t want to update all those and restart your containers and all that, right? So why don’t you kind of get a container image with the OpenTelemetry collector with this source and this receiver and this export, right? So that’s actually probably more applicable. Yeah, I think that makes sense, right? I think that is a really good end, especially because the static containers in the past were in the original idea was that the static gives you some kind of consistency and some security on how the container looks, but we figured out over time, that is not the best solution. So I’m really looking forward to that being probably a more general thing. To be honest, actually the idea, I call it dynamic containers, but in reality, from a user perspective, they’re the same static as before. They are dynamic from the registry perspective.

Chris Engelbert: Right, okay, fair enough. All right, thank you very much. It was a pleasure like always talking to you. And for the other ones, I see, hear, or read you next week with my next guest. And thank you to Álvaro, thank you for being here. It was appreciated like always.

Álvaro Hernández Tortosa: Thank you very much.

The post Production-grade Kubernetes PostgreSQL, Álvaro Hernández appeared first on simplyblock.