PaaSTA - Build, Deploy, Connect, and Monitor Services

PaaSTA Logo

PaaSTA is a highly-available, distributed system for building, deploying, and running services using containers and Kubernetes.

PaaSTA has been running production services at Yelp since 2016. It was originally designed to run on top of Apache Mesos but has subsequently been updated to use Kubernetes. Over time the features and functionality that PaaSTA provides have increased but the principal design remains the same.

PaaSTA aims to take a declarative description of the services that teams need to run and then ensures that those services are deployed safely, efficiently, and in a manner that is easy for the teams to maintain. Rather than managing Kubernetes YAML files, PaaSTA provides a simplified schema to describe your service and in addition to configuring Kubernetes it can also configure other infrastructure tools to provide monitoring, logging, cost management etc.

Want to know more about the opinions behind what makes PaaSTA special? Check out the PaaSTA Principles.

Components

Note: PaaSTA is an opinionated platform that uses a few un-opinionated tools. It requires a non-trivial amount of infrastructure to be in place before it works completely:

Docker for code delivery and containment
Kubernetes for code execution and scheduling (runs Docker containers)
Tron for running things on a timer (nightly batches)
SmartStack and Envoy for service registration and discovery
Sensu for monitoring/alerting
Jenkins (optionally) for continuous deployment
Prometheus and HPA for autoscaling services

One advantage to having a PaaS composed of components like these is you get to reuse them for other purposes. For example, at Yelp Sensu is not just for PaaSTA, it can be used to monitor all sorts of things. We also use Kubernetes to run other more complex workloads like Jolt and Cassandra. Our service mesh, which is a heavily customised version of SmartStack and Envoy, allows many systems at Yelp to communicate with PaaSTA services and each other.

On the other hand, requiring lots of components, means lots of infrastructure to setup before PaaSTA can work effectively! Realistacally, running PaaSTA outside of Yelp would not be sensible, because in addition to the integrations mentioned above we also have strong opinions encoded in other tooling that you would need to replicate. Nevertheless, we code PaaSTA in the open because we think it is useful to share our approach and hope that the code can at least help others understand or solve similar problems.

Integrations and Features

In addition to the direct integrations above PaaSTA also relies on other components to provide PaaSTA users with other features and to manage compute capacity at Yelp.

We use Karpenter to autoscale pools of EC2 instances to run PaaSTA. Formerly we used our own autoscaler Clusterman
We bake AMIs using Packer
We collect logs from services and send them via Monk to Kafka
We use StatefulSets to run a few stateful PaaSTA services
We autotune the resources needed by each service by monitoring usage (similar to VPA)

Design Goals

Declarative, rather than imperative, control
Fault tolerance
Service isolation
Efficient use of resources
No single points of failure
Pleasant interface

Getting Started

See the getting started documentation for how to deploy PaaSTA. This reference is intended to help understand how PaaSTA works but we don't advise that you use PaaSTA in production.