Technical Documentation: stefanprodan/podinfo
โน๏ธ Provenance: Multi-Tier Ingestion ยท CodeWiki Reference
Overview (`README.md`)
podinfo
Podinfo is a tiny web application made with Go that showcases best practices of running microservices in Kubernetes.
Podinfo is used by CNCF projects like Flux and Flagger
for end-to-end testing and workshops.
Specifications:
- Health checks (readiness and liveness)
- Graceful shutdown on interrupt signals
- File watcher for secrets and configmaps
- Instrumented with Prometheus and Open Telemetry
- Structured logging with zap
- 12-factor app with viper
- Fault injection (random errors and latency)
- Swagger docs
- Timoni, Helm and Kustomize installers
- End-to-End testing with Kubernetes Kind and Helm
- Multi-arch container image with Docker buildx and GitHub Actions
- Container image signing with Sigstore cosign
- SBOMs and SLSA Provenance embedded in the container image
- CVE scanning with govulncheck
Web API:
GET /prints runtime informationGET /versionprints podinfo version and git commit hashGET /metricsreturn HTTP requests duration and Go runtime metricsGET /healthzused by Kubernetes liveness probeGET /readyzused by Kubernetes readiness probePOST /readyz/enablesignals the Kubernetes LB that this instance is ready to receive trafficPOST /readyz/disablesignals the Kubernetes LB to stop sending requests to this instancePOST /fault_injection/enablemakes this instance respond with HTTP 500 to all application endpoints (probes, metrics, pprof and the/fault_injection/*control endpoints stay healthy) โ useful for testing client-side circuit breakers / outlier detection against a single "sick" replicaPOST /fault_injection/disablerestores normal responsesGET /fault_injection/statusreturns the current fault injection state (enabledordisabled)GET /status/{code}returns the status codeGET /paniccrashes the process with exit code 255POST /echoforwards the call to the backend service and echos the posted contentGET /envreturns the environment variables as a JSON arrayGET /headersreturns a JSON with the request HTTP headersGET /delay/{seconds}waits for the specified periodPOST /tokenissues a JWT token valid for one minuteJWT=$(curl -sd 'anon' podinfo:9898/token | jq -r .token)GET /token/validatevalidates the JWT tokencurl -H "Authorization: Bearer $JWT" podinfo:9898/token/validateGET /configsreturns a JSON with configmaps and/or secrets mounted in theconfigvolumePOST/PUT /cache/{key}saves the posted content to RedisGET /cache/{key}returns the content from Redis if the key existsDELETE /cache/{key}deletes the key from Redis if existsPOST /storewrites the posted content to disk at /data/hash and returns the SHA1 hash of the contentGET /store/{hash}returns the content of the file /data/hash if existsGET /ws/echoechos content via websocketspodcli ws ws://localhost:9898/ws/echoGET /chunked/{seconds}usestransfer-encodingtypechunkedto give a partial response and then waits for the specified periodGET /swagger.jsonreturns the API Swagger docs, used for Linkerd service profiling and Gloo routes discovery
gRPC API:
/grpc.health.v1.Health/Checkhealth checking/grpc.EchoService/Echoechos the received content/grpc.VersionService/Versionreturns podinfo version and Git commit hash/grpc.DelayService/Delayreturns a successful response after the given seconds in the body of gRPC request/grpc.EnvService/Envreturns environment variables as a JSON array/grpc.HeaderService/Headerreturns the headers present in the gRPC request. Any custom header can also be given as a part of request and that can be returned using this API/grpc.InfoService/Inforeturns the runtime information/grpc.PanicService/Paniccrashes the process with gRPC status code as '1 CANCELLED'/grpc.StatusService/Statusreturns the gRPC Status code given in the request body/grpc.TokenService/TokenGenerateissues a JWT token valid for one minute/grpc.TokenService/TokenValidatevalidates the JWT token
Web UI:
To access the Swagger UI open /swagger/index.html in a browser.
Guides
- Getting started with Timoni
- Getting started with Flux
- Progressive Deliver with Flagger and Linkerd
- Automated canary deployments with Kubernetes Gateway API
Install
To install Podinfo on Kubernetes the minimum required version is Kubernetes v1.23.
Timoni
Install with Timoni:
timoni -n default apply podinfo oci://ghcr.io/stefanprodan/modules/podinfoHelm
Install from github.io:
helm repo add podinfo https://stefanprodan.github.io/podinfo
helm upgrade --install --wait frontend \
--namespace test \
--set replicaCount=2 \
--set backend=http://backend-podinfo:9898/echo \
podinfo/podinfo
helm test frontend --namespace test
helm upgrade --install --wait backend \
--namespace test \
--set redis.enabled=true \
podinfo/podinfoInstall from ghcr.io:
helm upgrade --install --wait podinfo --namespace default \
oci://ghcr.io/stefanprodan/charts/podinfoKustomize
kubectl apply -k github.com/stefanprodan/podinfo//kustomizeDocker
docker run -dp 9898:9898 stefanprodan/podinfoContinuous Delivery
In order to install podinfo on a Kubernetes cluster and keep it up to date with the latest
release in an automated manner, you can use Flux.
Install the Flux CLI on MacOS and Linux using Homebrew:
brew install fluxcd/tap/fluxInstall the Flux controllers needed for Helm operations:
flux install \
--namespace=flux-system \
--network-policy=false \
--components=source-controller,helm-controllerAdd podinfo's Helm repository to your cluster and
configure Flux to check for new chart releases every ten minutes:
flux create source helm podinfo \
--namespace=default \
--url=https://stefanprodan.github.io/podinfo \
--interval=10mCreate a podinfo-values.yaml file locally:
cat > podinfo-values.yaml <<EOL
replicaCount: 2
resources:
limits:
memory: 256Mi
requests:
cpu: 100m
memory: 64Mi
EOLCreate a Helm release for deploying podinfo in the default namespace:
flux create helmrelease podinfo \
--namespace=default \
--source=HelmRepository/podinfo \
--release-name=podinfo \
--chart=podinfo \
--chart-version=">5.0.0" \
--values=podinfo-values.yamlBased on the above definition, Flux will upgrade the release automatically
when a new version of podinfo is released. If the upgrade fails, Flux
can rollback
to the previous working version.
You can check what version is currently deployed with:
flux get helmreleases -n defaultTo delete podinfo's Helm repository and release from your cluster run:
flux -n default delete source helm podinfo
flux -n default delete helmrelease podinfoIf you wish to manage the lifecycle of your applications in a GitOps manner, check out
this workflow example
for multi-env deployments with Flux, Kustomize and Helm.
Overview (`.cosign/README.md`)
Podinfo signed releases
Podinfo release assets (container image, Helm chart, Flux artifact, Timoni module)
are published to GitHub Container Registry and are signed with
Cosign v3 keyless & GitHub Actions OIDC.
Verify podinfo with cosign
Install the cosign CLI:
brew install cosignContainer image
Verify the podinfo container image hosted on GHCR:
cosign verify ghcr.io/stefanprodan/podinfo:6.12.0 \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.comVerify the podinfo container image hosted on Docker Hub:
cosign verify docker.io/stefanprodan/podinfo:6.12.0 \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.comHelm chart
Verify the podinfo Helm chart hosted on GHCR:
cosign verify ghcr.io/stefanprodan/charts/podinfo:6.12.0 \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.comFlux artifact
Verify the podinfo Flux artifact hosted on GHCR:
cosign verify ghcr.io/stefanprodan/manifests/podinfo:6.12.0 \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.comTimoni module
Verify the podinfo Timoni module hosted on GHCR:
cosign verify ghcr.io/stefanprodan/modules/podinfo:6.12.0 \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.comOverview (`.notation/README.md`)
Podinfo signed releases
Podinfo release assets such as the Helm chart and the Flux artifact
are published to GitHub Container Registry and are signed with
Notation.
Generate signing keys
Generate a new signing key pair:
openssl genrsa -out podinfo.key 2048
openssl req -new -key podinfo.key -out podinfo.csr -config codesign.cnf
openssl x509 -req -days 1826 -in podinfo.csr -signkey podinfo.key -out notation.crt -extensions v3_req -extfile codesign.cnfOverview (`deploy/README.md`)
Deploy demo webapp
Demo webapp manifests:
Deploy the demo in webapp namespace:
kubectl apply -f ./webapp/common
kubectl apply -f ./webapp/backend
kubectl apply -f ./webapp/frontendDeploy the demo in the dev namespace:
kustomize build ./overlays/dev | kubectl apply -f-Deploy the demo in the staging namespace:
kustomize build ./overlays/staging | kubectl apply -f-Deploy the demo in the production namespace:
kustomize build ./overlays/production | kubectl apply -f-Testing Locally Using Kind
NOTE: You can install kind from here
The following will create a new cluster called "podinfo" and configure host ports on 80 and 443. You can access the
endpoints on localhost. The example also deploys cert-manager within the cluster along with a self-signed cluster issuer
used to generate the certificate to validate the secure port.
./kind.shOverview (`otel/README.md`)
Tracing & Logging Demo
The directory contains sample OpenTelemetry Collector
and Jaeger / Loki configurations for a tracing and logging demo.
Configuration
The provided docker-compose.yaml sets up 6 containers:
- PodInfo Frontend on port 9898
- PodInfo Backend on port 9899
- OpenTelemetry Collector listening on port 4317 for GRPC
- Jaeger all-in-one with UI on port 16686
- Loki on port 3100
- Grafana on port 3000
How does it work?
The frontend pod is configured to call the backend pod. Both podinfo pods send traces
and logs to the collector at port 4317 using OTLP gRPC.
The collector forwards:
- Traces to Jaeger via OTLP gRPC on port 4317
- Logs to Loki via OTLP HTTP on port 3100
Jaeger exposes its UI on port 16686. Grafana exposes its UI on port 3000 and is
pre-configured with both Jaeger and Loki as datasources.
Running it locally
- Start all the containers
make run- Send some sample requests
curl -v http://localhost:9898/status/200
curl -X POST -v http://localhost:9898/api/echo- Visit
http://localhost:16686/to see traces in Jaeger - Visit
http://localhost:3000/to explore logs in Grafana (Explore โ Loki) and traces (Explore โ Jaeger) - Stop all the containers
make stopOverview (`charts/podinfo/README.md`)
Podinfo
Podinfo is a tiny web application made with Go
that showcases best practices of running microservices in Kubernetes.
Podinfo is used by CNCF projects like Flux
and Flagger
for end-to-end testing and workshops.
Installing the Chart
The Podinfo charts are published to
GitHub Container Registry
and signed with Cosign & GitHub Actions OIDC.
To install the chart with the release name podinfo from GHCR:
$ helm upgrade -i podinfo oci://ghcr.io/stefanprodan/charts/podinfoTo verify a chart version with Cosign:
$ cosign verify ghcr.io/stefanprodan/charts/podinfo:<VERSION> \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
--certificate-identity-regexp="^https://github\.com/stefanprodan/.*$"Alternatively, you can install the chart from GitHub pages:
$ helm repo add stefanprodan https://stefanprodan.github.io/podinfo
$ helm upgrade -i podinfo stefanprodan/podinfoThe command deploys podinfo on the Kubernetes cluster in the default namespace.
The configuration section lists the parameters that can be configured during installation.
Uninstalling the Chart
To uninstall the podinfo release:
$ helm uninstall podinfoThe command removes all the Kubernetes components associated with the chart and deletes the release.
Configuration
The following tables lists the configurable parameters of the podinfo chart and their default values.
| Parameter | Default | Description |
|---|---|---|
replicaCount |
1 |
Desired number of pods |
logLevel |
info |
Log level: debug, info, warn, error |
backend |
None |
Echo backend URL |
backends |
[] |
Array of echo backend URLs |
cache |
None |
Redis address in the format tcp://<host>: |
redis.enabled |
false |
Create Redis deployment for caching purposes |
redis.repository |
docker.io/redis |
Redis image repository |
redis.tag |
<VERSION> |
Redis image tag |
redis.imagePullSecrets |
[] |
Redis image pull secrets |
ui.color |
#34577c |
UI color |
ui.message |
None |
UI greetings message |
ui.logo |
None |
UI logo |
prefix |
/ |
URL path prefix for HTTP routes (e.g. /foo maps /api/info to /foo/api/info) |
faults.delay |
false |
Random HTTP response delays between 0 and 5 seconds |
faults.error |
false |
1/3 chances of a random HTTP response error |
faults.unhealthy |
false |
When set, the healthy state is never reached |
faults.unready |
false |
When set, the ready state is never reached |
faults.testFail |
false |
When set, a helm test is included which always fails |
faults.testTimeout |
false |
When set, a helm test is included which always times out |
image.repository |
ghcr.io/stefanprodan/podinfo |
Image repository |
image.tag |
<VERSION> |
Image tag |
image.pullPolicy |
IfNotPresent |
Image pull policy |
image.pullSecrets |
[] |
Image pull secrets |
service.enabled |
true |
Create a Kubernetes Service, should be disabled when using Flagger |
service.type |
ClusterIP |
Type of the Kubernetes Service |
service.metricsPort |
9797 |
Prometheus metrics endpoint port |
service.httpPort |
9898 |
Container HTTP port |
service.externalPort |
9898 |
ClusterIP HTTP port |
service.grpcPort |
9999 |
ClusterIP gPRC port |
service.grpcService |
podinfo |
gPRC service name |
service.nodePort |
31198 |
NodePort for the HTTP endpoint |
service.trafficDistribution |
"" |
Traffic distribution strategy |
service.additionalLabels |
{} |
Additional labels to add to the service |
service.externalTrafficPolicy |
None |
External traffic policy for LoadBalance service |
h2c.enabled |
false |
Allow upgrading to h2c (non-TLS version of HTTP/2) |
extraArgs |
[] |
Additional command line arguments to pass to podinfo container |
extraEnvs |
[] |
Extra environment variables for the podinfo container |
config.path |
"" |
config file path |
config.name |
"" |
config file name |
hpa.enabled |
false |
Enables the Kubernetes HPA |
hpa.maxReplicas |
10 |
Maximum amount of pods |
hpa.cpu |
None |
Target CPU usage per pod |
hpa.memory |
None |
Target memory usage per pod |
hpa.requests |
None |
Target HTTP requests per second per pod |
serviceAccount.enabled |
false |
Whether a service account should be created |
serviceAccount.name |
None |
The name of the service account to use, if not set a name is generated using the fullname template |
serviceAccount.imagePullSecrets |
[] |
List of image pull secrets if pulling from private registries |
securityContext |
{} |
The security context to be set on the podinfo container |
podSecurityContext |
{} |
The security context to be set on the pod |
podAnnotations |
{} |
Pod annotations |
serviceMonitor.enabled |
false |
Whether a Prometheus Operator service monitor should be created |
serviceMonitor.interval |
15s |
Prometheus scraping interval |
serviceMonitor.additionalLabels |
{} |
Add additional labels to the service monitor |
ingress.enabled |
false |
Enables Ingress |
ingress.className |
"" |
Use ingressClassName |
ingress.additionalLabels |
{} |
Add additional labels to the ingress |
ingress.annotations |
{} |
Ingress annotations |
ingress.hosts |
[] |
Ingress accepted hosts |
ingress.tls |
[] |
Ingress TLS configuration |
httpRoute.enabled |
false |
Enables Gateway API HTTPRoute |
httpRoute.additionalLabels |
{} |
Add additional labels to the HTTPRoute |
httpRoute.annotations |
{} |
HTTPRoute annotations |
httpRoute.parentRefs |
[] |
Gateways that this route is attached to |
httpRoute.hostnames |
["podinfo.local"] |
Hostnames matching HTTP header |
httpRoute.rules |
[] |
List of rules and filters applied |
hooks.<hookType>.job.enabled |
false |
Create a Helm hook job for testing (hookType: see values.yaml for available hooks) |
hooks.<hookType>.job.hookDeletePolicy |
hook-succeeded,hook-failed |
Helm hook delete policy |
hooks.<hookType>.job.ttlSecondsAfterFinished |
None |
Job TTL after finished |
hooks.<hookType>.job.sleepSeconds |
None |
Sleep duration before job exits |
hooks.<hookType>.job.exitCode |
0 |
Job exit code |
resources.requests.cpu |
1m |
Pod CPU request |
resources.requests.memory |
16Mi |
Pod memory request |
resources.limits.cpu |
None |
Pod CPU limit |
resources.limits.memory |
None |
Pod memory limit |
nodeSelector |
{} |
Node labels for pod assignment |
tolerations |
[] |
List of node taints to tolerate |
affinity |
None |
Node/pod affinities |
priorityClassName |
"" |
Pod priority class name |
Specify each parameter using the --set key=value[,key=value] argument:
$ helm upgrade -i podinfo oci://ghcr.io/stefanprodan/charts/podinfo \
--set=serviceMonitor.enabled=true,serviceMonitor.interval=5sTo add custom annotations you need to escape the annotation key string:
$ helm upgrade -i podinfo oci://ghcr.io/stefanprodan/charts/podinfo \
--set podAnnotations."toolkit\.fluxcd\.io\/tenant"=dev-teamAlternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart:
$ helm upgrade -i my-release oci://ghcr.io/stefanprodan/charts/podinfo -f values.yamlOverview (`timoni/podinfo/README.md`)
Podinfo
Podinfo is a tiny web application
made with Go that showcases the best practices of running microservices in Kubernetes.
Module Repository
This module is available on GitHub Container Registry at
ghcr.io/stefanprodan/modules/podinfo.
To list all available versions and their digests:
timoni mod list oci://ghcr.io/stefanprodan/modules/podinfoPrerequisites
- Kubernetes 1.25+
- Timoni 0.34+
Install
To create an instance using the default values:
timoni -n default apply podinfo oci://ghcr.io/stefanprodan/modules/podinfoTo install a specific module version:
timoni -n default apply podinfo oci://ghcr.io/stefanprodan/modules/podinfo -v 6.14.1To change the default configuration,
create one or more values.cue files and apply them to the instance.
For example, create a file my-values.cue with the following content:
values: {
resources: requests: {
cpu: "100m"
memory: "128Mi"
}
}And apply the values with:
timoni -n default apply podinfo oci://ghcr.io/stefanprodan/modules/podinfo \
--values ./my-values.cueUninstall
To uninstall an instance and delete all its Kubernetes resources:
timoni -n default delete podinfoBundle
For production deployments it is recommended to use a Timoni
bundle that offers a declarative way of
managing the lifecycle of applications and their infra dependencies.
The following bundle deploys a highly-available podinfo frontend
exposed through a Gateway API HTTPRoute, calling a backend instance
with Redis caching enabled:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
backend: {
module: url: "oci://ghcr.io/stefanprodan/modules/podinfo"
namespace: "podinfo"
values: {
caching: {
enabled: true
redisURL: "tcp://redis:6379"
}
hpa: {
enabled: true
minReplicas: 2
maxReplicas: 10
cpu: 90
}
}
}
frontend: {
module: url: "oci://ghcr.io/stefanprodan/modules/podinfo"
namespace: "podinfo"
values: {
ui: backend: "http://backend.podinfo.svc.cluster.local/echo"
replicas: 2
podDisruptionBudget: enabled: true
httpRoute: {
enabled: true
parentRefs: [{
name: "gateway"
namespace: "gateway-system"
}]
hostnames: ["podinfo.example.com"]
}
test: enabled: true
}
}
}
}Save the bundle as podinfo.cue and apply the stack with:
timoni bundle apply -f podinfo.cueConfiguration
All values are optional. The module complies with the restricted
Kubernetes pod security standard
by default: the container is denied privilege escalation, its root
filesystem is read-only, all its capabilities are dropped, and the pod
runs as the non-root user of the podinfo image with the RuntimeDefault
seccomp profile.
General values
| Key | Type | Default | Description |
|---|---|---|---|
image |
timoniv1.#Image |
ghcr.io/stefanprodan/podinfo:<latest version> |
Container image repository, tag, digest and pull policy; the digest takes precedence over the tag when specified |
metadata.labels / annotations |
{[string]: string} |
unset | Extra labels and annotations added to all resources |
replicas |
int |
1 |
Number of pod replicas; ignored when hpa is enabled |
revisionHistoryLimit |
int |
unset | Number of old ReplicaSets to retain |
strategy |
appsv1.#DeploymentStrategy |
RollingUpdate with maxUnavailable: 50% |
Rollout strategy |
logLevel |
string |
info |
Log level: debug, info, warn or error |
extraArgs |
[...string] |
[] |
Extra command line arguments appended to the podinfo command |
env |
[...corev1.#EnvVar] |
unset | Extra environment variables of the podinfo container |
resources |
timoniv1.#ResourceRequirements |
10m CPU, 32Mi memory requests |
Container resource requirements |
securityContext |
corev1.#SecurityContext |
hardened | Container security context; defaults: no privilege escalation, read-only rootfs, all capabilities dropped (the pod identity comes from podSecurityContext) |
livenessProbe / readinessProbe |
corev1.#Probe |
/healthz, /readyz |
Container probes on the http port |
startupProbe |
corev1.#Probe |
unset | Startup probe |
dataVolume |
corev1.#VolumeSource |
emptyDir |
Volume backing /data, where podinfo stores its runtime data (the root filesystem is read-only) |
extraVolumes / extraVolumeMounts |
[...] |
unset | Additional volumes and volume mounts |
initContainers / extraContainers |
[...corev1.#Container] |
unset | Additional containers |
serviceAccount.create |
bool |
true |
Create the service account; set to false to use an existing one |
serviceAccount.name |
string |
instance name, or default when create: false |
Service account name |
serviceAccount.labels / annotations |
{[string]: string} |
unset | Extra service account metadata (e.g. IRSA role annotations) |
serviceAccount.automountServiceAccountToken |
bool |
false |
Mount the token through the service account (the pod setting mounts it instead) |
test.enabled |
bool |
false |
Run the end-to-end test Jobs at install and upgrades: a curl request to the http port and a grpc-health-probe check of the grpc port |
test.image |
timoniv1.#Image |
ghcr.io/curl/curl-container/curl-multi:<version> |
Container image of the http test Job |
test.grpcImage |
timoniv1.#Image |
ghcr.io/grpc-ecosystem/grpc-health-probe:<version> |
Container image of the gRPC test Job |
Pod scheduling values
| Key | Type | Default | Description |
|---|---|---|---|
podLabels / podAnnotations |
{[string]: string} |
unset | Extra pod metadata |
securityContextPreset |
hardened or platform |
hardened |
Pod identity defaults: hardened pins the image's non-root UID 100 and GID 101, platform leaves the identity to the cluster (e.g. OpenShift SCCs) |
podSecurityContext |
corev1.#PodSecurityContext |
per securityContextPreset |
Pod security context |
imagePullSecrets |
[...] |
unset | Secrets for pulling from private registries |
priorityClassName |
string |
unset | Pod priority class |
affinity.podAntiAffinity |
soft, hard, none or raw rules |
soft |
Spread the replicas across nodes; raw rules replace the preset and need explicit label selectors |
affinity.nodeAffinity / affinity.podAffinity |
raw rules | unset | Node and pod affinity rules |
nodeSelector |
{[string]: string} |
Linux nodes | Node selection; a supplied value replaces the default |
tolerations / topologySpreadConstraints |
unset | Standard scheduling controls | |
hostAliases |
[...corev1.#HostAlias] |
unset | Extra entries for the pod /etc/hosts |
schedulerName |
string |
unset | Alternate scheduler |
dnsPolicy / dnsConfig |
unset | Pod DNS settings | |
terminationGracePeriodSeconds |
int |
unset | Pod termination grace period |
automountServiceAccountToken |
bool |
false |
Mount the API credentials in the pod; podinfo does not talk to the Kubernetes API |
deploymentLabels / deploymentAnnotations |
{[string]: string} |
unset | Extra Deployment metadata |
Service values
The http-metrics port (9797) is always exposed for scraping.
| Key | Type | Default | Description |
|---|---|---|---|
service.port |
int |
80 |
Service http port, the backend of ingress and httpRoute |
service.grpcPort |
int |
9999 |
Service grpc port, targeting the podinfo gRPC server |
service.type |
string |
ClusterIP |
Service type |
service.annotations / labels |
{[string]: string} |
unset | Extra Service metadata |
service.clusterIP / externalIPs / loadBalancerIP / loadBalancerClass / loadBalancerSourceRanges / externalTrafficPolicy |
unset | Service networking settings per type | |
service.nodePort / grpcNodePort |
int |
0 (auto) |
Node ports with the NodePort and LoadBalancer service types |
service.ipFamilies / ipFamilyPolicy |
unset | Service IP family settings |
Routing values
| Key | Type | Default | Description |
|---|---|---|---|
ingress.enabled |
bool |
false |
Create an Ingress for the http Service port |
ingress.className |
string |
unset | IngressClass name |
ingress.hosts |
[...#IngressHost] |
required when enabled | Ingress rules: each host takes paths with path and pathType (default / Prefix) |
ingress.tls |
[...networkingv1.#IngressTLS] |
unset | TLS termination settings |
ingress.annotations / labels |
{[string]: string} |
unset | Extra Ingress metadata |
httpRoute.enabled |
bool |
false |
Create a Gateway API HTTPRoute for the http Service port |
httpRoute.parentRefs |
[...] |
required when enabled | Gateways the route attaches to (name, namespace, optional sectionName to pin a listener); TLS is terminated by the Gateway listener |
httpRoute.hostnames |
[...string] |
unset | Hostnames the route matches |
httpRoute.rules |
[...] |
match all | Route rules with matches and filters; the backend reference is generated by the module |
httpRoute.annotations / labels |
{[string]: string} |
unset | Extra HTTPRoute metadata |
For example, to expose podinfo with an Ingress and a cert-manager issued certificate:
values: {
ingress: {
enabled: true
className: "nginx"
annotations: "cert-manager.io/cluster-issuer": "letsencrypt"
hosts: [{host: "podinfo.example.com"}]
tls: [{
secretName: "podinfo-tls"
hosts: ["podinfo.example.com"]
}]
}
}Or with a Gateway API HTTPRoute attached to an existing Gateway. TLS is
terminated by the Gateway listener the route attaches to (the certificate
is referenced in the Gateway spec, not in the route), and the route
hostnames must match the listener hostname:
values: {
httpRoute: {
enabled: true
parentRefs: [{
name: "gateway"
namespace: "gateway-system"
}]
hostnames: ["podinfo.example.com"]
}
}When waiting for the instance to become ready, Timoni waits for the
HTTPRoute to be accepted by the referenced Gateways.
Autoscaling and disruption values
| Key | Type | Default | Description |
|---|---|---|---|
hpa.enabled |
bool |
false |
Create a HorizontalPodAutoscaler owning the replica count |
hpa.minReplicas / maxReplicas |
int |
1 / minReplicas |
Autoscaling bounds |
hpa.cpu |
int |
99 |
CPU average utilization target (percentage); 0 disables the CPU metric |
hpa.memory |
string |
unset | Memory average value target (e.g. 1024Mi) |
hpa.metrics |
[...] |
[] |
Extra scaling metrics appended to the generated ones |
hpa.behavior |
unset | Scaling behavior tuning | |
podDisruptionBudget.enabled |
bool |
false |
Create a PodDisruptionBudget |
podDisruptionBudget.minAvailable |
int or percent |
1 |
Minimum available pods; mutually exclusive with maxUnavailable |
podDisruptionBudget.maxUnavailable |
int or percent |
unset | Maximum unavailable pods |
podDisruptionBudget.unhealthyPodEvictionPolicy |
string |
unset | IfHealthyBudget or AlwaysAllow; requires Kubernetes 1.27+ |
Monitoring values
When the ServiceMonitor is disabled, the metrics endpoint is advertised
to Prometheus through the prometheus.io/scrape and prometheus.io/port
pod annotations.
| Key | Type | Default | Description |
|---|---|---|---|
serviceMonitor.enabled |
bool |
false |
Create a Prometheus Operator ServiceMonitor for the http-metrics Service port, in the instance namespace |
serviceMonitor.additionalLabels / annotations |
{[string]: string} |
unset | Extra ServiceMonitor metadata, e.g. labels for Prometheus discovery |
serviceMonitor.jobLabel |
string |
app.kubernetes.io/name |
Service label used as the Prometheus job name |
serviceMonitor.interval / scrapeTimeout |
string |
unset | Scrape cadence; defaults to the Prometheus settings |
serviceMonitor.honorLabels |
bool |
false |
Keep scraped label values on collision |
serviceMonitor.enableHttp2 |
bool |
unset | Enable HTTP2 for scraping |
serviceMonitor.scheme / tlsConfig / bearerTokenFile / bearerTokenSecret / proxyUrl |
unset | Scrape scheme, TLS, authentication and proxy settings | |
serviceMonitor.metricRelabelings / relabelings |
[...] |
unset | Relabeling rules for the samples and the targets |
serviceMonitor.sampleLimit / targetLimit / labelLimit / labelNameLengthLimit / labelValueLengthLimit |
int |
unset | Scrape limits |
serviceMonitor.targetLabels / podTargetLabels |
[...string] |
unset | Service/pod labels copied onto the metrics |
Caching values
| Key | Type | Default | Description |
|---|---|---|---|
caching.enabled |
bool |
false |
Enable Redis caching |
caching.redisURL |
string |
required when enabled | Redis URL in the format tcp://[:password@]host[:port] |
UI values
| Key | Type | Default | Description |
|---|---|---|---|
ui.color |
string |
#34577c |
Background color |
ui.message |
string |
unset | Greeting message |
ui.backend |
string |
unset | Backend service URL called by the /api/echo endpoint |
Overview (`deploy/bases/database/README.md`)
Database Setup
This directory contains the Kubernetes manifests to simulate a database setup
with a primary database, read replicas, and scheduled maintenance tasks using CronJobs.
Components
Core Resources
| Resource | File | Description |
|---|---|---|
| ServiceAccount | serviceaccount.yaml |
Shared service account for all database workloads |
| PVC | pvc-primary.yaml |
1Gi persistent storage for primary database |
| StatefulSet | statefulset-primary.yaml |
Primary database with persistent storage at /data |
| Deployment | deployment-replica.yaml |
Read replica deployment |
| Service (Headless) | service-primary.yaml |
Headless service for StatefulSet |
| Service | service-replica.yaml |
ClusterIP service for replicas |
| HPA | hpa-replica.yaml |
Autoscaler for replicas (2-3 pods, 99% CPU) |
CronJobs
| CronJob | Schedule | Duration | TTL Cleanup | Description |
|---|---|---|---|---|
rollup-daily |
Every 10 min | ~1 min | 1 hour | Daily rollup simulation (6 iterations) |
rollup-weekly |
Every 30 min | ~2 min | 1 day | Weekly rollup simulation (12 iterations) |
backup-daily |
Daily at midnight | ~1 min | 1 day | Backup simulation (configured to fail) |
Scripts
Located in scripts/ directory:
rollup.sh- Rollup simulation script with configurable steps viaROLLUP_STEPSenv varbackup.sh- Backup simulation script with configurable exit code viaBACKUP_EXITenv var
Labels
All resources use Kubernetes recommended labels:
app.kubernetes.io/name- Component nameapp.kubernetes.io/part-of: database- Part of database application
Configuration
Primary Database
- Port: 3306 (MySQL standard)
- Storage: 1Gi PersistentVolumeClaim mounted at
/data - Service: Headless (
clusterIP: None) for StatefulSet
Replica Database
- Port: 3306
- Scaling: HPA with 2-3 replicas at 99% CPU utilization
- Service: ClusterIP
CronJob Scripts
The scripts check database-replica health before running:
podcli check http database-replica:3306/readyzUsage
Deploy with Kustomize:
kubectl apply -k deploy/bases/databaseOr include in an overlay:
# kustomization.yaml
resources:
- ../../bases/databasePkg Api Http Docs Swagger.Json (`pkg/api/http/docs/swagger.json`)
{
"schemes": [
"http",
"https"
],
"swagger": "2.0",
"info": {
"description": "Go microservice template for Kubernetes.",
"title": "Podinfo API",
"contact": {
"name": "Source Code",
"url": "https://github.com/stefanprodan/podinfo"
},
"license": {
"name": "MIT License",
"url": "https://github.com/stefanprodan/podinfo/blob/master/LICENSE"
},
"version": "2.0"
},
"basePath": "/",
"paths": {
"/": {
"get": {
"description": "renders podinfo UI",
"produces": [
"text/html"
],
"tags": [
"HTTP API"
],
"summary": "Index",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/api/echo": {
"post": {
"description": "forwards the call to the backend service and echos the posted content",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Echo",
"responses": {
"202": {
"description": "Accepted",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/api/info": {
"get": {
"description": "returns the runtime information",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Runtime information",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.RuntimeResponse"
}
}
}
}
},
"/cache/{key}": {
"get": {
"description": "returns the content from cache if key exists",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Get payload from cache",
"parameters": [
{
"type": "string",
"description": "Key to load from cache",
"name": "key",
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
},
"post": {
"description": "writes the posted content in cache",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Save payload in cache",
"parameters": [
{
"type": "string",
"description": "Key to save to",
"name": "key",
"in": "path",
"required": true
}
],
"responses": {
"202": {
"description": "Accepted"
}
}
},
"delete": {
"description": "deletes the key and its value from cache",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Delete payload from cache",
"parameters": [
{
"type": "string",
"description": "Key to delete",
"name": "key",
"in": "path",
"required": true
}
],
"responses": {
"202": {
"description": "Accepted"
}
}
}
},
"/chunked/{seconds}": {
"get": {
"description": "uses transfer-encoding type chunked to give a partial response and then waits for the specified period",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Chunked transfer encoding",
"parameters": [
{
"type": "integer",
"description": "seconds to wait for",
"name": "seconds",
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/delay/{seconds}": {
"get": {
"description": "waits for the specified period",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Delay",
"parameters": [
{
"type": "integer",
"description": "seconds to wait for",
"name": "seconds",
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/env": {
"get": {
"description": "returns the environment variables as a JSON array",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Environment",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
},
"/fault_injection/disable": {
"post": {
"description": "restores normal responses",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Fault Injection"
],
"summary": "Disable fault injection",
"responses": {
"202": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/fault_injection/enable": {
"post": {
"description": "makes the server respond with HTTP 500 for all non-probe endpoints",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Fault Injection"
],
"summary": "Enable fault injection",
"responses": {
"202": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/fault_injection/status": {
"get": {
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Fault Injection"
],
"summary": "Get fault injection status",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
}
}
},
"/headers": {
"get": {
"description": "returns a JSON array with the request HTTP headers",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Headers",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
},
"/healthz": {
"get": {
"description": "used by Kubernetes liveness probe",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Kubernetes"
],
"summary": "Liveness check",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/metrics": {
"get": {
"description": "returns HTTP requests duration and Go runtime metrics",
"produces": [
"text/plain"
],
"tags": [
"Kubernetes"
],
"summary": "Prometheus metrics",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/panic": {
"get": {
"description": "crashes the process with exit code 255",
"tags": [
"HTTP API"
],
"summary": "Panic",
"responses": {}
}
},
"/readyz": {
"get": {
"description": "used by Kubernetes readiness probe",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Kubernetes"
],
"summary": "Readiness check",
"responses": {
"200": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/readyz/disable": {
"post": {
"description": "signals the Kubernetes LB to stop sending requests to this instance",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Kubernetes"
],
"summary": "Disable ready state",
"responses": {
"202": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/readyz/enable": {
"post": {
"description": "signals the Kubernetes LB that this instance is ready to receive traffic",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"Kubernetes"
],
"summary": "Enable ready state",
"responses": {
"202": {
"description": "OK",
"schema": {
"type": "string"
}
}
}
}
},
"/status/{code}": {
"get": {
"description": "sets the response status code to the specified code",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Status code",
"parameters": [
{
"type": "integer",
"description": "status code to return",
"name": "code",
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/store": {
"post": {
"description": "writes the posted content to disk at /data/hash and returns the SHA1 hash of the content",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Upload file",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/store/{hash}": {
"get": {
"description": "returns the content of the file /data/hash if exists",
"consumes": [
"application/json"
],
"produces": [
"text/plain"
],
"tags": [
"HTTP API"
],
"summary": "Download file",
"parameters": [
{
"type": "string",
"description": "hash value",
"name": "hash",
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "file",
"schema": {
"type": "string"
}
}
}
}
},
"/token": {
"post": {
"description": "issues a JWT token valid for one minute",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Generate JWT token",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.TokenResponse"
}
}
}
}
},
"/token/validate": {
"post": {
"description": "validates the JWT token",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Validate JWT token",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.TokenValidationResponse"
}
},
"401": {
"description": "Unauthorized",
"schema": {
"type": "string"
}
}
}
}
},
"/version": {
"get": {
"description": "returns podinfo version and git commit hash",
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Version",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
},
"/ws/echo": {
"post": {
"description": "echos content via websockets",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"HTTP API"
],
"summary": "Echo over websockets",
"responses": {
"202": {
"description": "Accepted",
"schema": {
"$ref": "#/definitions/http.MapResponse"
}
}
}
}
}
},
"definitions": {
"http.MapResponse": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"http.RuntimeResponse": {
"type": "object",
"properties": {
"color": {
"type": "string"
},
"goarch": {
"type": "string"
},
"goos": {
"type": "string"
},
"hostname": {
"type": "string"
},
"logo": {
"type": "string"
},
"message": {
"type": "string"
},
"num_cpu": {
"type": "string"
},
"num_goroutine": {
"type": "string"
},
"revision": {
"type": "string"
},
"runtime": {
"type": "string"
},
"version": {
"type": "string"
}
}
},
"http.TokenResponse": {
"type": "object",
"properties": {
"expires_at": {
"type": "string"
},
"token": {
"type": "string"
}
}
},
"http.TokenValidationResponse": {
"type": "object",
"properties": {
"expires_at": {
"type": "string"
},
"token_name": {
"type": "string"
}
}
}
}
}
Pkg Api Http Docs Swagger.Yaml (`pkg/api/http/docs/swagger.yaml`)
basePath: /
definitions:
http.MapResponse:
additionalProperties:
type: string
type: object
http.RuntimeResponse:
properties:
color:
type: string
goarch:
type: string
goos:
type: string
hostname:
type: string
logo:
type: string
message:
type: string
num_cpu:
type: string
num_goroutine:
type: string
revision:
type: string
runtime:
type: string
version:
type: string
type: object
http.TokenResponse:
properties:
expires_at:
type: string
token:
type: string
type: object
http.TokenValidationResponse:
properties:
expires_at:
type: string
token_name:
type: string
type: object
info:
contact:
name: Source Code
url: https://github.com/stefanprodan/podinfo
license:
name: MIT License
url: https://github.com/stefanprodan/podinfo/blob/master/LICENSE
title: Podinfo API
version: "2.0"
paths:
/:
get:
produces:
- text/html
responses:
"200":
schema:
type: string
summary: Index
tags:
- HTTP API
/api/echo:
post:
consumes:
- application/json
produces:
- application/json
responses:
"202":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Echo
tags:
- HTTP API
/api/info:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.RuntimeResponse'
summary: Runtime information
tags:
- HTTP API
/cache/{key}:
delete:
consumes:
- application/json
parameters:
- description: Key to delete
in: path
name: key
required: true
type: string
produces:
- application/json
responses:
"202":
summary: Delete payload from cache
tags:
- HTTP API
get:
consumes:
- application/json
parameters:
- description: Key to load from cache
in: path
name: key
required: true
type: string
produces:
- application/json
responses:
"200":
schema:
type: string
summary: Get payload from cache
tags:
- HTTP API
post:
consumes:
- application/json
parameters:
- description: Key to save to
in: path
name: key
required: true
type: string
produces:
- application/json
responses:
"202":
summary: Save payload in cache
tags:
- HTTP API
/chunked/{seconds}:
get:
consumes:
- application/json
and then waits for the specified period
parameters:
- description: seconds to wait for
in: path
name: seconds
required: true
type: integer
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Chunked transfer encoding
tags:
- HTTP API
/delay/{seconds}:
get:
consumes:
- application/json
parameters:
- description: seconds to wait for
in: path
name: seconds
required: true
type: integer
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Delay
tags:
- HTTP API
/env:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
items:
type: string
type: array
summary: Environment
tags:
- HTTP API
/fault_injection/disable:
post:
consumes:
- application/json
produces:
- application/json
responses:
"202":
schema:
type: string
summary: Disable fault injection
tags:
- Fault Injection
/fault_injection/enable:
post:
consumes:
- application/json
produces:
- application/json
responses:
"202":
schema:
type: string
summary: Enable fault injection
tags:
- Fault Injection
/fault_injection/status:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
additionalProperties:
type: string
type: object
summary: Get fault injection status
tags:
- Fault Injection
/headers:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
items:
type: string
type: array
summary: Headers
tags:
- HTTP API
/healthz:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
type: string
summary: Liveness check
tags:
- Kubernetes
/metrics:
get:
produces:
- text/plain
responses:
"200":
schema:
type: string
summary: Prometheus metrics
tags:
- Kubernetes
/panic:
get:
responses: {}
summary: Panic
tags:
- HTTP API
/readyz:
get:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
type: string
summary: Readiness check
tags:
- Kubernetes
/readyz/disable:
post:
consumes:
- application/json
produces:
- application/json
responses:
"202":
schema:
type: string
summary: Disable ready state
tags:
- Kubernetes
/readyz/enable:
post:
consumes:
- application/json
traffic
produces:
- application/json
responses:
"202":
schema:
type: string
summary: Enable ready state
tags:
- Kubernetes
/status/{code}:
get:
consumes:
- application/json
parameters:
- description: status code to return
in: path
name: code
required: true
type: integer
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Status code
tags:
- HTTP API
/store:
post:
consumes:
- application/json
SHA1 hash of the content
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Upload file
tags:
- HTTP API
/store/{hash}:
get:
consumes:
- application/json
parameters:
- description: hash value
in: path
name: hash
required: true
type: string
produces:
- text/plain
responses:
"200":
schema:
type: string
summary: Download file
tags:
- HTTP API
/token:
post:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.TokenResponse'
summary: Generate JWT token
tags:
- HTTP API
/token/validate:
post:
consumes:
- application/json
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.TokenValidationResponse'
"401":
schema:
type: string
summary: Validate JWT token
tags:
- HTTP API
/version:
get:
produces:
- application/json
responses:
"200":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Version
tags:
- HTTP API
/ws/echo:
post:
consumes:
- application/json
produces:
- application/json
responses:
"202":
schema:
$ref: '#/definitions/http.MapResponse'
summary: Echo over websockets
tags:
- HTTP API
schemes:
- http
- https
swagger: "2.0"
--- METRICS ---
- Files Extracted: 10
- Estimated Token Budget: ~20032 tokens
- Canonical Reference: https://codewiki.google/github.com/stefanprodan/podinfo