What does it mean?

annotation comment,definition
ceph Delivers object, block, and file storage in one unified system.
crd custom resource definitions
cncf Cloud Native Computing Foundation
cni Container Network Interface (OVNKubernetes OpenShiftSDN) <
cri Container Runtime Interface
cri-o Lightweight container runtime for kubernetes.
csi Container Storage Interface
csv cluster service version
deployment You describe a desired state in a Deployment
Ephemeral Short lived
idp identity provider
ipi Installer-Provisioned Infrastructure
kubelet Kubelet is the primary "node agent" that runs on each node. Takes a set of PodSpecs (primarily through the apiserver) and ensures the containers described are running and healthy.
noobaa data service for cloud environments, providing S3 object-store interface with flexible tiering, mirroring, and spread placement policies, over any storage resource that allows GET/PUT including S3,GCS..
nsfs virtual filesystem making Linux-kernel namespaces available.
oadp openshift api data protection
ocp OpenShift Container Platform
ocs OpenShift Container Storage
ocm OpenShift Cluster Manager
odf OpenShift Data Foundation
olm Operator Lifecycle Manager
pvc Persistent volume claim.
pv Persistent volume. Persistent storage.
reconciling Restore friendly relations between.
replicaset Maintain a stable set of replica Pods running at any given time
rook File, block, and object storage for your cloud native environment and is based on battle tested ceph storage.
rosa Red Hat OpenShift Service on AWS
service Logical abstraction for a deployed group of pods in a cluster (which all perform the same function).
scc security context constraints
skopeo Command line utility used to interact with local and remote container images and container image registries
thanos Long-Term storage for your Prometheus Metrics on OpenShift
upi User-Provisioned Infrastructure
uWSGI Project aims at developing a full stack for building hosting services.
wwn world wide names. Fiber channel

where do I start

. <(oc completion bash)  Get bash completion running.
oc help                  Get commands
oc api-resources         What can you use commands on.
oc options               Which options apply to all commands

read

https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/

files of value

metadata.json         File created during install. Used by openshift-install destroy cluster

oc get

Available resources to ask about.

oc api-resources

login

oc login --username developerhttps://openshift:6443

switch user

oc login --username developer

which clusters have you logged into

oc config get-clusters

List projects

oc projects
oc get projects

select project

oc project $project

list pods

oc get pods
oc get pods --all-namespaces
oc get pods -o wide

wide will give you on which node pod is running.

oc get pods -o wide --all-namespaces

Get pods that are not runing.

oc get pods --field-selector status.phase!=Running --all-namespaces

Get pods matching two states

oc get pods --field-selector=status.phase!=Running,spec.restartPolicy=Always

get services

oc get svc

get shell on node

It is possible to debug more then nodes. (deployment, build, or job)

oc debug node/infra-2.ocpdev.lkl.ltkalmar.se

Get working env

chroot /host

get debug information from oc

oc debug --loglevel=10 node/$node

get nodes

oc get nodes
oc get nodes -o jsonpath='{.items[*].metadata.name}'

Get nodes without headears. name, cpu:s, disk size, mem, ip address.

oc get nodes --no-headers --selector="node-role.kubernetes.io/worker" -o=custom-columns='NAME:.metadata.name,CPU:.status.capacity.cpu,DISK:.status.capacity.ephemeral-storage,MEM:.status.capacity.memory,IP:.status.addresses[?(@.type=="InternalIP")].address'

connect to pod

oc rsh $pod bash

list containers in pod

oc get pod/router-default-6b76b87c6-5m7h6 -n openshift-ingress -o json | jq -r '.spec.containers[].name'
router
logs

connect to container in pod

oc rsh -c router pod/router-default-6b76b87c6-5m7h6

get logs from all containers

Get logs all pods containers.

for POD in $(oc get pods -o jsonpath='{.items[*].metadata.name}') ; do for CONTAINER in $(oc get pod/$POD -o json | jq -r '.spec.containers[].name') ; do echo '***' pod $POD, container $CONTAINER ;

Get logs all pods containers in all namespaces.

oc get namespaces --no-headers | awk '{print $1}' | while read NAMESPACE ; do oc project $NAMESPACE >/dev/null ; for POD in $(oc get pods -o jsonpath='{.items[*].metadata.name}') ; do for CONTAINER in $(oc get pod/$POD -o json | jq -r '.spec.containers[].name') ; do echo '***' namespace $NAMESPACE pod $POD, container $CONTAINER ; oc logs $POD $CONTAINER | grep vsphere.int.redbridge.se | tail -10 ; done; done ; done | tee /temp/vsphere.int.redbridge.se

search logs for all pods for string save to file

SEARCH="cosprod-m22s6-worker-m52c8" ; oc get namespaces --no-headers | awk '{print $1}' | while read NAMESPACE ; do oc project $NAMESPACE >/dev/null ; for POD in $(oc get pods -o jsonpath='{.items[*].metadata.name}') ; do for CONTAINER in $(oc get pod/$POD -o json | jq -r '.spec.containers[].name') ; do echo '***' namespace $NAMESPACE pod $POD, container $CONTAINER ; oc logs $POD $CONTAINER | grep "${SEARCH}" | tail -10 ; done; done ; done | tee /tmp/search_all_containers_"${SEARCH}".$(date '+%Y-%m-%d_%H-%M-%S').log

execute command in pod

oc exec pod/router-default-545ffb97db-4h9rx -- $command

describe pods

oc describe pods
oc describe pod stage-sales-62-qjd

To get (almost) all object with a specific label from the current project, execute:

oc get all -l '<label_name>=<label_value>'

logs

oc logs -f router-default-6b76b87c6-4lc4b logs > /tmp/tmp
oc logs -f --tail=0 router-default-6c666984fd-ct8zf logs
oc logs -f --namespace openshift-gitops deployment/openshift-gitops-server

get config from pod in yaml format

oc get pods router-default-545ffb97db-kgsdb -o yaml

get deployments

oc get deployments --all-namespaces

set environment variable in pod

oc set env dc/your-app-name COLOR=blue

unset environment variable in pod

oc set env dc/your-app-name COLOR-

list environment variables

oc set env pod/router-default-545ffb97db-lj2t5 --list

list templates

oc get templates -n openshift

Custom resource definitions.(crd)

oc get crd

sort

CREATED AT

oc get crd --sort-by=.metadata.creationTimestamp

edit

oc edit deployment.apps/router-default

Watch changes taking place.

watch -n1 oc get all

grant permission to project

oc adm policy add-role-to-user view developer -n mysecrets

crictl

List running containers

crictl ps

List all pods

crictl pods

List all images

crictl images

Execute a command in a running container

crictl exec -it 1f73f2d81bf98 /bin/sh

nsenter

run program in different namespaces

which version

oc version

copy files from pod

Copy session keys locally.

oc rsync caas-2-8s6cl:/tmp/sslkeylog .

tcpdump from nodes

ssh $node
toolbox

oc get route -A

get routing.

oc describe route sales -n hlt-prod

Name:                   sales
Namespace:              hlt-prod
Created:                13 months ago
Labels:                 <none>
Annotations:            haproxy.router.openshift.io/balance=roundrobin
                        haproxy.router.openshift.io/disable_cookies=true
Requested Host:         sales.prod.bobcat.hlt.se
                           exposed on router default (host apps.ocpprod.lkl.ltkalmar.se) 13 months ago
Path:                   <none>
TLS Termination:        edge
Insecure Policy:        <none>
Endpoint Port:          port-8000-tcp

Service:        sales
Weight:         100 (100%)
Endpoints:      10.160.7.166:8000, 10.160.7.167:8000, 10.160.7.168:8000 + 35 more...

oc get pods -o custom-columns=POD:.metadata.name --no-headers --all-namespaces

Only name without headers

get label:s

oc get pods --no-headers --all-namespaces|grep -i running | head -2 | while read  namespace pod blabla ; do echo '***' oc label pod/$pod --list=true -n $namespace ; oc label pod/$pod --list=true -n $namespace ; done

get subscriptions

oc get subscriptions -A

whoami

oc whoami
oc config current-context
oc whoami --show-console=true --show-context=true

get list of user

kubectl config view -o jsonpath='{.users[*].name}'

list contexts

oc config get-contexts

use-context

oc config use-context openshift-marketplace/api-ocp-02-cambio-rbcloud-net:6443/kube:admin

oc explain pv

oc explain pv

oc get configmap cluster-monitoring-config

put node offline

Mark a node as unschedulable.

oc adm cordon node1

Drain a node in preparation for maintenance.

oc adm drain <node> --force --delete-emptydir-data --ignore-daemonsets
oc adm drain <node> --ignore-daemonsets --force --grace-period=30 --delete-local-data

Mark node as online.

oc adm uncordon node1

Get pv:s

oc get pv

Get more info about a pv.

oc describe pv $PV

get pvc:s

oc get pvc --all-namespaces | less

which pods are using pvc

kubectl get pods --all-namespaces -o=json | jq -c '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName:.spec.volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'

kubectl

List contexts

kubectl config get-contexts

Select context

kubectl config use-context default/api-blabla-halfface-se:6443/kube:admin

list groups

oc get groups -o wide

scale

oc scale --replicas=2 rc/postgresql-1

top(disable wikipedia top

oc adm top pods --use-protocol-buffers --all-namespaces
oc adm top nodes --sort-by=cpu or memory

oc get crd

Get Custom Resource Definitions.

oc get crd

operators

Automatically setup of a instances.

oc adm upgrade --to-image=

Upgrade to version that you found on github okd

oc adm upgrade

Upgrade okd images.

oc adm must-gather

Launch a new instance of a pod for gathering debug information

oc adm certificate approve <csr_name>

Approve csr certificate

openshift upgrade path

https://access.redhat.com/labs/ocpupgradegraph/update_path?channel=stable-4.9&arch=x86_64&is_show_hot_fix=false&current_ocp_version=4.9.15&target_ocp_version=4.10.11

helm

List all helm charts in all namespaces

helm list -aA

cronjobs

oc get cj
oc get cronjobs -o wide -A

delete po

stop pod

oc delete po --all --force
oc delete pod openshift-gitops-server --namespace openshift-gitops
oc delete pods -n openshift-oauth-apiserver --all
oc get po -A | grep -v ^NAME | awk '$4 !~ /Running/' | sort -k4 | while read NAMESPACE POD READY STATUS END ; do echo '****' $POD $STATUS ; echo oc delete po $POD -n $NAMESPACE --force --grace-period=0 ; done
oc get pods -A --field-selector=status.phase!=Running --no-headers | while read NAME_SPACE POD REST_OF_LINE ; do echo oc delete pod $POD -n "${NAME_SPACE}" --force --grace-period=0 ; done

use other namespace

oc rsh  --namespace namespace-name pod-name
oc rsh --namespace namespace-name-operator pod-name bash -c 'echo $PATH $HOSTNAME'

list namespaces

oc get namespace

use namespace

oc rsh  --namespace openshift-gitops openshift-gitops-application-controller-0

kubectl get netnamespace

Command line utility used to configure network. Egress address can be used to define outgoing address. Which can also cause other issues.

oc get netnamespace openshift-gitops -oyaml

oc get routes

oc get routes --namespace openshift-gitops

oc get oauth

Describe authentication methods.

oc get oauth cluster -o yaml

decode token. base64

https://jwt.io/

view secrets

oc get secret ca-key-pair -o go-template='Template:Range $k,$v := .dataTemplate:"Template:$kTemplate:"\n"Template:$vTemplate:"\n\n"Template:End'

delete cluster

openshift-install destroy cluster

storageclasses(sc)

oc get storageclasses

get service accounts

oc get serviceaccounts -A
oc get sa -A

which permissions do I have

oc auth can-i --as=fjuza --list
oc get groups -o wide

alerts

View Alertmanager configured alerts.

oc get prometheusrules -A -o yaml | grep alert: | sort | less

view alerts firing

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s   'http://localhost:9090/api/v1/alerts' | jq . | less -ISRM

View historical alerts.

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s   'http://localhost:9090/api/v1/query_range?query=ALERTS&start=2022-08-08T00:00:00.781Z&end=2022-08-09T00:00:00.781Z&step=1m'

Talk to api with Bearer.

HOST=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.spec.host})
TOKEN=$(oc whoami -t)
curl -skH "Authorization: Bearer $TOKEN" "https://$HOST/api/v2/alerts" | jq .

bash completion

. <(oc completion bash)

machineconfig

view settings

oc describe machineconfigpool

set ntp servers

echo 'variant: openshift
version: 4.9.0
metadata:
  name: 99-master-chrony 
  labels:
    machineconfiguration.openshift.io/role: master 
storage:
  files:
  - path: /etc/chrony.conf
    mode: 0644 
    overwrite: true
    contents:
      inline: |
        server ntp.lio.se iburst
        driftfile /var/lib/chrony/drift
        makestep 1.0 3
        rtcsync
        logdir /var/log/chrony' | butane | oc apply -f -

get users

oc get users

work with oc without login

export KUBECONFIG=auth/kubeconfig

Add the following if cert is not trusted

- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:443
  name: my-cluster

etcdctl

oc exec -it pod/etcd-ocp-03-lm8km-master-1 -n openshift-etcd -- bash
Defaulted container "etcdctl" out of: etcdctl, etcd, etcd-metrics, etcd-health-monitor, setup (init), etcd-ensure-env-vars (init), etcd-resources-copy (init)
[root@ocp-03-lm8km-master-1 /]# etcdctl --write-out=table endpoint status
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| htt://172.19.14.36:2379   | c4f7b42b92713818 |   3.5.0 |  105 MB |     false |      false |         6 |    2632074 |            2632074 |        |
| htt://172.19.14.37:2379   | 5dea668b432969fc |   3.5.0 |  105 MB |     false |      false |         6 |    2632074 |            2632074 |        |
| htt://172.19.14.41:2379   | 51cecd971b657ee5 |   3.5.0 |  105 MB |      true |      false |         6 |    2632074 |            2632074 |        |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

create troubleshooting pod

oc run abjorklund-redhat-ubi8 --image=redhat/ubi8 -i --tty -- sh
oc run abjorklund-curlimage-curl --image=curlimages/curl -i --tty -- sh
oc run -it busybox --image=busybox --restart=Never -- ash

proxy settings

oc get proxy cluster -o yaml

Change ca

oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

oc proxy

Run a proxy to the Kubernetes API server

Install additional ca certificate

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-redbridge-ca-cert
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURrVENDQW5tZ0F3SUJBZ0lFSC93Skh6QU5CZ2txaGtpRzl3MEJBUXNGQURBM01SVXdFd1lEVlFRS0RBeFMKUlVSQ1VrbEVSMFV1VTBVeEhqQWNCZ05WQkFNTUZVTmxjblJwWm1sallYUmxJRUYxZEdodmNtbDBlVEFlRncweQpNVEF5TWpNd056RTVOVFphRncwME1UQXlNak13TnpFNU5UWmFNRGN4RlRBVEJnTlZCQW9NREZKRlJFSlNTVVJIClJTNVRSVEVlTUJ3R0ExVUVBd3dWUTJWeWRHbG1hV05oZEdVZ1FYVjBhRzl5YVhSNU1JSUJJakFOQmdrcWhraUcKOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5mY1F3YURwcEdzNWJxaUc5ajE5aFJVaG1sMzhjb2JGT2tzRQpsZFo3Y3RkV1d6VHJqSTFCRGxZSEd5SXBYMEo4ZU1PaDhvbUZqbVR6VTEzTkpWSnJrWm5RaDRhTzA1UGtKRlJRCkg1ZVA2N3R0S2pEb0txOFZVWXRZUldxRlFaalNxY2lQMzJobXZSNG42QVZDWDdCaUVBZjd2Y05ZVys0a1k5OUsKbTluV1BNbEpGU056M1puRnlWc1BtR1ZWeVN2RmFVL0dBTmt1Z25uSGdUM1VUUTNsc2NidU5keUpBcVEya3dHSwpKbkdZKzBSajVrUWpvdXptUjBDZ3pJN0hWSmhwK2Z6R1lyenRYQXA1Zkt0Z3ZTZFRtTndVVXZJR3pLTmU4WklGCmY0WVVUUDFPdU9jUmNIRDJQclVodDgzWlRLYzNwOUhLYk5CazIzWFFtYU85QVBqeEl3SURBUUFCbzRHa01JR2gKTUI4R0ExVWRJd1FZTUJhQUZMbWFrNHdDamtuakZvWkd6M1daRGErY2N4RGxNQjBHQTFVZERnUVdCQlM1bXBPTQpBbzVKNHhhR1JzOTFtUTJ2bkhNUTVUQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BNEdBMVVkRHdFQi93UUVBd0lCCnhqQStCZ2dyQmdFRkJRY0JBUVF5TURBd0xnWUlLd1lCQlFVSE1BR0dJbWgwZEhBNkx5OXBjR0V0WTJFdWNtVmsKWW5KcFpHZGxMbk5sTDJOaEwyOWpjM0F3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQURabURvUytJY1ZMcERBRwpiSXM0SWRJKzcxY0xINk90NjNkYWhBT25QRDJnMUhvVUFIZFdUcGdobER3TkFQWjg3UXQybFc4Q1B4eDhCQVZOCnlrZWlEN2paeVA5dmVCcDRxNjBiSTVYSENndWV5U2lGdjBBKzloKzMzekMrYy9WbStJVHJNTkZ0dlZMNE1kRWQKaVE4UVBhaFJEWW1qVkJVb1VIZWErMDdkWEY3TzQxY2t2YzZRb0lad2F5Y1Zhc0gvd05lVGNrdzl1TlNiajNTQwoyNHdpOUthQnpxdDZsWlF3TG5uUjVnNjNWUDZNZUprR2FXMTBxdExiQVM4NGZwQ1NWTUx3U051MGZqeFU2d2lPCkRjaWlKKzNZOG5ldjM5NGJHRkwxcG5ZVmM4YmpoL0xaaHM1dTRQUnhlNFBLRER2Y09NZUhpUkN1M1YySWRRTTgKbDl3enBQZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQoK
        mode: 0644
        overwrite: true
        path: /etc/pki/ca-trust/source/anchors/redbridge-ca.crt

get raw api data

oc get --raw "/api/v1/nodes/[node]/proxy/stats/summary"

Via proxy.

oc proxy &
Starting to serve on 127.0.0.1:8001
curl -s http://localhost:8001/api/v1/nodes/crc-lgph7-master-0/proxy/stats/summary
curl -s http://localhost:8001/api/v1/nodes/crc-lgph7-master-0/proxy/metrics/resource

explain

Get documentation for a resource

oc explain deployment

events

Get events.

oc get events -A --sort-by=.metadata.creationTimestamp

yq

Select specific values.

oc get mcp worker -o yaml | yq '.spec.configuration.source.[].name'

Delete specific values.

oc get catalogsources -n openshift-marketplace -o yaml | yq 'del(.items.[].status)'

jsonpath

Get names of MachineConfigs one value per line.

oc get mc -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' --no-headers

endpoints

look to see that pods are defined in

oc get endpoints -n default

ImageStreamTag

ImageStreamTag represents an Image that is retrieved by tag name from an ImageStream.

BuildConfig

Build configurations define a build process for new container images.

setup openshift cluster

Download binary

cd /tmp/ ; curl -L -O https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.10.47/openshift-install-linux.tar.gz && sudo tar xf openshift-install-linux.tar.gz -C /usr/local/bin/

Add vmware certs if using that backend.

(cd /tmp/ ; curl -sk https://${vspherer_server}/certs/download.zip -O) ; cd /etc/pki/ca-trust/source/anchors ; sudo unzip -oj /tmp/download.zip certs/lin/\* ; sudo update-ca-trust

Create config file

install-config.yaml

Then fire of install

openshift-install create cluster

argocd

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

metrics

Search strings.

Cpu usage per node.

100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[30m])) * 100)

oc kubectl

Download openshift client.

wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/openshift-client-linux.tar.gz; tar -xzvf openshift-client-linux.tar.gz; chmod +x oc; sudo rm /usr/local/bin/oc 2>/dev/null ; sudo mv oc /usr/local/bin

time and timezone in first pod

oc get pods --no-headers -o 'custom-columns=:.metadata.namespace,:.metadata.name' | head -1 | while read NAMESPACE POD ; do oc rsh -n $NAMESPACE $POD  bash -c 'date "+%Y-%m-%d %H:%M:%S %Z"' 2>/dev/null ; don

oc get installplan

InstallPlan defines the installation of a set of operators.

oc get installplan install-bk8hw -n openshift-operators -o yaml

Openshift