DRAFT CNCF Conformance WG Intro Kubecon Seattle

2020-02-27 55浏览

1.CNCF Conformance Working Group Intro Aaron Crickenberger, Google (@spiffxp) Dan Kohn, CNCF (@dankohn)
2.Agenda Goals of Conformance / Goals of Conformance Group (Dan) Overview of Certification Program ● ● Requirements How to run tests and submit results Progress we have made ● ● ● Tests added Coverage improved There was no reason to let it last this long and get this bad
3.Certification Program Certified Kubernetes is a software conformance program. Vendors can certify with CNCF that their Kubernetes products conform to the community test suite. Launched in Nov 2017, by now more than 70 platforms have been certified
4.Over 77 offeringsSource:CNCF
5.Participants Must 1. Determine goods and services 2. Self-testing for Qualifying Offerings 3. Submit signed Participation Form to LF 4. Submit self-testing results and get accepted a. Send pull request to github.com/cncf/k8s-conformance 5. Pay fees (no charge for CNCF members and non-profit organizations) 6. Abide by the terms and conditions
6.Key Conditions ● Eligible Versions ○ ○ ● Latest two minor releases E.g. k8s v1.12.2, v1.11.4 Conformance time Period ○ ○ Remains valid up to 12 months after release version that certified Platforms must complete recertification each year with latest two minor versions E.g. A company “Acme” initially certified its product with v1.7.0 (released on June 30, 2017). On June 30, 2018, latest minor releases are v1.11.0 and v1.10.5 On or before June 30, 2018, company “Acme” needs to recertify with v1.11.0 or v1.10.5 (or v1.9.9 before v1.11.0 released) ● End user reproducibility ○ ● End users can run self-tests on a Qualifying Offering themselves More details ○ Referto:Terms and Conditions
7.Conformance FAQ ● What’s the cost of certification? ○ ○ ○ CNCFmembers:free of charge Commercial organizations (not a CNCF member): same as joining the CNCF Non-profitorganizations:free of charge ● How to run self-tests? ○ ○ With kubetest With sonobuoy
8.Running via kubetest # ensure kubetest is installed go get -u k8s.io/test-infra/kubetest # build test binaries, ginkgo, and kubectlfirst:make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" # setup for conformance tests export KUBECONFIG=/path/to/kubeconfig export KUBERNETES_CONFORMANCE_TEST=y # OptionA:run all conformance tests serially kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" # OptionB:run parallel conformance tests first, then serial conformance tests serially GINKGO_PARALLEL=y kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" Moredetails:Conformance Testing in Kubernetes
9.Running via sonobuoy ● Prerequisites ○ ○ ○ Have your k8s cluster ready Have kubectl installed An admin kubeconfig file, and KUBECONFIG environment variable set. ● Run sonobuoy CLI ○ ○ go get -u -v github.com/heptio/sonobuoy sonobuoy run ● Checkout results ○ ○ Wait until `sonobuoy status` show the run as completed Copy output to a local directory `sonobuoy retrieve {local-dir}`
10.Running takes a while Why it takes long time to run conformance tests?Pains:● Can take 1-3 hours to pass (If everything goes fine) ● It takes more time to finish ○ In case of any failure encountered during testing ○ Some tests don’t fail fast
11.Running takes a while Why it takes long time to run conformance tests?Cures:● run parallel conformance tests first, then serial conformance tests serially ● Extend default timeout configuration (in sonobuoy) to get full report of failing tests
12.Progress v1.11.0 4 new test cases v1.12.0 Conformance test requirements API Coverage via audit logging 22 new test cases v1.13 Code coverage of conformance tests Upstream kube-conformance image 35 new test cases
13.Add Tests Write an e2e test, get it merged by the owning SIG Document the e2e test Demonstrate that it meets these requirements* - Tests only GA, non-optional features or APIs Works for all providers Is non-privileged Works without public internet access Binaries used are required for Linux kernel or kubelet to run Images used support all architectures for which Kubernetes releases are built Passes against versions of Kubernetes consistent with version skew policy Provides consistent results without flakes Propose to SIG Architecture that the e2e test be promoted to Conformance
14.ProgressSource:github.com/spiffxp/adventures-in-k8s-conformance
15.Add Tests - 1.9 to 1.10 [sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance] [sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance] [sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance] [sig-api-machinery] Garbage collector should not delete dependents that have both valid owner and owner that's waiting for dependents to be deleted [Conformance] [sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance] [sig-api-machinery] Garbage collector should orphan pods created by rc if delete options say so [Conformance] [sig-apps] Daemon set [Serial] should retry creating failed daemon pods [Conformance] [sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance] [sig-apps] Daemon set [Serial] should run and stop complex daemon [Conformance] [sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance] [sig-apps] Daemon set [Serial] should update pod when spec was updated and update strategy is RollingUpdate [Conformance] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Burst scaling should run to completion even with unhealthy pods [Conformance] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy [Conformance] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Should recreate evicted statefulset [Conformance] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform canary updates and phased rolling updates of template modifications [Conformance] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance]
16.Add Tests - 1.10 to 1.11 [sig-api-machinery] Watchers should be able to restart watching from the last resource version observed by the previous watch [Conformance] [sig-api-machinery] Watchers should be able to start watching from a specific resource version [Conformance] [sig-api-machinery] Watchers should observe add, update, and delete watch notifications on configmaps [Conformance] [sig-api-machinery] Watchers should observe an object deletion if it stops meeting the requirements of the selector [Conformance]
17.Add Tests - 1.11 to 1.12 [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart exec hook properly [NodeConformance] [Conformance] [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop exec hook properly [NodeConformance] [Conformance] [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop http hook properly [NodeConformance] [Conformance] [k8s.io] InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod [Conformance] [k8s.io] InitContainer [NodeConformance] should invoke init containers on a RestartNever pod [Conformance] [k8s.io] InitContainer [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod [Conformance] [k8s.io] InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [Conformance] [sig-api-machinery] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted [Conformance] [sig-api-machinery] Namespaces [Serial] should ensure that all services are removed when a namespace is deleted [Conformance] [sig-apps] Deployment RecreateDeployment should delete old pods and create new ones [Conformance] [sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [sig-apps] Deployment deployment should delete old replica sets [Conformance] [sig-apps] Deployment deployment should support proportional scaling [Conformance] [sig-apps] Deployment deployment should support rollover [Conformance] [sig-storage] ConfigMap binary data should be reflected in volume [NodeConformance] [Conformance] [sig-storage] Secrets should be able to mount in a volume regardless of a different secret existing with same name in different namespace [NodeConformance] [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod with mountPath of existing file [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with projected pod [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with secret pod [Conformance]
18.Add Tests - 1.12 to 1.13 [k8s.io] Container Runtime blackbox test when starting a container that exits should run with the expected status [NodeConformance] [Conformance] [k8s.io] Kubelet when scheduling a busybox Pod with hostAliases should write entries to /etc/hosts [NodeConformance] [Conformance] [k8s.io] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] [k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should be possible to delete [NodeConformance] [Conformance] [k8s.io] Kubelet when scheduling a busybox command that always fails in a pod should have an terminated reason [NodeConformance] [Conformance] [k8s.io] Kubelet when scheduling a read only busybox container should not write to root filesystem [NodeConformance] [Conformance] [k8s.io] Pods should support remote command execution over websockets [NodeConformance] [Conformance] [k8s.io] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] [sig-apps] ReplicaSet should adopt matching pods on creation and release no longer matching pods [Conformance] [sig-apps] ReplicationController should adopt matching pods on creation [Conformance] [sig-apps] ReplicationController should release no longer matching pods [Conformance] [sig-cli] Kubectl client [k8s.io] Guestbook application should create and stop a working application [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl api-versions should check if v1 is in available api versions [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl cluster-info should check if Kubernetes master services is included in cluster-info [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl describe should check if kubectl describe prints relevant information for rc and pods [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl label should update the label on a resource [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl logs should be able to retrieve and filter logs [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl patch should add annotations for pods in rc [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl replace should update a single-container pod's image [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl rolling-update should support rolling-update to same image [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run --rm job should create a job from an image, then delete the job [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run default should create an rc or deployment from an image [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run deployment should create a deployment from an image [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run job should create a job from an image when restart is OnFailure [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run pod should create a pod from an image when restart is Never [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl run rc should create an rc from an image [Conformance] [sig-cli] Kubectl client [k8s.io] Kubectl version should check is all data is printed [Conformance] [sig-cli] Kubectl client [k8s.io] Proxy server should support --unix-socket=/path [Conformance] [sig-cli] Kubectl client [k8s.io] Proxy server should support proxy with --port 0 [Conformance] [sig-cli] Kubectl client [k8s.io] Update Demo should create and stop a replication controller [Conformance] [sig-cli] Kubectl client [k8s.io] Update Demo should do a rolling update of a replication controller [Conformance] [sig-cli] Kubectl client [k8s.io] Update Demo should scale a replication controller [Conformance] [sig-storage] EmptyDir wrapper volumes should not cause race condition when used for configmaps [Serial] [Slow] [Conformance] [sig-storage] EmptyDir wrapper volumes should not conflict [Conformance]
19.ProgressSource:github.com/spiffxp/adventures-in-k8s-conformance
20.API Coverage - by %Source:github.com/spiffxp/adventures-in-k8s-conformance
21.API Coverage v1.13.0 conformance (e2e.test only) If we are looking solely at API coverage caused by the e2e.test binary, it looks as though we have a very long way to go This is why we decided measuring api coverage client side wasn’t an effective representation of behavior exercised within kubernetesSource:apisnoop.cncf.io
22.API Coverage v1.13.0 release If every potentially qualifying e2e test we had today was promoted to conformance, this is how far we would get in terms of API coverage We need to be writing new test cases in order to get the rest of the way thereSource:apisnoop.cncf.io
23.How You Can Help Contribute tests that exercise Pod functionality Categorize existing tests Run Conformance tests against your Kubernetes cluster Participate in discussion at CNCF Conformance WG Participate in discussion at SIG Architecture
24.References - - What is Kuberneteshttps://github.com/kubernetes/community/tree/master/iconshttps://github.com/kubernetes/community/blob/master/contributors/devel/arch-roadmap-1.pngUsing Conformancehttps://github.com/cncf/k8s-conformancehttps://testgrid.k8s.io/conformance-allConformance Requirementshttps://github.com/kubernetes/community/blob/master/contributors/devel/conformance-tests.mdhttps://github.com/kubernetes/community/tree/master/sig-architecture#conformance-definition'>https://github.com/kubernetes/community/tree/master/sig-architecture#conformance-definition