Question: clientset "k8s.io/client-go/kubernetes" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/test/e2e/framework" <del> e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet" </del> e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2erc "k8s.io/kubernetes/test/e2e/framework/rc"

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: podName := "pod-dualstack-host-ips" <del> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: map[string]string{"test": "dualstack-host-ips"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "dualstack-host-ips", Image: imageutils.GetE2EImage(imageutils.Agnhost), }, }, }, } </del> <ins> pod := genPodHostIPs(podName+string(uuid.NewUUID()), false) </ins> ginkgo.By("submitting the pod to kubernetes") podClient := e2epod.NewPodClient(f)

4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


Question: "context" "fmt" "os" <ins> "os/exec" </ins> "path/filepath" "strconv" "time"

Failing on cri-o serial job. Looks like the dbus command isn't available for graceful shutdown tests that aren't running on Ubuntu. /assign /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hmm, is this still issue? The testgrid job is still failing even after the fix appears to be merged? !
Please take a look when you get a chance, the error still seems to be that dbus-send is unavailable... /cc
oops... I forgot to add this to the other


Question: # TODO(mbforbes): Refactor setting scope flags. local -a scope_flags=() if (( "${#MINION_SCOPES[@]}" > 0 )); then <del> scope_flags=("--scopes" "${MINION_SCOPES[@]}") </del> <ins> scope_flags=("--scopes" "$(join_csv ${MINION_SCOPES[@]})") </ins> else scope_flags=("--no-scopes") fi

This happens over and over. Scary messages. WARNING: Lists should be separated by commas, try "--allow=tcp,udp,icmp,esp,ah,sctp". If you intend to use udp, icmp, esp, ah, sctp for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--scopes=storage-ro,compute-rw". If you intend to use compute-rw for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--metadata-from-file=startup-script=/usr/local/google/home/thockin/src/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/gce/configure-,kube-env=/tmp/kubernetes.MURmzX/master-kube-". If you intend to use kube-env=/tmp/kubernetes.MURmzX/master-kube- for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--disk=name=e2e-test-thockin-master-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no". If you intend to use device-name=master-pd, mode=rw, boot=no, auto-delete=no for positional arguments, put them before the flag.
That's a new one. gcloud started getting a little more anal here. Exciting. cc
They changed list argument syntax a while back, but the warnings are new. Time to update startup scripts I guess.


Question: package iptables import ( <ins> "bufio" </ins> "bytes" "context" "fmt" "regexp" <ins> "strconv" </ins> "strings" "sync" "time"

my k8s cluster version is 1.17.x this cluster has 10 master only 2 nodes get problem. it make my svc sometime can usesometime can timeout what can i do can resolve this problem ? this 2 node's svc is nodeportType
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can increase kube-proxy verbosity adding flag and you should get more information about the cause of the error Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least
/sig network
hello bromy cluster is use kubeadm init .so what should i do .i know this is use configmap but in this i can not found how edit this configmap is availableabout upgrade this cluster .this time is impossible
about stack. i only get this log.
To increase the verbosity on kube-proxy on a kubeadm deployment you can do and wait until pods are restarted. Despite that, if there is a bug, it will not be backported to 1.17, so you have to go to a supported version first
I swear we code to dump the ruleset when this happens, but I don't see it any more.
I'm going to re-purpose this as a feature :)
Hi, I would like to pick this up. Can someone give me some code pointers on where to start?
Let me know if I can help in anyway.
/assign
I'd start with - It's worth thinking about the cleanest solution here, but maybe something like: When you get an error from exec, parse the error for and if it matches, return a . Then let the callers check for a by interface assertion, extract the Line(), extract their input's data at -3 and +3 from that line and log it. E.g. You'll proably want a helper func to do that, which could live in that same iptables package. And a test.
/assign
hi everyone! Is anyone doing this?

I am willing to do this
at least I didn't see any PR from , in case of collision please sync between both to avoid duplicate efforts
hi did you work on it?
Hi apologies I was busy with some other work. You must please continue.
thanks
/assign
I'm working on this now, but I'm stuck in here, I got the line, and then how I extract at -3 and +3 from that line?
you may need to write a helper function to extract them from the logs. my 2 cents.. here is my version for reference. you could add some tests, to cover the boundary conditions. to play around :)


Question: return err } <ins> // store generation in order to avoid repeated updates for the same generation (and potential // fights of API server in HA environments). c.lastSeenGenerationLock.Lock() defer c.lastSeenGenerationLock.Unlock() c.lastSeenGeneration[crd.Name] = crd.Generation </ins> return nil }

/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


Question: "k8s.io/kubernetes/pkg/api/unversioned" "k8s.io/kubernetes/pkg/apis/policy" policyclient "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/policy/internalversion" <ins> "k8s.io/kubernetes/pkg/client/retry" </ins> "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/registry/generic/registry" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> ) const (

What happened: When I try to test after implementing the 1st approach according to , I found pod disruption budgets cannot handle concurrent requests. The error message is: It is an Optimistic Lock error message. It means PDB can't handle concurrent requests. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment with 6 replicas and schedule them to the same node, say node A. You can leave only one node scheduling enabled, and then create the deployment. 2) Create the PDB that match the deployment's label 3) Uncordon another node, say node B 4) Drain node A using concurrent POST eviction requests. Ref: I'm not sure if this behavior is OK for PDB. I think I can make work by waiting for a while if current pod label(s) match previous pod label(s).
Oh, that's interesting. I think the problem is that this function doesn't retry if UpdateStatus() fails due to version conflict. I think the solution is to modify the caller, i.e. this function so that instead of just it instead does something like This will loop re-doing checkAndDecrement() and getPodDisruptionBudgets() until checkAndDecrement() succeeds, or fails with a non-conflict error. Does that make sense? cc/ for sanity check
Or use PATCH?
Yes, PATCH should work also. I haven't written code that uses PATCH, so I can't advise exactly the code to write here (there are examples by grepping for patch, but I'm not sure exactly what is the right way to do it from inside of API server code, which is where we are here). can you do a PR to implement this, either using the approach I described above or using PATCH? I think this is important to get into 1.5.
I think PATCH may NOT be safe here. For example: 1) current pdb 2) creating eviction A: get 3) creating eviction B: get 4) creating eviction A: PATCH pdb 5) creating eviction B: PATCH pdb 6) pdb's . It's wrong. Retry loop make more sense to me.
Good point! You're right. Retry loop should be OK.
I can help to create a PR using retry loop.

Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
Done.
who is actually working on this issue? Could one of you set yourselves as an assignee for this issue? Thanks! cc:
retry logic with PR needs reviews/lgtm


Question: storageframework.CapOnlineExpansion:     true, storageframework.CapNodeExpansion:       true, storageframework.CapSnapshotDataSource:  true, <ins> storageframework.CapReadWriteOncePod:    true, </ins> }, RequiredAccessModes: []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce}, TopologyKeys:        []string{GCEPDCSIZoneTopologyKey},

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: } return opts } <ins> type typeCheckingCompiler struct { compositionEnv *plugincel.CompositionEnv typeOverwrite  typeOverwrite } // CompileCELExpression compiles the given expression. // The implementation is the same as that of staging/src/k8s.io/apiserver/pkg/admission/plugin/cel/compile.go // except that: // - object, oldObject, and params are typed instead of Dyn // - compiler does not enforce the output type // - the compiler does not initialize the program func (c *typeCheckingCompiler) CompileCELExpression(expressionAccessor plugincel.ExpressionAccessor, options plugincel.OptionalVariableDeclarations, mode environment.Type) plugincel.CompilationResult { resultError := func(errorString string, errType apiservercel.ErrorType) plugincel.CompilationResult { return plugincel.CompilationResult{ Error: &apiservercel.Error{ Type:   errType, Detail: errorString, }, ExpressionAccessor: expressionAccessor, } } env, err := c.compositionEnv.Env(mode) if err != nil { return resultError(fmt.Sprintf("fail to build env: %v", err), apiservercel.ErrorTypeInternal) } ast, issues := env.Compile(expressionAccessor.GetExpression()) if issues != nil { return resultError(issues.String(), apiservercel.ErrorTypeInvalid) } // type checker does not require the program, however the type must still be set. return plugincel.CompilationResult{ OutputType: ast.OutputType(), } } var _ plugincel.Compiler = (*typeCheckingCompiler)(nil) </ins>

When I load a policy definition like this The object is updated with the following status While this error is reported, the policy does seem to be fully functioning This is similar to the documented examples No error reported on a valid spec Create a a policy that uses variables Observe the status of the newly created policy No response <details</details<detailsn/a </details<details</details<detailskind </details<details</details<details</details
/sig api-machinery
/assign Oops. Good find. Thank you. /triage accepted


Question: fakeDocker.ContainerList = []docker.APIContainers{ { // format is // k8s_<container-id>_<pod-fullname>_<pod-uid> <del> Names: []string{"/k8s_bar_foo_new_12345678_0"}, </del> <ins> Names: []string{"/k8s_bar1_foo1_new_12345678_0"}, </ins> ID:    "1234", }, <ins> { // format is // k8s_<container-id>_<pod-fullname>_<pod-uid> Names: []string{"/k8s_bar2_foo2_new_87654321_0"}, ID:    "5678", }, { // format is // k8s_<container-id>_<pod-fullname>_<pod-uid> Names: []string{"/k8s_POD_foo2_new_87654321_0"}, ID:    "8765", }, </ins> } kubelet.pods = []api.BoundPod{ { ObjectMeta: api.ObjectMeta{ UID:       "12345678", <del> Name:      "foo", </del> <ins> Name:      "foo1", </ins> Namespace: "new", }, Spec: api.PodSpec{ Containers: []api.Container{ <del> {Name: "bar"}, </del> <ins> {Name: "bar1"}, }, }, }, { ObjectMeta: api.ObjectMeta{ UID:       "87654321", Name:      "foo2", Namespace: "new", }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "bar2"}, </ins> }, }, }, } <del> waitGroup.Add(1) </del> <ins> waitGroup.Add(2) </ins> err := kubelet.SyncPods(kubelet.pods, emptyPodUIDs, time.Now()) if err != nil { t.Errorf("unexpected error: %v", err)

Observed in testing somehting unrelated - prio We need a test that captures this - we can't let this slip through, Maybe same as ?
I just reproduced this with newly built cluster from HEAD. This is definitely a regression introduced lately. Will look into it in detail.


Question: - key: config path: ip-masq-agent tolerations: <ins> - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists </ins> - key: "CriticalAddonsOnly" operator: "Exists"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: ip-masq-agent should be scheduled on all the nodes and has the same toleration as the deamon set . See the difference: What you expected to happen: ip-masq-agent tolerate NoSchedule and NoExecute so that it will be scheduled in all nodes. How to reproduce it (as minimally and precisely as possible): Create a node with taint operation=NoSchedule. ip-masq-agent will not be scheduled in that node. Anything else we need to know?: I don't believe this is intended behavior. But if it is, let me know and I will close this issue. Environment: Kubernetes version (use ): HEAD /assign /assign /assign
/sig cluster-lifecycle


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package schema import ( "testing" ) // TestStructuralUnfoldNilValueValidation tests that Unfold() does not crash // on a IntOrString pattern with nil ValueValidation. func TestStructuralUnfoldIntOrString(t *testing.T) { schema := Structural{ Extensions: Extensions{ XIntOrString: true, }, } schema.Unfold() } </ins>

What happened: API server panics handling requests for a CRD with OpenAPI validation with What you expected to happen: CRD is applied successfully. How to reproduce it (as minimally and precisely as possible): kind create cluster --image kindest/node:v1.16.1 then kubectl apply: Partial logs from API server: Anything else we need to know?: Environment: Kind Kubernetes version (use ):OS (e.g: ):
/sig api-machinery
/cc
where It seems that s.AnyOf was referencing some invalid address.
yes, which is very strange, given s was dereferenced earlier in the same method: is investigating
reproduced and has a fix
I had the same issue with prometheus (kube 1.16.1). Deploying is fine but when trying to delete, it crashes the api server. For those who can't upgrade to 1.16.2+ (fixed version) there is a workaround : using etcd to delete the resources that crash the api server : here I grep on coreos because I know it is prometheus that crashes the api server Remove every item in the list that causes the crash :
Note this is also resolved in 1.15.5+ and 1.16.2+, upgrading to 1.17.x is not required to resolve this


Question: var retryErr error err := wait.ExponentialBackoff(az.requestBackoff(), func() (bool, error) { machine, retryErr = az.getVirtualMachine(name) <ins> if retryErr == cloudprovider.InstanceNotFound { return true, cloudprovider.InstanceNotFound } </ins> if retryErr != nil { glog.Errorf("backoff: failure, will retry,err=%v", retryErr) return false, nil

Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug /sig azure What happened: When performing a rolling update of my nodes (i.e., cordon, drain, delete VM on Azure side) one after the other the corresponding objects of the deleted VMs are not being deleted from etcd. This affects all patch versions while it is working fine with . What you expected to happen: The objects are removed from etcd when the backing virtual machine does not exist any more. How to reproduce it (as minimally and precisely as possible): Create a Kubernetes cluster with version on Azure, cordon and drain the node and delete the backing VM afterwards. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.9.X Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): CoreOS Stable 1632.3.0 Others: machine type
It requires some time for controller manager to detect the removed VM on Azure. Could you share the logs of kube-controller-manager?
Hi thank's for taking a a look. Well, sure, it may take some time, however, it's really fast for Kubernetes clusters on Azure with versions prior than and also for clusters on other cloud providers (even with ). I have waited more than 30 minutes, but the "old" objects are still there (which leads to a weird state/view on pods, etc.). I've seen this problem only with Azure . I just reproduced the problem: At this point in time the backing VMs for and are already gone/do not exist any more. The relevant logs of the kube-controller-manager:
Thanks of providing the information. I'm checking whether there is something wrong inside cloud provider
confirmed it's a bug in azure cloud provider
Thank you for investigating. Do you already know whether the fix will be backported or only part of 1.10?
By the way, have you configured backoff with azure cloud provider? what params have you configured?
This is the relevant part of the current cloud provider configuration: Does it look good to you?
There is a problem in backoff logic which results in retrying even the VM has already removed from Azure. With the params you set, the backoff timeout will be about 40 hours. I recommend setting a smaller cloudProviderBackoffDuration (ensure 1) since this is an exponential backoff.
Okay, thank's a lot!


Question: // NOTE: It's not recommended to be used concurrently from multiple threads - // if first user takes the whole timeout, the second one will get 0 timeout // even though the first one may return something later. <del> type timeBudget struct { </del> <ins> type timeBudget interface { takeAvailable() time.Duration returnUnused(unused time.Duration) } type timeBudgetImpl struct { </ins> sync.Mutex budget time.Duration

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: } // newAuthenticatorFromTokenFile returns an authenticator.Request or an error <del> func newAuthenticatorFromKeystoneURL(keystoneURL string) (authenticator.Request, error) { keystoneAuthenticator, err := keystone.NewKeystoneAuthenticator(keystoneURL) </del> <ins> func newAuthenticatorFromKeystoneURL(keystoneURL string, keystoneCAFile string) (authenticator.Request, error) { keystoneAuthenticator, err := keystone.NewKeystoneAuthenticator(keystoneURL, keystoneCAFile) </ins> if err != nil { return nil, err }

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: } else if config.SubnetworkName != "" { subnetURL = gceSubnetworkURL(config.ApiEndpoint, netProjID, config.Region, config.SubnetworkName) } else { <del> // Attempt to determine the subnetwork in case it's an automatic network. // Legacy networks will not have a subnetwork, so subnetworkURL should remain empty. </del> <ins> // Determine the type of network and attempt to discover the correct subnet for AUTO mode. // Gracefully fail because kubelet calls CreateGCECloud without any config, and minions // lack the proper credentials for API calls. </ins> if networkName := lastComponent(networkURL); networkName != "" { <del> if n, err := getNetwork(service, netProjID, networkName); err != nil { // Gracefully fail because kubelet calls CreateGCECloud without any config, and API calls will fail coming from minions. glog.Warningf("Could not retrieve network %q in attempt to determine if legacy network or see list of subnets, err %v", networkURL, err) </del> <ins> var n *compute.Network if n, err = getNetwork(service, netProjID, networkName); err != nil { glog.Warningf("Could not retrieve network %q; err: %v", networkName, err) </ins> } else { <del> // Legacy networks have a non-empty IPv4Range if len(n.IPv4Range) > 0 { glog.Infof("Determined network %q is type legacy", networkURL) </del> <ins> switch typeOfNetwork(n) { case netTypeLegacy: glog.Infof("Network %q is type legacy - no subnetwork", networkName) </ins> isLegacyNetwork = true <del> } else { // Try to find the subnet in the list of subnets subnetURL = findSubnetForRegion(n.Subnetworks, config.Region) if len(subnetURL) > 0 { glog.Infof("Using subnet %q within network %q & region %q because none was specified.", subnetURL, n.Name, config.Region) </del> <ins> case netTypeCustom: glog.Warningf("Network %q is type custom - cannot auto select a subnetwork", networkName) case netTypeAuto: subnetURL, err = determineSubnetURL(service, netProjID, networkName, config.Region) if err != nil { glog.Warningf("Could not determine subnetwork for network %q and region %v; err: %v", networkName, config.Region, err) </ins> } else { <del> glog.Warningf("Could not find any subnet in region %q within list %v.", config.Region, n.Subnetworks) </del> <ins> glog.Infof("Auto selecting subnetwork %q", subnetURL) </ins> } } }

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: // If nodePort is specified, user should be able to use nodeIP:nodePort to reach the backend endpoints if svcInfo.nodePort > 0 { <ins> // If the preserve-destination service annotation is present, we will disable routing mesh for NodePort. // This means that health services can use Node Port without falsely getting results from a different node. nodePortEndpoints := hnsEndpoints if svcInfo.preserveDIP { nodePortEndpoints = hnsLocalEndpoints } </ins> hnsLoadBalancer, err := hns.getLoadBalancer( <del> hnsEndpoints, </del> <ins> nodePortEndpoints, </ins> loadBalancerFlags{localRoutedVIP: true}, sourceVip, "",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When preserve-destination annotation is set to true the routing mesh is getting disabled for actual traffic flow but the route mesh isn’t getting disabled for LB liveness probes. As a result the VM's which doesn't have active pods are looking healthy from the point of view of LB and getting active customer traffic. The VM's are dropping the traffic since the routing mesh is disabled. What you expected to happen: When preserve-destination annotation is set routing mesh should be disabled for LB liveness probes as well as end user traffic. How to reproduce it (as minimally and precisely as possible): Enable preserve-destination=true annotation in Azure k8s service and deploy an app that is registered with the service. Anything else we need to know?: Environment: Kubernetes version (use ): v1.15.1 Cloud provider or hardware configuration: Azure (ACS) OS (e.g: ): Windows 1903 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Fixed by
Hey Yu-Ju this missed the 1.17 deadline and it’s a bug fix for the Windows preserve-destination feature. Can we back port this?
/sig node windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, { Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       "/dont-record-oom-event", VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "kubepods/some-container", VictimContainerName: recordEventContainerName + "kubepods", </ins> }, } numExpectedOomEvents := len(oomInstancesToStream) - numOomEventsWithIncorrectContainerName

What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon


Question: } func (r *SimpleWatchReactor) Handles(action Action) bool { <del> resourceCovers := r.Resource == "*" || r.Resource == action.GetResource().Resource if !resourceCovers { return false } return true </del> <ins> return resourceCovers(r.Resource, action) </ins> } func (r *SimpleWatchReactor) React(action Action) (bool, watch.Interface, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Can't match "serviceaccounts/token", can only match "serviceaccounts" Can match "serviceaccounts/token"
/kind feature
/sig testing
/assign
/triage accepted


Question: case ${DEB_ARCH} in s390x|arm64|ppc64el) <del> sed -i '/debian-security/d' /etc/apt/sources.list ;; </del> <ins> # we have to use the archive mirrors. # See: https://lists.debian.org/debian-devel-announce/2019/03/msg00006.html echo "deb http://archive.debian.org/debian/ jessie main contrib non-free" | tee /etc/apt/sources.list echo "deb-src http://archive.debian.org/debian/ jessie main contrib non-free" | tee -a /etc/apt/sources.list ;; </ins> esac exit 0

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The image ` cannot be built anymore, since its mirrors has been archived. For more details, see: What you expected to happen: The image should be buildable. How to reproduce it (as minimally and precisely as possible): cd kubernetes/tests/images make all-container WHAT=jessie-dnsutils Anything else we need to know?: /sig testing /area conformance /priority important-soon Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/assign
ref:


Question: } cmd.Flags().BoolP("client", "c", false, "Client version only (no server required).") cmd.Flags().BoolP("short", "", false, "Print just the version number.") <ins> cmd.Flags().String("output", "", "output format, options available are yaml and json") </ins> cmd.Flags().MarkShorthandDeprecated("client", "please use --client instead.") return cmd } func RunVersion(f cmdutil.Factory, out io.Writer, cmd *cobra.Command) error { <del> v := fmt.Sprintf("%#v", version.Get()) if cmdutil.GetFlagBool(cmd, "short") { v = version.Get().GitVersion </del> <ins> var serverVersion *apimachineryversion.Info = nil var serverErr error = nil vo := Version{nil, nil} clientVersion := version.Get() vo.ClientVersion = &clientVersion if !cmdutil.GetFlagBool(cmd, "client") { serverVersion, serverErr = retrieveServerVersion(f) vo.ServerVersion = serverVersion </ins> } <del> fmt.Fprintf(out, "Client Version: %sn", v) if cmdutil.GetFlagBool(cmd, "client") { return nil </del> <ins> switch of := cmdutil.GetFlagString(cmd, "output"); of { case "": if cmdutil.GetFlagBool(cmd, "short") { fmt.Fprintf(out, "Client Version: %sn", clientVersion.GitVersion) if serverVersion != nil { fmt.Fprintf(out, "Server Version: %sn", serverVersion.GitVersion) } } else { fmt.Fprintf(out, "Client Version: %sn", fmt.Sprintf("%#v", clientVersion)) if serverVersion != nil { fmt.Fprintf(out, "Server Version: %sn", fmt.Sprintf("%#v", *serverVersion)) } } case "yaml": y, err := yaml.Marshal(&vo) if err != nil { return err } fmt.Fprintln(out, string(y)) case "json": y, err := json.Marshal(&vo) if err != nil { return err } fmt.Fprintln(out, string(y)) default: return errors.New("invalid output format: " + of) </ins> } <del> discoveryclient, err := f.DiscoveryClient() if err != nil { return err </del> <ins> if serverErr != nil { return serverErr </ins> } <del> serverVersion, err := discoveryclient.ServerVersion() </del> <ins> return nil } func retrieveServerVersion(f cmdutil.Factory) (*apimachineryversion.Info, error) { discoveryClient, err := f.DiscoveryClient() </ins> if err != nil { <del> return err </del> <ins> return nil, err </ins> } <del> v = fmt.Sprintf("%#v", *serverVersion) if cmdutil.GetFlagBool(cmd, "short") { v = serverVersion.GitVersion </del> <ins> serverVersion, err := discoveryClient.ServerVersion() if err != nil { return nil, err </ins> } <del> fmt.Fprintf(out, "Server Version: %sn", v) return nil </del> <ins> return serverVersion, nil </ins> }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "kubectl version", "kubectl version" json Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request Kubernetes version (use ): $ kubectl version --client Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-12T05:22:15Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS 10.12.2 I find it difficult to parse the output of programmatically. I have to use a YAML parser to do it. I'd prefer to use JSON, or to be given just the string and asked to parse it with a regex. This is obviously a relatively low-priority request.
I wouldnt mind helping out on this, if there are no objections?
are you working on this? If not, I would like to work on it.
I plan to, but if i'll keep you posted if i get too delayed.
How can I direct message you?
didnt see this till now, felt bad i had not kept up on this so i decided to just push through it. If my email is not available in my profile, please let me know, otherwise AT gmail would be the best way to contact me.
Thanks!
Included below are some examples. Let me know what you think, my only concern is the the original output of included the key and then the object type which is a little strange. Note the output flag is optional so we don't break previous behavior and contract, which is now baked and part of e2e testing. here are the couple of samples:


Question: return o.writeConfigFile() } <del> proxyServer, err := NewProxyServer(o.config, o.CleanupAndExit, o.CleanupIPVS, o.scheme, o.master) </del> <ins> proxyServer, err := NewProxyServer(o) </ins> if err != nil { return err }

https://k8s- https://k8s- cc: /kind bug /priority failing-test
This is at least twice in a week we've had cross break due to a public method signature change not being brought to all platforms () We should do:
I'm lost - the PR in question adds an argument and fixes the call-site - what is broken?
Ahh, I see.
How did that PR get through, if it is so obviously broken on Cross?
Cross build doesn't run 100% of the time as a pre-submit (because it takes so long), and it can't block. It's something sig-testing is tracking, but there isn't a quick fix yet.
Looks like was skipped for that PR: "pull-kubernetes-cross Skipped" not sure why yet.
Cross runs when a file matching: changes, (which needs to be fixed, opening a test-infra issue to expand the regex), but also some files could have build tags without the file name matching () Additionally cross is not blocking (), the underlying bug should be fixed now though so once we get the fix in to k/k we can make it blocking when it does run.
I find has already sent a PR. Thanks!


Question: return nil } <ins> func extractTransportname(ifaceOutput string) (iscsiTransport string) { re := regexp.MustCompile(`iface.transport_name = (.*)n`) rex_output := re.FindStringSubmatch(ifaceOutput) if rex_output != nil { iscsiTransport = rex_output[1] } else { return "" } // While iface.transport_name is a required parameter, handle it being unspecified anyways if iscsiTransport == "<empty>" { iscsiTransport = "tcp" } return iscsiTransport } </ins> func extractDeviceAndPrefix(mntPath string) (string, string, error) { ind := strings.LastIndex(mntPath, "/") if ind < 0 {

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: } } <ins> // Be conservative in what you do, be liberal in what you accept from others. // If it's non-zero, we mark only out of window RST segments as INVALID. // Ref: https://docs.kernel.org/networking/nf_conntrack-sysctl.html conntrackTCPLiberal := false if val, err := sysctl.GetSysctl(sysctlNFConntrackTCPBeLiberal); err == nil && val != 0 { conntrackTCPLiberal = true klog.InfoS("nf_conntrack_tcp_be_liberal set, not installing DROP rules for INVALID packets") } </ins> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. However, there is no corresponding session link on the host, and the host sends a reset packet, causing the session link to be interrupted connection not reset This problem can be solved by command iptables -t filter -I INPUT -p tcp -m conntrack --ctstate INVALID -j DROP Similar But this issue "drop" is placed on the forward chain, our scenario needs to be placed on the input chain. $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} <details</details<details</details<details</details<details</details<details</details
/sig network is this about pods with hostNetwork: true ? if not, why is this a kube-proxy issue if the problem is in INPUT?
/assign
Yeah, to be clear - filter INPUT applies to packets which are going to be delivered to a process. Is that kube-proxy's job to define?
The "How can we reproduce" sounds a bit like this always happens. Here is a better instruction
IMO is the best description of the problem:
This may be a better solution than let add rules in the INPUT chain
hehe, ironically we should decide to or , one is a sysctl call the other is one (or two) iptables rules. The iptables rule has also the side effect of breaking people (cc ) If we switch to set the option, the change does not seem to be backportable. I tend to agree with Lars, remove the iptables INVALID drop rule and set the sysctl, we are already setting more host sysctls what do you think?

it will be nice to have some iptables hackers opinion too
I tried to recreate this problem on virtual en (kvm/qemu) by requsting a 100MB file with curl and do; to provoke some packet loss. The transfer took longer time, but never failed.
IMO kube-proxy should not set any sysctls that are not literally required for functionality that the user has explicitly opted into (eg for most network plugins). Kube-proxy does not own the host network namespace, and it should not be doing things that will affect other people's host-network traffic, because if we do it's going to break some users. (See also: .) We can document that we think it's a good idea for users/distros to set , and if we think it's almost always a better idea than not, we can warn at startup if it's not set, but (IMO) we shouldn't set it ourselves. If we can get the same effect with an iptables rule that only affects kube-proxy's own traffic, then I think that's better than setting a sysctl that will also affect non-kube-proxy traffic. If I understand the situation here correctly, if kube-proxy a drop rule for the invalid conntrack packets, but then the administrator set , then the result would be that conntrack would not mark some packets as invalid, and so our drop rule would just not get hit, and so our drop rule wouldn't interfere with the "better" sysctl-based solution?
Agree in general. We have done it in the past (e.g. route-localnet) and mostly it seems like a not-great idea. Also agree.
If we see this as a known bug, please consider documenting it in - even if latest Kubernetes includes a fix (we can nevertheless point that out!)
I'll fix this in the next few days.
From further googling, it seems like there have been situations in the past where people were setting that sysctl to work around bugs in conntrack, which were later fixed. I think this may be another such situation; it seems like our problem is that there is some state involving dropped or retransmitted packets which linux would cope with fine in the non-conntrack case, but which it doesn't handle in the conntrack case, unless "be_liberal" is set, and that seems like a bug.

Every time I look away from this set of issues/PRs for longer than 15 minutes I forget the exact scenario that causes the bug, but if someone could summarize it very clearly we could try dragging some kernel developers in. (The regression test in the e2e suite creates a packet with an intentionally-bad sequence number, but it's not clear to me what sort of real-world issue that's supposed to be representing.)
Thanks now that was an excellent article! Well written (not TL;DR that is), and IMO perfect technical level. If there is some collection of recommended reading for persons who want to know about K8s networking, this article should be in it.
Hi thank you for the , it's really helpful! I have some questions: , should we update this sentence accordingly ==we use the above workaround, would there any side effect for the cluster network or cluster nodes?
That says "this problem will occur if conntrack is unable to recognize a packet", but it doesn't explain the real-world scenario that causes conntrack to be unable to recognize a packet. If we go to the kernel developers and say "manually injecting random incorrect packets into a conntrack'ed connection causes it to get confused", then they'll likely respond either "don't do that then" or "use ". But if we can tell them something like "if a connection has a duplicated packet after a dropped packet, then conntrack gets confused", then that sounds a lot more like a bug in conntrack that they ought to fix, such that then everything will just work for everyone in the future without needing any DROP rules or sysctls.
So do we need to revert the drop rules? As mentioned in , it now seems that this makes sense.
There are three potential "fixes" here: this is unambiguously a conntrack bug, then we should figure out the details, and get the kernel devs to fix it. However, getting kernel bug fixes into all k8s clusters in the world takes "a long time", so even if it is a kernel bug, we should still think about the other fixes. cluster admins can use . We should advertise this better, but we feel that it would be dubious to have kube-proxy set this flag itself ().

per we could probably tweak the existing rule so that it didn't interfere with non-k8s packets ~and as suggested by the OP of this issue, we could add a similar rule to the INPUT chain (though this would need a bit of further thinking about to make sure we weren't introducing any new conflicts with non-k8s traffic)~.
We could revert the DROP rules and tell people to use the sysctl instead (and actually, probably that's a better answer for this issue than adding a new DROP rule to the INPUT chain is). But we'd need to warn users in advance if we were going to do that, since it would break existing clusters.
Totally agree, This is a better fix for this issue. If this is a kernel bug and it gets fixed someday, we can easily tell the user that you don't need to set anymore. Also, If we revert the DROP rule, which can solve the routing asymmetry issue mentioned in , which is a big fix for us Yes, We need to document it and tell the user. and In order not to break the existing cluster, Do we need a flag to decide if we need the DROP rule? I'll make it default to , and make it consistent with the current logic.
but the change to only apply the DROP rule to pod traffic should also fix that We should not require administrators to choose between "high-traffic TCP services work" and "asymmetric routing works". The current DROP rule is broken. We should either fix it to work with asymmetric routing, or else remove it and tell people who needed it to use the sysctl instead. In the latter case, we might want to make the transition via a feature gate. Then people who need asymmetric routing to work can enable the feature gate in Alpha (to make kube-proxy stop creating the DROP rule), and people who need high-traffic services to work but who didn't get the memo about using the sysctl can disable the feature gate in Beta until they have a chance to update their node configs to set the right sysctl.
Refer to , The DROP rule to pod doesn't fix our case. In our case, the pods of the external cluster subnet failed to access the pods of the cluster subnet to connect due to asymmetric routing, So the packet still matches The DROP rule to pod. Do we need a KEP for this or only a feature gate?

In general, feature gates imply KEPs (and any change that will take multiple releases to complete, where we want to make sure we don't forget about it halfway through, should get a KEP, because we have processes for tracking those). It doesn't have to be a big KEP, and some of the sections of the KEP template will be irrelevant and can be omitted, because you're not introducing a new feature. (See for example.)
We are going to break users, unfortunatly features gates does not have the effect that we are assuming, and we know that all the users have to set the "ipconntracktcpbeliberal" because eventually some connections will be broken. I do no like to set sysctl, but 'm not in favor of replacing the rule and leaving the problem to users that may not even know what a sysctl is ... agree the DROP rule is broken and breaks some esoteric but legit scenarios, but we are already setting sysctls, so I think that if we don't find any alternative to the DROP rule we should try to set the sysctl for the user by default
If we skip a KEP, make sure that the release team know to track the details (comms, docs, etc) as if it were a KEP. This is an important thing to track.
Should we at least triage this as Accepted?
/triage accepted Some alternatives DROP rule and ask users to set sysctl (I'm not in favor per my previous comment and the least surprise principle) DROP rule and we set sysctl set DROP rule if is not set the DROP rule with a flag/config so users can opt-out if don't want to set it (they can decide to use the sysctl or live with that risk) ...
I can accept KEP, but I think this may take multiple releases to complete. how about this? We remove the DROP rule Control the sysctl with a flag or a feature-gate. If it's a flag, I believe it's relatively simple and we can finish it in 1.29. In the case of feature-gate, this may take multiple releases to complete.
If sets the to true by default, we can safely remove the DROP rule
Yes, but if we were doing it that way, that would be because we considered that to be a feature rather than a bug.

We want to avoid breaking working clusters on upgrade. Maybe just having a release note saying "you may need to pass to kube-proxy" is enough. I think we agreed we don't want to change any sysctls except by explicit admin request (because that affects the behavior of non-kubernetes networking as well).
Agree, but we need to think in the users, this is a problem where the fix that solve a sublte and complex bug that may affect 100% of kubernetes users and most probably they don't know they are affected, impact a esoteric network configuration on the cluster networking, and the solution we are offering is "please read the release notes", that we know only 20% of users do ... I think that advanced users can opt-out to the DROP rule if we don't want to mess with the sysctl,
If setting was a no-brainer good idea, then the kernel would just have that behavior by default. There must be some reason why they don't, and so I worry that if we set that flag by default, then we're just going to start breaking some other subset of users. Also, if it turns out that this is all just a conntrack bug, which eventually gets fixed, then we wouldn't want to be setting the sysctl after it got fixed. But at that point it would be likely that there were some users unknowingly relying on the sysctl for some other side effect.
We could potentially have an iptables rule to count invalid conntrack packets, and expose the count as a metric
I completely and absolutely agree. we should not make that sysctl a default in kube-proxy ... what if we read the sysctl and install the DROP rule only if it is not set? Users that apply the sysctl we no longer have the DROP rule
/assign let me send a PR , I think that approach covers all cases
FWIW, my kernel conntrack informant tells me that there really aren't any bad side effects of setting it ("I don't see anything that could break by enabling this mode"), and in fact OVN sets it unconditionally on hosts it's running on. He also tells me that the bug (at least in ) is fixed in kernel ~6.2~ 6.1 ("").

So I'm not sure if that means "we should just go ahead and set the sysctl because it's harmless" or "we should just keep it like it is now and not set the sysctl since it won't be needed in the future"... (NB: if the latter we still need a docs update to go along with this PR, telling admins about the sysctl.)
If we can determine that the kernel fixes the issue in a certain version, then I think we can do this: Do not install the DROP rule.If the kernel version of the host is greater than the safe kernel version, we do not set sysctl.If less than, we have a flag to set sysctl. I'll make it default to 1. (refer to ), And let the user know that we may remove this sysctl in the future.
Exactly my thought. I made this mistake with route_localnet, and we still pay for it. Let me do some internal outreach too, see if I can get a concurring opinion?
Kernel ppl here concur: "I say : if kernel can not be fixed, set tcpbeliberal to one " "having Kubernetes set net.netfilter.nfconntracktcpbeliberal=1 always SGTM, since you want it to work on any kernel version" argues for a flag, default to true (or a hegative flag, defsult false) which seems a bit paranoid, but probably smart :) Are we all in agreement? Who wants to do the PR?
already has a PR to add to the existing set of configurable conntrack sysctls in kube-proxy () so it's just a matter of flipping the default
If the sysctl defaults to true, I think this still keeps backward compatibility, So we don't need the DROP rule anymore.
I just realized that we need this to be a flag because there are cases that run kube-proxy on environments with readonly /sys subsystem, so trying to set it always will fail will and make kube-proxy unusable for those environments
if people does not set it, they will hit the bug until they use kernel 6.1
I mean this sysctl defaults to true in kube-proxy, I can't imagine why users would need to set it to false unless the kernel version is safe (6.1). Of course, we can also document that the user that it is better not to do this.

There are cases where we CAN'T set sysctls but should not hard-fail kube-proxy. On Wed, Sep 6, 2023, 11:25 PM Cyclinder
after setting the jobs for nftables and hitting the conntrack bug continuously I think we should default the flag to true and set the sysctl nfconntracktcpbeliberal to 1 , We are already setting some sysctl and for rootless or other environment they can always opt out, this is what we do in kind
Sounds OK to me. Risk should be ~zero, right?
Yes, I assumed the risk is when to 1, we do not set it by default, so default to 0(see ). I'd be happy with it, Can I have this one?
Talked with offline, I don't have very strong arguments to make this enable by default , so let's wait for more user feedback, nftables will move to beta this release, we can revisit if we have more feedback from users
we are considering turning on by default in our kube-proxy 1.29+ deployments. Based on the recent discussion here that seems fine and the risk should "be ~zero"(), correct? And you are all even discussing turning this on by default in kube-proxy in some future version? One thing I'm a little confused on: is doing the same thing as turning on ? So by setting you're just turning on behavior that will be on by default in 6.1. Did I get that right?
Not exactly. affects a handful of situations. 6.1 only "normalized" one of them. The general intent of is something like "make conntrack work better with packets from broken TCP implementations". But the netfilter devs realized that one of the places it was being used was actually something that could happen due to totally normal TCP packet drops/retransmissions. And so in 6.1+, that particular situation is just handled correctly automatically, whether is enabled or not. My argument against this is that once we do it, we have to keep doing it forever, because people might start unwittingly depending on the other side effects of that sysctl. And particularly given that people never hit the bug anyway.

(It seems like we got more reports of the "my cluster is broken because of the DROP rule" bug than we ever got of the "my cluster is broken because of conntrack invalid packet handling" bug.) day once everyone has moved to 6.1 the bug will be gone forever ISTM that it's better to still not set it by default.
Thanks for the clarification on the 6.1 bugfix That's true and they probably will. I'm just trying to leverage some of the earlier discussion here to make a decision about the safety of flipping this to true, as it seems like the concern was not too high / basically nil.


Question: Help:      fmt.Sprintf("A metric measuring the saturation of the rate limiter for %v", ownerName), }) rateLimiterMetrics[ownerName] = metric <del> prometheus.MustRegister(metric) </del> <ins> if err := prometheus.Register(metric); err != nil { return fmt.Errorf("error registering rate limiter usage metric: %v", err) } </ins> return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Shows that prometheus.MustRegister will panic on things like dashes in the metric name. There are currently 138 calls to prometheus.MustRegister in the Kubernetes code base almost exclusively from controllers. We should audit the code to ensure that panic is the desirable response to a bad metric name for these calls vs. something like prometheus.Register which will instead return an error. What you expected to happen: See above How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: No Environment: All Kubernetes version (use ): 1.9 Cloud provider or hardware configuration**: All OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I did an analysis on this issue, and this is what I found: of the calls use a static metric name (hardcoded at the method invocation via a string and/or constant); is only a single case where the metric name is constructed dynamically (from a parameter), at the file (and happens to be where panicked); My proposal is: a. For static metric names (1), leave them as they are; b. For dynamic metric names (2), where the caller might not know that a parameter will be used to construct a metric name: Update the file to use instead of , and return the error if the metric cannot be registered (already submitted PR ); A quick search from the method invocation (from the above file) results in 17 invocations, mostly controllers, but although returns an error, no one of those 17 results is catching the error. So, we need to update all invocations to catch the error (I plan to submit new PRs to address this).

/sig instrumentation
We can trace all the task for update metrics.RegisterMetricAndTrackRateLimiterUsage invocations, and I am very happy to help fix it. [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [ ] [ ] [x] [x] [x]
Please go ahead with the ones not yet updated if you want.
OK, thanks!
now, we catch the error and when there return an error, we return in the function . But in the test file we use function for many times and Then there will be a lot of error in the test file, such as: then the test will always go fail. I think we should distinguish the error and when the error is Duplicate registration we dhould ignore it and return the normal controller.
Can you please provide an example where this is happening? I ran the tests on several controllers and haven't found that error. Ignoring this error doesn't seem a good idea IMO, if there're some tests that fail, we should investigate why are failing and fix them appropriately.
Sorry, I thought you were talking about the unit tests, didn't see the integration tests problem. I've submitted to fix this and ran the integration tests to check that creating multiple controllers don't fail anymore.
actually in some cases the unit tests is also failed such as endpoint controllerreplicaset controller. because different set up before test. So we should take a action to resolve it.


Question: e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment" e2eendpoints "k8s.io/kubernetes/test/e2e/framework/endpoints" e2eendpointslice "k8s.io/kubernetes/test/e2e/framework/endpointslice" <del> e2ekubesystem "k8s.io/kubernetes/test/e2e/framework/kubesystem" </del> e2enetwork "k8s.io/kubernetes/test/e2e/framework/network" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> "k8s.io/kubernetes/test/e2e/framework/providers/gce" </del> e2erc "k8s.io/kubernetes/test/e2e/framework/rc" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/storage/utils" testutils "k8s.io/kubernetes/test/utils" imageutils "k8s.io/kubernetes/test/utils/image" <del> gcecloud "k8s.io/legacy-cloud-providers/gce" </del> "github.com/onsi/ginkgo" "github.com/onsi/gomega"

Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: Services should be able to change the type and ports of a service [Slow] Testgrid link: https://k8s- Reason for failure: Test timeouts in ~half of the runs since it was enabled here: /sig scalability /sig network
/cc /cc
The context is - the problems are strictly related to scale - 5k-node jobs are timing out. I very briefly skimmed through the test and didn't see anything obviously suspicious (apart from the fact there are gazillions long timeouts there, which alltogether sum up long hours). That said - it seems to me that there are effectively two separate tests inside it: one for TCP one for UDP They don't seem to share anything with each other. - would it make sense to split this test two effectively two separate tests: one for TCP and one for UDP. (I guess it would also allow us to share some code there).
[And potentially it could help with debugging the problem if it is strictly related to only one of those cases...]
I wanted to focus on this past week but I didn´t have time, this is my list of more expensive times and approx average duration, all of the sig-network ones use LoadBalancers. I wanted to time the test to understand if there are places where can save time ... I don´t have much context on those tests, maybe Tim can help us more, but this comment makes me think that the use of both protocols is intended
/triage accepted
/triage accepted It's been a very long time since I touched this, and I recall holding my nose on it in the first place. Splitting TCP from UDP seems reasonable if we can run them in parallel. we'll want to share the flow, and parameterize some shared funcs. I don't know that this will prevent test timeouts, though.
We could use a volunteer to go poke at this, and see how hard it will really be.
You mean "forcing them to run in parallel"? Or "allowing them to run in parallel"? The later doesn't require any additiona work. The former I think mean that it will still have to be a single test (just the code will be organized better) and that won't really help with debugging (nor preventing timeouts).

this was discussed yesterday in sig-net meeting and my understanding was to split current test in 2 different so the test can run in parallel, if you want to do so and one protocol doesn´t have to wait for the other. I think that is what you meant here
As long as we CAN parallelize. Creating LBs can be sloooooow. :P Antonio Ojea <:
Reopening - I don't think just splitting will address the problem. It may help us with debugging though...
With the split we moved from 60 mins to run the test, to run 2 tests in [30-40] mins each https://k8s- Seems a win :smile:
Awesome - that's something I was silently counting for TBH :) [I doubt it will fix flakiness, but maybe it will make it easier to debug...]
Actually - I think we're getting close thanks to the two changes ( + this one). Here is the run: This particular one filed on UDP (not sure if it's just UDP-related, or just manifested with UDP now). And we have a clear failure now: In my PR I decreased the timeout from 2h to 45m, as it seemed to me that in the past it should be enough. But it failed on 45m timeout here. So I think the next question now is: is it that slow - that we need more than 45m or something isn't working as expected (either in the k8s itself or in the test) But I think we have a clear single place to look at now.
Actually - the full stacktrace: So it's this line that failed: What is interesting is that something was changing during those 45 minutes:
Note that it's 4th time we actually wait for LB in this test. The failure happens after the pod were scaled down to 0 and then to 1. So that makes me think, that's it's actually not LB-related, but rather something internally in k8s, becuase L4 LB doesn't depend on pods - it targets nodes...
If it is always UDP, I start to see ghosts between GCE and UDP ... we only have KIND to compare against, or do we have some other non gce enviroment?

I mean, there are other tests failing in UDP , and show much more failures in GCE than in kind
- this does not look "normal". Can we give some prio to helping track this flake?
hmm interesting. we will investigate.
/assign
- any progress on this?
I haven't had a chance to look into this yet. I will prioritize it this week.
Digging through the logs I believe this was a bug in the test code that was fixed by . In the test, the service port changes around 12:33. However due to the bug, the test continues to use the old service port as seen in the test output: The port should be 81. I believe the following sequence occurs from this point: 1) The reachability test fails in the beginning because the service has changed the port and the load balancer has not been updated yet resulting in the error. 1) Around 12:38, the service controller is syncing the service and notices that the service port has changed and starts to delete the forwarding rule causing the error to change to . 1) Once the forwarding rule is recreated, the error goes back to being because the forwarding rule exists but the wrong port is used. Looking at , the test runs after the one linked have all passed. Is there anything else that needs to be investigated as part of this issue?
OK - the test was quite flaky in the past but indeed it didn't flake for the last 3 weeks (neither this nor TCP one). I'm fine with closing this one - we can open a new one if this happens again. - thanks for debugging. - it seems that the splitting of the test actually made things waay better. Which is great.


Question: - --memory={{ base_metrics_memory }} - --extra-memory={{metrics_memory_per_node}}Mi - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=heapster - --poll-period=300000 - --estimator=exponential

I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue?

This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Question: if res == nil { continue } <ins> if len(wc.resultChan) == outgoingBufSize { glog.Warningf("Fast watcher, slow processing. Number of buffered events: %d."+ "Probably caused by slow dispatching events to watchers", outgoingBufSize) } </ins> // If user couldn't receive results fast enough, we also block incoming events from watcher. // Because storing events in local will cause more memory usage. // The worst case would be closing the fast watcher.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


Question: createdRS, err := dc.client.Extensions().ReplicaSets(namespace).Create(&newRS) switch { // We may end up hitting this due to a slow cache or a fast resync of the deployment. <del> case errors.IsAlreadyExists(err): return dc.rsLister.ReplicaSets(namespace).Get(newRS.Name) </del> <ins> // TODO: Restore once https://github.com/kubernetes/kubernetes/issues/29735 is fixed // ie. we start using a new hashing algorithm. // case errors.IsAlreadyExists(err): //	return dc.rsLister.ReplicaSets(namespace).Get(newRS.Name) </ins> case err != nil: msg := fmt.Sprintf("Failed to create new replica set %q: %v", newRS.Name, err) if deployment.Spec.ProgressDeadlineSeconds != nil {

Env: Kubernetes 1.3.3 on AWS I ran into the same problem described into but now it's happening on every deploy. I would have reopened the issue, but I was not the author so I could not. The bug presents itself because I have a build that does just after a and it gets stuck forever because the rollout is never reported as finished (it does finish tho).
could you provide more detail info? like what's your resource file and how to reproduce it.
yes, of course: We have an internal tool that downloads some manifests from S3 in a temporary folder and then calls . It's harder to narrow it down because that's all there is to it. It caused the issue above for 3/4 builds in a row but in the last build it worked. The three deployment files are fairly standard deployment templates, this is an example of one of them: The is filled by jenkins at build time before the manifest is uploaded on s3. I am not sure about how to narrow this down further, any suggestions?
can somebody triage this?
We just started seeing this as well. kube 1.3.6 on AWS. The cluster was running fine for two weeks, then the problem started on the last CI build yesterday, and shows no signs of fixing itself. Our deploy process is a series of commands called from a bash script during CI.
should fix the hotloop by using a rate limiter in the deployment controller. Seems like it was included in 1.3.7. Also the "already exists" shouldn't be treated as an error but instead we should return the replica set from the cache at this point (since it already exists). I was handling this error in a recent PR of mine, will update this issue back with the #
Are you sure the fix is in 1.3.7? I just hit this problem in a build of 1.3.7, and it kills the master node as it pegs a cpu at 100% in this error loop. edit: it doesn't seem to be on the release-1.3 branch, but is in release-1.4
Hm, git fooled me. The fix is included in 1.4.

From the code, it looks like deployment api generates an adler-32 checksum of the pod template, then uses that to create a new replicaset . If that hash already exists, it barfs with the controller-manager loop. So it looks to me, from a quick read, as simply a hash collision handled poorly. This seems to mirror what I'm seeing, since it only happens on deployments with a history 100 deploys. I'm guessing there isn't enough data change in the pod spec to ensure a decent amount of uniqueness with adler-32 - in my case just the version changes. It should probably use a better hash function (that doesn't duplicate on such a small number of deployments), and/or handle the collision more gracefully.
I don't think you would get a hash collision even with 100 deploys (not sure where Adler starts to fail though). Even the slightest change in the pod template changes the hash. Can you provide the pod template of the existing replica set that blocks the deployment controller and also the latest state of your deployment? I think this issue is simply us not handling AlreadyExists errors when we try to create new replica sets (the replica set cache may get stale).
sure... I admit I may be wrong - that was my just my super quick analysis. :) Although, adler is pretty terrible as a hash function, and I was a little surprised to see it used - which is why I leapt to that conclusion. I need to try reproducing to see if it's actually the case though - which I'll try now. Here's the replicaset that it complains already exists: And the current failing deployment (although, I believe it started failing at generation 204, version 0.612.0).
Hash collisions are definitely a problem. I wrote up a test and can get a collision within a couple hundred versions (only version changes). Here's I'm guessing we can't improve the hash function at this point, to maintain backwards compatibility? We should at least handle the collision somehow - what do you suggest?
Further, in the example I gave above, the hash output only has about 200 unique values, after which all newer versions collide with previous hashed values.
I suppose the immediate workaround for users is to limit deployment revision history to small values ().
thanks for the test! I run it as well and I see the same results.

can you have a look? An immediate workaround is to use revisionHistoryLimit, yes. In reality most users won't care about the history of 200 replica sets but it is concerning that Adler breaks that fast.
cc
Adler was only used because we were originally overly concerned about the hash performance. We need to be using a reasonably (but not excessively) fast hash where collision chance is 1 in 1e5, and preferably 1e10 or more.
fnv would be an easy change
Tangentially... having ~200 RS resources hanging around per-deployment seems like a lot and I imagine could result in triggering unknown-unknowns. IMHO the revision history limit should be bounded by default before moving out of beta. I can image things like automated hourly builds creating lots of history.
Generally bounds would be enforced by limit ranges, which already bound number of replica sets On Oct 18, 2016, at 11:31 PM, Phillip Wittrock wrote: Tangentially... having ~200 RS resources hanging around per-deployment seems like a lot and I imagine could result in triggering unknown-unknowns. IMHO the revision history limit should be bounded by default before moving out of beta. I can image things like automated hourly builds creating lots of history. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Thanks
cc
sent a PR to change the defaults (), but we'd need a new apiversion to do that.
We just ran into this issue with a much smaller history (only 10 RS resources). We're using the revisionHistoryLimit workaround but it seems like the real fix would be using a different hashing algorithm. Any thoughts on when a fix may come out?
hash collision is not the only case where you can stumble upon this issue. Are you sure this was a hash coliision? Did you check that the deployment pod template is different from the rs pod template that you get in the error?
yes, here are two templates that we are seeing hash to the same value: vs In this case there's only a small change for the version bump, and the template is rather short.
Can we even change this in a backwards compatible manner on the cluster safely?

On Nov 2, 2016, at 5:52 PM, Lucas Chi wrote: yes, here are two templates that we are seeing hash to the same value: apiVersion: extensions/v1beta1 kind: Deployment metadata: name: exworker spec: replicas: 2 template: metadata: labels: run: exworker language: elixir spec: containers: name: exworker image: teacherspayteachers/exworker:release-2.1.2 env: name: ENVIRONMENT value: "prod" name: COREAPIKEY valueFrom: secretKeyRef: name: core-api-secret-key key: secretkey imagePullSecrets: name: "tptreadonly-docker" vs apiVersion: extensions/v1beta1 kind: Deployment metadata: name: exworker spec: replicas: 2 template: metadata: labels: run: exworker language: elixir spec: containers: name: exworker image: teacherspayteachers/exworker:release-1.3.1 env: name: ENVIRONMENT value: "prod" name: COREAPIKEY valueFrom: secretKeyRef: name: core-api-secret-key key: secretkey imagePullSecrets: name: "tptreadonly-docker" In this case there's only a small change for the version bump, and the template is rather short. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
New hash algo? Unlikely.
So we don't want to blow out an existing cluster immediately on upgrade, so we need a way to stage this so an admin can slowly roll this out under their control. One possible algo: a flag to the controller manager config that decides hash behavior "legacy", "migrate-to-X", "X" the deployment controller calculates hashes, calculate old and new hash there are no existing replicas for a deployment, use the new hash the "old" hash matches and a new hash is available, put the deployment in a queue N at a time from the queue and have them use the new hash time a deployment is successfully moved to a new hash, draw a replacement from the queue all deployments are migrated, admin can change the hash behavior We could potentially get away without config, but I think admins need to opt in to this change in 1.N and be forced to do it in 1.

(N+1) , Michail Kargakis wrote:
What about old replica sets? What happens if a new hash collides with an old hash (latest template hashed == old replica set computed with old hashing algo)?
Old replica sets is covered by 4. If you have a hash collision its game over anyway, so not worried about that. , Michail Kargakis wrote:
Bumping to P1 because this makes deployments unpredictable - we need to assess the change required for 1.5 next week and whether we can come up with a backwards compatible, safe, and non-disruptive change.
This is quite a big change for 1.5 and since the problem was always there I would move a fix for this in 1.6.
Tested with the unit test provided by above and it didn't break for 10^3 up to 10^5 (10^6 was timing out the test and I didn't bother to test more).
Results from benchmarking Fnv is a bit slower but much more stable.
so my understanding is that the migration of old replica sets will happen during the normal operation of the cluster, is that correct? We can migrate all replica sets with zero replicas in the background, and migrate running sets once a new rollout happens. We may not even need a separate queue. I am still not sure if we can notify admins somehow when all existing deployments have migrated.
We could expose a metric that is "how many old replication controllers we still have"
For 1.6, we moved Deployments into the apps api group[1] so we can change the defaults (revisionHistoryLimit is now set to 3 by default for newly created deployments) and we also made the cleanup policy run independent of a rollout (so a rollout that gets stuck won't block deletiion of older replica sets)[2]. I've also opened an update to the Deployment proposal for moving away from hashing: Moving the milestone to 1.7 [1] [2]
Changing revision history doesn't seem to alleviate hash collisions, right? Looking at - are we using a poor hash for this?
Actually, it does. If all you have is 3 old replica sets, it's less likely that the controller will break you because of a hash collision. 200 seems to be the limit with adler. Yes.

There is that changes us to fnv but there is also that moves us away from hashing. I am going to open an alternative proposal to 384 that covers the transition to fnv and we need to decide with what do we want to proceed.
Opened as an alternative to kubernetes/community
We saw this in a related configuration. Marking --record=true on the deployment [1] and setting the revision history [2] configuration seemed to alleviate the stress. Is there a status on the hash collision bug? [1] [2] Thanks, Brugz
the fix will be included in 1.7 release