Question: } type serviceInfo struct { <ins> isAliveAtomic       int32 // Only access this with atomic ops </ins> portal              portal protocol            api.Protocol proxyPort           int

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: name:       "forbid create of other node", podsGetter: noExistingPods, attributes: admission.NewAttributesRecord(othernodeObj, nil, nodeKind, othernodeObj.Namespace, othernodeObj.Name, nodeResource, "", admission.Create, mynode), <del> err:        "cannot modify other nodes", </del> <ins> err:        "cannot modify node", }, { name:       "forbid create of other node pulling name from object", podsGetter: noExistingPods, attributes: admission.NewAttributesRecord(othernodeObj, nil, nodeKind, othernodeObj.Namespace, "", nodeResource, "", admission.Create, mynode), err:        "cannot modify node", </ins> }, { name:       "forbid update of other node", podsGetter: existingPods, attributes: admission.NewAttributesRecord(othernodeObj, othernodeObj, nodeKind, othernodeObj.Namespace, othernodeObj.Name, nodeResource, "", admission.Update, mynode), <del> err:        "cannot modify other nodes", </del> <ins> err:        "cannot modify node", </ins> }, { name:       "forbid delete of other node", podsGetter: existingPods, attributes: admission.NewAttributesRecord(nil, nil, nodeKind, othernodeObj.Namespace, othernodeObj.Name, nodeResource, "", admission.Delete, mynode), <del> err:        "cannot modify other nodes", </del> <ins> err:        "cannot modify node", </ins> }, { name:       "forbid update of other node status", podsGetter: existingPods, attributes: admission.NewAttributesRecord(othernodeObj, othernodeObj, nodeKind, othernodeObj.Namespace, othernodeObj.Name, nodeResource, "status", admission.Update, mynode), <del> err:        "cannot modify other nodes", </del> <ins> err:        "cannot modify node", </ins> }, // Unrelated objects

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know: None


Question: for { _, _, err := conn.ReadMessage() if err != nil { <del> t.Logf("server err reading message: %v", err) return </del> <ins> break </ins> } } }))

FAIL: TestWebSocketClientHeartbeatSucceeds also flake in ci-kubernetes-unit use of closed network connection /cc /sig api-machinery network cli
/cc /triage accepted
seems it got better since
The new flake in this test is like I also see the failure in some PR: for instance.
yeah, it is racing somewhere , at least we have a different symptom now
/help someone want a great challenge and make a very useful contribution?
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
something odd is with the test, the panic is because it logs on the goroutine, but I'm not completely sure the test is working as expected
my apologies, I found the problem and I'm submitting a patch /assign /unassign you can help me with the review if you want


Question: clusterCache.serviceQueue.Add(key) return err } <del> var needUpdate bool </del> <ins> var needUpdate, isDeletion bool </ins> if exists { service, ok := serviceInterface.(*v1.Service) if ok {

As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.


Question: "k8s.io/apimachinery/pkg/fields" "k8s.io/client-go/tools/cache" <ins> "encoding/json" </ins> appsv1 "k8s.io/api/apps/v1" v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> unstructuredv1 "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" appsclient "k8s.io/client-go/kubernetes/typed/apps/v1" watchtools "k8s.io/client-go/tools/watch"

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, <ins> { Type:               v1.NodeMemoryPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, { Type:               v1.NodeDiskPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, </ins> }, }, },

I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)


Question: func (h *hostpathCSIDriver) PrepareTest(f *framework.Framework) (*storageframework.PerTestConfig, func()) { // Create secondary namespace which will be used for creating driver driverNamespace := utils.CreateDriverNamespace(f) <del> ns2 := driverNamespace.Name ns1 := f.Namespace.Name </del> <ins> driverns := driverNamespace.Name testns := f.Namespace.Name </ins> ginkgo.By(fmt.Sprintf("deploying %s driver", h.driverInfo.Name)) cancelLogging := utils.StartPodLogs(f, driverNamespace)

Tracking bug for /sig storage /important-longterm /assign
/sig storage /triage accepted
/priority important-longterm


Question: prefix := fmt.Sprintf("port forwarding for %s", addr) ctx, cancel := context.WithCancel(ctx) l := &listener{ <del> connections: make(chan *connection), ctx:         ctx, cancel:      cancel, addr:        addr, </del> <ins> ctx:    ctx, cancel: cancel, addr:   addr, </ins> } <del> var connectionsCreated, connectionsClosed int32 </del> <ins> var connectionsCreated int </ins> runForwarding := func() { klog.V(2).Infof("%s: starting connection polling", prefix) defer klog.V(2).Infof("%s: connection polling ended", prefix) <del> // This delay determines how quickly we notice when someone has // connected inside the cluster. With socat, we cannot make this too small // because otherwise we get many rejected connections. With the mock // driver as proxy that doesn't happen as long as we don't // ask for too many concurrent connections because the mock driver // keeps the listening port open at all times and the Linux // kernel automatically accepts our connection requests. tryConnect := time.NewTicker(100 * time.Millisecond) </del> <ins> tryConnect := time.NewTicker(connectionPollInterval) </ins> defer tryConnect.Stop() for { select { case <-ctx.Done(): return case <-tryConnect.C: <del> currentClosed := atomic.LoadInt32(&connectionsClosed) openConnections := connectionsCreated - currentClosed if openConnections >= maxConcurrentConnections { break } klog.V(5).Infof("%s: trying to create a new connection #%d, %d open", prefix, connectionsCreated, openConnections) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) break } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) l.connections <- &connection{ stream:  stream, addr:    addr, counter: connectionsCreated, closed:  &connectionsClosed, } connectionsCreated++ </del> <ins> func() { l.mutex.Lock() defer l.mutex.Unlock() for i, c := range l.connections { if c == nil { klog.V(5).Infof("%s: trying to create a new connection #%d", prefix, connectionsCreated) stream, err := dial(ctx, fmt.Sprintf("%s #%d", prefix, connectionsCreated), dialer, addr.Port) if err != nil { klog.Errorf("%s: no connection: %v", prefix, err) return } // Make the connection available to Accept below. klog.V(5).Infof("%s: created a new connection #%d", prefix, connectionsCreated) c := &connection{ l:       l, stream:  stream, addr:    addr, counter: connectionsCreated, } l.connections[i] = c connectionsCreated++ return } } }() </ins> } } }

ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see


Question: config.createTestPods() epCount := len(config.EndpointPods) <ins> // Note that this is not O(n^2) in practice, because epCount SHOULD be < 10.  In cases that epCount is > 10, this would be prohibitively large. // Check maxNetProxyPodsCount for details. </ins> config.MaxTries = epCount*epCount + testTries framework.Logf("Setting MaxTries for pod polling to %v for networking test based on endpoint count %v", config.MaxTries, epCount) }

What happened: In CAPA clusters, NodePorts are disabled for UDP but not for TCP, so when testing, we see a small hole in coverage that was luckily caught by the tests.... So, I noticed that we thus don't have a targetted ginkgo test for UDP + nodeports, but we do seem to have the Contract one, which I guess tests two things at once: UDP NodePort service works Conntrack can persist the service We do have but this test is obviously testing many other things as well. So, would like to have a test.... What you expected to happen: running the sig-network test suite, while skipping with UDP not working would result in a test failure specific to UDP and NodePorts. How to reproduce it (as minimally and precisely as possible): Run the test suite, skip , with a UDP firewall up - you'll see no relevant failures. Anything else we need to know?: Environment: Kubernetes version (use ): master
/sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
There is already one testing NodePort UDP in
However, the more tests the better :+1:
Hiya Antonio. I think the one your talking about in granular checks is also tagged as "Slow"..... Is there a FAST one? If so I'll close this issue don't want to make a redundant test . :)
I honestly don't know the history of the Slow tag and I don't see it consistently applied, but if you think that you can improve the Granular check and have an additional faster test, then just go for it :muscle:
ok cool


Question: // set because it cannot be automatically converted. optional string apiVersion = 3; <del> // Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply' </del> <ins> // Time is the timestamp of when the ManagedFields entry was added. The // timestamp will also be updated if a field is added, the manager // changes any of the owned fields value or removes a field. The // timestamp does not update when a field is removed from the entry // because another manager took it over. </ins> // +optional optional Time time = 4;

Godoc says that Time should be empty on mangedFields entry created by "Apply" operation, but kubernetes populates time on apply. Godoc consistent with kubernetes behaviour Godoc: Kubernetes: No response <details</details<details</details<details</details<details</details<details</details<details</details
/wg api-expression /sig api-machinery
/cc
/cc /triage accepted
Yeah, good catch. The comment matches the behavior from alpha, which we changed later on. Thanks for letting us now :-) I think we could do a better job describing how this value is changed now.


Question: safe-bind-mount "${ephemeral_mountpoint}/log_pods" "/var/log/pods" } <ins> # set journald configuration function setup-journald() { if [[ "${SET_JOURNALD_CONFIGURATION:-true}" = "true" ]]; then cat <<EOF > /etc/systemd/journald.conf [Journal] Storage=persistent SystemMaxUse=1G SystemMaxFileSize=100M RuntimeMaxUse=100M EOF systemctl restart systemd-journald.service fi } </ins> # Installs logrotate configuration files function setup-logrotate() { mkdir -p /etc/logrotate.d/

Example using a random PR for , take a look at: in this set of logs, at least 4 files are empty kube-container-runtime- kube-node- kube- kubelet- The reason tracked down using was that the journald configuration was defaulting to something that wipes the logs quickly!
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Extract the paths handled using mux handler. handledPaths = append(handledPaths, muxHelper.RegisteredPaths...) sort.Strings(handledPaths) <del> writeRawJSON(http.StatusOK, api.RootPaths{Paths: handledPaths}, w) </del> <ins> writeRawJSON(status, api.RootPaths{Paths: handledPaths}, w) </ins> } }

Authenticated requests to master with an invalid URL path should return a non-status 200 response code. Preferably a 404 (not found). Sample output: { "paths": [ "/api", "/api/v1beta1", "/api/v1beta2", "/api/v1beta3", "/healthz", "/healthz/ping", "/logs/", "/metrics", "/static/", "/swagger-ui/", "/swaggerapi/", "/validate", "/version" ] }


Question: }, "selector": { "$ref": "#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", <del> "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace.", "x-kubernetes-patch-strategy": "replace" </del> <ins> "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace." </ins> } }, "type": "object"

PDB was upgraded to v1 in 1.21. Despite the changelog saying that the behavior of the v1beta1 implementation should have stayed the same, it appears that the for the field was set to . I believe this is the change in question: When running with a starting config that had a with both a key and a key, and a patch that had a with just a key, the behavior between 1.20 and 1.21 differs. What we get is the key gets deleted, rather than merged with the patch's key to make one unified selector. In 1.20, the behavior we'd get is thus, which is desirable for us because it lets us inject a base assumption about our PDBs into everything we do: This is my repro case: : Compare by using the command from earlier in this report: I also brought this up in Slack: I don't think we want to switch to using just for PDBs, because that wouldn't solve issues where our users want to use without having to re-specify the from our base. Putting aside the fact that changing the behavior of the v1beta1 implementation was probably not intentional, it also seems undesirable to change how selectors work in general. I don't know how widely scoped this change will be across various resource types, too, so we're starting down quite a bit of nasty discovery time. <detailsThis is the version I used to demonstrate my repro case, but I confirmed that this occurs even in kubectl v1.21.0. For my "happy" repro case, this is what the version output looks like: </details<detailsAWS (the hard way) </detailsNo response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as requested, I am tagging you and on this one
We should narrowly revert the strategic merge patch selector behavior for v1beta1 to match pre-1.21 behavior for backwards compatibility. More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB was already inconsistent w.r.t. merging portions of the selector and replacing others (it would merge and replace ) For that reason, server-side apply treats selector writes as owning the entire selector, and the v1 PodDisruptionBudget API will do the same for strategic merge patch. edit: reworded the inconsistencies with the < 1.21 selector handling
More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB has problems with data getting left in the selector incorrectly if the same user submits a patch with two keys, and then submits a patch with only a single key. Would that be considered a bug in the implementation, or was there some inherent, design-caused race condition in patch application that I don't understand? Cuz if I were to sequentially apply patches with the old strategic merge, I would think the selector would just accumulate keys, which seems fine to me–but I can see how a race condition would turn it into chaos
edit: I was wrong about the previous behavior... the issue was that it already inconsistently merged the selector... it would merge and replace , which is still pretty incoherent If you really want to tweak field-level things, jsonpatch can still do surgical insertion/appending/removal of specific fields.
Should we expect to use jsonpatch for anything that can have a selector, going forward? If it changed for PDBs then it should change for everything else, I would assume edit: we discussed more in slack. The answer is no, the fields for other resource types' label selectors are not changing, so we only need to treat PDBs as a special case.


Question: func updatePod(client clientset.Interface, pod *v1.Pod, condition *v1.PodCondition, nominatedNode string) error { klog.V(3).InfoS("Updating pod condition", "pod", klog.KObj(pod), "conditionType", condition.Type, "conditionStatus", condition.Status, "conditionReason", condition.Reason) <del> podCopy := pod.DeepCopy() </del> <ins> podStatusCopy := pod.Status.DeepCopy() </ins> // NominatedNodeName is updated only if we are trying to set it, and the value is // different from the existing one. <del> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && </del> <ins> if !podutil.UpdatePodCondition(podStatusCopy, condition) && </ins> (len(nominatedNode) == 0 || pod.Status.NominatedNodeName == nominatedNode) { return nil } if nominatedNode != "" { <del> podCopy.Status.NominatedNodeName = nominatedNode </del> <ins> podStatusCopy.NominatedNodeName = nominatedNode </ins> } <del> return util.PatchPod(client, pod, podCopy) </del> <ins> return util.PatchPodStatus(client, pod, podStatusCopy) </ins> } // assume signals to the cache that a pod is already in the cache, so that binding can be asynchronous.

I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted


Question: return false } <ins> func getServiceTags(service *v1.Service) ([]string, error) { if serviceTags, found := service.Annotations[ServiceAnnotationAllowedServiceTag]; found { tags := strings.Split(strings.TrimSpace(serviceTags), ",") for _, tag := range tags { // Storage and Sql service tags support setting regions with suffix ".Region" if strings.HasPrefix(tag, "Storage.") || strings.HasPrefix(tag, "Sql.") { continue } if !supportedServiceTags.Has(tag) { return nil, fmt.Errorf("only %q are allowed in service tags", supportedServiceTags.List()) } } return tags, nil } return nil, nil } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When using the loadBalancerSourceRanges property of a service in a manifest, we are force to use a CIDR block. This was discovered while debugging the following issue. What you expected to happen: Azure allows us to use Service Tags (e.g. VirtualNetwork, Internet, etc) WE should be able to specify them form a k8s service manifest. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.11 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): centOS 7.3 Kernel (e.g. ): 3.10 Install tools: Others:
/sig azure
Here's the requested Feature.
SourceRanges is part of service spec, while serviceTag is azure specific. I think we don't expect to add cloud-specific features into Kubernetes API. What about adding a new annotation for this? e.g.
I believe that at this stage, (no out of tree cloud provider logic), using an annotation would be the only way to go unfortunately.
/priority important-soon
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: expected: []*v1.Node{nodes[0], nodes[4], nodes[2]}, }, { <del> name: "Remove pod from non-existing node", </del> <ins> name: "Add pod before its node", operations: []operation{ addNode(0), addPod(1), updatePod(1), addNode(1), }, expected: []*v1.Node{nodes[1], nodes[0]}, }, { name: "Remove node before its pods", </ins> operations: []operation{ <del> addNode(0), addPod(0), addNode(2), updateSnapshot(), </del> <ins> addNode(0), addNode(1), addPod(1), addPod(11), removeNode(1), updatePod(1), updatePod(11), removePod(1), removePod(11), </ins> }, <del> expected: []*v1.Node{nodes[2], nodes[0]}, </del> <ins> expected: []*v1.Node{nodes[0]}, </ins> }, { name: "Add Pods with affinity",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:


Question: listeningPort := "" var l net.Listener <ins> var err error </ins> BeforeEach(func() { <del> l, err := net.Listen("tcp", ":0") </del> <ins> l, err = net.Listen("tcp", ":0") </ins> if err != nil { framework.Failf("Failed to open a new tcp port: %v", err) }

Failure cluster: This is the most prevalent flake for release-blocking postsubmit kubelet builds, eg: (notably, it happens far less often here) ref: we're probably going to need to ignore kubelet-1.8 for cutting a release
/kind flake /priority test-failure
could you take a look? /assign
Looking to the problem


Question: <del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package priorities import ( "fmt" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" schedulernodeinfo "k8s.io/kubernetes/pkg/scheduler/nodeinfo" ) // CalculateNodePreferAvoidPodsPriorityMap priorities nodes according to the node annotation // "scheduler.alpha.kubernetes.io/preferAvoidPods". func CalculateNodePreferAvoidPodsPriorityMap(pod *v1.Pod, meta interface{}, nodeInfo *schedulernodeinfo.NodeInfo) (framework.NodeScore, error) { node := nodeInfo.Node() if node == nil { return framework.NodeScore{}, fmt.Errorf("node not found") } var controllerRef *metav1.OwnerReference if priorityMeta, ok := meta.(*priorityMetadata); ok { controllerRef = priorityMeta.controllerRef } else { // We couldn't parse metadata - fallback to the podspec. controllerRef = metav1.GetControllerOf(pod) } if controllerRef != nil { // Ignore pods that are owned by other controller than ReplicationController // or ReplicaSet. if controllerRef.Kind != "ReplicationController" && controllerRef.Kind != "ReplicaSet" { controllerRef = nil } } if controllerRef == nil { return framework.NodeScore{Name: node.Name, Score: framework.MaxNodeScore}, nil } avoids, err := v1helper.GetAvoidPodsFromNodeAnnotations(node.Annotations) if err != nil { // If we cannot get annotation, assume it's schedulable there. return framework.NodeScore{Name: node.Name, Score: framework.MaxNodeScore}, nil } for i := range avoids.PreferAvoidPods { avoid := &avoids.PreferAvoidPods[i] if avoid.PodSignature.PodController.Kind == controllerRef.Kind && avoid.PodSignature.PodController.UID == controllerRef.UID { return framework.NodeScore{Name: node.Name, Score: 0}, nil } } return framework.NodeScore{Name: node.Name, Score: framework.MaxNodeScore}, nil } </del>

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: // IsNodeUnmanagedByProviderID returns true if the node is not managed by Azure cloud provider. // All managed node's providerIDs are in format 'azure:///subscriptions/<id>/resourceGroups/<rg>/providers/Microsoft.Compute/.*' func (az *Cloud) IsNodeUnmanagedByProviderID(providerID string) bool { <del> return azureNodeProviderIDRE.Match([]byte(providerID)) </del> <ins> return !azureNodeProviderIDRE.Match([]byte(providerID)) </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a Kubernetes 1.12.1 cluster in Azure with Availability Set backed masters and VMSS backed agents, the cloud-controller-manager is unable to parse the ProviderID returned by the Azure SDK. What you expected to happen: The providerID should be split and the cloud-controller-manager should be able to find a kubelet node IP from the cloud provider How to reproduce it (as minimally and precisely as possible): Stand up a master node in Azure within an availability set with the kubelet configured with Run the configured for Azure Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
When switching over to vmType the following was encountered;
The fix in resolved the issue that was seen in this issue, closing


Question: return nil, err } } return cadvisorClient, nil } <ins> func (cc *cadvisorClient) Start() error { return cc.Manager.Start() } </ins> func (cc *cadvisorClient) exportHTTP(port uint) error { mux := http.NewServeMux() err := cadvisorHttp.RegisterHandlers(mux, cc, "", "", "", "", "/metrics")

This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?

I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.


Question: // Lookup implements PluginHandler func (h *defaultPluginHandler) Lookup(filename string) (string, error) { <del> // if on Windows, append the "exe" extension // to the filename that we are looking up. if runtime.GOOS == "windows" { filename = filename + ".exe" } </del> return exec.LookPath(filename) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the kubectl plugins system - when running under Windows - will determine that a plugin is executable by looking at the plugin's extension, comparing it to '.exe'. I think we should allow plugins w/ a few more known extensions (.bat, .com, .cmd; for example). Why is this needed: This would give more of a similar experience to plugins under *nix, where shell scripts are often used.
/sig cli
Please see the PR below:


Question: k8s-app: elasticsearch-logging addonmanager.kubernetes.io/mode: Reconcile subjects: <del> - kind: ServiceAccount name: elasticsearch-logging namespace: kube-system apiGroup: "" </del> <ins> - kind: ServiceAccount name: elasticsearch-logging namespace: kube-system apiGroup: "" </ins> roleRef: kind: ClusterRole name: elasticsearch-logging

What happened: I download fluentd-elasticsearch in addons. The following error occurs then I use "kubectl apply -f es- " Then, i find the elasticsearch-logging pod restart lots of time, and keep status with CrashLoopBackOff. [root ~]# kubectl logs -f elasticsearch-logging-0 -n kube-system export NODENAME=elasticsearch-logging-0 NODENAME=elasticsearch-logging-0 export NODEMASTER=true NODEMASTER=true export NODEDATA=true NODEDATA=true export HTTPPORT=9200 HTTPPORT=9200 export TRANSPORTPORT=9300 TRANSPORTPORT=9300 export MINIMUMMASTERNODES=2 MINIMUMMASTERNODES=2 chown -R elasticsearch:elasticsearch /data ./bin/elasticsearchloggingdiscovery I0902 14:25:06. 9 ] Kubernetes Elasticsearch logging discovery I0902 14:25:06. 9 ] Found [] I0902 14:25:16. 9 ] Found [] I0902 14:25:26. 9 ] Found [] I0902 14:25:36. 9 ] Found [] I0902 14:25:46. 9 ] Found [] I0902 14:25:56. 9 ] Found []
unclear whether the addon is up to date and supported at this point. /sig instrumentation
cc
/assign I will take a look this week.
I have the same problem k8s version:v1.18.8
same
I've pushed a change to the stateful set that should unblock people as there was a 1.18 compatibility problem. however, please make sure you are also creating the service via , the discovery program finds the other elasticsearch nodes via service registration.
please feel free to re-open this issue if the latest version doesn't work for you.
Why was this PR to v1.20?
probably because that is the next release. most people who use this addon do so from master. Do you need this packaged in some other way?


Question: // purposes for the legacy object counting names in quota.  Unless its supporting // backward compatibility, alias should not be used. func NewObjectCountEvaluator( <del> allowCreateOnUpdate bool, </del> groupResource schema.GroupResource, listFuncByNamespace ListFuncByNamespace, alias api.ResourceName) quota.Evaluator {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/discovery/v1beta1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", "//staging/src/k8s.io/client-go/tools/record:go_default_library",

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: import ( "flag" "fmt" <ins> "os" </ins> "strings" "time"

Now elasticsearchloggingdiscovery will only look for the endpoints in kube-system namespace due to hard-coded api.NamespaceSystem value. Maybe let user to decide in which namespace elasticsearchloggingdiscovery binary will look for endpoints ?


Question: }, }) <del> pod := &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{}}, </del> <ins> v1PodIn := &v1.Pod{ Spec: v1.PodSpec{ Containers: []v1.Container{{}}, </ins> }, } <ins> v1defaults.SetObjectDefaults_Pod(v1PodIn) pod := &api.Pod{} if err := v1defaults.Convert_v1_Pod_To_core_Pod(v1PodIn, pod, nil); err != nil { t.Fatal(err) } </ins> attrs := admission.NewAttributesRecord(pod, nil, api.Kind("Pod").WithVersion("version"), ns, "myname", api.Resource("pods").WithVersion("version"), "", admission.Create, &metav1.CreateOptions{}, false, nil) err := admissiontesting.WithReinvocationTesting(t, admit).Admit(context.TODO(), attrs, nil) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --my mutating webhook calculates a hash of the , allows reinvocation; no side effects my validating webhook validates the hash of the and fails because the has changed webhook auditing indicates that my webhooks are the only ones processing the upon serializing the spec to JSON for both the mutator and validator, the only change is related to the "kube-api-access-xyz" projected volume, managed by a built-in k8s admission plugin the aforementioned plugin does not appear to mutate the reinvocation context, even through it may mutate a pod (robbing other mutators of the chance to observe the change): it appears that both my hooks see the injected volume, but the validation hook sees the injected SA volume with a "defaultMode" field set, and my mutation hook does NOT see this field set my validation hook should calculate the same hash mutating, validating webhooks are implemented atop controller-runtime v0.8.3, pod objects are being decoded into targets Kubernetes version (use ): 1.21.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
interesting... defaults are applied when going from versioned → internal server decodes v1 Pod request → applies defaults → converts to internal Pod -in admission control operates on the internal Pod service account admission injects the kube-api-access volume admission is called internal Pod → converts to v1 Pod → serializes → sends to mutating webhook mutating webhook returns patch patch is applied to v1 Pod JSON resulting v1 Pod JSON is decoded → defaults applied (defaultMode is set on the api-token volume here) → converted to internal Pod webhook admission is called internal Pod → converts to v1 Pod → serializes → sends to validating webhook That means if any built-in admission plugin sets up data that would be modified by applying defaulting to the pod, the data sent to mutating webhooks will be missing those applied defaults until the first mutating webhook that returns a patch that changes the object
Possible solutions: defaults are applied to the internal → versioned object before invoking webhooks built-in admission always populates data matching default values for unset fields
As an aside, I'm not sure what you're using the hashing for, but be aware that hashing is fragile in the presence of defaults and new fields over time. A hash you add to a pod spec in one release might not match a hash computed in a different release, or with different admission in place.
/triage accepted /assign /priority backlog


Question: store CheckpointStore } <del> func NewPersistentCheckpointHandler() CheckpointHandler { return &PersistentCheckpointHandler{store: &FileStore{path: filepath.Join(dockershimRootDir, sandboxCheckpointDir)}} </del> <ins> func NewPersistentCheckpointHandler() (CheckpointHandler, error) { fstore, err := NewFileStore(filepath.Join(dockershimRootDir, sandboxCheckpointDir)) if err != nil { return nil, err } return &PersistentCheckpointHandler{store: fstore}, nil </ins> } func (handler *PersistentCheckpointHandler) CreateCheckpoint(podSandboxID string, checkpoint *PodSandboxCheckpoint) error {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: package mount <del> type Mounter struct{} </del> <ins> type Mounter struct { mounterPath string } </ins> func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error { return nil

Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
, it seems we forgot to add to ; I'll create a PR for that.
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!


Question: "k8s.io/kubernetes/pkg/client/record" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" containertest "k8s.io/kubernetes/pkg/kubelet/container/testing" <del> "k8s.io/kubernetes/pkg/kubelet/events" </del> "k8s.io/kubernetes/pkg/kubelet/images" "k8s.io/kubernetes/pkg/kubelet/network" "k8s.io/kubernetes/pkg/kubelet/network/mock_network"

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: // Represents a projected volume source message ProjectedVolumeSource { // list of volume projections <ins> // +optional </ins> repeated VolumeProjection sources = 1; // Mode bits used to set permissions on created files by default.

What happened: I created a Pod with a ProjectedVolume with an empty array of , but Kubernetes transformed that value to . Some clients are happy with either value (e.g. Go) while others are not (e.g. Python). For example, the following is accepted and stored by Kubernetes, but cannot be read by the Python client: The field has been transformed to : Even a JSON patch has no effect: What you expected to happen: I expected the required (OpenAPI) field of core/v1.ProjectedVolumeSource to be returned as the empty array I submitted. How to reproduce it (as minimally and precisely as possible): Create the above Pod: See that has been transformed to : See that the officially-supported Python client is unable to deserialize the Pod: Anything else we need to know?: I recognize there's a lot of layers involved in the steps above. The ineffectiveness of the JSON patch is what makes me think the culprit is somewhere in the field spec or API somehow. /sig api-machinery Environment: Kubernetes version (use ): [official-clients]:
/remove-sig api-machinery /sig storage the issue is in the API definition, which does not indicate the sources field is optional (even though it is)
/cc


Question: IP address. release: v1.9 file: test/e2e/common/node/pods.go <ins> - testname: Pods, patching status codename: '[sig-node] Pods should patch a pod status [Conformance]' description: A pod is created which MUST succeed and be found running. The pod status when patched MUST succeed. Given the patching of the pod status, the fields MUST equal the new values. release: v1.25 file: test/e2e/common/node/pods.go </ins> - testname: Pods, completes the lifecycle of a Pod and the PodStatus codename: '[sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance]'

[X] test approval issue : [X] test pr : 28/07/2022 [X] two weeks soak end date : 12/07/2022 [X] test promotion pr : The endpoint as seen on the website is tested but not part of conformance. The APIsnoop query below shows that there is no conformance test for this endpoint. patchCoreV1NamespacedPodStatus Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the pod status endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale targeting this for 1.25
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: controlPlaneConfig.GenericConfig.Authorization.Authorizer = authorizerfactory.NewAlwaysDenyAuthorizer() _, s, closeFn := framework.RunAnAPIServer(controlPlaneConfig) defer closeFn() ns := framework.CreateTestingNamespace("auth-always-deny", s, t) defer framework.DeleteTestingNamespace(ns, s, t) <del> transport := http.DefaultTransport </del> <ins> transport := resttransport.NewBearerAuthRoundTripper(framework.UnprivilegedUserToken, http.DefaultTransport) </ins> for _, r := range getTestRequests(ns.Name) { bodyBytes := bytes.NewReader([]byte(r.body))

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: } // remove old KMS provider <del> // verifyIfKMSTransformersSwapped sometimes passes even before the changes in the encryption config file are observed. // this causes the metrics tests to fail, which validate two config changes. // this may happen when an existing KMS provider is already running (e.g., new-kms-provider-for-secrets in this case). // to ensure that the changes are observed, we added one more provider (kms-provider-to-encrypt-all) and are validating it in verifyIfKMSTransformersSwapped. </del> encryptionConfigWithoutOldProvider := ` kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } <ins> func Test_WhenNilPersister_NoOpPersisterIsAssigned(t *testing.T) { if err := RegisterAuthProviderPlugin("anyPlugin", pluginPersistProvider); err != nil { t.Errorf("unexpected error: failed to register 'anyPlugin': %v", err) } cfg := &clientcmdapi.AuthProviderConfig{ Name:   "anyPlugin", Config: nil, } plugin, err := GetAuthProvider("127.0.0.1", cfg, nil) if err != nil { t.Errorf("unexpected error: failed to get 'anyPlugin': %v", err) } anyPlugin := plugin.(*pluginPersist) if _, ok := anyPlugin.persister.(*noopPersister); !ok { t.Errorf("expected to be No Operation persister") } } </ins> // emptyTransport provides an empty http.Response with an initialized header // to allow wrapping RoundTrippers to set header values. type emptyTransport struct{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: no panicking How to reproduce it (as minimally and precisely as possible): use azure or OIDC auth plugin with a or without a Anything else we need to know?: a similar OIDC panic issue is registered here
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig auth

Using with a without a () will result in a panic. What happened: What you expected to happen: Expected a successful kube-api server call, or at the very least an error to be returned. Although i think it should just not call and continue with the kube-api server call. Anything else we need to know?: The reason for this is due to where has no check.
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ExtraArgs are extra arguments provided to the etcd binary // when run inside a static pod. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> // +optional ExtraArgs map[string]string `json:"extraArgs,omitempty"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: } wait.Wait() } else if diff > 0 { <del> glog.V(2).Infof("Too many "%s" replicas, deleting %dn", controller.Name, diff) </del> <ins> glog.V(2).Infof("Too many %q replicas, deleting %d", controller.Name, diff) // Sort the pods in the order such that not-ready < ready, unscheduled // < scheduled, and pending < running. This ensures that we delete pods // in the earlier stages whenever possible. sort.Sort(activePods(filteredList)) </ins> wait := sync.WaitGroup{} wait.Add(diff) for i := 0; i < diff; i++ {

I've noticed a issue where if you have multiple replicated slave pods that don't have a HOST assigned and you reduce the number of replicas in the replicationController definition for those pods, sometimes the currently assigned (and running) pods get killed and the unassigned pod will get assigned to run in the killed pods place. The issue can be reproduced using the guestbook example provided in the repo. Steps to reproduce: (Note this is not always deterministic and may need multiple attempts) the master pod the master service the replicated slaves the Slave Controller definition to only have 1 replicas the Slave Controller with new definition The expected behavior should be that the scheduler is able to determine that unassigned pods are more viable candidates when reducing number of replicas. It seems to be more likely that since this issue is not deterministic, the ordering in which the command returns result is not always deterministically in the same order
: please can you look at this or suggested someone else? Thank you.
Unassign myself from this bug since I cannot get working on this soon. Also remove kubelet from the label since it is not kubelet issue. This is obviously a replicaController issue. cc/ Instead of destroying pod in pending state, replicaController unbound running pod from a node. Kubelet running on such node has this new desire state and kill unwanted pod to match it.
Yes, the replication controller should be smarter, taking into account Phase, Readiness, recency, ... This will be a significant problem for auto-scalers, but I expect manual scaling to be infrequent. I doubt we'll get to this soon. A PR would be welcome. Place to change is here:
Hi, I am final year Master's student of International Institute of Information Technology, Hyderabad. I am familiar with docker and containers. I have basic understanding of Go language with some hands on Docker API. This project seems interesting and a solution to this will also be a good addition to Kubernetes. So I would like to pick this as my GSoC project. I have started reading about Replication Controller and Pods and going though the code. I would appreciate if someone has any inputs for me on this. Thanks, Swapnil Patil


Question: // this feature currently supported only on GCE/GKE/AWS framework.SkipUnlessProviderIs("gce", "gke", "aws") <ins> loadBalancerLagTimeout := framework.LoadBalancerLagTimeoutDefault if framework.ProviderIs("aws") { loadBalancerLagTimeout = framework.LoadBalancerLagTimeoutAWS } </ins> loadBalancerCreateTimeout := framework.LoadBalancerCreateTimeoutDefault if nodes := framework.GetReadySchedulableNodesOrDie(cs); len(nodes.Items) > framework.LargeClusterMinNodesNumber { loadBalancerCreateTimeout = framework.LoadBalancerCreateTimeoutLarge

Sub-issue for For AWS ELB, when an ELB is first created, is made it is very normal that it takes 2 minutes to be created, but the DNS name is reported fairly quickly.


Question: embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//staging/src/k8s.io/apiserver/pkg/endpoints/metrics:go_default_library", "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", </ins> ], )

What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: tests := []struct { name           string mountPoint     string <ins> selinuxEnabled bool </ins> expectedResult bool }{ { <ins> "ext4 on / with disabled SELinux", "/", false, false, }, { </ins> "ext4 on /", "/", true, <ins> true, </ins> }, { "tmpfs on /var/lib/bar", "/var/lib/bar", <ins> true, </ins> false, }, { "nfsv4", "/media/nfs_vol", <ins> true, </ins> false, }, } for _, test := range tests { <del> out, err := GetSELinux(test.mountPoint, filename) </del> <ins> out, err := GetSELinux(test.mountPoint, filename, func() bool { return test.selinuxEnabled }) </ins> if err != nil { t.Errorf("Test %s failed with error: %s", test.name, err) }

On a busy system, kubelet failed to tell the container runtime that a pod's CSI volume supports SELinux and should be relabeled with pod's SELinux label. The pod then can't access the volume. Kubelet correctly detects that the CSI volume supports SELinux and tells the container runtime to relabel it. Unknown. Get a busy system and try hard? CSI volume plugin tried to detect that the volume supports SELinux during SetupAt. This check failed: Current CSI volume plugin code silently consumes the error and assumes that the volume does not support SELinux at all: Therefore, the volume is marked as mounted and given to the container runtime without the right SELinux options and the runtime does not relabel volume. <details22.0 </details<detailsNone </details<details</details<details</details<detailsCRI-O, but I think it's not really relevant. </details<details</details
/sig storage /triage accepted
We've updated kubelet in our clusters to v1.20.13 (which contains the cherry picked fix), but still observe the issue


Question: // mountPropagation determines how mounts are propagated from the host // to container and the other way around. // When not set, MountPropagationHostToContainer is used. <del> // This field is alpha in 1.8 and can be reworked or removed in a future // release. </del> <ins> // This field is beta in 1.10. </ins> // +optional MountPropagation *MountPropagationMode }