Question: supports, err := fakeSupportsQuotas(testcase.path) if err != nil { fail = true <del> t.Errorf("Case %v (%s, %v) Got error in fakeSupportsQuotas: %v", seq, testcase.path, true, err) </del> <ins> t.Errorf("Case %v (%s, %s, %v) Got error in fakeSupportsQuotas: %v", seq, testcase.name, testcase.path, true, err) </ins> } if supports != testcase.supportsQuota { fail = true <del> t.Errorf("Case %v (%s, %v) fakeSupportsQuotas got %v, expect %v", seq, testcase.path, true, supports, testcase.supportsQuota) </del> <ins> t.Errorf("Case %v (%s, %s, %v) fakeSupportsQuotas got %v, expect %v", seq, testcase.name, testcase.path, true, supports, testcase.supportsQuota) </ins> } return fail case "Set":

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: } } <ins> func TestApplyRetry(t *testing.T) { initTestErrorHandler(t) nameRC, currentRC := readAndAnnotateReplicationController(t, filenameRC) pathRC := "/namespaces/test/replicationcontrollers/" + nameRC firstPatch := true retry := false getCount := 0 f, tf, codec := NewAPIFactory() tf.Printer = &testPrinter{} tf.Client = &fake.RESTClient{ Codec: codec, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { case p == pathRC && m == "GET": getCount++ bodyRC := ioutil.NopCloser(bytes.NewReader(currentRC)) return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: bodyRC}, nil case p == pathRC && m == "PATCH": if firstPatch { firstPatch = false statusErr := kubeerr.NewConflict(unversioned.GroupResource{Group: "", Resource: "rc"}, "test-rc", fmt.Errorf("the object has been modified. Please apply at first.")) bodyBytes, _ := json.Marshal(statusErr) bodyErr := ioutil.NopCloser(bytes.NewReader(bodyBytes)) return &http.Response{StatusCode: http.StatusConflict, Header: defaultHeader(), Body: bodyErr}, nil } retry = true validatePatchApplication(t, req) bodyRC := ioutil.NopCloser(bytes.NewReader(currentRC)) return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: bodyRC}, nil default: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } }), } tf.Namespace = "test" buf := bytes.NewBuffer([]byte{}) cmd := NewCmdApply(f, buf) cmd.Flags().Set("filename", filenameRC) cmd.Flags().Set("output", "name") cmd.Run(cmd, []string{}) if !retry || getCount != 2 { t.Fatalf("apply didn't retry when get conflict error") } // uses the name from the file, not the response expectRC := "replicationcontroller/" + nameRC + "n" if buf.String() != expectRC { t.Fatalf("unexpected output: %snexpected: %s", buf.String(), expectRC) } } </ins> func TestApplyNonExistObject(t *testing.T) { nameRC, currentRC := readAndAnnotateReplicationController(t, filenameRC) pathRC := "/namespaces/test/replicationcontrollers"

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: .PP fB--generatorfP="" <del> The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'. </del> <ins> The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'.  This will happen only for cluster version at least 1.2, for olders we will fallback to 'run/v1' for --restart=Alwyas, 'run-pod/v1' for others. </ins> .PP fB--hostportfP=-1

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: Password:         password, } <del> _, err := openstack.AuthenticatedClient(opts) </del> <ins> _, err := keystoneAuthenticator.AuthenticatedClient(opts) </ins> if err != nil { <del> glog.Info("Failed: Starting openstack authenticate client") </del> <ins> glog.Info("Failed: Starting openstack authenticate client:" + err.Error()) </ins> return nil, false, errors.New("Failed to authenticate") } return &user.DefaultInfo{Name: username}, true, nil } <ins> // AuthenticatedClient logs in to an OpenStack cloud found at the identity endpoint specified by options, acquires a // token, and returns a Client instance that's ready to operate. func (keystoneAuthenticator *KeystoneAuthenticator) AuthenticatedClient(options gophercloud.AuthOptions) (*gophercloud.ProviderClient, error) { client, err := openstack.NewClient(options.IdentityEndpoint) if err != nil { return nil, err } if keystoneAuthenticator.transport != nil { client.HTTPClient.Transport = keystoneAuthenticator.transport } err = openstack.Authenticate(client, options) return client, err } </ins> // NewKeystoneAuthenticator returns a password authenticator that validates credentials using openstack keystone <del> func NewKeystoneAuthenticator(authURL string) (*KeystoneAuthenticator, error) { </del> <ins> func NewKeystoneAuthenticator(authURL string, caFile string) (*KeystoneAuthenticator, error) { </ins> if !strings.HasPrefix(authURL, "https") { return nil, errors.New("Auth URL should be secure and start with https") } if authURL == "" { return nil, errors.New("Auth URL is empty") } <ins> if caFile != "" { roots, err := certutil.NewPool(caFile) if err != nil { return nil, err } config := &tls.Config{} config.RootCAs = roots transport := netutil.SetOldTransportDefaults(&http.Transport{TLSClientConfig: config}) return &KeystoneAuthenticator{authURL, transport}, nil } </ins> <del> return &KeystoneAuthenticator{authURL}, nil </del> <ins> return &KeystoneAuthenticator{authURL: authURL}, nil </ins> }

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: // start serving go func(hcI *hcInstance, listener net.Listener, httpSrv httpServer) { <del> // Serve() will exit when the listener is closed. </del> <ins> // Serve() will exit and return ErrServerClosed when the http server is closed. </ins> klog.V(3).InfoS("Starting goroutine for healthcheck", "service", hcI.nsn, "address", listener.Addr()) <del> if err := httpSrv.Serve(listener); err != nil { </del> <ins> if err := httpSrv.Serve(listener); err != nil && err != http.ErrServerClosed { </ins> klog.ErrorS(err, "Healthcheck closed", "service", hcI.nsn) return } klog.V(3).InfoS("Healthcheck closed", "service", hcI.nsn, "address", listener.Addr()) }(hcI, listener, httpSrv) <del> hcI.listeners = append(hcI.listeners, listener) </del> hcI.httpServers = append(hcI.httpServers, httpSrv) }

When a LoadBalancer Service with Local ExternalTrafficPolicy was deleted normally, an error log "use of closed network connection" was always generated in kube-proxy log, which confused people that there was something wrong: Do not generate error log when it's expected. a LoadBalancer Service with Local ExternalTrafficPolicy it No response <detailsv1.26.0 </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network


Question: } now := unversioned.Now() oldNamespace := &api.Namespace{ <del> ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "10"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "10", DeletionTimestamp: &now}, </ins> Spec:       api.NamespaceSpec{Finalizers: []api.FinalizerName{"kubernetes"}}, Status:     api.NamespaceStatus{Phase: api.NamespaceActive}, }

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: return fmt.Errorf("port %s still open", proxyPort) } <del> // a simple echoServer that only accepts one connection. Returns port actually // being listened on, or an error. func echoServer(t *testing.T, addr string) (string, error) { l, err := net.Listen("tcp", addr) </del> <ins> var port string func init() { ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte(r.URL.Path[1:])) })) u, err := url.Parse(ts.URL) </ins> if err != nil { <del> return "", fmt.Errorf("failed to start echo service: %v", err) </del> <ins> panic(fmt.Sprintf("failed to parse: %v", err)) } _, port, err = net.SplitHostPort(u.Host) if err != nil { panic(fmt.Sprintf("failed to parse: %v", err)) </ins> } <del> go func() { defer l.Close() conn, err := l.Accept() if err != nil { t.Errorf("failed to accept new conn to echo service: %v", err) } io.Copy(conn, conn) conn.Close() }() _, port, err := net.SplitHostPort(l.Addr().String()) return port, err </del> } func testEchoConnection(t *testing.T, address, port string) { <del> conn, err := net.Dial("tcp", net.JoinHostPort(address, port)) </del> <ins> path := "aaaaa" res, err := http.Get("http://" + address + ":" + port + "/" + path) </ins> if err != nil { <del> t.Fatalf("error connecting to proxy: %v", err) } magic := "aaaaa" if _, err := conn.Write([]byte(magic)); err != nil { t.Fatalf("error writing to proxy: %v", err) </del> <ins> t.Fatalf("error connecting to server: %v", err) </ins> } <del> buf := make([]byte, 5) if _, err := conn.Read(buf); err != nil { t.Fatalf("error reading from proxy: %v", err) </del> <ins> defer res.Body.Close() data, err := ioutil.ReadAll(res.Body) if err != nil { t.Errorf("error reading data: %v %v", err, string(data)) </ins> } <del> if string(buf) != magic { t.Fatalf("bad echo from proxy: got: %q, expected %q", string(buf), magic) </del> <ins> if string(data) != path { t.Errorf("expected: %s, got %s", path, string(data)) </ins> } } func TestProxy(t *testing.T) { <del> port, err := echoServer(t, "127.0.0.1:0") if err != nil { t.Fatalf("Unexpected error: %v", err) } </del> lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{ {JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}})

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: * Connections from Pods to Services with 0 endpoints will now ICMP reject immediately, rather than blackhole and timeout. ([#72534](https://github.com/kubernetes/kubernetes/pull/72534), [@thockin](https://github.com/thockin)) * Services of type=LoadBalancer which have no endpoints will now immediately ICMP reject connections, rather than time out. ([#74394](https://github.com/kubernetes/kubernetes/pull/74394), [@thockin](https://github.com/thockin)) * Ensure the backend pools are set correctly for Azure SLB with multiple backend pools (e.g. outbound rules) ([#76691](https://github.com/kubernetes/kubernetes/pull/76691), [@feiskyer](https://github.com/feiskyer)) <del> * Connections from Pods to Services with 0 endpoints will now ICMP reject immediately, rather than blackhole and timeout. ([#72534](https://github.com/kubernetes/kubernetes/pull/72534), [@thockin](https://github.com/thockin)) </del> * Services of type=LoadBalancer which have no endpoints will now immediately ICMP reject connections, rather than time out. ([#74394](https://github.com/kubernetes/kubernetes/pull/74394), [@thockin](https://github.com/thockin)) * fix race condition issue for smb mount on windows ([#75371](https://github.com/kubernetes/kubernetes/pull/75371), [@andyzhangx](https://github.com/andyzhangx)) * fix smb unmount issue on Windows ([#75087](https://github.com/kubernetes/kubernetes/pull/75087), [@andyzhangx](https://github.com/andyzhangx))

What happened: There is a duplicate row in the Changelog for v1.12.7 How to reproduce it (as minimally and precisely as possible): the first and the fourth row are exactly the same. /sig docs


Question: func isRegistryEqual(registryX, registryY Registry) bool { for name, pluginFactory := range registryY { if val, ok := registryX[name]; ok { <del> if reflect.ValueOf(pluginFactory).Pointer() != reflect.ValueOf(val).Pointer() { </del> <ins> p1, _ := pluginFactory(nil, nil) p2, _ := val(nil, nil) if p1.Name() != p2.Name() { </ins> // pluginFactory functions are not the same. return false }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: } func TestDescribeNamespace(t *testing.T) { <del> fake := fake.NewSimpleClientset(&corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: "myns", </del> <ins> exampleNamespaceName := "example" testCases := []struct { name      string namespace *corev1.Namespace expect    []string }{ { name: "no quotas or limit ranges", namespace: &corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: exampleNamespaceName, }, Status: corev1.NamespaceStatus{ Phase: corev1.NamespaceActive, }, }, expect: []string{ "Name", exampleNamespaceName, "Status", string(corev1.NamespaceActive), "No resource quota", "No LimitRange resource.", }, }, { name: "has conditions", namespace: &corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: exampleNamespaceName, }, Status: corev1.NamespaceStatus{ Phase: corev1.NamespaceTerminating, Conditions: []corev1.NamespaceCondition{ { LastTransitionTime: metav1.NewTime(time.Date(2014, time.January, 15, 0, 0, 0, 0, time.UTC)), Message:            "example message", Reason:             "example reason", Status:             corev1.ConditionTrue, Type:               corev1.NamespaceDeletionContentFailure, }, }, }, }, expect: []string{ "Name", exampleNamespaceName, "Status", string(corev1.NamespaceTerminating), "Conditions", "Type", string(corev1.NamespaceDeletionContentFailure), "Status", string(corev1.ConditionTrue), "Reason", "example reason", "Message", "example message", "No resource quota", "No LimitRange resource.", }, </ins> }, <del> }) c := &describeClient{T: t, Namespace: "", Interface: fake} d := NamespaceDescriber{c} out, err := d.Describe("", "myns", DescriberSettings{ShowEvents: true}) if err != nil { t.Errorf("unexpected error: %v", err) </del> } <del> if !strings.Contains(out, "myns") { t.Errorf("unexpected out: %s", out) </del> <ins> for _, testCase := range testCases { t.Run(testCase.name, func(t *testing.T) { fake := fake.NewSimpleClientset(testCase.namespace) c := &describeClient{T: t, Namespace: "", Interface: fake} d := NamespaceDescriber{c} out, err := d.Describe("", testCase.namespace.Name, DescriberSettings{ShowEvents: true}) if err != nil { t.Errorf("unexpected error: %v", err) } for _, expected := range testCase.expect { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out) } } }) </ins> } }

should output Conditions. For example: Including Conditions in the output is useful. In this example, a Condition can explain why a namespace deletion has not completed. While the output below includes Conditions, I believe the output of would be easier to read.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign


Question: }, }, }, <ins> { name: "pod with init containers", opts: &DebugOptions{ Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, pod: &corev1.Pod{ Spec: corev1.PodSpec{ InitContainers: []corev1.Container{ { Name: "init-container-1", }, { Name: "init-container-2", }, }, Containers: []corev1.Container{ { Name: "debugger", }, }, }, }, expected: &corev1.EphemeralContainer{ EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name:                     "debugger-1", Image:                    "busybox", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, { name: "pod with ephemeral containers", opts: &DebugOptions{ Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, pod: &corev1.Pod{ Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "debugger", }, }, EphemeralContainers: []corev1.EphemeralContainer{ { EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name: "ephemeral-container-1", }, }, { EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name: "ephemeral-container-2", }, }, }, }, }, expected: &corev1.EphemeralContainer{ EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name:                     "debugger-1", Image:                    "busybox", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, </ins> } { t.Run(tc.name, func(t *testing.T) { tc.opts.IOStreams = genericclioptions.NewTestIOStreamsDiscard()

What happened: When using to create a debug container for a pod that has multiple init containers or ephemeral containers, fails with . What you expected to happen: should create a debug container in pod successfully. How to reproduce it (as minimally and precisely as possible): The EphemeralContainers feature gate must be enabled by using flag in your cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): NAME="CentOS Linux" VERSION="7 (Core)" Kernel (e.g. ): Linux node-1 3.10.0-957.el7.x8664 SMP Thu Nov 8 23:39:32 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cli
I have investigated this issue and it looks like this issue happens in the following cases. When using without option When using with option In those cases, the containerNameToRef function is called and each for-loop in the function refers to the same despite the range is , or .
cc If this issue needs to be fixed, I would like to work on this.
/assign
Thanks for the report and for fixing it!


Question: return isTerminal, nil } <del> // If the pod should not be running, we request the pod's containers be stopped. This is not the same // as termination (we want to stop the pod, but potentially restart it later if soft admission allows // it later). Set the status and phase appropriately runnable := kl.canRunPod(pod) if !runnable.Admit { // Pod is not runnable; and update the Pod and Container statuses to why. if apiPodStatus.Phase != v1.PodFailed && apiPodStatus.Phase != v1.PodSucceeded { apiPodStatus.Phase = v1.PodPending } apiPodStatus.Reason = runnable.Reason apiPodStatus.Message = runnable.Message // Waiting containers are not creating. const waitingReason = "Blocked" for _, cs := range apiPodStatus.InitContainerStatuses { if cs.State.Waiting != nil { cs.State.Waiting.Reason = waitingReason } } for _, cs := range apiPodStatus.ContainerStatuses { if cs.State.Waiting != nil { cs.State.Waiting.Reason = waitingReason } } } </del> // Record the time it takes for the pod to become running // since kubelet first saw the pod if firstSeenTime is set. existingStatus, ok := kl.statusManager.GetPodStatus(pod.UID)

Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into


Question: return s, fwk, nil } <ins> func TestInitPluginsWithIndexers(t *testing.T) { tests := []struct { name string // the plugin registration ordering must not matter, being map traversal random entrypoints map[string]frameworkruntime.PluginFactory wantErr     string }{ { name: "register indexer, no conflicts", entrypoints: map[string]frameworkruntime.PluginFactory{ "AddIndexer": func(obj runtime.Object, handle framework.Handle) (framework.Plugin, error) { podInformer := handle.SharedInformerFactory().Core().V1().Pods() err := podInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ "nodeName": indexByPodSpecNodeName, }) return &TestPlugin{name: "AddIndexer"}, err }, }, }, { name: "register the same indexer name multiple times, conflict", // order of registration doesn't matter entrypoints: map[string]frameworkruntime.PluginFactory{ "AddIndexer1": func(obj runtime.Object, handle framework.Handle) (framework.Plugin, error) { podInformer := handle.SharedInformerFactory().Core().V1().Pods() err := podInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ "nodeName": indexByPodSpecNodeName, }) return &TestPlugin{name: "AddIndexer1"}, err }, "AddIndexer2": func(obj runtime.Object, handle framework.Handle) (framework.Plugin, error) { podInformer := handle.SharedInformerFactory().Core().V1().Pods() err := podInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ "nodeName": indexByPodAnnotationNodeName, }) return &TestPlugin{name: "AddIndexer1"}, err }, }, wantErr: "indexer conflict", }, { name: "register the same indexer body with different names, no conflicts", // order of registration doesn't matter entrypoints: map[string]frameworkruntime.PluginFactory{ "AddIndexer1": func(obj runtime.Object, handle framework.Handle) (framework.Plugin, error) { podInformer := handle.SharedInformerFactory().Core().V1().Pods() err := podInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ "nodeName1": indexByPodSpecNodeName, }) return &TestPlugin{name: "AddIndexer1"}, err }, "AddIndexer2": func(obj runtime.Object, handle framework.Handle) (framework.Plugin, error) { podInformer := handle.SharedInformerFactory().Core().V1().Pods() err := podInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ "nodeName2": indexByPodAnnotationNodeName, }) return &TestPlugin{name: "AddIndexer2"}, err }, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { fakeInformerFactory := NewInformerFactory(&fake.Clientset{}, 0*time.Second) var registerPluginFuncs []st.RegisterPluginFunc for name, entrypoint := range tt.entrypoints { registerPluginFuncs = append(registerPluginFuncs, // anything supported by TestPlugin is fine st.RegisterFilterPlugin(name, entrypoint), ) } // we always need this registerPluginFuncs = append(registerPluginFuncs, st.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), st.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), ) _, err := st.NewFramework(registerPluginFuncs, "test", frameworkruntime.WithInformerFactory(fakeInformerFactory)) if len(tt.wantErr) > 0 { if err == nil || !strings.Contains(err.Error(), tt.wantErr) { t.Errorf("got error %q, want %q", err, tt.wantErr) } return } if err != nil { t.Fatalf("Failed to create scheduler: %v", err) } }) } } func indexByPodSpecNodeName(obj interface{}) ([]string, error) { pod, ok := obj.(*v1.Pod) if !ok { return []string{}, nil } if len(pod.Spec.NodeName) == 0 { return []string{}, nil } return []string{pod.Spec.NodeName}, nil } func indexByPodAnnotationNodeName(obj interface{}) ([]string, error) { pod, ok := obj.(*v1.Pod) if !ok { return []string{}, nil } if len(pod.Annotations) == 0 { return []string{}, nil } nodeName, ok := pod.Annotations["node-name"] if !ok { return []string{}, nil } return []string{nodeName}, nil } </ins>

I'm writing a scheduler plugin using the scheduler framework, and in my plugin I'd like to add a using the provided by the . Adding such a Indexer lead to a runtime crash like The reason for this is that I'm not sure the scheduler framework wants to explicitely disallow adding Indexers using the facilities. If this is the case, it should be more explicit somehow because the current behaviour is surprising. If the scheduler framework doesn't want to add this limitation, then adding indexers should not lead to runtime crashes add a new scheduler plugin trying to add indexers. Sample code (relevant minimal portions only) I'm not sure this is a real bug and not a deliberate decision; a research was inconcusive, and a (admittedly quick) search in the past issues was not fruitful either. The fix could be as trivial as <detailsconsole $ kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-10T16:30:48Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"} </detailsbut applies also on latest master (see above) <detailsany </details<detailsany </details<detailsnot relevant </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: newService("s4", "123", false), }, expectedUpdateCalls: []fake_cloud.FakeUpdateBalancerCall{ <del> {Name: lbName("s1"), Region: region, Hosts: []string{"node0", "node1", "node73"}}, {Name: lbName("s3"), Region: region, Hosts: []string{"node0", "node1", "node73"}}, </del> <ins> {Name: "a888", Region: region, Hosts: []string{"node0", "node1", "node73"}}, {Name: "a999", Region: region, Hosts: []string{"node0", "node1", "node73"}}, </ins> }, }, }

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: Use the [master.yaml](cloud-configs/master.yaml) and [node.yaml](cloud-configs/node.yaml) cloud-configs to provision a multi-node Kubernetes cluster. <ins> *Attention:* This requires at least CoreOS version 593.0.0/598.0.0. </ins> ## Overview * Provision the master node

Hello, I have installed the latest kubernetes binaries on CoreOS running on GCE. The cloud-config is as per instructions on "getting started guides". After I installed 3 Pods, the state on all were constantly "pending". I looked around but none of the previous solutions worked. After messing around a lot for 3 days it turned out to be the following line. I had 3 in pending status now one Pod is "successful" the others are "running". Change the config for docker daemon from ExecStart=/usr/bin/docker -d -s=overlay -H fd:// --bip={FLANNELMTU} To ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// --bip={FLANNELMTU}
Usually this means that you have a scheduling conflict. What does print? My guess is that you have set on your pods, and you have more replicas than you have machines, and so some pods can not schedule due to hostPort conflicts.
The event dump as follows. I have the entire cluster in a stand alone machine. The cloud-config is this one
So it looks like you are trying to get node level logging up using Flunetd -Elasticsearch? Let me know if there is something I can do to help.
satnam, I will contact you if I have any questions with Flunetd -Elasticsearch. Much appreciated.
Not sure if related but CoreOS instructions are borked for 0.10.x or 0.11.x. Will PR changes today. See .
Actually, changed the layer driver from to which means he's using an older version of CoreOS. I can't reproduce this behavior with CoreOS 593.0.0/598.0.0 and Kubernetes 0.9.3, 0.10.1 or 0.11.0.
I am using coreos-beta-584-0-0-v20150218 . Perhaps a warning should go on getting-started-guides to say it only applies to CoreOS after 593.0.0/598.0.0.


Question: fldPath := field.NewPath("issuer", "discoveryURL") testCases := []struct { <del> name      string in        string issuerURL string want      string </del> <ins> name                          string in                            string issuerURL                     string want                          string structuredAuthnFeatureEnabled bool </ins> }{ { <del> name: "url is empty", in:   "", want: "", </del> <ins> name:                          "url is empty", in:                            "", want:                          "", structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "url parse error", in:   "https://oidc.oidc-namespace.svc:invalid-port", want: `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc:invalid-port": parse "https://oidc.oidc-namespace.svc:invalid-port": invalid port ":invalid-port" after host`, </del> <ins> name:                          "url parse error", in:                            "https://oidc.oidc-namespace.svc:invalid-port", want:                          `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc:invalid-port": parse "https://oidc.oidc-namespace.svc:invalid-port": invalid port ":invalid-port" after host`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "url is not https", in:   "http://oidc.oidc-namespace.svc", want: `issuer.discoveryURL: Invalid value: "http://oidc.oidc-namespace.svc": URL scheme must be https`, </del> <ins> name:                          "url is not https", in:                            "http://oidc.oidc-namespace.svc", want:                          `issuer.discoveryURL: Invalid value: "http://oidc.oidc-namespace.svc": URL scheme must be https`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "url user info is not allowed", in:   "https://user:pass@oidc.oidc-namespace.svc", want: `issuer.discoveryURL: Invalid value: "https://user:pass@oidc.oidc-namespace.svc": URL must not contain a username or password`, </del> <ins> name:                          "url user info is not allowed", in:                            "https://user:pass@oidc.oidc-namespace.svc", want:                          `issuer.discoveryURL: Invalid value: "https://user:pass@oidc.oidc-namespace.svc": URL must not contain a username or password`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "url raw query is not allowed", in:   "https://oidc.oidc-namespace.svc?query", want: `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc?query": URL must not contain a query`, </del> <ins> name:                          "url raw query is not allowed", in:                            "https://oidc.oidc-namespace.svc?query", want:                          `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc?query": URL must not contain a query`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "url fragment is not allowed", in:   "https://oidc.oidc-namespace.svc#fragment", want: `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc#fragment": URL must not contain a fragment`, </del> <ins> name:                          "url fragment is not allowed", in:                            "https://oidc.oidc-namespace.svc#fragment", want:                          `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc#fragment": URL must not contain a fragment`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "valid url", in:   "https://oidc.oidc-namespace.svc", want: "", </del> <ins> name:                          "valid url", in:                            "https://oidc.oidc-namespace.svc", want:                          "", structuredAuthnFeatureEnabled: true, </ins> }, { <del> name: "valid url with path", in:   "https://oidc.oidc-namespace.svc/path", want: "", </del> <ins> name:                          "valid url with path", in:                            "https://oidc.oidc-namespace.svc/path", want:                          "", structuredAuthnFeatureEnabled: true, </ins> }, { <del> name:      "discovery url same as issuer url", issuerURL: "https://issuer-url", in:        "https://issuer-url", want:      `issuer.discoveryURL: Invalid value: "https://issuer-url": discoveryURL must be different from URL`, </del> <ins> name:                          "discovery url same as issuer url", issuerURL:                     "https://issuer-url", in:                            "https://issuer-url", want:                          `issuer.discoveryURL: Invalid value: "https://issuer-url": discoveryURL must be different from URL`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name:      "discovery url same as issuer url, with trailing slash", issuerURL: "https://issuer-url", in:        "https://issuer-url/", want:      `issuer.discoveryURL: Invalid value: "https://issuer-url/": discoveryURL must be different from URL`, </del> <ins> name:                          "discovery url same as issuer url, with trailing slash", issuerURL:                     "https://issuer-url", in:                            "https://issuer-url/", want:                          `issuer.discoveryURL: Invalid value: "https://issuer-url/": discoveryURL must be different from URL`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name:      "discovery url same as issuer url, with multiple trailing slashes", issuerURL: "https://issuer-url", in:        "https://issuer-url///", want:      `issuer.discoveryURL: Invalid value: "https://issuer-url///": discoveryURL must be different from URL`, </del> <ins> name:                          "discovery url same as issuer url, with multiple trailing slashes", issuerURL:                     "https://issuer-url", in:                            "https://issuer-url///", want:                          `issuer.discoveryURL: Invalid value: "https://issuer-url///": discoveryURL must be different from URL`, structuredAuthnFeatureEnabled: true, </ins> }, { <del> name:      "discovery url same as issuer url, issuer url with trailing slash", issuerURL: "https://issuer-url/", in:        "https://issuer-url", want:      `issuer.discoveryURL: Invalid value: "https://issuer-url": discoveryURL must be different from URL`, </del> <ins> name:                          "discovery url same as issuer url, issuer url with trailing slash", issuerURL:                     "https://issuer-url/", in:                            "https://issuer-url", want:                          `issuer.discoveryURL: Invalid value: "https://issuer-url": discoveryURL must be different from URL`, structuredAuthnFeatureEnabled: true, }, { name: "discovery url set but structured authn feature disabled", in:   "https://oidc.oidc-namespace.svc", want: `issuer.discoveryURL: Invalid value: "https://oidc.oidc-namespace.svc": discoveryURL is not supported when StructuredAuthenticationConfiguration feature gate is disabled`, </ins> }, } for _, tt := range testCases { t.Run(tt.name, func(t *testing.T) { <del> got := validateIssuerDiscoveryURL(tt.issuerURL, tt.in, fldPath).ToAggregate() </del> <ins> got := validateIssuerDiscoveryURL(tt.issuerURL, tt.in, fldPath, tt.structuredAuthnFeatureEnabled).ToAggregate() </ins> if d := cmp.Diff(tt.want, errString(got)); d != "" { t.Fatalf("URL validation mismatch (-want +got):n%s", d) }

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question: kubectl taint nodes foo dedicated:NoSchedule- # Remove from node 'foo' all the taints with key 'dedicated' <del> kubectl taint nodes foo dedicated-`)) </del> <ins> kubectl taint nodes foo dedicated- # Add a taint with key 'dedicated' on nodes having label mylabel=X kubectl taint node -l myLabel=X  dedicated=foo:PreferNoSchedule`)) </ins> ) func NewCmdTaint(f cmdutil.Factory, out io.Writer) *cobra.Command {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: } if runtime.GOOS == "windows" { <del> if strings.HasSuffix(info.Name(), ".exe") { </del> <ins> fileExt := strings.ToLower(filepath.Ext(fullPath)) switch fileExt { case ".bat", ".cmd", ".com", ".exe": </ins> return true, nil } return false, nil

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the kubectl plugins system - when running under Windows - will determine that a plugin is executable by looking at the plugin's extension, comparing it to '.exe'. I think we should allow plugins w/ a few more known extensions (.bat, .com, .cmd; for example). Why is this needed: This would give more of a similar experience to plugins under *nix, where shell scripts are often used.
/sig cli
Please see the PR below:


Question: } func (obj InfoObject) Last() (map[string]interface{}, error) { <del> if obj.Info.Object == nil { </del> <ins> if obj.Remote == nil { </ins> return nil, nil // No object is live, return empty } <del> accessor, err := meta.Accessor(obj.Info.Object) </del> <ins> accessor, err := meta.Accessor(obj.Remote) </ins> if err != nil { return nil, err }

/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.


Question: hostNetwork: true containers: - name: fluentd-gcp <del> image: gcr.io/google-containers/fluentd-gcp:2.0.5 </del> <ins> image: gcr.io/google-containers/fluentd-gcp:2.0.7 </ins> # If fluentd consumes its own logs, the following situation may happen: # fluentd fails to send a chunk to the server => writes it to the log => # tries to send this message to the server => fails to send a chunk and so on.

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  <del> "${_pkg}"  </del> <ins> "${_pkg}"  </ins> ${testargs[@]:+${testargs[@]}} go test ${goflags[@]:+${goflags[@]}}  ${KUBE_RACE}  ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  <del> "${_pkg}"  </del> <ins> "${_pkg}"  </ins> ${testargs[@]:+${testargs[@]}}  | tee ${junit_filename_prefix:+"${junit_filename_prefix}-$_pkg_out.stdout"}  | grep "${go_test_grep_pattern}""

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master (SHA ) What happened: What you expected to happen: Generate code coverage report on Mac How to reproduce it (as minimally and precisely as possible):
/cc
does it work for you prior to this commit? There may be an escaping issue, currently on mobile and unable to verify that hypothesis.
, just create PR to fix that, PTAL. That should be a typo: some missed for in embedded script.


Question: // CronJobs and rcs have the same name validation allErrs := apivalidation.ValidateObjectMeta(&scheduledJob.ObjectMeta, true, apivalidation.ValidateReplicationControllerName, field.NewPath("metadata")) allErrs = append(allErrs, ValidateCronJobSpec(&scheduledJob.Spec, field.NewPath("spec"))...) <ins> if len(scheduledJob.ObjectMeta.Name) > apimachineryvalidation.DNS1035LabelMaxLength-11 { // The cronjob controller appends a 11-character suffix to the cronjob (`-$TIMESTAMP`) when // creating a job. The job name length limit is 63 characters. // Therefore cronjob names must have length <= 63-11=52. If we don't validate this here, // then job creation will fail later. allErrs = append(allErrs, field.Invalid(field.NewPath("metadata").Child("name"), scheduledJob.ObjectMeta.Name, "must be no more than 52 characters")) } </ins> return allErrs }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We have some cron jobs with longish names (~60 characters), and noticed that the corresponding jobs weren't being created by the cronjob controller because the name was too long. Here's what happened: There's a 63-character limit on job names (because for some reason on job creation the name gets copied into a label, where the limit is 63 chars). For example, if I try to create a job with a long name, I get the error message: By itself, this is fine. I can just make a job with a shorter name. However! If I create a cron job, the cron job controller will create jobs named . So the job's name is 11 characters longer than the job name. This means that if I create a cron job with a 63-character name, it'll result in a job name that's 74 characters, which the API will refuse to accept. This means the cron job never gets scheduled. Some possible resolutions: validation on cron job creation that refuses to let you create jobs with names 52 characters the cron job controller truncate cronjobs names remove the 63-character limit on job names (this seems difficult) There's actually a deeper problem here at play: Kubernetes' resource name limit is 253 characters. But the limit for resouce labels is 63 characters. So when a job is created, the name (which is allowed to be up to 255 chars) gets copied into a label, which has a lower limit. So the job creation fails. (these are the ) What you expected to happen: When I create a cron job, I expect it to either to create the cron job, or jobs correctly from that cron job How to reproduce it (as minimally and precisely as possible): a cronjob with a name that's more than 52 characters that it does not schedule jobs, with an error message like Anything else we need to know?: Kubernetes version: 1.7.3 Cloud provider: AWS OS: Ubuntu 16.04 Kernel: 4.4.0 example of a job definition with a long name This job definition is invalid:
There are no sig labels on this issue.

Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
cc


Question: } } <del> func TestValidateIPVSSchedulerMethod(t *testing.T) { newPath := field.NewPath("KubeProxyConfiguration") successCases := []kubeproxyconfig.IPVSSchedulerMethod{ kubeproxyconfig.RoundRobin, kubeproxyconfig.WeightedRoundRobin, kubeproxyconfig.LeastConnection, kubeproxyconfig.WeightedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnectionWithReplication, kubeproxyconfig.SourceHashing, kubeproxyconfig.DestinationHashing, kubeproxyconfig.ShortestExpectedDelay, kubeproxyconfig.NeverQueue, "", } for _, successCase := range successCases { if errs := validateIPVSSchedulerMethod(successCase, newPath.Child("Scheduler")); len(errs) != 0 { t.Errorf("expected success: %v", errs) } } errorCases := map[string]struct { mode         kubeproxyconfig.IPVSSchedulerMethod expectedErrs field.ErrorList }{ "non-existent scheduler method": { mode:         kubeproxyconfig.IPVSSchedulerMethod("non-existing"), expectedErrs: field.ErrorList{field.Invalid(newPath.Child("ProxyMode.Scheduler"), "non-existing", "must be in [rr wrr lc wlc lblc lblcr sh dh sed nq ], blank means the default algorithm method (currently rr)")}, }, } for _, errorCase := range errorCases { errs := validateIPVSSchedulerMethod(errorCase.mode, newPath.Child("ProxyMode")) if len(errorCase.expectedErrs) != len(errs) { t.Fatalf("Expected %d errors, got %d errors: %v", len(errorCase.expectedErrs), len(errs), errs) } for i, err := range errs { if err.Error() != errorCase.expectedErrs[i].Error() { t.Fatalf("Expected error: %s, got %s", errorCase.expectedErrs[i], err.Error()) } } } } </del> func TestValidateKubeProxyNodePortAddress(t *testing.T) { newPath := field.NewPath("KubeProxyConfiguration")

The ipvs mode limit the scheduler algorithm to : now in new kernel have this new algorithm : the are specialy interesting... that be in kernel 4.18...
/sig network /area ipvs
/assign
This seems non-controversial on its face, but a) how do we know whether the kernel supports it or not; and b) if every node makes independent decisions, doesn't this basically become random? /assign uablrek
a) It should be the user's responsibility to ensure that the configured ipvs sheduler is available on all nodes This sounds rude, but please read for some rationale (and ). Exception may be the which is the default, but an installation with but without the sheduler is extremely unlikely. If that relaxing of the sheduler pre-test is acceptable, I would say that any current and future ipvs sheduler shall be allowed. Just treat them as a string. But it should be documented in the release-notes and some place else, but I don't know where.
b) For all shedulers that make decisions based on local state, yes. (that includes btw) Well, not really random ... For instance if is used and a new endpoint is addred, then all ipvs instances will get an "empty" target and will select it for new connections and may overload the newcomer, or it will become "statistically correct" over time. But IMHO users should be allowed to try. Some sheduler may be very useful in their environment. That will (already has) of course spawn some issues when it doesnt work as expected (for the reason in b) for instance), but allowing all shedulers will not change that.
So, my proposal is: Drop the scheduler module check Treat the configured scheduler as a string in the ipvs proxier Document
The module check can be replaced with a function check, e.g. insert/delete an ipvs dummy rule.
people is trying to reduce privileges on containers these days, does it requires more privileges than today or is the same?
The same. Kube-proxy configures ipvs already. This would only add an extra probing insert/delete on start. Even on first start (if ipvs is empty) with some care.
Could even be less privileges if "modprobe" requires any privileges (now or in the future)
What is your use-case?

I have wanted to check the sceduler in ipvs, but havn't got time yet. The load-balancer is stateless, but is not (well, at least not normally). A problem Maglev tries to solve is if the number of Application Servers (AS) (endpoints in K8s) changes, the updated hash buckets must be as few as possible. But if the load-balancer is stateful and stores the connections (like ipvs or the Linux conntrack used in proxy-mode=iptables), no existing connections are affected, no matter what hash algorithm is used for new connections. So, for both proxy-mode ipvs and iptables existing connection are not affected by adding or removing endpoints. But... if external traffic is routed to different nodes, e.g. a GW rehashes ECMP targets after a cluster scaling, then traffic is affected. This is easily tested: Setup a service of type LoadBalancer Route external traffic to the loadBalancerIP to node A Setup a number of TCP connections Re-route external traffic to the loadBalancerIP to node B All connections are lost (I have tested). But if the load-balancer would have been a stateless Maglev no connections would be lost. Google Maglev stores connections but only as a cache and getting the best of stateful, existing connections never lost on AS scaling, and stateless, the load-balancing tier can be scaled with minimum impact. This can be the case in ipvs , but is it?
Lars: proposal SGTM. IPVS impl is pretty much your call to make :) On Thu, Dec 22, 2022, 1:37 AM Lars Ekman
I have started working on a PR to remove the module checks. You are the only active reviewer and I will try my best to divide it into small reviewable commits as you wrote in :smile:
PR to remove the module checks I also intend to remove the constraint on ipvs schedulers, which will fix this issue. If you can create a VS with your configured scheduler you can use it, whatever that scheduler may be.
Looks like it is. Please read I have tested "mh" in kube-proxy (with a patch) and first problem is that is hashes on address only by default, so a traffic generator using the same source all the time will end up on the same endpoint.

This can be configured but will require special treatment in kube-proxy.
Should we close this now that is in?
/triage accepted
No, just removes the module ckeck. The validator still restricts the usable schedulers
to add afinity without using states. is the finality of maglev to avoid perf on managing state... and is more robust on target lost than other consistant hashing method... is normal is what you whant when you use maglev... to have afinity on ip src... but without need of storing states... is afinity by consistant hashing... cilium use this algorithm too....
the google research paper on the algorythm:
If you can build and use an own please try PR I have tested and "mh" can be configured, and with many src-addresses load-balancing works. However if external ingress traffic is re-routed from one K8s node to another, all connections are lost, same as "rr".
with ipvs the node who receive the traffic is considered a director, if you LB to other node... this node must have default route to the director because of nat masquerading of ipvs... when you use MH/RR with Direct Server Return avoid that problem (no more nat masquerading)... the node respond to the client directly (no need to return back to the first node who receive the traffic). but you need to tunnel the traffic from node who receive the request to the node you forward the traffic (cloud flare use fou protocol for that)... i dont know if kubeproxy already support that sort of configuration... cilium for LB DSR + Maglev use tcp option field to store the Virtual IP that receive the traffic in the packet and nat the destination with the pod ip before sending to the node who host the node (that work in full routing mode), and the node respond to the client directly in replacing thaire ip with the Virtual IP from tcp option field.... in that way no state needed... no tunneling... no complex network configuration.... some has been done on kube-router project: and i think you must use in conjonction with MH/RR to use Direct Server Return if i understand whell... and good exemple of the use case : that use ipvs (use ipip tunnel) to do that....
Regarding "mh" algorithm, I submitted PR a few weeks ago which adds the new load distribution method to kube-proxy.

As this commit supports 'mh" with hashing on srcIP and srcPort by default, connections from the same srcIP with different srcPorts will be distributed equally to all EndPoints. If users want to have an affinity on srcIP, they can use "mh" with "Session Affinity" option enabled. The PR is targeted for the main use-cases below: Connection failover from a worker node to another. Scale-in/out worker nodes Rerouting from GWs or external LoadBalancer changes receiving worker nodes of existing traffic flows. I would need to rebase the commit after your fix for this issue PR . PR is now waiting triage state, please verify if it is reasonable to add this new algorithm to kube-proxy.
No, and it probably never will. The proper (only) way to get this into is to write a (KEP). To :
I would also like to see a demo or e2e test working with this setup, it doesn't seem obvious from the comments that this will work out of the box ... The project Kubernetes, as Lars correctly mentions , use KEPs to discuss and document feature, I agree with him and I think this will require to write a KEP
What I meant was your use-case in K8s. Having worked most of 2021 with the Maglev hash algorithm when developing the I know what it can do, but I also know K8s and I think I can say right now; To allow the "mh" in kube-proxy=ipvs will not give you what you hope for.
And a side-remark on DSR. Cilium can do this because they modify packets with eBPF. The traditional way of doing DSR is to actually assign the Virtual IP address (VIP) to an interface in the server (the POD in the K8s case). This must have support in the CNI-plugin, which I guess is the case for as it handles load-balancing and is a CNI-plugin. To add generic support for DSR in K8s is not simple.
I'm new to this community. Thanks for your suggestion about the process of adding a new feature to kube-proxy. I'll prepare a KEP for the "mh" load distribution.
But in any case I would like PR to be merged. For sanity reasons if not others.

The for scheduler=mh can/should be in a separate issue+PR. This will allow users to test "mh" in kube-proxy, because; Nobody will take my word for it :smile:
I don't want to sound discouraging but IMO you should not spend too much time on a KEP before presenting it to get some early feedback. I can't see how "mh" can provide the statelessness unless DSR is used, and that may be very hard to get into the . So, I suggest you describe a use-case in the KEP, and what it takes to support it, and present it on a bi-weekly sig/network meeting. And don't worry, people are nice there :smile:
Perhaps DSR is not needed to use with "mh" since I cannot really see any problem with nat masquerading of ipvs if subsequence packets are sent to the second node. With net/ipv4/vs/sloppy_* enabled, ipvs on the second node accepts the packets as an existing connection and then keeps forwarding them to the original endpoint thanks to the mh algorithm. IPVS of course performs snat and "remember" the client IP of the packets as if they are sent to the first node. When receiving replies from the endpoint, ipvs on the second node will be able to forward the replies to the client with the cached client IP in its NAT table. Please correct me if I'm missing something in this regard.
So even if the kube-proxy forwards packets to the same POD (assuming the same mh configuration in node A and B) the receiving POD will reject with a TCP-RST packet.
Since you have written PR you should be able to test this with you own-built kube-proxy. For me all connections are teminated when re-directing ingress traffic from one K8s node to another.
Thanks for your clarification. Now I could see this problem is on the pod side, not ipvs. It seems that having DSR does not help to solve the problem. Though, I suppose this issue will not happen with the following scenarios: UDP traffic Traffic is sent to Cluster-IP instead of NodeIP
With DSR the src=VIP all the way, so it should be OK, even if masq (DNAT) is used.

UDP same thing, unless your UDP server has no session and doesn't care about addresses, just the server port. This is however not the usual case. Traffic sent to the Cluster-IP will be have src=pod, yes, but it will always be hadled by the local kube-proxy, so the use-case to move traffic to another node doesn't exist for clusterIP.
Me neither :) Agree - you will have the best experience if do this iteratively. Start with an email to the SIG list. Then a doc discussing the peoblem and high-level solution details. THEN a KEP makes sense.
cilium assign vip also but more higher in the kernel stack with ebpf... in that way is more simple to implemente than in ipvs... juste simple packet manipulation (plus the fact that is working only in full routed mode)... ipvs can do that, helped with some config on each node (configuring tunnel for dsr)... for mh usage without dsr we need to add default route on other node to the actual dedicated kubeproxy as l4 director (one node can be designated as director and move to another node when the first one die). is generaly more for loadbalancer/external ip usage....
Is this an acceptable way? merged. This will allow "mh" to be configured as a ipvs scheduler, but only with L3 hashing (no ports). a new issue like "Ipvs scheduler=mh only hashes on addresses, not ports", and implement it in a new (small) PR with FlagSourceHash as have already done in this issue as "done" since "mh" can be configured and as it's implemented in another way. the work towards a KEP to make "mh" work as you want it to work. Including a K8s use-case, DSR (if needed in some form), setting of sysctl's, etc. I leave item 4. to you and . Item 1-3 will enable "mh" but because of NAT, connections are still lost when traffic is re-directed from one node to another (this can be tested without too much effort. I have done it).
Agree with the approach. fixes the raised issue that it enables all distribution methods supported by ipvs in kube-proxy (including "mh" one).

A separate issue for enabling source port hash when using 'mh' sounds reasonable. This is not only applicable for "mh", but also for the "sh" method which kube-proxy is currently supporting as stated in k8s documents. Agree. Sure. I'll need to verify the issue further and check if any easy way possible to avoid the NAT issue on different nodes. IMHO, replacing VIP (virtual IP) with NodeIP seems to be the first option to overcome the issue, but we need to verify its feasibility and impact in k8s.
To allow the "mh" in kube-proxy=ipvs will not give you what you hope for. Nobody will take my word for it :smile: Ok, ok, please let me rephrase: To just allow "mh" in the current ipvs proxier will not give you what you hope for. So, is not enough.
/kind feature


Question: return utilfeature.DefaultMutableFeatureGate.SetFromMap(o.config.FeatureGates) } <ins> // copyLogsFromFlags applies the logging flags from the given flag set to the given // configuration. Fields for which the corresponding flag was not used are left // unmodified. For fields that have multiple values (like vmodule), the values from // the flags get joined so that the command line flags have priority. // // TODO (pohly): move this to logsapi func copyLogsFromFlags(from *pflag.FlagSet, to *logsapi.LoggingConfiguration) error { var cloneFS pflag.FlagSet logsapi.AddFlags(to, &cloneFS) vmodule := to.VModule to.VModule = nil var err error cloneFS.VisitAll(func(f *pflag.Flag) { if err != nil { return } fsFlag := from.Lookup(f.Name) if fsFlag == nil { err = fmt.Errorf("logging flag %s not found in flag set", f.Name) return } if !fsFlag.Changed { return } if setErr := f.Value.Set(fsFlag.Value.String()); setErr != nil { err = fmt.Errorf("copying flag %s value: %v", f.Name, setErr) return } }) to.VModule = append(to.VModule, vmodule...) return err } </ins> // Creates a new filesystem watcher and adds watches for the config file. func (o *Options) initWatcher() error { fswatcher := filesystem.NewFsnotifyWatcher()

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: } if a.usernameClaim == "email" { <del> // Check the email_verified claim to ensure the email is valid. </del> <ins> // If the email_verified claim is present, ensure the email is valid. </ins> // https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims <del> var emailVerified bool if err := c.unmarshalClaim("email_verified", &emailVerified); err != nil { return nil, false, fmt.Errorf("oidc: parse 'email_verified' claim: %v", err) } if !emailVerified { return nil, false, fmt.Errorf("oidc: email not verified") </del> <ins> if hasEmailVerified := c.hasClaim("email_verified"); hasEmailVerified { var emailVerified bool if err := c.unmarshalClaim("email_verified", &emailVerified); err != nil { return nil, false, fmt.Errorf("oidc: parse 'email_verified' claim: %v", err) } // If the email_verified claim is present we have to verify it is set to `true`. if !emailVerified { return nil, false, fmt.Errorf("oidc: email not verified") } </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: This is related to I am trying to integrate Cloud Foundry UAA () with kubernetes using the OIDC flow and the JWT validation is failing on the kubernetes side with an error of emailverified is required. emailverified is not a required claim in the OIDC core spec From a security point of view emailverified is useful when Self Service is enabled which is true of internet identity providers User Provisioning workflow in almost all cases is admin driven and not self service and emailverified becomes an obsolete claim. With things as they stand today, this flow is not conducive to Enterprise Identity Providers. What you expected to happen: Authentication should succeeed How to reproduce it (as minimally and precisely as possible): Pass a JWT which is missing email_verified claim using the OIDC flow into kubernetes Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-16T03:15:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's not marked optional either, which was the original motivation for taking the safe route and assuming false if it wasn't provided. This would be a feature for us to detect the presence of the claim and act differently if it isn't there. Do any other providers do this? Almost all that I've worked with do support this claim.
Hi , Enterprise providers like ADFS , Okta don't support this as part of the user schema. There is the concept of whether a use is active/enabled which is not the same as email verified always.

If a user is not active, a token should not be minted in the first place. I am trying to understand what additional security does email_verified provided in the context of entire identity providers where there is no self self service use creation. Thanks, Sree
Okta supports , but it looks like ADFS doesn't: We don't know when the provider is self-service or not. Not assuming makes the client logic uniform and less error prone instead of trying to reason about the intent of the provider. There's some discussion when we enabled this validation, including some UAA notes. , specifically
More than 80% of the enterprise customers of Cloud Foundry are using either AD/ADFS . This becomes a breaking change. UAA can add a email_verified claim. However its going to be of little value if the underlying AD doesn't propagate that. My two cents is this check doesn't make sense in an enterprise set up
has agreed to take this (can't assign her because she's not part of the org) /sig auth /kind feature /milestone v1.11 /priority important-longterm /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: est.lock.Lock() defer est.lock.Unlock() <del> rrv := est.relatedResourceVersions(endpointSlice) _, ok := rrv[endpointSlice.Name] </del> <ins> rrv, ok := est.relatedResourceVersions(endpointSlice) if !ok { return false } _, ok = rrv[endpointSlice.Name] </ins> return ok }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: klog.V(3).Infof("ensureVMSSInPool: found vmss %s being deleted, skipping", vmssName) continue } <del> if err != nil { return err } </del> if vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations == nil { klog.V(4).Infof("EnsureHostInPool: cannot obtain the primary network interface configuration, of vmss %s", vmssName) continue

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-controller-manager panic when wrong Azure VMSS name is configured: This is because the error is not validated first in the following code path: Azure cloud provider should check whether err happens and whether vmss is nil here. What you expected to happen: Azure cloud provider should report errors on such case, so that customers could get the root raw error and then fix the configuration. How to reproduce it (as minimally and precisely as possible): Create Kuberenetes cluster with basic load balancer, configure a wrong VMSS name and then create/delete LoadBalancer typed services. Anything else we need to know?: The issue also happens on older versions, so the fixes need to cherry-pick on other branches. /area provider/azure /sig cloud-provider /milestone v1.20 Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign


Question: if [[ "${NETWORK_PROVIDER:-}" == "kubenet" ]] ||  [[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then #TODO(andyzheng0831): We should make the cni version number as a k8s env variable. <del> local -r cni_tar="cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz" download-or-bust "" "https://storage.googleapis.com/kubernetes-release/network-plugins/${cni_tar}" </del> <ins> local -r cni_tar="cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz" local -r cni_sha1="19d49f7b2b99cd2493d5ae0ace896c64e289ccbb" download-or-bust "${cni_sha1}" "https://storage.googleapis.com/kubernetes-release/network-plugins/${cni_tar}" </ins> local -r cni_dir="${KUBE_HOME}/cni" mkdir -p "${cni_dir}" tar xzf "${KUBE_HOME}/${cni_tar}" -C "${cni_dir}" --overwrite

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: saw_fatal = true panic(e) }) <del> cmd.SetArgs([]string{test.arg}) </del> <ins> cmd.SetArgs(strings.Split(test.arg, " ")) </ins> cmd.Execute() }()

They should be consistent with all other commands that allow multiple arguments (since they already allow label selectors). should cordon all of them, not return an error.
Hi , can I take this if no one has been working on it?
Please also refer . These two and should take multiple arguments as it is very common to get subset of nodes in this way . So it would be very nice if these would work:


Question: glog.V(2).Info("Using iptables Proxier.") execer := exec.New() <del> proxierIptables, err := iptables.NewProxier(utiliptables.New(execer, protocol), execer, s.SyncPeriod) </del> <ins> proxierIptables, err := iptables.NewProxier(utiliptables.New(execer, protocol), execer, s.SyncPeriod, s.MasqueradeAll) </ins> if err != nil { glog.Fatalf("Unable to create proxier: %v", err) }

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: // Here we'll firstly ensure service do not lie in the opposite LB. serviceName := getServiceName(service) glog.V(5).Infof("ensureloadbalancer(%s): START clusterName=%q", serviceName, clusterName) <del> flippedService := flipServiceInternalAnnotation(service) if _, err := az.reconcileLoadBalancer(clusterName, flippedService, nil, false /* wantLb */); err != nil { return nil, err } if _, err := az.reconcilePublicIP(clusterName, service, true /* wantLb */); err != nil { return nil, err } </del> lb, err := az.reconcileLoadBalancer(clusterName, service, nodes, true /* wantLb */) if err != nil {

/kind bug What happened: ILB Frontend IP configuration does not change while updating service What you expected to happen: LB Frontend IP configuration should get updated How to reproduce it (as minimally and precisely as possible): Source issue : AKS service is created using .yaml manifest. ILB is described as it is shown in the documentation: annotations: "true" spec: ports: port: 80 type: LoadBalancer loadBalancerIP: 10.240.200.2 ILB instance with the "kubernetes-internal" name is created with a correct private IP address pointing to the service described. If I change the "loadBalancerIP" to another internal IP address and re-deploy the service, the last is deployed with the correct new IP address, but "kubernetes-internal" ILB still has the old Frontend IP configuration. Environment: Cloud provider or hardware configuration: Azure
/sig azure
This also not works for public LB. /cc Could you fix the bug and cherry pick to old versions?
I got this bug today, I changed the external LoadBalancerIP on a LoadBalancer service. The service event log showed that it immediately recognised the change ("oldIP =newIP") but I had to delete/recreate the service to see the change in the Azure loadbalancer.