Question: if util.NewStringSet(fields...).IsSuperset(argset) { return true, nil } <ins> glog.V(5).Infof("DBG: fields is not a superset of args: fields=%v  args=%v", fields, args) </ins> } }

I am running kubernetes 0.5.2 + and I've noticed that the number of entries in iptables increases every few seconds. I started with a clean install, no pods or replicationControllers, there is now two services by default (kubernetes-ro and kubernetes). I ran and got . Next, I turned up 7 services and 3 replicationControllers which resulted in 3 pods also being turned up. I started a watch... I saw "38", "43, "50, "64", "66", "68", "70", ..., "115" and on until I stopped . It appears the proxy continues to add REDIRECT's for the services, for example the lines that keep being look like: A similar line for each service gets every second or so, it appears to all be in the table. I'm thinking the newly function might be the culprit, since calls and in the log I see every few seconds (looks like does a sync every 5 seconds).
I noticed that the 0.5.3 and 0.5.4 releases don't include the , though it's in master. Perhaps the issue I've filed is already known and I missed it when searching the issues.
I have not heard this issue. Can you paste a listing of iptables-save -t nat when you are seeing this problem? Alternatively, I can cut you a branch with some extra debug instrumentation if you're willing to test it out today...
I'm not able to test it out today. As far as the iptables output, I'd have to type it in (can't copy/paste -- different physical network). However, the output has lines, like the one given in the issue, which are every 5 seconds to the "nat" table. It seems like the check to see if the rule already exists is perhaps not functioning properly. I am working on a fix for (because it uses the iptables -C flag which isn't supported in older iptables), in that fix I created a function to check if an existing rule already exists: Example call: Using may not be ideal, perhaps breaking up the output of the iptables command into a slice and checking each line would be better(?) sure yet.
Fix is pending - would be great if someone could verify it against an actually old iptables (I just faked the version string).

should be able to try it out friday, maybe monday
Feel free to rip off our checks with out echeck function :) On Nov 26, 2014 7:36 PM, "Craig Wickesser" wrote:
that it actually works. On Nov 26, 2014 7:40 PM, "Tim Hockin" wrote:
I fired up kuberenetes 0.5.2 (plus + your changes) and things looked good initially (when starting up kubernetes, no pods/services, etc). The number of entries iptables-save remained constant. I then turned up some services and watched the length of iptables-save increase. I turned down the services and the number of entries in iptables-save continues to increase. At the moment, every 5 seconds it's back to adding two entries (one for the kubernetes service and one for the kubernetes-ro service). With log level set to 0, I see "Opened iptables portal for service.." messages in the proxy log every 5 seconds ().
Are you sure you have the right proxy version running? I could repro the initial problem, but not this - it works just fine. To be clear when you say it increases after creating a service - do you mean it increases by 1 rule or that it increases continually? , Craig Wickesser wrote:
On Monday/Tuesday I'll try the latest from master, perhaps I messed something up. To clarify, the increase was continuous. When I initially setup kubernetes (with a fresh/clean etcd), things looked normal with regards to iptables. I then deployed some pods and services and kept watching iptables-save. The number of entries increased every 5 seconds. I deleted pods, replication controllers and services and the number of entries continued to increase, but only by two (for the kubernetes and kubernetes-ro services) every 5 seconds.
Did you paste the output of iptables-save? Its possible that it is slightly different from my version. This is why parsing text is no fun :( On Nov 28, 2014 3:42 PM, "Craig Wickesser" wrote:
I did not include the output of iptables-save. I'll see if I can get additional output besides the line included in the initial issue.
Just a couple lines so I can see the format.

On Nov 28, 2014 7:32 PM, "Craig Wickesser" wrote:
here's a few lines from : All of the lines follow the exact same format. Looks like the proxier (to fit with the output I provided) would need to look like this:
FWIW - I'm using CentOS 6.5 with iptables version 1.4.7.
I recompiled with changes to proxier as mentioned earlier (changed the order of args) and did not resolve the issue. I continue seeing (in the proxy log):
I've some additional logging to log the and the . I am logging them using these two lines: When I reduce the output between the and the (when it should return "true", that fields is a superset), the result looks like: Notice has quotes around it in the set, and the does not? I think perhaps that's the issue and that's why the check is false.
I just tried changing to: and that's working. Basically, strip out quotes from the line. There might be a better alternative.
Uggh, seriously? Your iptables add quotes around the comment, mine omits the quotes for single tokens. I guess I can force it to not be a single word, something like "service: " service name. What a hack. Any better ideas? , Craig Wickesser wrote:
Hmm, it seems worse than that. If I feed in a comment with quotes, I get ""foo"", so I can't do that to be compatible with your implementation. Can you try something? iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment "foo1" iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment '"foo2"' iptables-save | grep foo -A PREROUTING -d 1.2.3.4/32 -m comment --comment ""foo2"" -A PREROUTING -d 1.2.3.4/32 -m comment --comment foo1 , Tim Hockin wrote:
i'll try that in about 8 hours :)
here's what I got:
ugh, that is SO BROKEN. I'm going to have to do something really unpleasant.

, Craig Wickesser wrote:
I have a potential fix, but airport wifi prevents me from pushing git <shrug. will publish it later.
sounds good, I appreciate the effort.
PR posted - please test before we commit.
I've tried the PR and a comment to it.
got the same problem here using Kubernetes v0.6.0-12-gd927031aecdeb6 on redhat 6.5
It looks like 0.6 was cut before - - can we patch that in, or should we wait of 0.7? , polariss0i wrote:
got the same problem on kubernetes v1.0.3 and iptables-v1.4.7. Should I file a new issue?


Question: <del> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "os" "os/exec" "path/filepath" "strings" "testing" ) // Can't just call Chroot() because it will affect other tests in the // same process.  Golang makes it hard to just call fork(), so instead // we exec ourselves again, and run the test in the new subprocess. func testChrootReal(t *testing.T) { testfile := filepath.FromSlash("/" + filepath.Base(os.Args[0])) dir := filepath.Dir(os.Args[0]) if dir == "." { t.Skip("skipping: running test at root somehow") } if err := Chroot(dir); err != nil { if strings.Contains(err.Error(), "operation not permitted") { t.Skip("skipping: insufficient permissions to chroot") } t.Fatalf("chroot error: %v", err) } // All file access should now be relative to `dir` if _, err := os.Stat(testfile); err != nil { t.Errorf("Expected file %q to exist, but got %v", testfile, err) } } func TestChroot(t *testing.T) { if os.Getenv("GO_TEST_CHROOT_FOR_REALZ") == "1" { testChrootReal(t) return } cmd := exec.Command(os.Args[0], "-test.v", "-test.run=TestChroot") cmd.Env = []string{"GO_TEST_CHROOT_FOR_REALZ=1"} out, err := cmd.Output() t.Logf("subprocess output:n%s", out) if err != nil { t.Errorf("subprocess error: %v", err) } } </del>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: () Since when has it been failing: Failing 4/27. Testgrid link: Reason for failure: /kind failing-test /priority critical-urgent /milestone v1.15 /sig cluster-lifecycle /cc
cc
testing locally
PR sent: feel free to LGTM.
Thank you very much and for jumping on this so quickly


Question: func TestDecoratedWatcher(t *testing.T) { w := watch.NewFake() decorator := func(obj runtime.Object) { <del> pod := obj.(*example.Pod) pod.Annotations = map[string]string{"decorated": "true"} </del> <ins> if pod, ok := obj.(*example.Pod); ok { pod.Annotations = map[string]string{"decorated": "true"} } </ins> } <del> dw := newDecoratedWatcher(w, decorator) </del> <ins> ctx, cancel := context.WithCancel(context.Background()) dw := newDecoratedWatcher(ctx, w, decorator) </ins> defer dw.Stop() <del> go w.Add(&example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) </del> <ins> go func() { w.Error(&metav1.Status{Status: "Failure"}) w.Add(&example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) w.Error(&metav1.Status{Status: "Failure"}) w.Modify(&example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) w.Error(&metav1.Status{Status: "Failure"}) w.Delete(&example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) }() expectErrorEvent(t, dw) // expect error is plumbed and doesn't force close the watcher expectPodEvent(t, dw, watch.Added) expectErrorEvent(t, dw) // expect error is plumbed and doesn't force close the watcher expectPodEvent(t, dw, watch.Modified) expectErrorEvent(t, dw) // expect error is plumbed and doesn't force close the watcher expectPodEvent(t, dw, watch.Deleted) // cancel the passed-in context to simulate request timeout cancel() // expect the decorated channel to be closed select { case e, ok := <-dw.ResultChan(): if ok { t.Errorf("expected result chan closed, got %#v", e) } case <-time.After(wait.ForeverTestTimeout): t.Errorf("timeout after %v", wait.ForeverTestTimeout) } // expect the underlying watcher to have been stopped as a result of the context cancellation if !w.IsStopped() { t.Errorf("expected underlying watcher to be stopped") } } func expectPodEvent(t *testing.T, dw *decoratedWatcher, watchType watch.EventType) { </ins> select { case e := <-dw.ResultChan(): pod, ok := e.Object.(*example.Pod) if !ok { <del> t.Errorf("Should received object of type *api.Pod, get type (%T)", e.Object) return </del> <ins> t.Fatalf("Should received object of type *api.Pod, get type (%T)", e.Object) </ins> } if pod.Annotations["decorated"] != "true" { <del> t.Errorf("pod.Annotations["decorated"], want=%s, get=%s", "true", pod.Labels["decorated"]) </del> <ins> t.Fatalf("pod.Annotations["decorated"], want=%s, get=%s", "true", pod.Labels["decorated"]) } if e.Type != watchType { t.Fatalf("expected type %s, got %s", watchType, e.Type) </ins> } case <-time.After(wait.ForeverTestTimeout): <del> t.Errorf("timeout after %v", wait.ForeverTestTimeout) </del> <ins> t.Fatalf("timeout after %v", wait.ForeverTestTimeout) } } func expectErrorEvent(t *testing.T, dw *decoratedWatcher) { select { case e := <-dw.ResultChan(): _, ok := e.Object.(*metav1.Status) if !ok { t.Fatalf("Should received object of type *metav1.Status, get type (%T)", e.Object) } if e.Type != watch.Error { t.Fatalf("expected type %s, got %s", watch.Error, e.Type) } case <-time.After(wait.ForeverTestTimeout): t.Fatalf("timeout after %v", wait.ForeverTestTimeout) </ins> } }

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: glog.V(2).Infof("Deleted obsolete listener: %s", listener.ID) } <del> status := &v1.LoadBalancerStatus{} status.Ingress = []v1.LoadBalancerIngress{{IP: loadbalancer.VipAddress}} </del> portID := loadbalancer.VipPortID floatIP, err := getFloatingIPByPortID(lbaas.network, portID) if err != nil && err != ErrNotFound {

When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).


Question: func (rl *respLogger) Log() { latency := time.Since(rl.startTime) if glog.V(2) { <del> glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) </del> <ins> if !rl.hijacked { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } else { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) hijacked [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } </ins> } }

From kube-, Kube-apiserver returns 200 as Success code upon some requests, otherwise return 0 as Success. I1102 20:11:33. 5 ] GET /api/v1/nodes/kubernetes-minion-3io1: (2.683448ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:33. 5 ] PUT /api/v1/nodes/kubernetes-minion-3io1/status: (5.474235ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:34. 5 ] GET /api/v1/nodes: (2.139174ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:37. 5 ] GET /api/v1/nodes/kubernetes-minion-jmxr: (1.452673ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:37. 5 ] PUT /api/v1/nodes/kubernetes-minion-jmxr/status: (4.833358ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:38. 5 ] GET /healthz: (104.272µs) 0 [[Go 1.1 package http] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998412938s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39.

5 ] GET /api/v1/watch/persistentvolumeclaims?resourceVersion=1687: (10.000876825s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998155747s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/namespaces?resourceVersion=1667: (59.977954152s) 0 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:39. 5 ] GET /api/v1/nodes: (1.471292ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (884.225µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumeclaims: (942.355µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (1.065416ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/namespaces: (1.441822ms) 200 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:40.

5 ] GET /api/v1/nodes: (1.958739ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:]
If it were only the watches I would understand, but it's pretty weird for /healthz.
Looking into this, this is likely a false alarm due to an error in pkg/httplog. It assumes that everything calls does. In this case, the status is defaulted to 200. From the So when you just write, the httplog package doesn't see the status and assumes it's 0. I'm going to work on reproducing, confirming, then fixing this.
I confirmed with curl that this is just a logging error at least for healthz. Status code actually returns was 200. This is also pretty obvious since someone writing a 0 status code would result in a stacktrace in the log.
Watches are a different issue than healthz. I think watches are hijacked, and that kinda disables all status recording ability by this package. Do we care to fix that?
websockets hijack-- I thought watches did ordinary HTTP chunked encoding? , krousey wrote:
Probably, but the way it's currently implemented, hijacking completely bypasses this middle-ware logging layer we have.
Maybe we detect hijacking and log that instead of false data? e.g. would become
works for me! , krousey wrote:
Ok out for review in


Question: dnsImage := GetGenericArchImage(cfg.ImageRepository, "k8s-dns-kube-dns", constants.KubeDNSVersion) if features.Enabled(cfg.FeatureGates, features.CoreDNS) { <del> dnsImage = GetGenericArchImage(cfg.ImageRepository, "coredns", constants.CoreDNSVersion) </del> <ins> dnsImage = fmt.Sprintf("%s/%s:%s", cfg.ImageRepository, constants.CoreDNS, constants.CoreDNSVersion) </ins> } imgs = append(imgs, dnsImage) return imgs

Is this a BUG REPORT or FEATURE REQUEST?: What happened: The output of my kubeadm: When I pull the image with docker pull, I get such output: But other images are successfully pulled as we can see from previous log. Why?
/sig cluster-lifecycle cc
Ugh can you fix this please? I missed this in the PR that just got merged, should be without the arch suffix as this is a manifest list
Getting on with this ASAP.
fyi ,


Question: import ( "fmt" <ins> "sync" </ins> "testing" "github.com/coreos/go-etcd/etcd"

Detected by
Create a Pod as follows: Name: jenkins Using image: jenkins In a new Kubenetes namespace named website-frontend kubectl create ns website-frontend apiVersion: v1 kind: Pod metadata: name: jenkins namespace: website-frontend spec: containers: name: jenkins image: jenkins kubectl create -f


Question: - name: kube-addon-manager # When updating version also bump it in cluster/images/hyperkube/static-pods/addon-manager.json image: gcr.io/google-containers/kube-addon-manager:v5.2 <ins> command: - /bin/bash - -c - /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </ins> resources: requests: cpu: 5m

Addon Manager was broken a few times recently by some indirect-related changes(like ) made in the code base. Addon Manager's logs would be very helpful for narrowing down the problem. And I don't think there is an obvious way to retrieve Addon Manager's logs for Jenkins tests currently. This can be achieved in the same way as what we did for the components like and (mounting /var/log). I will try it through this way for addon manager. cc


Question: k.BirthCry() go kubelet.GarbageCollectLoop(k) <del> go kubelet.MonitorCAdvisor(k) </del> <ins> go kubelet.MonitorCAdvisor(k, kc.CAdvisorPort) </ins> kubelet.InitHealthChecking(k) return k

Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?


Question: "k8s.io/apimachinery/pkg/types" utilsets "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apiserver/pkg/audit" <ins> "k8s.io/apiserver/pkg/authentication/user" </ins> "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"

Currently we don't have an easy way of determining what number of requests are apiserver self-requests. This has historically bitten us, since we used to have a 50(?) QPS limit on apiserver self-requests (which thus has been removed). Even so, it would be a good idea to know this generally, since this may be helpful in pinpointing performance regressions for apiserver. /sig api-machinery instrumentation
Can this issue be considered a newbie one? I'm hunting for issues which might be a good first PR.
In the triage meeting was mentioned that would be good to have similar issues to track other well known actors like controller-manager and kubelet. But let's keep this one for api-server calling itself to have it done. /cc
/cc


Question: "//pkg/apis/core/validation:go_default_library", "//pkg/kubelet/types:go_default_library", "//pkg/securitycontext:go_default_library", <ins> "//vendor/github.com/stretchr/testify/assert:go_default_library", </ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a static pod for etcd as following with vsphere as cloud provider: But the hostname is using upper name, and when kubelet start the etcd static pod, it will report error as follows: I think that the vsphere should not return upper case hostname, but only lower case hostname. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area cloudprovider
Here is a quick
/sig node


Question: m.allocFunc = f } <del> // Start starts the gRPC server of the device plugin </del> <ins> // Start starts the gRPC server of the device plugin. Can only // be called once. </ins> func (m *Stub) Start() error { err := m.cleanup() if err != nil {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: | xargs -n1 dirname                                              | LC_ALL=C sort -u                                               ) CONVERSION_FILES := $(addsuffix /$(CONVERSION_FILENAME), $(CONVERSION_DIRS)) <del> CONVERSION_EXTRA_PEER_DIRS := k8s.io/kubernetes/pkg/apis/core,k8s.io/kubernetes/pkg/apis/core/v1,k8s.io/api/core/v1 </del> <ins> CONVERSION_EXTRA_PEER_PKGS :=  k8s.io/kubernetes/pkg/apis/core  k8s.io/kubernetes/pkg/apis/core/v1  k8s.io/api/core/v1 CONVERSION_EXTRA_PKGS := $(addprefix $(PRJ_SRC_PATH)/, $(CONVERSION_DIRS)) </ins> # Reset the list of packages that need generation. $(shell mkdir -p $$(dirname $(META_DIR)/$(CONVERSION_GEN)))

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: } secondaryAllocatorCIDR := r.secondaryServiceIPs.CIDR() <del> if netutil.IsIPv6String(service.Spec.ClusterIP) && netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) { </del> <ins> if netutil.IsIPv6String(service.Spec.ClusterIP) == netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) { </ins> return r.secondaryServiceIPs }

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" <ins> "github.com/onsi/gomega" </ins> ) <del> var _ = SIGDescribe("Ephemeral Containers", func() { </del> <ins> var _ = SIGDescribe("Ephemeral Containers [NodeFeature:EphemeralContainers]", func() { </ins> f := framework.NewDefaultFramework("ephemeral-containers-test") var podClient *framework.PodClient ginkgo.BeforeEach(func() {

E2eNode Suite.[sig-node] Ephemeral Containers will start an ephemeral container in an existing pod node-kubelet-orphans Unknown, possibly was never green No response KEP: /sig node
Also need to fish this test out of the orphans by adding the label
can you please take a look?
yup, happy to /assign
could you explain this more? This label should exist even though the feature is no longer alpha?
Looks like the tests are still passing in the presubmits It looks like the kubectl invocation is asking for a username?
Moved to node features and now the error message is: kubectl isn't available to the node tests? Hmm..
/triage accepted
/reopen The test is flaking now, can you please take a look?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ah, I think I know what the problem is. Didn't get a chance to look at it today but will get to it soon.
FYI, I spot checked several of the recent failures and so far none of them include the fix commit. Will keep an eye on it.
Yes, thank you. Looks very green now


Question: import ( "context" <ins> corev1 "k8s.io/api/core/v1" </ins> discoveryv1beta1 "k8s.io/api/discovery/v1beta1" apiequality "k8s.io/apimachinery/pkg/api/equality" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation/field" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/storage/names" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/api/legacyscheme" <ins> apivalidation "k8s.io/kubernetes/pkg/apis/core/validation" </ins> "k8s.io/kubernetes/pkg/apis/discovery" "k8s.io/kubernetes/pkg/apis/discovery/validation" "k8s.io/kubernetes/pkg/features"

On the first update of an EndpointSlice using the v1 API, we drop deprecatedTopology with EndpointSlice strategy. Although we only do this if at least 1 endpoint has been modified, there is no guarantee that node name has been populated for all endpoints within the EndpointSlice. This is most noticeable if the first update using the v1 API is only adding endpoint(s). In that case the EndpointSlice controller does not update EndpointSlices and instead just adds the new endpoint(s). That will cause node names to be dropped from the deprecated topology field without having the newer node name field populated. Node name to be retained in deprecated topology until the new field was populated. an EndpointSlice with the v1beta1 API. an endpoint in that EndpointSlice: No response This is most notable after upgrading a 1.20 cluster to 1.21, but can technically occur in any 1.21 or newer cluster. Any Any Any N/A N/A
I'm working on a fix for this, hoping to have it out later today. /triage accepted /sig network /cc
I think this is better characterized as a controller bug, but is the safest mitigation I can think of in the short term, since we don't want to rewrite 100% of endpointslices on startup. Can you open a separate issue to work through the controller reconciliation so that v1 clients can depend on the nodeName field more reliably?


Question: A pod containing containers that specify readiness probes will also report the Ready condition. Condition status values may be `True`, `False`, or `Unknown`. <ins> ## Container Probes A [Probe](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes/pkg/api/v1#Probe) is a diagnostic performed periodically by the kubelet on a container. Specifically the diagnostic is one of three [Handlers](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes/pkg/api/v1#Handler): * `ExecAction`: executes a specified command inside the container expecting on success that the command exits with status code 0. * `TCPAction`: performs a tcp check against the container's IP address on a specified port expecting on success that the port is open. * `HTTPGetAction`: performs an HTTP Get againsts the container's IP address on a specified port and path expecting on success that the response has a status code greater than or equal to 200 and less than 400. Each probe will have one of three results: * `Success`: indicates that the container passed the diagnostic. * `Failure`: indicates that the container failed the diagnostic. * `Unknown`: indicates that the diagnostic failed so no action should be taken. Currently, the kubelet optionally performs two independent diagnostics on running containers which trigger action: * `LivenessProbe`: indicates whether the container is *live*, i.e. still running. The LivenessProbe hints to the kubelet when a container is unhealthy. If the LivenessProbe fails, the kubelet will kill the container and the container will be subjected to it's [RestartPolicy](#restartpolicy). The default state of Liveness before the initial delay is "Success". The state of Liveness for a container when no probe is provided is assumed to be "Success". * `ReadinessProbe`: indicates whether the container is *ready* to service requests. If the ReadinessProbe fails, the endpoints controller will remove the pod's IP address from the endpoints of all services that match the pod. Thus, the ReadinessProbe is sometimes useful to signal to the endpoints controller that even though a pod may be running, it should not receive traffic from the proxy (e.g. the container has a long startup time before it starts listening or the container is down for maintenance). The default state of Readiness before the initial delay is "Failure". The state of Readiness for a container when no probe is provided is assumed to be "Success". </ins> ## Container Statuses <del> More detailed information about the current (and previous) container statuses can be found in `containerStatuses`. The information reported depends on the current ContainerState, which may be Waiting, Running, or Termination (sic). </del> <ins> More detailed information about the current (and previous) container statuses can be found in `containerStatuses`. The information reported depends on the current ContainerState, which may be Waiting, Running, or Termination (sic). </ins> ## RestartPolicy

There don't appear to be any explanations of how to use readiness or liveness probes, with exception of one or two sentences in and . Can we get a rough page up with the basics of what these probes are and how they are meant to be used ? What is a probe? What is a livenessProbe vs a readinessProbe ? How are either of these normally used. There is of course the connection to "Readiness" here which is the important part we want to explain. Im playing around some with these things by looking at the api + the examples stuff. But I won't be a good candidate for writing the doc page. Possibly might be able to provide a quick sketch up of how to get the best info on these , if busy, then if you want to dump some links in this thread, I or someone else can investigate some more and write up a more formal docs update.
Ack. I have some downtime today so I can write up a sketch.


Question: podName       string } <del> // func that attempts suicide type jumper func(bindings.ExecutorDriver, <-chan struct{}) type suicideWatcher interface { Next(time.Duration, bindings.ExecutorDriver, jumper) suicideWatcher Reset(time.Duration) bool Stop() bool } </del> type podStatusFunc func() (*api.PodStatus, error) // KubeletInterface consists of the kubelet.Kubelet API's that we actually use

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Followed the documentation on rules to limit group Id's to non-superuser (root) using the rule. The creation attempt failed with: From PSP yaml: What you expected to happen: Successfully create a new PSP for non-superuser (root) group privileges using available rule values (which actually seems to be , , according the the ) How to reproduce it (as minimally and precisely as possible): Attempt to create a PSP with a rule set to Anything else we need to know?: I don't think so Environment: Kubernetes version (use ): , looks to affect , and doc as well Cloud provider or hardware configuration: IBM Cloud OS (e.g: ):Kernel (e.g. ): Install tools: Others:
/sig docs
/assign
thanks will send out a fix next week
I wonder whether this is a bug in implementation or the document. Is this really a documentation issue?
yes, RunAsGroup does not support MustRunAsNonRoot
Got it. Thank you!
/priority important-soon


Question: // Not federated namespace, ignoring. return } <del> baseNamespace := baseNamespaceObj.(*api_v1.Namespace) </del> <ins> // Create a copy before modifying the namespace to prevent race condition with // other readers of namespace from store. namespaceObj, err := conversion.NewCloner().DeepCopy(namespaceObjFromStore) baseNamespace, ok := namespaceObj.(*api_v1.Namespace) if err != nil || !ok { glog.Errorf("Error in retrieving obj from store: %v, %v", ok, err) nc.deliverNamespace(namespace, 0, true) return } </ins> if baseNamespace.DeletionTimestamp != nil { if err := nc.delete(baseNamespace); err != nil { glog.Errorf("Failed to delete %s: %v", namespace, err)

Unit tests on PR are failing due to data race in federation-controller/namespace unittest. More logs: https://k8s-
CC
Sent to fix this


Question: } func (g *gcpAuthProvider) WrapTransport(rt http.RoundTripper) http.RoundTripper { <del> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}} </del> <ins> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}, g.persister} </ins> } func (g *gcpAuthProvider) Login() error { return nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl "does not allow access" Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP GKE 1.4.5 OS (e.g. from /etc/os-release): Not sure, using gci Kernel (e.g. ): Not sure, using gci Install tools: GKE web UI Others: N/A What happened: In juggling multiple gcloud accounts with GKE clusters I accidentally used with the wrong application default credentials and it cached bad creds in After pointing gcloud at the right application default credentials and re-running , it continued trying and failing to use the bad creds cached in ; at this point I was stuck and confused for a few hours To repair, I had to manually edit to remove , and then started working again (or else wait ~1h for the access token to expire and refresh itself) What you expected to happen: After pointing gcloud at the right application default credentials should just work, or at least provide an error msg to the user that can help them figure out that they need to revoke the bad access token and how to do it How to reproduce it (as minimally and precisely as possible): Good scenario: just works Bad scenario: user makes one mistake and gets stuck
+1 Removing helped, thanks.
I can look into fixing this if no-one else has already made a start on it?
feel free to take it, thanks!

Ok, I think I have an understanding of what is happening call to is storing credentials in a json file in a known location. a subsequent credential-requiring kubectl command, the GCP plugin is being loaded, and the application default credentials are being exchanged for a token. That token is then cached in the .kube/config file. So where do things go wrong? As far as I can tell, if I switch to a cluster-2 without refreshing the application default credentials from it's separate account, those existing credentials will still be exchanged successfully for an oauth2 token and cached against cluster-2. Those tokens will then fail against a call to cluster-2 - because they were intended for cluster-1. The only sure-fire fix I can think of would be to ensure the application default credentials client ID is stored in users[].user.authprovider. Doing this would at least enable the ability to verify that the default credentials were valid for that particular cluster, and if not, prompt the user with a message to correctly aquire the application default credentials for it. Lets call this option 1. Of course, there is a problem here with determining what the Client ID is in the first place. I think this would require that a call to would store the client ID in the generated config. This would require exposing the client id from oauth2/google. Otherwise, perhaps storing the client ID the first time around would be enough - assuming the user at least manages to authenticate successfully against the cluster in the first instance. Then only oauth2/google would require a change, and not the gcloud command. The other option (option 2) would be, upon failure, check if is true and if-so, prompt the user that their application default credentials may be invalid for the target cluster, and to either prompt or clear the currently cached credentials in .kube/config. This would probably mean extending the auth provider interface such that a function may be provided that can be called upon failure that enables the auth provider plugin to do some checking of the state and to provide a reasonable message as to what might be wrong with the credentials that have been provided. I feel like option 2 is probably the better option in the long term. Different auth plugins are likely to have different failure modes. Providing a way to communicate back to the user about possible edge cases seems a little more flexible. Any thoughts?

edit: On closer inspection this seems possible by wrapping the oauth2.Transport call in WrapTransport. I would then be able to check for an authorization failure and provide a reasonable error message based on the existing configuration.
Just come across this, its particularly strange to have to edit the config file, when there is a folder in the folder.
Is it really fixed? I still can reproduce it on k8s 1.14 with description provided in initial comment.
I also reproduced this, unfortunately. Tried to use kubectl to access a cluster from an unauthorised account. Then, switched to the authorised account but the invalid authentication was cached. To resolve it, I set the to an expired date.
Believe I'm hitting this issue as well
I just ran into this as well, gcloud 311.0.0, kubectl 1.18.6.
I've got the same issue. I need to change the expiry time to 1 year back as pointed out.
Solution that usually works for me So it will use different credentials when switching contexts
You'll need  installed ().


Question: if esTracker.Stale(tc.checksParam) != tc.expectStale { t.Errorf("tc.tracker.Stale(%+v) == %t, expected %t", tc.checksParam, esTracker.Stale(tc.checksParam), tc.expectStale) } <ins> assert.Equal(t, tc.expectResourceVersionsByService, esTracker.resourceVersionsByService) </ins> }) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: newVM := compute.VirtualMachine{ Location: vm.Location, VirtualMachineProperties: &compute.VirtualMachineProperties{ <ins> HardwareProfile: vm.HardwareProfile, </ins> StorageProfile: &compute.StorageProfile{ DataDisks: &disks, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: vm sku restriction policy does not work in azure disk attach/detach If user set up a vm sku(vm size) restriction policy, only allow set up specified VM size in current subscription, disk attach/detach would fail. Error is like following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign


Question: # cleanup kubectl delete svc prune-svc 2>&1 "${kube_flags[@]:?}" <ins> ## kubectl apply --prune can prune resources not in the defaulted namespace # Pre-Condition: namespace nsb exists; no POD exists kubectl create ns nsb kube::test::get_object_assert pods "{{range.items}}{{${id_field:?}}}:{{end}}" '' # apply a into namespace nsb kubectl apply --namespace nsb -f hack/testdata/prune/a.yaml "${kube_flags[@]:?}" kube::test::get_object_assert 'pods a -n nsb' "{{${id_field:?}}}" 'a' # apply b with namespace kubectl apply --namespace nsb -f hack/testdata/prune/b.yaml "${kube_flags[@]:?}" kube::test::get_object_assert 'pods b -n nsb' "{{${id_field:?}}}" 'b' # apply --prune must prune a kubectl apply --prune --all -f hack/testdata/prune/b.yaml # check wrong pod doesn't exist output_message=$(! kubectl get pods a -n nsb 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' # check right pod exists kube::test::get_object_assert 'pods b -n nsb' "{{${id_field:?}}}" 'b' # cleanup kubectl delete ns nsb ## kubectl apply -n must fail if input file contains namespace other than the one given in -n output_message=$(! kubectl apply -n foo -f hack/testdata/prune/b.yaml 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'the namespace from the provided object "nsb" does not match the namespace "foo".' </ins> ## kubectl apply -f some.yml --force # Pre-condition: no service exists

Not sure if this is a bug, feature requests, or a documentation issue. What happened: I am trying to apply some configs with --prune, and was surprised by the behavior. I have a config file with objects in multiple namespaces, and apply with (note: no flag). By kube context was set to a namespace that was not in the manifests. Here is what I found: If my context is set to a random namespace, nothing is pruned If its set to namespace X, only objects in namespace X are applied If I pass a flag I can control which namespace gets pruned, but that will break the apply because the file has multiple namespaces in it What you expected to happen: I expect some way to be able to apply a config file with objects in multiple namespace and having pruning across all namespaces. Seems there are a few options: all namespaces referenced in the file, prune these across all namespaces warn about why we are not pruning due to namespace mismatch How to reproduce it (as minimally and precisely as possible): Run the script from namespace default, a, and b. You will see different behavior on each run. Related? Environment: Kubernetes version (use ): 1.17.0 Others: /sig cli
I encountered this issue. I think has the bug that o.namespace gets "default" if no namespace is given. Then only a single namespace is visited with or without namespace. If my observation is correct, it changes the bahaviour drastically. In my work, pruning just stops working.


Question: for _, test := range tests { oldNode, expectNewNode := generateNodeAndTaintedNode(test.oldTaints, test.newTaints) new_node := &v1.Node{} tainted := false f, tf, codec, ns := cmdtesting.NewAPIFactory()

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: ) const ( <del> // Taken from lmctfy https://github.com/google/lmctfy/blob/master/lmctfy/controllers/cpu_controller.cc MinShares     = 2 </del> <ins> // These limits are defined in the kernel: // https://github.com/torvalds/linux/blob/0bddd227f3dc55975e2b8dfa7fc6f959b062a2c7/kernel/sched/sched.h#L427-L428 MinShares = 2 MaxShares = 262144 </ins> SharesPerCPU  = 1024 MilliCPUToCPU = 1000

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with.

unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e.

if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.


Question: Description: The ValidatingAdmissionPolicy should type check the expressions defined inside policy. */ <del> framework.ConformanceIt("should type check validation expressions", func(ctx context.Context) { </del> <ins> framework.It("should type check validation expressions", func(ctx context.Context) { </ins> var policy *admissionregistrationv1.ValidatingAdmissionPolicy ginkgo.By("creating the policy with correct types", func() { policy = newValidatingAdmissionPolicyBuilder(f.UniqueName+".correct-policy.example.com").

<img width="1015" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
/priority critical-urgent Looking at the last green and first red shows the following: xref : /assign
hrmm, looks super solid in
yea something screwy going on with the kubeadm jobs :(
my main concern what that this was a test ..
kinder or kind jobs have been mostly solid across the board for a few years. if something is failing in kind / kinder then likely it would also fail in resource constraint scenarios. sorry but my 2c are that this is worth investigating by owning sigs
i don't know what else could be the cause here. kubeadm is crd agnostic.
completely agree this needs resolution, I'm just confused since is a really similar setup and looks completely solid The test waits until the status is set, so even a resource-constrained env should just wait a little longer ... the failures I see are bailing out of that loop after only one wait (105ms), so something is turning that field non-nil without populating the expected results I'm also pretty confused
the diff between the two jobs (kinder/kind) reveals that the kind job has much higher resource limits/requests (for CPU) full diff: <details</details
i'm not familiar with this area of the code. it does appear to be waiting for the TypeChecking struct to be populated, but not for the Warnings slice to be populated. logically (without trying to find the code): if TypeChecking is expected to be populated with Warnings during a Get call after the Create call, we are catching a flake bug. if the Warnings in the TypeChecking are populated async of the TypeChecking then we are catching a test flake and the test must be updated to wait for the Warnings too.
well, the code is here: it appears that the only way for this to happen is if the Check() call returns nil warnings under "some conditions". the problem is not reproducible in the unit test here if the interval is set to a lower value <= 100ms and tested with the tool. but that's with a fake client.

this extra err check seems redundant: it might be worth temporary adding a print of the crd and policy before this line: then allow this test to flake again and inspect the results.
/milestone v1.30
Thanks for raising the issue and thanks for the investigation! I don't know exactly why the solid test in kind would be flaky in kubeadm by a quick scan but agree it needs attention.(The tests look less flaky though..) /assign the type checking code owner. Would you mind adding a printout as suggested above and see if we could get more info? Thanks!
note, both kind and kinder (from the compared jobs) use kubeadm.
and i wonder if this is an informer cache issue.
Same guess as failed for missing event for service patch since 03/06(similar time). [Not sure; just guess]
shows that the test failed for a different reason. failure in ci-kubernetes-e2e-ec2-alpha-enabled-default: from 03/04 to 03/06.
/remove-kind failing-test /kind flake
the API field doc indicates a non-nil value means type checking is complete: It doesn't look like the type checker has any way to signal that it was lacking some information needed to complete type checking, and will record an empty set of warnings if it is: are there any scenarios where a laggy REST mapper or informer can make the controller think it completed type checking but be missing some info that makes it produce an empty list of warnings? I think our options are: the test to wait for non-empty warnings in the status (not just non-nil typeChecking) the controller to not write an empty typeChecking stanza if it fails to type check in some way option 2 seems better for observers, but I don't know how invasive that is
option 2 feels like we are creating a hole? (folks will see warnings sometimes and sometimes they don't?)
Option 2 might cause confusion especially when deleting existing warnings from typechecking field.

(We will remove the typechecking warning if observed the fix in schema)
I'm not suggesting deleting existing warnings, but it appears we are currently reporting an empty typeChecking stanza when we have not actually finished type checking... that doesn't match the API docs and is likely confusing to API clients (it's confusing our own e2e test we wrote based on the documented behavior)
Ignoring errors while finishing the type checking is intended behavior defined by the original KEP. Can we go back to change the tests to do a retry loop until the CRD appears in the RESTMapper? The root problem of laggy RESTMapper seems out of the scope of this test case.
if is fixing the test then it's masking a state not guaranteed by the field comment: i guess already explained that above.
The type checking is completed with this field set to non-nil. The problem is that the type checking finishes with error, and it cause some of the referred types to be ignored.
should this error be surfaced to the user and does that need an e2e test?
No. The KEP defines that errors will be ignored. However, there is a next step to make the type checker reacts on CRD changes, which handles the case when the CRD schema is late for the type checking.
i think users might be confused by this behavior (the same way people on this thread are) even if it's documented in a KEP, so a field comment update feels worth it, at minimum.
The current type checking is informational and only works on ValidatingAdmissionPolicy. If the CRD is created after type checking or CRD created but not showing in schema watcher yet, the error would be ignored. And it will not retry as long as ValidatingAdmissionPolicy remains unchanged. There is a PR for adding retry with schema resolver raised in which will solve the issue entirely. In current case, the guess would be schema watcher is slow so it didn't have crd info yet when performing typechecking. The temporary fix by in will resolve the test flaky by adding manually retry. The field comment of makes sense to me since it indicates the completion of the type checking but not handling the case where the CRD has not been created or created but not showing in schema watcher yet. FYI The says I agree it might not fit in conformance test. Not sure about the conformance test criteria.
cc
/triage accepted


Question: return err } for ix := range objs { <del> if err := s.printOneObject(objs[ix], parsers, w); err != nil { </del> <ins> if err := s.printOneObject(objs[ix], parsers, out); err != nil { </ins> return err } } } else { <del> if err := s.printOneObject(obj, parsers, w); err != nil { </del> <ins> if err := s.printOneObject(obj, parsers, out); err != nil { </ins> return err } } <del> return w.Flush() </del> <ins> return nil </ins> } func (s *CustomColumnsPrinter) printOneObject(obj runtime.Object, parsers []*jsonpath.JSONPath, out io.Writer) error {

Compare the following two outputs (note the bad alignment in the last line): The custom-columns string is exactly the same in both cases, but something messes up the column widths when using openapi print columns. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Columns printed when using are misaligned. What you expected to happen: Columns in all rows should be aligned. How to reproduce it (as minimally and precisely as possible): Deploy Service Catalog, patched with Run the above two commands Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-15T23:06:17Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} (same behavior with kubectl built from current master branch)
/sig api-machinery
/remove-sig api-machinery /sig cli PR is ready:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } func (p *Poller) sync(e Enumerator) { <del> current := p.store.Contains() </del> <ins> current := p.store.ContainedIDs() </ins> for i := 0; i < e.Len(); i++ { id, object := e.Get(i) p.store.Update(id, object)

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: signalToNodeCondition map[evictionapi.Signal]v1.NodeConditionType // signalToResource maps a Signal to its associated Resource. signalToResource map[evictionapi.Signal]v1.ResourceName <del> // resourceToSignal maps a Resource to its associated Signal resourceToSignal map[v1.ResourceName]evictionapi.Signal </del> <ins> // resourceClaimToSignal maps a Resource that can be reclaimed to its associated Signal resourceClaimToSignal map[v1.ResourceName][]evictionapi.Signal </ins> ) func init() {

It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge


Question: if notBestEffort { return lifecycle.PodAdmitResult{Admit: true} } <ins> // When node has memory pressure and TaintNodesByCondition is enabled, check BestEffort Pod's toleration: // admit it if tolerates memory pressure taint, fail for other tolerations, e.g. OutOfDisk. if utilfeature.DefaultFeatureGate.Enabled(features.TaintNodesByCondition) && v1helper.TolerationsTolerateTaint(attrs.Pod.Spec.Tolerations, &v1.Taint{ Key:    algorithm.TaintNodeMemoryPressure, Effect: v1.TaintEffectNoSchedule, }) { return lifecycle.PodAdmitResult{Admit: true} } </ins> } // reject pods when under memory pressure (if pod is best effort), or if under disk pressure.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: According to the preemption's doc, we'll take priority over Qos; but in , the higher priority BestEffort pod maybe rejected because of memory pressure. If TaintNodesByCondition enabled, the pod need memory pressure toleration. /cc , correct me if any mis-understanding.
/sig scheduling /sig node
If the node has memory pressure, it rejects all pods that do not tolerate it. This is orthogonal to QoS and Priority.
Preemption and node evictions do not work with the same exact metrics. Preemption is unaware of usage, while node eviction logic has pod and node resource usage information. Node eviction takes usage into account both when admitting a new pod and when evicting an existing pod. has recently sent out to update node eviction doc.
, but will also reject it if pod tolerate it :(
, please refer to for what I mean :). BTW, you need to configure out and predicates, and enable alpha feature. /cc
I'm not familiar with the Preemption/Priority design, so I don't know where is preemption happening (Node or control plane). But despite that I wouldn't read what you wrote as inconsistency. Priority takes precedence over QoS when it can. But if some QoS Pods are not allowed to run somewhere it's not an issue with preemption/priority or QoS. It's a separate predicate. Slight OT, but related to your proposal, I think we should go another way. We should prevent BE pods to get MemoryPressure toleration. Having this can result with bad user experience, as Pod will get scheduled on a Node, and then immediately killed, because Kubelet ignores tolerations. Or we can teach kubelet respect tolerations for NC created taints. I'd consider it a blocker for beta of TaintNodeByCondition.
Yes; but if BE pod has higher priority, the other pods with lower priority will be evicted by kubelet. This maybe better, no idea on which pod will be BE but with higher priority :) , do you think we should reject BE pod with MemoryPressure toleration, even it has a higher priority?

I would like to mention several points here: of all, I should say that a high priority BE pod is a result of bad configuration and we should not optimize for it, but given that it is not rejected by the system, we must handle it. BE pod will never cause preemption due to lack of resources, because a BE pod needs zero resources from scheduler's point of view. of point , I think we should not worry about a high priority BE pod getting rejected by kubelet or causing other pods to be evicted from a node. We should document the system behavior and have clear guidelines and point out the problems that could be caused by bad configurations. After that we should leave it to the user to decide if a configuration makes sense for their use-case.
Overall, I agree :). Personally, I'd like to keep components behavior align: reject by admission/kubelet or accept by them. Currently, apiserver/scheduler accept and handle it and kubelet will reject it.
Kubelet does not reject a BE pod with high priority either.
I mean BE pod with Memory Pressure toleration :)
I see. The pod may still be scheduled without any problem if another node in the cluster is found that does not have memory pressure. So, I wouldn't call this inconsistent behavior of components. Anyway, I leave it up to SIG node to decide whehter the current behavior should be changed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: } runner := New(&fexec) // Success <del> ok, err := runner.TestEntry(testEntry.String(), "FOOBAR") </del> <ins> ok, err := runner.TestEntry(testEntry.String(), setName) </ins> if err != nil { t.Errorf("expected success, got %v", err) } if fcmd.CombinedOutputCalls != 1 { t.Errorf("expected 2 CombinedOutput() calls, got %d", fcmd.CombinedOutputCalls) } <del> if !sets.NewString(fcmd.CombinedOutputLog[0]...).HasAll("ipset", "test", "FOOBAR", "10.120.7.100,tcp:8080") { </del> <ins> if !sets.NewString(fcmd.CombinedOutputLog[0]...).HasAll("ipset", "test", setName, "10.120.7.100,tcp:8080") { </ins> t.Errorf("wrong CombinedOutput() log, got %s", fcmd.CombinedOutputLog[0]) } if !ok {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--The method only finds string in output of , but when the name of the set contains , it will return no matter the entry exists in ipset or not. What happened: returns false whenever entry exists if the name of set contains , e,g: , , What you expected to happen: returns whether the entry exists whatever the set name is How to reproduce it (as minimally and precisely as possible): replace the set name to any name containing , for example in of , then cd to and run , and it fails on . Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.6, but the bug still exists in Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /kind bug
/assign
This page says that returns non-zero when entry does not exist in set, and zero when entry exists: and I tested the output of (ipset v6.29 on Ubuntu 16.04 LTS, kernel 4.4.0-116-generic),the result goes as follow: so maybe we can judge by return value of command and the separated output of stdout and stderr,


Question: } options.CleanupAndExit = true <del> proxyserver, err := NewProxyServer(options.config, options.CleanupAndExit, options.CleanupIPVS, options.scheme, options.master) </del> <ins> proxyserver, err := NewProxyServer(options) </ins> assert.Nil(t, err, "unexpected error in NewProxyServer, addr: %s", addr) assert.NotNil(t, proxyserver, "nil proxy server obj, addr: %s", addr)

https://k8s- https://k8s- cc: /kind bug /priority failing-test
This is at least twice in a week we've had cross break due to a public method signature change not being brought to all platforms () We should do:
I'm lost - the PR in question adds an argument and fixes the call-site - what is broken?
Ahh, I see.
How did that PR get through, if it is so obviously broken on Cross?
Cross build doesn't run 100% of the time as a pre-submit (because it takes so long), and it can't block. It's something sig-testing is tracking, but there isn't a quick fix yet.
Looks like was skipped for that PR: "pull-kubernetes-cross Skipped" not sure why yet.
Cross runs when a file matching: changes, (which needs to be fixed, opening a test-infra issue to expand the regex), but also some files could have build tags without the file name matching () Additionally cross is not blocking (), the underlying bug should be fixed now though so once we get the fix in to k/k we can make it blocking when it does run.
I find has already sent a PR. Thanks!


Question: return err } <ins> // visitNode handles debugging for node targets by creating a privileged pod running in the host namespaces. // Returns an already created pod and container name for subsequent attach, if applicable. func (o *DebugOptions) visitNode(ctx context.Context, node *corev1.Node) (*corev1.Pod, string, error) { pods := o.podClient.Pods(o.Namespace) newPod, err := pods.Create(ctx, o.generateNodeDebugPod(node.Name), metav1.CreateOptions{}) if err != nil { return nil, "", err } return newPod, newPod.Spec.Containers[0].Name, nil } </ins> // visitPod handles debugging for pod targets by (depending on options): //   1. Creating an ephemeral debug container in an existing pod, OR //   2. Making a copy of pod with certain attributes changed

This issue tracks implementing node debugging with privileged containers for , as described in the This feature allows administrators to start a privileged container on a node in host namespaces rather than having to SSH to the node. For example: /kind feature /sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } func runPausePod(ctx context.Context, f *framework.Framework, conf pausePodConfig) *v1.Pod { <del> return runPausePodWithTimeout(ctx, f, conf, framework.PollShortTimeout) } func runPausePodWithTimeout(ctx context.Context, f *framework.Framework, conf pausePodConfig, timeout time.Duration) *v1.Pod { </del> pod := createPausePod(ctx, f, conf) <del> framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(ctx, f.ClientSet, pod.Name, pod.Namespace, timeout)) </del> <ins> framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(ctx, f.ClientSet, pod.Name, pod.Namespace, f.Timeouts.PodStartShort)) </ins> pod, err := f.ClientSet.CoreV1().Pods(pod.Namespace).Get(ctx, conf.Name, metav1.GetOptions{}) framework.ExpectNoError(err) return pod

master-blocking kind-ipv6-master-parallel RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance] 01-27-2023 16:08 PST https://k8s- { failed [FAILED] timed out while waiting for pod runtimeclass-3517/without-label to be running In [It] at: @ 01/28/23 22:33:45.607 } No response kind cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It looks like that “Kubernetes e2e suite: . Would we want to reopen this issue? I will keep a close eye on to see if it continues to flake.
Reopen to monitor failures cc
/sig node /cc
/retitle [It] [sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taint
Example failure:
Failure The pod was created at , from the kubelet logs: so the pod started at , but it was deleted at because the 1 minute timeout was hit. It looks like the container startup took a while based on containerd logs: there is a gap there of and between and
It is hitting this timeout - That uses which is 1 min which seems to be too short. We should probably use or which is 5min or 2min, because the whole pod needs to start and it seems like 1min is too short in this case.