Question: declTypes     []*apiservercel.DeclType paramGVK      schema.GroupVersionKind paramDeclType *apiservercel.DeclType <ins> variables []v1beta1.Variable </ins> } type typeOverwrite struct {

When I load a policy definition like this The object is updated with the following status While this error is reported, the policy does seem to be fully functioning This is similar to the documented examples No error reported on a valid spec Create a a policy that uses variables Observe the status of the newly created policy No response <details</details<detailsn/a </details<details</details<detailskind </details<details</details<details</details
/sig api-machinery
/assign Oops. Good find. Thank you. /triage accepted


Question: fs.StringVar(&s.NetworkPluginName, "network-plugin", s.NetworkPluginName, "<Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle") fs.StringVar(&s.NetworkPluginDir, "network-plugin-dir", s.NetworkPluginDir, "<Warning: Alpha feature> The full path of the directory in which to search for network plugins") fs.StringVar(&s.VolumePluginDir, "volume-plugin-dir", s.VolumePluginDir, "<Warning: Alpha feature> The full path of the directory in which to search for additional third party volume plugins") <del> fs.StringVar(&s.CloudProvider, "cloud-provider", s.CloudProvider, "The provider for cloud services.  Empty string for no provider.") </del> <ins> fs.StringVar(&s.CloudProvider, "cloud-provider", s.CloudProvider, "The provider for cloud services. By default, kubelet will attempt to auto-detect the cloud provider. Specify empty string for running with no cloud provider. [default=auto-detect]") </ins> fs.StringVar(&s.CloudConfigFile, "cloud-config", s.CloudConfigFile, "The path to the cloud provider configuration file.  Empty string for no configuration file.") fs.StringVar(&s.KubeletCgroups, "resource-container", s.KubeletCgroups, "Optional absolute name of the resource-only container to create and run the Kubelet in.")

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: func TestPersistentVolumeDescriber(t *testing.T) { block := corev1.PersistentVolumeBlock file := corev1.PersistentVolumeFilesystem <ins> foo := "glusterfsendpointname" </ins> deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(-10, 0, 0)} testCases := []struct { name               string

What happened: What you expected to happen: EndpointNamespace should be the string name of the namespace used. How to reproduce it (as minimally and precisely as possible): Create a glusterfs PV using an EndpointNamespace entry and run 'kubectl describe pv' on the resulting PV. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T17:09:13Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): Linux bionic-vm 4.15.0-58-generic -Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.15
/sig cli
/assign


Question: h := &Helper{ Out:                os.Stdout, GracePeriodSeconds: 10, <ins> OnPodDeletionOrEvictionStarted: func(pod *corev1.Pod, usingEviction bool) { if tc.evictionSupported && !tc.disableEviction { if !usingEviction { t.Errorf("%s: OnPodDeletionOrEvictionStarted callback failed while evicting; actualnt%vnexpectednt%v", tc.description, usingEviction, !usingEviction) } } else if tc.evictionSupported && tc.disableEviction { if usingEviction { t.Errorf("%s: OnPodDeletionOrEvictionStarted callback failed while deleting; actualnt%vnexpectednt%v", tc.description, !usingEviction, usingEviction) } } }, OnPodDeletedOrEvicted: func(pod *corev1.Pod, usingEviction bool) { if tc.evictionSupported && !tc.disableEviction { if !usingEviction { t.Errorf("%s: OnPodDeletedOrEvicted callback failed while evicting; actualnt%vnexpectednt%v", tc.description, usingEviction, !usingEviction) } } else if tc.evictionSupported && tc.disableEviction { if usingEviction { t.Errorf("%s: OnPodDeletedOrEvicted callback failed while deleting; actualnt%vnexpectednt%v", tc.description, !usingEviction, usingEviction) } } }, OnPodDeletionOrEvictionFinished: func(pod *corev1.Pod, usingEviction bool, err error) { if tc.evictionSupported && !tc.disableEviction { if !usingEviction { t.Errorf("%s: OnPodDeletionOrEvictionFinished callback failed while evicting; actualnt%vnexpectednt%v", tc.description, usingEviction, !usingEviction) } } else if tc.evictionSupported && tc.disableEviction { if usingEviction { t.Errorf("%s: OnPodDeletionOrEvictionFinished callback failed while deleting; actualnt%vnexpectednt%v", tc.description, !usingEviction, usingEviction) } } }, </ins> } // Create 4 pods, and try to remove the first 2

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: func NewDockerService(client dockertools.DockerInterface, seccompProfileRoot string, podSandboxImage string, streamingConfig *streaming.Config, pluginSettings *NetworkPluginSettings, cgroupsName string, kubeCgroupDriver string, execHandler dockertools.ExecHandler) (DockerService, error) { c := dockertools.NewInstrumentedDockerInterface(client) <ins> checkpointHandler, err := NewPersistentCheckpointHandler() if err != nil { return nil, err } </ins> ds := &dockerService{ seccompProfileRoot: seccompProfileRoot, client:             c,

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: // It's possible that the watcher is already not in the structure (e.g. in case of // simultaneous Stop() and terminateAllWatchers(), but it is safe to call stopLocked() // on a watcher multiple times. <del> c.watchers.deleteWatcher(index, scope, triggerValue, triggerSupported) c.stopWatcherLocked(w) </del> <ins> c.watchers.deleteWatcher(index, scope, triggerValue, triggerSupported, c.stopWatcherLocked) </ins> } }

A data race manifested in job link: NA NA No response No response No response /sig api-machinery
/cc
/assign
ack, fixing ...
/triage accepted


Question: } __EOF__ # Post-condition:redis-master-service service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:service-.*-test:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:service-.*-test:' </ins> # Command kubectl update service "${kube_flags[@]}" service-${version}-test --patch="{"selector":{"my":"test-label"},"apiVersion":"v1beta1"}" # Post-condition: selector.version == ${version} # This test works only in v1beta1 and v1beta2 # https://github.com/GoogleCloudPlatform/kubernetes/issues/4771 <del> kube::test::get_object_assert "service service-${version}-test" "{{range.selector}}{{.}}{{end}}" "test-label" </del> <ins> kube::test::get_object_assert "service service-${version}-test" "{{range.$service_selector_field}}{{.}}{{end}}" "test-label" </ins> ### Identity kubectl get service "${kube_flags[@]}" service-${version}-test -o json | kubectl update "${kube_flags[@]}" -f - ### Delete services by id # Pre-condition: redis-master-service service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:service-.*-test:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:service-.*-test:' </ins> # Command kubectl delete service redis-master "${kube_flags[@]}" kubectl delete service "service-${version}-test" "${kube_flags[@]}" # Post-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> ### Create two services # Pre-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> # Command kubectl create -f examples/guestbook/redis-master-service.json "${kube_flags[@]}" kubectl create -f examples/guestbook/redis-slave-service.json "${kube_flags[@]}" # Post-condition: redis-master and redis-slave services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:redisslave:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:redisslave:' </ins> ### Delete multiple services at once # Pre-condition: redis-master and redis-slave services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:redisslave:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:redisslave:' </ins> # Command kubectl delete services redis-master redisslave "${kube_flags[@]}" # delete multiple services at once # Post-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> ###########################

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: // owner: @gnufied // alpha: v1.18 <ins> // beta: v1.20 </ins> // Allows user to configure volume permission change policy for fsGroups when mounting // a volume in a Pod. ConfigurableFSGroupPolicy featuregate.Feature = "ConfigurableFSGroupPolicy"

We need to move this field to beta in 1.20. /sig storage
/triage accepted
According to , is working on a better design on this.
Nope. The current design is the design we are going to use to handle fsgroup. we discussed this at length in sig-storage call.
Is it okay if I make this change?


Question: cadvisor:                       cadvisorInterface, diskSpaceManager:               diskSpaceManager, cloud:                          cloud, <del> nodeRef:                        nodeRef, nodeLabels:                     nodeLabels, nodeStatusUpdateFrequency:      nodeStatusUpdateFrequency, </del> <ins> autoDetectCloudProvider:   autoDetectCloudProvider, nodeRef:                   nodeRef, nodeLabels:                nodeLabels, nodeStatusUpdateFrequency: nodeStatusUpdateFrequency, </ins> os:                         osInterface, oomWatcher:                 oomWatcher, cgroupRoot:                 cgroupRoot,

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: // Record the response going to a given host. func (at *affinityTracker) recordHost(host string) { at.hostTrace = append(at.hostTrace, host) <ins> Logf("Received response from host: %s", host) </ins> } // Check that we got a constant count requests going to the same host.

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-master-slow Which test(s) are failing: [sig-network] Services should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] Since when has it been failing: This test started to be flaky on 11/18 Testgrid link: https://k8s- Reason for failure: Looks to be a pretty straightfowards test not passing: If we look at triage, this test has always been flaky, but it got significantly more flaky (2X failures) on 10/27: Anything else we need to know: This job is non-blocking for 1.13, except if this represents an actual bug in Affinity the 1.13 team needs to know ASAP. This failure may or may not be related to . It looks like this test was disabled for large clusters, but maybe the problem wasn't the cluster size. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig network /priority important-soon attn:
FWIW this seems to be a long standing issue, atleast as far bck as 1.12 timeframe
Sure. But the failures have become 3X as frequent, so I still want a sign-off by sig-network.
cc
Ack. Will work on this.
we plan to lift code freeze for 1.13 tomorrow,11/28 5pm PST. If there is a new regression in 1.13 thats increased the flakes and needs addressing we need to get the fix in asap. Can you please let us know if this is a blocker for 1.13 by tomorrow, 11/28, morning 10am PST ?
Great observation. I checked on the commits around that date but wasn't able to find any change that may cause regression. Hasn't been able to reproduce the same failure locally as well.
I just noticed that the same test is flaking a lot more on 1.12 branches as well https://k8s-
Good catch on that, so the recent increase of flakiness likely is not caused by 1.13 changes so we shouldn't block release on this either. cc
But we will be looking deeper into the root cause.
Several observations: It only happens when is false.

When it's true, the test does more stuff before actually check service session affinity. It doesn't happen when executing in pod, probably due to the same reason above, the test start test pod before actually testing service. I wrote a script which does the same thing in my cluster of version 1.11.3-gke.18 in us-central1-a. cannot reproduce it. The hypothesis is that the service session affinity spec takes some time to reconcile after the service is created. It could happen either on the node or on the GCLB. Since I haven't been able to reproduce it's hard to tell. But it's more likely on GCLB side, even it could be regional issue. Looks like a sleep of seconds will deflake it. but it's just hiding issues. I would rather open another issue to track the real issue.
The most recent failure is here: Basically what happened is GCLB doesn't honor session affinity and still sends the request to different VMs. The log shows that 3 requests are routed to 3 different VMs. Dec 19 07:40:58.094: INFO: [service-fsn65 service-bf8nc] Dec 19 07:40:58.094: INFO: Affinity should hold but didn't. Dec 19 07:40:58.196: INFO: [pod,node] pairs: [{Pod:service-bf8nc Node:bootstrap-e2e-minion-group-f3nb} {Pod:service-fsn65 Node:bootstrap-e2e-minion-group-9sp7} {Pod:service-rnrcw Node:bootstrap-e2e-minion-group-m5rj}]; err: <nil
More recent failures: Logs:
Hello, could this issue be prioritized to help with stabilizing the release blocking upgrade tests? They're currently flaking ~23% of the time in () ~30% of the time in ()
/assign can you take a look
GitHub didn't allow me to assign the following users: take, a, look, can, you. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign can you take a look Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am preparing a PR, basically allow Affinity some time to be configured, instead of failing out instantly.
Thanks! Will keep an eye out for it.
The test is no longer flaky: https://k8s- Thanks everyone!
Thanks! Will keep an eye on it for a few days, on the following dashboards:
It's stabler now, but still flaking once/twice a day, e.g. in . - Is this the same issue as you looked at? Is there more to be done here? /reopen
Reopened this issue. <detailsIn response to : It's stabler now, but still flaking once/twice a day, e.g. in . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we should probably cherrypick into 1.13 in case of deflaking upgrade/downgrade job, because those jobs would run 1.13 tests against 1.14 (upgraded) cluster or 1.13 (downgraded) cluster. FYI we've hit similar situation for DNS test (ref ).
Good point. Cherrypick PR prepared.
14: <img width="1110" alt="Screen Shot 2019-03-16 at 12 11 07 PM" src="https://user-"13: <img width="1091" alt="Screen Shot 2019-03-16 at 12 11 22 PM" src="https://user-"Do we want to cherry-pick this back further?
Thanks for the nice graph. Look like the fix works. I'll create cherrypick PRs for 1.11 and 1.12 right away.
AFAICT this has been significantly improved: ! Closing this and thanks to /close
Closing this issue. <detailsIn response to : AFAICT this has been significantly improved: ! /close Instructions for interacting with me using PR comments are available .


Question: "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/conversion" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apiserver/pkg/endpoints/metrics"

When continuously re-applying the same object via a controller or kubectl with SSA the resourceVersion increases every time. I expected the resourceVersion to stay the same if the client always sends the same object via SSA, especially if the resulting object doesn't change (apart from resourceVersion & managedFields, but these are managed by the apiserver). Apply this CRD: <details</summary</detailsApply this CR: <details</summary</detailsGet state of the CR after first apply Apply the CR again Get state of the CR after second apply Diff =The resourceVersion and the "time" field of the managedField entry of the "test-manager" increases. The use case we have is a lot more complex than this simple scenario. The tl;dr is that we want to implement caching for our SSA calls, so that we don't have SSA calls on every reconcile of our controller if nothing actually changes. This is relatively hard to do if the resourceVersion increases after every apply. I took a closer look at the APIserver and ~ the following happens: "incoming" SSA patch is applied over the live object from etcd this leads to a change because has which means the field will be dropped Because of that change the "time" field in the managedField entry gets set to "time.Now()" later during admission the namespace field gets defaulted again (I also tried this with a mutating webhook, same result) On a high-level: client always applies the same object defaulting always results in the same object resourceVersion and managedFields[].time continuously go up I think there are many similar cases that would lead to this behavior. Basically whenever the admission chain mutates a field previously set via SSA. Notes: This only happens if there is 1s between each apply, otherwise the managedField[].time is identical and then the resourceVersion stays the same (because we then don't write to etcd) <detailsI also tried this with client-go v0.30 and kube-apiserver v1.30. Same result. </details<detailsRunning this natively on Apple Silicon M2. But can't imagine this makes a difference. Also got the same results in Docker Desktop on Mac. </details
(cc might be interesting for you)
/sig api-machinery
It looks less a bug but more of a intended behavior to me.

/cc /triage accepted
Bypassing the writes triggered by noop applies would break storage version migration (), so we're committed to supporting this behavior as a feature.
I am confused. Are you saying this is intended behavior for a NOP SSA apply to increment resourceVersion? This seems to not actually happen in all cases, only for certain objects/edge cases (see ). For example, try this: No resourceVersion is incremented above). Surprisingly I couldn't find any clear statement that SSA doesn't increment the resource version, but does say "a no-op apply is not very much more work for the API server than an extra GET" which seems impossible if resourceVersion is incremented
is probably only updating it because either the fieldManager changes or in the patch body the apiVersion is different (or both, perhaps)?
We need this in storage version migration to make sure we correctly handles the CRD schema changes. See Joe's comments here - If it helps to understand this better, we have tests to validate this.
I may have misspoke. When storage versions changes, a NOP SSA apply will trigger a resourceVersion bump. If the storage version is unchanged and an update would be a NOP for etcd, I believe we short-circuit the write (?). This might explain the configMap case.
For the reported issue: We've seen cases like this before where the time gets bumped causing a NOP to become a write. This could be improved. Note that this isn't technically a bug because the API server is allowed to bump the resourceVersion for any write (the discussion about storage version changes causing this is a good example of where this can happen in a way that the caller wouldn't expect). But I'd like to see the etcd write optimized away for cases like this.
Sounds similar to perhaps which had some investigation. FWIW as a controller author my expectation is that repeatedly applying does not repeatedly trigger resourceVersion increments. It's fine if it happens rarely (storage version change, etc) but if it happens consistently that means our own writes re-trigger reconciliation and we have an infinite loop of writes. AFAIK all the marketing around writing SSA controllers has been that you just apply what you want and don't try to do a get+diff+apply. Even if we did want to do that, it doesn't really work well currently ().

If we aren't able to rely on NOP writes (usually) being NOPs, I don't see how we are supposed to use SSA to write a controller at all?
This is a really good point. I agree.
looks like it was intended fix this sort of problem. I'm looking into why it doesn't work for this case.
doesn't work for CRs since the managed fields are inside of a unstructured object and are not typed as (for the slow path equality check in IgnoreManagedFieldsTimestampsTransformer).
is a draft proposal for a fix. Feedback welcome on how to make this fix cleaner. I'll test carefully before moving it out of draft.


Question: "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/conversion" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apiserver/pkg/endpoints/metrics"

Every time I server-side apply a with , the updates. Nothing about my manifest changes between applies; if I map to the empty map, the same manifest updates the . With client-side apply, the stays the same as long as my manifest remains constant, even if . I read in that Kubernetes should treat absent keys, empty values (in this case, an empty map), and values the same. I observe that leaving the key out and behave as expected, not modifying the and leaving the same, but empty map behaves as described above. If we cannot change this behavior, I'd at least like to understand the rationale behind it. I expected my object to stay the same between identical server-side applies, where the manifest file content was exactly the same, regardless of the value of . A different every apply indicates that the object was event type . I expect no events to fire on applying the same manifest. contains In actuality, any is fine. . Every successive repeat of steps 3-4 should show different values. Example of with : This is especially a problem when I server-side apply with code in my operator. My operator is built on and watches changes to the child objects it deploys. If I create a with , the operator constantly reconciles because every server-side apply of the manifest registers as a type event on the object, which triggers the watching operator to reconcile. N/A <details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
Potentially similar case as


Question: if !errors.IsConflict(err) { return nil, err } <ins> prevNamespace := latestNamespace </ins> latestNamespace, err = kubeClient.Core().Namespaces().Get(latestNamespace.Name) if err != nil { return nil, err } <ins> if prevNamespace.UID != latestNamespace.UID { return nil, fmt.Errorf("namespace uid has changed across retries") } </ins> } }

See for full dumps. The line number of corresponds to:
From a quick look, perhaps the namespace object returned from retryOnConflictError() just above no longer has a DeletionTimestamp and thus "*namespace.DeletionTimestamp" triggers the panic?
Thanks for the report, I will investigate.


Question: SchedulerQueueIncomingPods, SchedulerGoroutines, PermitWaitDuration, <ins> CacheSize, </ins> } )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track scheduler cache size. metric name Description -- -- schedulercachesize Number of elements in scheduler cache. Why is this needed: Gives better observability of scheduler cache. For, the number of "assumedpods" gives us an idea of how many pods are "assumed" and therefore being "bound". We can compare this with number of to make sure assume-bind works as expected. /sig scheduling
Is anyone already working on these metrics? If not I'd like to help out /assign
Thanks . Feel free to work on this one. Just a reminder, we may need feedback/approval from sig-instrumentation.


Question: <ins> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import "testing" func TestCleanUserAgent(t *testing.T) { for _, tc := range []struct { In  string Out string }{ { In:  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36", Out: "Browser", }, { In:  "kubectl/v1.2.4", Out: "kubectl/v1.2.4", }, } { if cleanUserAgent(tc.In) != tc.Out { t.Errorf("Failed to clean User-Agent: %s", tc.In) } } } </ins>

Some of the exported prometheus metrics, e.g. include high-cardinality fields like the User-Agent. This creats a large number of time-series on the prometheus server is ususally not advised to do. Please consider reducing the cardinality of the exported metrics.


Question: cd /build/ <del> wget "$2" -O input.json wget "$3" -O register.go </del> <ins> wget "$2" -O register.go # gendocs takes "input.json" as the input swagger spec. cp /swagger-source/"$1".json input.json </ins> ./gradle-2.5/bin/gradle gendocs --info

From gen-swagger-docs should read the local swagger spec rather than fetching it from That way it generates release api-reference when run-gen-swagger-docs is run in release branch and it generates api-reference for HEAD code, when the script is run on HEAD.
Assigning to as per discussion.
Blocks


Question: // Handle ExternalTraiffc related fields during service creation. if utilfeature.DefaultFeatureGate.Enabled(features.ExternalTrafficLocalOnly) { if apiservice.NeedsHealthCheck(service) { <del> if err := rs.allocateHealthCheckNodePort(service); err != nil { </del> <ins> if err := rs.allocateHealthCheckNodePort(service, nodePortOp); err != nil { </ins> return nil, errors.NewInternalError(err) } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: healthcheck nodeport allocation doesn't have roll-back, this may lead to nodeport leak. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig network /assign Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network /assign
Could you clarity "roll-back"? It seems that we basically in the same way.
I didn't mean the HealthCheck NodePort release process in . I meant the allocation process in and . Take for example, please see: There is a defer for common NodePorts in : Once fails, the above defer will finally exec. Then the allocated NodePorts will roll-back, i.e. these NodePorts will be released. But the HealthCheck NodePort allocation process doesn't use this roll-back. This will lead to leak. I run a simple test and see the HealthCheck NodePort leak. Will paste the result later.
Testing service manifest, . some simple testing code. See: . Note that I add this error intentionally: kube-apiserver with flag . . We will get below output. This is expected. again. We will get below output. Note the error . That means that when creating service fails at step 3, the NodePort and HealthCheck NodePort should be both released, but the was NOT.
Fortunately, several minute later, the leak was repaired and cleaned up. But anyway, we should release the HealthCheck NodePort when fails. We shouldn't depend on the later repair.
Thanks for tracing down, sent for fixing.
We're having this issue on service deletion. It looks like healthcheck nodeport is not deallocated immediately when deleting the service. This is causing issues for backup and restore of services with heathcheck node port.


Question: fi ret="${ret} --can-ip-forward" <del> if [[ -n ${address:-} ]]; then </del> <ins> if [[ -n ${address:-} ]] && [[ "$address" != "no-address" ]]; then </ins> ret="${ret} --address ${address}" fi fi

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) <ins> ginkgo.It("should provide Internet connection for Linux containers using DNS [Feature:Networking-DNS]", func() { linuxPod := createTestPod(f, linuxBusyBoxImage, linuxOS) ginkgo.By("creating a linux pod and waiting for it to be running") linuxPod = f.PodClient().CreateSync(linuxPod) ginkgo.By("verifying pod external connectivity to the internet") ginkgo.By("checking connectivity to 8.8.8.8 53 (google.com) from Linux") assertConsistentConnectivity(f, linuxPod.ObjectMeta.Name, linuxOS, linuxCheck("8.8.8.8", 53)) }) ginkgo.It("should provide Internet connection for Windows containers using DNS [Feature:Networking-DNS]", func() { windowsPod := createTestPod(f, windowsBusyBoximage, windowsOS) ginkgo.By("creating a windows pod and waiting for it to be running") windowsPod = f.PodClient().CreateSync(windowsPod) ginkgo.By("verifying pod external connectivity to the internet") ginkgo.By("checking connectivity to 8.8.8.8 53 (google.com) from Windows") assertConsistentConnectivity(f, windowsPod.ObjectMeta.Name, windowsOS, windowsCheck("www.google.com")) }) </ins> }) })

We pinging to the original E2Es, however in airgapped environments, these tests fail for obvious reasons: In general, we need to parameterize this if possible. Running the or tests should pass in air-gapped environments, especially those that dont explicitly declare external connectivity. For example: Run the test suites inside of a kuberentes cluster which has no internet connection. pointing finger at :) <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
some of these were recently removed by :) but some more to remove i think
They should have the corresponding tags so they can be filtered
and do we need to add a parameter in function as and replace from link. If not, what are we trying to achieve here ? I would like to collaborate.
independent test cases with their own feature tags as in
/assign
/sig testing windows network
/triage accepted


Question: const ( parallelCreateServiceWorkers = 1 maxServicesPerCluster        = 10000 <ins> maxServicesPerNamespace      = 5000 </ins> checkServicePercent          = 0.05 )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: For these 2 test suites visible in the testgrid at network-gce : http://k8s- http://k8s- the testsuite cannot setup correctly the cluster, due to a firewall issue (I think) See part of the build log below: What you expected to happen: I would think that setup the cluster is part of the process of the test-infra job, not the e2e test itself. Any missing flag or option on the job definition (see below) ? Need guidance on how to find the root cause of this error. How to reproduce it (as minimally and precisely as possible): Just look at the testgrid, it happen every 6 hours. http://k8s- http://k8s- Anything else we need to know?: job definition is: NOTE: this job definition was copied from another one that seems working fine: Environment: Kubernetes version (use ): current (branch master) Cloud provider or hardware configuration: gce
: I finally create this issue to get help. , : any idea on why the firewall rules cannot be set ? Could you indicate how to investigate this issue ? Thanks..
/sig network /sig scalability
I think it's because the name is too long. There's a limit of 64 chars or sth for resource names on GCP. Try changing the --cluster arg to or sth.
proposed a fix here : it is now merge. Need to verify the next run of these testsuites within 0 to 6 hours from now. : Thanks for your advise !
After updating the name of the cluster (shorter). The Cluster is now setup correctly. The PerformanceDNS test itself is failing - unable to create he 10k services. We believe that is an issue on the number of 10k services that k8s cannot support with this configuration of cluster. Next step is to decrease the number of services to 5k - a number we tested locally and know that k8s support.
Can you tell the reason why it's unable to create so many services?
I suspect that the cluster cannot operate correctly with that many services.

It can create the services. But the cluster acts undesirably after they are created. In my local tests on a single node cluster when I ramped up the number of services to 10K, I saw the same mode of failure. With 10K services, the cluster is unable to spin up the "util" pod. The "util" pod is required in the test to query the DNS from the context of a pod - if it's not running then the test cannot proceed.
But do you know what exactly is happening? What do you mean by "cluster is unable to spin up the "util" pod"? Is the pod not even being created? Or maybe is it scheduled, but kubelet is not starting it? Having more insight would help debugging it (maybe it's just size of master machine that is too small).
I dont for sure. But... According to the logs, I think the pod is successfully created. But when we try to run DNS queries, it's the container in the pod named "util" that cannot be found - not necessarily the pod itself. So, it seems the pod may be still starting up at the time - perhaps slowed by the number of services present. Maybe if I create the pod before creating the services it will help.
Let's try to understand this better. If it's that creating services is limiting us on the api qps, then starting the pod may potentially be slowed down (due to shortage of qps). Your idea sounds good, or you might alternatively try to slow down the rate of service creation. In general, I believe it shouldn't be a problem to create so many empty services in the cluster.
Please paste link to logs from that cluster (run from testgrid) - so that others can take a look into it.
OK, these are logs of the failure when pod is created after services... creates services OK I believe (this step takes just under 10 minutes). Then it creates the client pod... Looks OK? - client pod is created... So we start querying DNS... But the "util" container is not found, possibly because the pod is not ready yet. Perhaps the helper func that I am using doesn't wait for the pod to be ready.
I looked into kubelet logs - pods was succesfully started but then dies (crashed?) - something seems to be wrong on your side.

Logs from here:
It waits for it to be ready, but it dies after that.
OK, thanks but this isnt a pod I have invented for this test. It is already used for testing DNS in other existing tests. So I was expecting it to work. And it does work fine if i reduce the number of services - which I'll go ahead and do.
Let's not down the number of service without understanding why it fails. We need to understand that.
Does anyone know if the 10K limit on services is a hard/enforced limit? I didnt think it was (but thats just an assumption) - but if it is enforced somewhere, and exceeding it causes silent failures, then i'll need to reduce the number of services by a "few" to account for the other already existing services (e.g. the cluster dns service, api, etc...)
It's not a hard limit.
I managed to get this tested locally with 10K services on a single node.... where I create the pod before the services, and it passes. OK if I submit a PR with that change? Or do we want to first try to understand why creating the client pod after services causes a failure?
If you can explain why this works now and didn't work before, I'm fine - otherwise there is a risk that it may e.g. be flaky. We should understand it before blindly changing that.
OK - so I've run it and catched that pod for you. The reason why it is crashlooping is the following log: It just can't initialize itself due to too long argument list. - So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? - it's the limitation that recently poped up in our conversations ^^
Thanks Wojtek!

And that is why I said we should make use of the experiments dns folks are already doing, to catch that limitation (and we indeed seem to have caught it) :) On Thu, Jun 7, 2018, 10:47 AM Wojciech Tyczynski <:
I'll update that discussion once the value is figured out. Btw - thanks for digging. On Thu, Jun 7, 2018, 11:03 AM Shyam Jeedigunta <: The reason why it is crashlooping is the following log: exec user process caused "argument list too long" It just can't initialize itself due to too long argument list. So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? it's the limitation that recently poped up in our conversations ^^ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine.
But I got it to work with 10k services in one namespace... by creating the client pod before creating the services. On Jun 7, 2018, at 5:47 AM, Wojciech Tyczynski <mailto:: FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread
- IIUC that's because you created the pod even before creating the services (so it wasn't passed those service env vars while creation).
OK cool.

Seems like the limit would need to be here ... Several of the entries are "per ns" but there isnt one for services. As an unrelated side note: I just noticed that the limit on the number of deployments per ns in the thresholds doc lists 20K, with a long term goal of 10K. Is that a typo? Or is the long term goal to reduce the limit?
Sent for it. Seems so to me (should've probably been 10^5?). cc
Yeah - I think two values are incorrect wrt it: pods/namespace - I think we've never run 15k deployments/namespace - it doesn't make much sense to support more deployments than pods Will send out a PR for it.


Question: // since whether the process is running in a TTY cannot be changed after it has started.  We // need the api.Pod to get the TTY status. pod, found := kl.GetPodByFullName(podFullName) <del> if !found || pod.UID != podUID { </del> <ins> if !found || (string(podUID) != "" && pod.UID != podUID) { </ins> return nil, fmt.Errorf("pod %s not found", podFullName) } containerSpec := kubecontainer.GetContainerSpec(pod, containerName)

Meta from When enabled frakti with kubelet, failed with errors: Kubelet logging: This is because the param in could be empty string:
assigning to you since you already have a PR up


Question: oldModifyResponse := proxy.ModifyResponse proxy.ModifyResponse = func(response *http.Response) error { code := response.StatusCode <del> if code >= 300 && code <= 399 { </del> <ins> if code >= 300 && code <= 399 && len(response.Header.Get("Location")) > 0 { </ins> // close the original response response.Body.Close() msg := "the backend attempted to redirect this request, which is not permitted"

Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(


Question: expectedOutput        []*v1.Pod } podProvider := newFakePodProvider() <del> podKiller := newFakePodKiller() </del> <ins> podKiller := newFakePodKiller(false) </ins> criticalPodAdmissionHandler := getTestCriticalPodAdmissionHandler(podProvider, podKiller) allPods := getTestPods() runs := []testRun{

What happened: Kops runs from a pod manifest stored on the node, having the annotation. We've experienced the case where the cluster autoscaler turned on a new node and, once the node has been to the cluster, application pods have been admitted before . Once it was the turn of , all system resources were utilized, so the pod has not been regularly admitted. However, since it's a critical pod, the executed the , which has tried to kill () a pod whose container wasn't started yet (it was downloading the Docker image), which has caused to timeout and return the error . The admission of failed with the following error: The pod execution has not been retried, so we've got a node without running and its networking broken. What you expected to happen: The should be eventually admitted. Anything else we need to know?: A minimal log extraction for comprehension: From the "Failed to admit pod" log I have been able to trace back the core code involved: () Logs: "Failed to admit pod [POD] - [ERROR]" () Error message: "preemption: pod [POD] failed to evict [ERROR]" () -() Error message: "timeout waiting to kill pod" Environment: Kubernetes version: Cloud provider: AWS OS: ubuntu 16.04.6 LTS (Xenial Xerus) Kernel: Install tools: kops Network plugin and version: kubenet
/sig node
I think the following change may alleviate the situation you encountered:
Thanks for your reply. To my understanding, when the is called it will return the minimum set of pods required to free up resources, so even if only 1 pod eviction fails it's likely that at the end of the the will not have enough resources anyway to admit the critical pod. In your opinion / experience: be an option introducing retries in the in case the eviction of 1 pod fails? yes, do you have any suggestion of a similar design adopted somewhere else in the ?
Looking at getPodsToPreempt, I am not sure it returns the minimum set of Pods. For , retrying may not help much since the pod to be killed may not respond to the subsequent kill command.
Sorry to reiterate on this, but I double checked the code and - if I'm not missing anything - returns the minimum set of pods.

Let's look at logic: pods by QOS class using that the total sum of evictable pods resources is enough to satify the requirements (resources to evict for the critical pod) the list of pods to evict, as the merge of three sublists: : the minimum set of pods to evict assuming we've already evicted all and all : the minimum set of pods to evict assuming we've already evicted all and the required : the minimum set of pods to evict to satify the requirements left Am I missing anything? A possible different approach to solve the issue is to refactor to return all input pods sorted by "should be evicted first". Then iterates over the full sorted list of potential pods to evict until the requirements are not satisfied (skipping to subtract pod resources from requirements if a pod fails on ) and breaking the loop as soon as the requirements are satisfied. What's your take on this? /cc who originally implemented this
w.r.t. approach outlined above, it makes sense.
Whether this evicts the "minimum set" depends on how you define the "minimum set". At the time this was implemented (k8s v1.5.0 IIRC), we didn't have PodPriority, so we used QoS to mirror the kubelet's eviction algorithm. It should probably prioritize low-priority pods for eviction today given the recent changes to the eviction algorithm to incorporate PodPriority. The current approach minimizes the eviction of guaranteed pods, and prioritizes evicting few larger pods over many smaller ones. Since eviction marks the evicted pod as failed, it should suffice to just admit the pod regardless of what is returned by the evictPod function. In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. It would eliminate this class of bugs entirely, which was the purpose of the feature in the first place.
See if the following aligns with your suggestion:
Something like that, yeah. I'm not sure the "Failed to admit pod" log warning is still required, but we can iron those things out during code review.
Doing like this it means we do always admit pods, while we should do it only for critical pods, correct?
Please see this The change is for critical pod(s).


Question: args = append(args, func() { storageframework.DefineTestSuites(driver, testsuites.CSISuites) }) <del> framework.Describe(args) </del> <ins> framework.Describe(args...) </ins> return nil }

Any test that invokes with the parameter October 21, 2023 No response /sig-storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Minimal Repro:
/sig storage


Question: if len(buckets) == 0 || buckets[len(buckets)-1].upperBound != math.Inf(+1) { // The list of buckets in dto.Histogram doesn't include the final +Inf bucket, so we <del> // add it here for the reset of the samples. </del> <ins> // add it here for the rest of the samples. </ins> buckets = append(buckets, bucket{ count:      float64(hist.GetSampleCount()), upperBound: math.Inf(+1),

The new scheduler performance test was introduced back in 1.18, back then, the metric it supported are all . However, along with , some metrics are updated to be , and hence it's no longer appropriate to return the first Histogram: However, it's very common (for instance Preemption) that the metrics may hold multiple histograms: Additionally, metrics and have been removed, we should update them to : /sig scheduling
/triage accepted /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: type responseUpgrader struct { } <ins> // connWrapper is used to wrap a hijacked connection and its bufio.Reader. All // calls will be handled directly by the underlying net.Conn with the exception // of Read and Close calls, which will consider data in the bufio.Reader. This // ensures that data already inside the used bufio.Reader instance is also // read. type connWrapper struct { net.Conn closed    int32 bufReader *bufio.Reader } func (w *connWrapper) Read(b []byte) (n int, err error) { if atomic.LoadInt32(&w.closed) == 1 { return 0, io.EOF } return w.bufReader.Read(b) } func (w *connWrapper) Close() error { err := w.Conn.Close() atomic.StoreInt32(&w.closed, 1) return err } </ins> // NewResponseUpgrader returns a new httpstream.ResponseUpgrader that is // capable of upgrading HTTP responses using SPDY/3.1 via the // spdystream package.

https://k8s- Failed: TestServeExec {streaming}
Should be fixed by

https://k8s- Failed: TestRoundTripAndNewConnection/redirect=false {spdy}
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=false {spdy}
Should be fixed by
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=false {spdy}

https://k8s- Failed: TestServeExecInContainer {server}
Should be fixed by
https://k8s- Failed: TestServeExecInContainer {server}

https://k8s- Failed: TestServeAttachContainer {server}
https://k8s- Failed: TestServeAttachContainer {server}
Should be fixed by

https://k8s- Failed: TestRoundTripAndNewConnection/redirect=true {spdy}
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=true {spdy}
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=true {spdy}
Should be fixed by
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=true {spdy}
https://k8s- Failed: TestRoundTripAndNewConnection/redirect=true {spdy}

seen in https://k8s-
Should be fixed by
I believe this is fixed?
Yes, I missed this one.


Question: t.Fatalf("unexpected error: %v", err) } } <ins> func TestLabelMsg(t *testing.T) { tests := []struct { obj             runtime.Object overwrite       bool resourceVersion string labels          map[string]string remove          []string expectObj       runtime.Object expectMsg       string expectErr       bool }{ { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, labels:    map[string]string{"a": "b"}, expectMsg: MsgNotLabeled, expectErr: true, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{}, }, labels: map[string]string{"a": "b"}, expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, expectMsg: MsgLabeled, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, labels:    map[string]string{"a": "c"}, overwrite: true, expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "c"}, }, }, expectMsg: MsgLabeled, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, labels: map[string]string{"c": "d"}, expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b", "c": "d"}, }, }, expectMsg: MsgLabeled, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, labels:          map[string]string{"c": "d"}, resourceVersion: "2", expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:          map[string]string{"a": "b", "c": "d"}, ResourceVersion: "2", }, }, expectMsg: MsgLabeled, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b"}, }, }, labels: map[string]string{}, remove: []string{"a"}, expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{}, }, }, expectMsg: MsgUnLabeled, }, { obj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"a": "b", "c": "d"}, }, }, labels: map[string]string{"e": "f"}, remove: []string{"a"}, expectObj: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "c": "d", "e": "f", }, }, }, expectMsg: MsgLabeled, }, } for _, test := range tests { oldData, err := json.Marshal(test.obj) if err != nil { t.Errorf("unexpected error: %v %v", err, test) } err = labelFunc(test.obj, test.overwrite, test.resourceVersion, test.labels, test.remove) if test.expectErr && err == nil { t.Errorf("unexpected non-error: %v", test) continue } if !test.expectErr && err != nil { t.Errorf("unexpected error: %v %v", err, test) } newObj, err := json.Marshal(test.obj) if err != nil { t.Errorf("unexpected error: %v %v", err, test) } dataChangeMsg := updateDataChangeMsg(oldData, newObj) if dataChangeMsg != test.expectMsg { t.Errorf("unexpected dataChangeMsg: %v != %v, %v", dataChangeMsg, test.expectMsg, test) } } } </ins>

Currently, the cmd output msg is not distinguished between label and unlabel (using suffix ), both of them response . To distinguish between label and unlabel msg: , just like .
/sig cli
/triage accepted


Question: // APIServer runs a kubernetes api server. type APIServer struct { <del> Port                       int </del> <ins> WideOpenPort               int </ins> Address                    util.IP PublicAddressOverride      util.IP ReadOnlyPort               int

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package auth import ( "testing" "github.com/Azure/go-autorest/autorest/adal" "github.com/Azure/go-autorest/autorest/azure" "github.com/stretchr/testify/assert" ) func TestGetServicePrincipalTokenFromMSIWithUserAssignedID(t *testing.T) { configs := []*AzureAuthConfig{ { UseManagedIdentityExtension: true, UserAssignedIdentityID:      "UserAssignedIdentityID", }, // The Azure service principal is ignored when // UseManagedIdentityExtension is set to true { UseManagedIdentityExtension: true, UserAssignedIdentityID:      "UserAssignedIdentityID", TenantID:                    "TenantID", AADClientID:                 "AADClientID", AADClientSecret:             "AADClientSecret", }, } env := &azure.PublicCloud for _, config := range configs { token, err := GetServicePrincipalToken(config, env) assert.NoError(t, err) msiEndpoint, err := adal.GetMSIVMEndpoint() assert.NoError(t, err) spt, err := adal.NewServicePrincipalTokenFromMSIWithUserAssignedID(msiEndpoint, env.ServiceManagementEndpoint, config.UserAssignedIdentityID) assert.NoError(t, err) assert.Equal(t, token, spt) } } func TestGetServicePrincipalTokenFromMSI(t *testing.T) { configs := []*AzureAuthConfig{ { UseManagedIdentityExtension: true, }, // The Azure service principal is ignored when // UseManagedIdentityExtension is set to true { UseManagedIdentityExtension: true, TenantID:                    "TenantID", AADClientID:                 "AADClientID", AADClientSecret:             "AADClientSecret", }, } env := &azure.PublicCloud for _, config := range configs { token, err := GetServicePrincipalToken(config, env) assert.NoError(t, err) msiEndpoint, err := adal.GetMSIVMEndpoint() assert.NoError(t, err) spt, err := adal.NewServicePrincipalTokenFromMSI(msiEndpoint, env.ServiceManagementEndpoint) assert.NoError(t, err) assert.Equal(t, token, spt) } } func TestGetServicePrincipalToken(t *testing.T) { config := &AzureAuthConfig{ TenantID:        "TenantID", AADClientID:     "AADClientID", AADClientSecret: "AADClientSecret", } env := &azure.PublicCloud token, err := GetServicePrincipalToken(config, env) assert.NoError(t, err) oauthConfig, err := adal.NewOAuthConfigWithAPIVersion(env.ActiveDirectoryEndpoint, config.TenantID, nil) assert.NoError(t, err) spt, err := adal.NewServicePrincipalToken(*oauthConfig, config.AADClientID, config.AADClientSecret, env.ServiceManagementEndpoint) assert.NoError(t, err) assert.Equal(t, token, spt) } func TestParseAzureEngironment(t *testing.T) { cases := []struct { cloudName               string resourceManagerEndpoint string identitySystem          string expected                *azure.Environment }{ { cloudName:               "", resourceManagerEndpoint: "", identitySystem:          "", expected:                &azure.PublicCloud, }, { cloudName:               "AZURECHINACLOUD", resourceManagerEndpoint: "", identitySystem:          "", expected:                &azure.ChinaCloud, }, } for _, c := range cases { env, err := ParseAzureEnvironment(c.cloudName, c.resourceManagerEndpoint, c.identitySystem) assert.NoError(t, err) assert.Equal(t, env, c.expected) } } func TestAzureStackOverrides(t *testing.T) { env := &azure.PublicCloud resourceManagerEndpoint := "https://management.test.com/" azureStackOverrides(env, resourceManagerEndpoint, "") assert.Equal(t, env.ManagementPortalURL, "https://portal.test.com/") assert.Equal(t, env.ServiceManagementEndpoint, env.TokenAudience) assert.Equal(t, env.ResourceManagerVMDNSSuffix, "cloudapp.test.com") assert.Equal(t, env.ActiveDirectoryEndpoint, "https://login.microsoftonline.com/") azureStackOverrides(env, resourceManagerEndpoint, "adfs") assert.Equal(t, env.ManagementPortalURL, "https://portal.test.com/") assert.Equal(t, env.ServiceManagementEndpoint, env.TokenAudience) assert.Equal(t, env.ResourceManagerVMDNSSuffix, "cloudapp.test.com") assert.Equal(t, env.ActiveDirectoryEndpoint, "https://login.microsoftonline.com") } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are no unit tests yet for pkg , we should add unit tests. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign


Question: func (t *unstructuredList) ConvertToNative(typeDesc reflect.Type) (interface{}, error) { switch typeDesc.Kind() { case reflect.Slice: <del> return t.elements, nil </del> <ins> switch t.itemsSchema.Type() { // Workaround for https://github.com/kubernetes/kubernetes/issues/117590 until we // resolve the desired behavior in cel-go via https://github.com/google/cel-go/issues/688 case "string": var result []string for _, e := range t.elements { s, ok := e.(string) if !ok { return nil, fmt.Errorf("expected all elements to be of type string, but got %T", e) } result = append(result, s) } return result, nil default: return t.elements, nil } </ins> } return nil, fmt.Errorf("type conversion error from '%s' to '%s'", t.Type(), typeDesc) }

Added this test to : this resulted in: The test should pass. Add the above test No response All versions where CEL is used. All Doesn't matter No response No response No response_
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Opened to discuss the expected behavior in cel-go for this.


Question: } func (rc *reconciler) updateLastSyncTime() { <ins> rc.timeOfLastSyncLock.Lock() defer rc.timeOfLastSyncLock.Unlock() </ins> rc.timeOfLastSync = time.Now() } func (rc *reconciler) StatesHasBeenSynced() bool { <ins> rc.timeOfLastSyncLock.Lock() defer rc.timeOfLastSyncLock.Unlock() </ins> return !rc.timeOfLastSync.IsZero() }

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: // GetWaitingPod returns a waiting pod given its UID. GetWaitingPod(uid types.UID) WaitingPod <ins> // RejectWaitingPod rejects a waiting pod given its UID. RejectWaitingPod(uid types.UID) </ins> // ClientSet returns a kubernetes clientSet. ClientSet() clientset.Interface

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: decision.Message = fmt.Sprintf("failed messageExpression: %s", err) } else if evalResult.EvalResult != celtypes.True { decision.Action = ActionDeny <ins> decision.Evaluation = EvalDeny </ins> if validation.Reason == nil { decision.Reason = metav1.StatusReasonInvalid } else {

This issue is about the function in . In this function, 's is evaluated in this line: and if is not , 's is set to : But remains empty string. should be set to . No response I imported public library <details</summary</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
I am new to contributing to this repo can you please tell me what all things I should be familiar with to start contributing in kubernetes specifically this issue.
/cc /triage accepted


Question: emitDeltaTypeReplaced bool } <ins> // DeltaType is the type of a change (addition, deletion, etc) type DeltaType string // Change type definition const ( Added   DeltaType = "Added" Updated DeltaType = "Updated" Deleted DeltaType = "Deleted" // Replaced is emitted when we encountered watch errors and had to do a // relist. We don't know if the replaced object has changed. // // NOTE: Previous versions of DeltaFIFO would use Sync for Replace events // as well. Hence, Replaced is only emitted when the option // EmitDeltaTypeReplaced is true. Replaced DeltaType = "Replaced" // Sync is for synthetic events during a periodic resync. Sync DeltaType = "Sync" ) // Delta is a member of Deltas (a list of Delta objects) which // in its turn is the type stored by a DeltaFIFO. It tells you what // change happened, and the object's state after* that change. // // [*] Unless the change is a deletion, and then you'll get the final //     state of the object before it was deleted. type Delta struct { Type   DeltaType Object interface{} } // Deltas is a list of one or more 'Delta's to an individual object. // The oldest delta is at index 0, the newest delta is the last one. type Deltas []Delta // NewDeltaFIFO returns a Queue which can be used to process changes to items. // // keyFunc is used to figure out what key an object should have. (It is // exposed in the returned DeltaFIFO's KeyOf() method, with additional handling // around deleted objects and queue state). // // 'knownObjects' may be supplied to modify the behavior of Delete, // Replace, and Resync.  It may be nil if you do not need those // modifications. // // TODO: consider merging keyLister with this object, tracking a list of //       "known" keys when Pop() is called. Have to think about how that //       affects error retrying. // NOTE: It is possible to misuse this and cause a race when using an //       external known object source. //       Whether there is a potential race depends on how the consumer //       modifies knownObjects. In Pop(), process function is called under //       lock, so it is safe to update data structures in it that need to be //       in sync with the queue (e.g. knownObjects). // //       Example: //       In case of sharedIndexInformer being a consumer //       (https://github.com/kubernetes/kubernetes/blob/0cdd940f/staging/ //       src/k8s.io/client-go/tools/cache/shared_informer.go#L192), //       there is no race as knownObjects (s.indexer) is modified safely //       under DeltaFIFO's lock. The only exceptions are GetStore() and //       GetIndexer() methods, which expose ways to modify the underlying //       storage. Currently these two methods are used for creating Lister //       and internal tests. // // Also see the comment on DeltaFIFO. // // Warning: This constructs a DeltaFIFO that does not differentiate between // events caused by a call to Replace (e.g., from a relist, which may // contain object updates), and synthetic events caused by a periodic resync // (which just emit the existing object). See https://issue.k8s.io/86015 for details. // // Use `NewDeltaFIFOWithOptions(DeltaFIFOOptions{..., EmitDeltaTypeReplaced: true})` // instead to receive a `Replaced` event depending on the type. // // Deprecated: Equivalent to NewDeltaFIFOWithOptions(DeltaFIFOOptions{KeyFunction: keyFunc, KnownObjects: knownObjects}) func NewDeltaFIFO(keyFunc KeyFunc, knownObjects KeyListerGetter) *DeltaFIFO { return NewDeltaFIFOWithOptions(DeltaFIFOOptions{ KeyFunction:  keyFunc, KnownObjects: knownObjects, }) } // NewDeltaFIFOWithOptions returns a Queue which can be used to process changes to // items. See also the comment on DeltaFIFO. func NewDeltaFIFOWithOptions(opts DeltaFIFOOptions) *DeltaFIFO { if opts.KeyFunction == nil { opts.KeyFunction = MetaNamespaceKeyFunc } f := &DeltaFIFO{ items:        map[string]Deltas{}, queue:        []string{}, keyFunc:      opts.KeyFunction, knownObjects: opts.KnownObjects, emitDeltaTypeReplaced: opts.EmitDeltaTypeReplaced, } f.cond.L = &f.lock return f } </ins> var ( _ = Queue(&DeltaFIFO{}) // DeltaFIFO is a Queue )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While reviewing for I noticed that introduced a second type of Delta that is specific to DeltaFIFO but did not update the relevant documentation at , which was introduced by concurrent PR . What you expected to happen: I expected the documentation on DeltaFIFO to be current. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I think it would be more reader-friendly if the definitions of Deltas and related types were closer to their mentions near the top of the file rather than near the end.
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
Hi, is someone working on this? If no, can I pick it up? Thanks.
: go ahead, and thanks!
Thanks! /assign


Question: // NewAPIServer creates a new APIServer object with default parameters func NewAPIServer() *APIServer { s := APIServer{ <del> Port:                   8080, </del> <ins> WideOpenPort:           8080, </ins> Address:                util.IP(net.ParseIP("127.0.0.1")), PublicAddressOverride:  util.IP(net.ParseIP("")), ReadOnlyPort:           7080, APIRate:                10.0, APIBurst:               200, <del> SecurePort:             8443, </del> <ins> SecurePort:             6443, </ins> APIPrefix:              "/api", EventTTL:               48 * time.Hour, AuthorizationMode:      "AlwaysAllow",

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: #### Security Context Lifecycle <del> The lifecycle of a security context will be tied to that of a service account. It is expected that a service account with a default security context will be created for every Kubernetes namespace (without administrator intervention). If resources need to be allocated when creating a security context (for example, assign a range of host uids/gids), a pattern such as [finalizers](https://github.com/GoogleCloudPlatform/kubernetes/issues/3585) can be used before declaring the security context / service account / namespace ready for use. </del>  No newline at end of file <ins> The lifecycle of a security context will be tied to that of a service account. It is expected that a service account with a default security context will be created for every Kubernetes namespace (without administrator intervention). If resources need to be allocated when creating a security context (for example, assign a range of host uids/gids), a pattern such as [finalizers](https://github.com/GoogleCloudPlatform/kubernetes/issues/3585) can be used before declaring the security context / service account / namespace ready for use. </ins>

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
/assign /area kubeadm
/sig cluster-lifecycle
i went trough all the flakes we have at the moment and adjusted a number timeouts. here is the PR:
was green for a bit since merge, but noticed a timeout on this morning. Do you think it is worth bumping the timeout there as well? It looks like the timeout was quite a large outlier compared to the successful runs (more than double the duration), so maybe we should address underlying problem.
lets wait for some more singlal before bumping the timeouts on init. locally for me takes a couple of minutes, so i would like to know if this is a temporary GCE/networking issue.
actually something is not right, because in this job: also this job showed that even 10m is not enough for it errored with: but looking at the log timestamps only ~2 minutes have passed. i will have another look later today or tomorrow.
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-master (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-master) /priority important-soon /remove-kind failing-test /kind flake the test seems to be timeouting occasionally now. this appeared to be a temporary network speed issue. in the meantime i have created this PR which attempts to catch network flakes early and reduce overall network traffic:
the kubeadm e2e jobs started requiring more than 40 minutes: https://k8s- so a lot of test jobs are now failing. this could be due to: resource limits in Prow more conformance tests were during the weekend or next week i will try bumping all jobs to ~60 minutes.
/remove-priority important-soon (prio is already critical)
xref
hi, the jobs seem to be now at the state where they were before the ticket was logged - i.e. mostly green. the Prow issue that was discussed in is definitely a factor, so i'm not going to bump the jobs to 60minutes for now.

we do still see 2 occasional types of flakes: one of them is known and we don't have a fix for it as it's complicated. for the other one i'm going to send a PR and ping you for a quick lgtm. thanks.
thanks for following up! I will make sure to take a look at your incoming PR :+1:
i haven't seen the same failure in the last 10 runs: https://k8s- please re-open this ticket if needed. thanks.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am seeing some pretty regular failures here again. Could you take a look? /reopen
Reopened this issue. <detailsIn response to : I am seeing some pretty regular failures here again. Could you take a look? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the job is taking more than 40 minutes. so this is related to:
The runs in last few days have been well under 60m, except for
i'm suspecting slow CPU/network for some of the recent failures.
/remove-sig testing SIG testing does not own kinder or upgrades
looks like the most prominent recent flakes are caused by the inability to prepare the node image or upgrade all nodes in less than 10 minutes. i will send a PR to increase these timeouts.
Is that test now named ?
yes. i see some occasional flakes of not being able to complete the conformance suite in 25 minutes (parallel).
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
i can conform that the suite requires mostly more than 25 minutes in later releases (either due to more tests or slower infra). this PR increases the e2e test suite in kubeadm to 35 minutes and will close this ticket: we should create new / separate tickets with flaky tasks in the job.


Question: } iptInterface := fakeiptables.NewFake() sysctl := fakesysctl.NewFake() <del> execer := &fakeexec.FakeExec{} </del> <ins> execer := &fakeexec.FakeExec{ LookPathFunc: func(_ string) (string, error) { return "", errors.New("fake execer") }, } </ins> eventBroadcaster := record.NewBroadcaster() recorder := eventBroadcaster.NewRecorder(legacyscheme.Scheme, v1.EventSource{Component: "kube-proxy", Host: config.NodeName})

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: when i run hollow-node deploymenthollow-proxy will cause panic. hollow-node deployment is as follows: hollow-proxy container logs: I1012 12:37:13. 7 ] syncProxyRules took 622.278µs I1012 12:37:13. 7 asmamd64.s:573] Shutting down service config controller E1012 12:37:13.

7 ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:573 /usr/local/go/src/runtime/asmamd64.s:2361 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c68ad9] goroutine 74 [running]: (0x0, 0x0, 0x0) +0x107 panic(0x32dc9e0, 0x5c20d20) +0x229 (FakeExec).LookPath(0xc420ad4450, 0x3ac6dba, 0x9, 0xc4208c51d0, 0x0, 0xc4200a4800, 0x4e7e83) +0x39 (0x3e3c000, 0xc420ad4450, 0xc4208c51d0, 0x5, 0x5, 0x5, 0x5) +0x5e (0x3e3c000, 0xc420ad4450, 0xc420768c70, 0xc, 0x3abdd6d, 0x3, 0x0, 0x0) +0x17d (Proxier).syncProxyRules(0xc420d8fe40) +0xb643 (Proxier).OnServiceSynced(0xc420d8fe40) +0x62 (ServiceConfig).Run(0xc420ad5080, 0xc4200b62a0) +0x183 created by (*ProxyServer).Run +0x283 What you expected to happen: How to reproduce it (as minimally and precisely as possible): boot up hollow-proxy container with "--use-real-proxier=true" Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g.

): Install tools: Others: I found that in "kubemark/hollow-" we init a 'FakeExec' struct, but we don't init LookPathFunc in 'FakeExec' struct. So, execer.LookPathFunc is a vild pointer, so it cause panic.
/sig Testing
maybe we need init:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale I met this issue as well.
/sig scalability


Question: return upgradeRoundTripper.NewConnection(resp) } <del> // DoRaw executes a raw request which is not subject to interpretation as an API response. func (r *Request) DoRaw() ([]byte, error) { </del> <ins> // request connects to the server and invokes the provided function when a server response is // received. It handles retry behavior and up front validation of requests. It wil invoke // fn at most once. It will return an error if a problem occured prior to connecting to the // server - the provided function is responsible for handling server errors. func (r *Request) request(fn func(*http.Request, *http.Response)) error { if r.err != nil { return r.err } // TODO: added to catch programmer errors (invoking operations with an object with an empty namespace) if (r.verb == "GET" || r.verb == "PUT" || r.verb == "DELETE") && r.namespaceSet && len(r.resourceName) > 0 && len(r.namespace) == 0 { return fmt.Errorf("an empty namespace may not be set when a resource name is provided") } if (r.verb == "POST") && r.namespaceSet && len(r.namespace) == 0 { return fmt.Errorf("an empty namespace may not be set during creation") } </ins> client := r.client if client == nil { client = http.DefaultClient

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: Pod:         st.MakePod().Name("test-prefilter").UID("test-prefilter").Obj(), NumAllNodes: 3, Diagnosis: framework.Diagnosis{ <del> NodeToStatusMap:      framework.NodeToStatusMap{}, </del> <ins> NodeToStatusMap: framework.NodeToStatusMap{ "node1": framework.NewStatus(framework.Unschedulable, "node(s) didn't satisfy plugin(s) [FakePreFilter2 FakePreFilter3] simultaneously"), "node2": framework.NewStatus(framework.Unschedulable, "node(s) didn't satisfy plugin(s) [FakePreFilter2 FakePreFilter3] simultaneously"), "node3": framework.NewStatus(framework.Unschedulable, "node(s) didn't satisfy plugin(s) [FakePreFilter2 FakePreFilter3] simultaneously"), }, </ins> UnschedulablePlugins: sets.Set[string]{}, PreFilterMsg:         "node(s) didn't satisfy plugin(s) [FakePreFilter2 FakePreFilter3] simultaneously", },

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: tolerations: - key: "CriticalAddonsOnly" operator: "Exists" <ins> - operator: "Exists" effect: "NoExecute" - operator: "Exists" effect: "NoSchedule" </ins> nodeSelector: kubernetes.io/os: linux containers:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Konnectivity Service gets evicted when node gets tainted with NoExecute. Just like CoreDNS and KubeProxy, Konnectivity-agent should not get evicted when Node gets a taint. Have a port-forward running from a service and taint a node with a NoExecute option. Konnectivity-agent is deleted and can't get scheduled on the Node again until taint is removed. All other system pods use a wildcard toleration and are not effected by node taints. Kubernetes version (use ): 1.19.11 Cloud provider or hardware configuration: Azure OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
As I know, only kube-proxy tolerates NoExecute. coredns does not. konnectivity agent's setting is same as coredns.
/cc Like dns-node-cache and kube-proxy, daemonset seems to have the toleration. /assign kubeadm kube-proxy is using below.
/sig cloud-provider
/assign
GitHub didn't allow me to assign the following users: jkh52. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
As long as this stays DaemonSet, these broader tolerations seem acceptable. With GKE we have switched to Deployment, and intentionally do not have these broad tolerations (to avoid poor interaction with cluster-autoscaler).
/triage accepted