Question: "//pkg/kubelet/server/stats:go_default_library", "//pkg/kubelet/types:go_default_library", "//pkg/kubelet/util/format:go_default_library", <ins> "//pkg/scheduler/algorithm:go_default_library", </ins> "//pkg/scheduler/util:go_default_library", "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: According to the preemption's doc, we'll take priority over Qos; but in , the higher priority BestEffort pod maybe rejected because of memory pressure. If TaintNodesByCondition enabled, the pod need memory pressure toleration. /cc , correct me if any mis-understanding.
/sig scheduling /sig node
If the node has memory pressure, it rejects all pods that do not tolerate it. This is orthogonal to QoS and Priority.
Preemption and node evictions do not work with the same exact metrics. Preemption is unaware of usage, while node eviction logic has pod and node resource usage information. Node eviction takes usage into account both when admitting a new pod and when evicting an existing pod. has recently sent out to update node eviction doc.
, but will also reject it if pod tolerate it :(
, please refer to for what I mean :). BTW, you need to configure out and predicates, and enable alpha feature. /cc
I'm not familiar with the Preemption/Priority design, so I don't know where is preemption happening (Node or control plane). But despite that I wouldn't read what you wrote as inconsistency. Priority takes precedence over QoS when it can. But if some QoS Pods are not allowed to run somewhere it's not an issue with preemption/priority or QoS. It's a separate predicate. Slight OT, but related to your proposal, I think we should go another way. We should prevent BE pods to get MemoryPressure toleration. Having this can result with bad user experience, as Pod will get scheduled on a Node, and then immediately killed, because Kubelet ignores tolerations. Or we can teach kubelet respect tolerations for NC created taints. I'd consider it a blocker for beta of TaintNodeByCondition.
Yes; but if BE pod has higher priority, the other pods with lower priority will be evicted by kubelet. This maybe better, no idea on which pod will be BE but with higher priority :) , do you think we should reject BE pod with MemoryPressure toleration, even it has a higher priority?

I would like to mention several points here: of all, I should say that a high priority BE pod is a result of bad configuration and we should not optimize for it, but given that it is not rejected by the system, we must handle it. BE pod will never cause preemption due to lack of resources, because a BE pod needs zero resources from scheduler's point of view. of point , I think we should not worry about a high priority BE pod getting rejected by kubelet or causing other pods to be evicted from a node. We should document the system behavior and have clear guidelines and point out the problems that could be caused by bad configurations. After that we should leave it to the user to decide if a configuration makes sense for their use-case.
Overall, I agree :). Personally, I'd like to keep components behavior align: reject by admission/kubelet or accept by them. Currently, apiserver/scheduler accept and handle it and kubelet will reject it.
Kubelet does not reject a BE pod with high priority either.
I mean BE pod with Memory Pressure toleration :)
I see. The pod may still be scheduled without any problem if another node in the cluster is found that does not have memory pressure. So, I wouldn't call this inconsistent behavior of components. Anyway, I leave it up to SIG node to decide whehter the current behavior should be changed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: cmd.AddCommand(NewCmdConfigGetContexts(out, pathOptions)) cmd.AddCommand(NewCmdConfigGetClusters(out, pathOptions)) cmd.AddCommand(NewCmdConfigDeleteCluster(out, pathOptions)) <del> cmd.AddCommand(NewCmdConfigDeleteContext(out, pathOptions)) </del> <ins> cmd.AddCommand(NewCmdConfigDeleteContext(out, errOut, pathOptions)) </ins> return cmd }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, this is a bug report. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: MacBook Pro Mid-2015 OS (e.g. from /etc/os-release): macOS Kernel (e.g. ): Install tools: Others: N/A What happened: When deleting the context that was my current context, I entered a state where I am seemingly unable to set my context. The shows none as the current: Also, when trying to take any actions that require a context the following error is presented: When viewing the full config () I can see that my selected context is still the one I deleted. What you expected to happen: I believe there are two expectations I had: should allow me to delete my current context, but not enter an inconsistent state in the process -or- should either provide a warning or disallow removing your current context How to reproduce it (as minimally and precisely as possible): I believe that you just need a configuration with multiple contexts. I had three before deleting the context that has me unable to set a new context.

list all contexts to find your current context () delete your current context list your current contexts again, and you'll see no contexts are created try to set your context with one of the ones that exist () listing your contexts again will show none being set Anything else we need to know: 3 contexts were configured before starting to delete one 2 of them were GKE clusters, automatically set up by the binary 1 is a context set up by the context I deleted was one of the contexts The issue was resolved by manually setting my context by editing .
I think this is the confusion of doesn't switch contexts, it lets you set fields on a context. is what switches contexts.
It's technically valid to have no current context, but I agree a warning message like "warning: this removed your active context, use to select a different one" would go a long way.
SGTM.


Question: test/integration/discoverysummarizer test/integration/examples test/integration/federation <ins> test/integration/metrics </ins> test/integration/openshift test/soak/cauldron test/soak/serve_hostnames

When running , I got the following errors. Did I miss anything?
This problem will show up on non-linux platforms since the test/integration/metrics has only one file and that is marked as not runnable in non-linux platforms:


Question: return false, []algorithm.PredicateFailureReason{ErrNodeUnknownCondition}, nil } <del> if nodeInfo.Node().Spec.Unschedulable { </del> <ins> // If pod tolerate unschedulable taint, it's also tolerate `node.Spec.Unschedulable`. podToleratesUnschedulable := v1helper.TolerationsTolerateTaint(pod.Spec.Tolerations, &v1.Taint{ Key:    algorithm.TaintNodeUnschedulable, Effect: v1.TaintEffectNoSchedule, }) // TODO (k82cn): deprecates `node.Spec.Unschedulable` in 1.13. if nodeInfo.Node().Spec.Unschedulable && !podToleratesUnschedulable { </ins> return false, []algorithm.PredicateFailureReason{ErrNodeUnschedulable}, nil }

https://k8s- This test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all. /kind bug /priority failing-test /sig storage /assign /cc /milestone v1.12
looks like some permissions issue after upgrade /assign
has triaged this to an understanding that the failures happen because the pod is scheduled on the master and getting logs through the method they're using is disabled on master nodes. This makes me think this is similar to and we have non-release-blocking issues in test cases not being quite robust enough now in the face of node taints.
It is interesting that in some places this test is always failing, some it is always succeeding, and even in one it flakes but with a high success rate: https://k8s-
Adding on to comment: The pod that we are creating does not have any special "master schedulable" annotations or tolerations, so from my understanding this should still be not scheduled to the master for any reason. The Pod Spec in question: Potentially relevant scheduler logs.
The fix mentioned in is only handling DaemonSets. I don't think it will fix this issue.
Do we need to get sig-scheduling involved to see why the master is showing up as a choice for this pod?
/sig scheduling Can you triage this for 1.12 release blocking?
It seems that this pvc-tester- pod is only scheduled to master in every run which caused the test to fail consistently.
However in this test suite, it seems like the test passes most of the time (pod is scheduled to a node), then occasionally flakes (pod scheduled to master).: https://k8s- Whereas in this one it doesn't seem to fail at all!: https://k8s- I talked to about why the behavior might be different across different test suites but we were unable to come up with anything concrete.
Is the master node in your test environment tainted properly?
When we were doing unrelated development with DaemonSets, we noticed a similar issue where our DaemonSet pod would get scheduled on gce environment using hack/cluster-, and we noticed masters brought up using that script were not tainted
What release was the tainting master logic ?
cc we had similar issue that gpu driver installer is being scheduled on master at , can also be related?

Not sure if this is a source of truth on the cluster but the test artifacts here on a successful run show that the master node doesn't have a taint: and on a failed run ALSO no taint: But I'm not really sure where this file comes from. And whether this is the cluster before or after the upgrade
ie:
Don't think will help. It looks specific to gpu labeling
Also ^^ is GKE specific, and GKE docu says no taints are inherited to a pool from the cluster, so shouldn't be something from that direction.
Some log snippets around the area of interest:
/priority critical-urgent
I think the fix is obvious: we need to taint the master node on the gce master setup in cluster/. What I still don't understand is how did we get this far without anything else failing, and why only this one test?
Do taints persist across master upgrades, or do they need to be re-applied again?
Based on Slack discussion and the triage so far, this looks like it's a case of unfortunate cluster/* directory bit rot. We can add the taint and have time before the release still to see clear CI signal, but I would not rate this as a release blocker, especially considering node-"" appears to have been introduced in 1.6.
/assign I'm going to contribute in my daytime :)
I think there's what happened: only update status when upgrading from 1.11 to 1.13, as the node was previously registered (1.11); so is not updated, including unschedulable taint. scheduler is 1.13 now, TaintNodesByCondition is enabled by default and NodeConditionCheck is disabled by default, so it ignores controller manager is 1.13 now, it'll taint nodes according to node's configurations, e.g. . Because (including unschedulable) is not updated by kubelet when upgrading, it has to dependent on controller manager (nodelifecycle) to update it which has race condition with kube-scheduler.

There's an another issue that, we'll reject taints update to Node in noderestriction admission controller :( For the solution, I think we need to patch (at least ) and add a whitelist to noderestriction admission controller for known taints key; we can not use to start the kubelet, as it also be skipped (only update node status). Refer to the following logs for what happened: All logs are from https://k8s- Build- The taint of master; the taint will include if tainted by controller manager. scheduler-
/sig node
if no comments on the solution, i will open a pr for it.
cc on relaxing node restriction for this upgrade case Could another potential solution be to keep the previous scheduler node.unschedulable logic for one more release in order to handle this 2 version upgrade scenario?
Actually to clarify, is this both a 1.10 to 1.12 and 1.11 to 1.12 issue?
please get a PR started so we can get review happening ASAP
Thanks, for the investigation. Initially, I wanted to suggest to keep the logic of checking node condition in the scheduler along side TaintNodeByCondition for one more release, but I guess there would be conflicts. For example, a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition. So, looks like we'll have to fix the tainting logic.
can you clarify on what fixing the tainting logic would entail? If I understand your example, wouldn't the Pod object need to be updated to also tolerate the taint that the condition got converted to?
I was specifically referring to what suggested: "I think we need to patch node (at least node.Spec.Taints) and add a whitelist to noderestriction admission controller for known taints key." Yes, the pod needs to have the right toleration for the taint. We already create our critical system pods with proper tolerations for certain node taints, such as unreachable, etc.
I'm going back to feeling this is actually a release blocking upgrade issue. please clarify in more detail your concern for breakages, the possible paths for resolution, and preferred path.
I agree with you that this is release blocking. We commented at the same time.

I mentioned above that I think we need to address tainting of the nodes in kubelet.
Who's the right owner for this now? I'm assuming we've lost Klaus to the weekend in his timezone, but some of us still have a few work hours yet on Friday...
I already pinged Dawn as our Node guru, but didn't get any reply. She might be on vacation. Let me see if I can find other folks here.
cc for what we can do on the node side. cc for the security/policy related matters associated with adding taints for node condition
help please
regarding your example "a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition", could this be an API compatibility issue? Will a pod that tolerates a node condition in 1.11 fail to schedule in 1.12 because the scheduler no longer looks at node conditions and only node taints?
Pods were not able to tolerate node conditions by setting some API fields. We had some annotations, such as pod critical annotation and other hard-coded logic in DS controller, etc. that would have the same effect of tolerating conditions, but they were not in our API.
Thanks for the explanation. It looks like the critical pod annotation was alpha, so does not have a backwards compatibility requirement, and we've fixed all the known users of it.
xref: - cc Should this be handled by the node controller? I.e. if the Unschedulable bool is set on the node, add the unschedulable taint?
I don't believe the kubelet has the ability to update its taints anymore. Assuming this is the case, this seems like a version skew problem between master components. I.E. the 1.11 controller manager isn't compatible with the 1.13 scheduler because it doesn't add taints that the scheduler requires. This means that if the scheduler is upgraded before the controller manager (or before it has the chance to perform updates), we can run into issues. I think making the scheduler respect conditions in 1.13, and thus making it compatible with the 1.11 controller manager, seems like the best remedy to this.

Regardless of this issue, user pods would need to include both the critical annotation as well as the toleration in order to maintain the same behavior across releases that transition from conditions to taints.
Oh, I see that the node controller is already doing that, but there's a race condition (), sorry I missed that earlier.
Yes, translation of conditions and unschedulable field to taints is done by the controller
I wonder if we can change our upgrade flow to ensure that the controller manager is upgraded and becomes stable (adds the taints) before the scheduler is upgraded.
The kubelet has never required the ability to update taints. Was that a typo, supposed to be 1.12? Have we ever supported two version skew between those components? Under what circumstances do we upgrade to 1.13 scheduler while a 1.11 controller manager is still running? Same question about two version skew between these two components. That's more concerning… is that something all user pods would be exposed to and required to change?
+1 to suggestion in to unblock the release
What about HA master upgrade? scheduler and controller-manager each have independent leader elections, so during a HA rolling upgrade, the current leaders for scheduler and controller managers could be on different releases in either direction
I wouldn't expect an upgrade from 1.12 to 1.13 to begin until the upgrade from 1.11 to 1.12 was complete/successful
Thanks for correcting me. It wasn't a typo, but just a lack of knowledge about how multi-version upgrades are performed. In that case we need to support 1 version skew. Yeah, it should be 1.12, not 1.13. But generally, the newer scheduler should still be compatible with the state left by the previous version's controller manager, even if it isn't running anymore. That's more concerning… is that something all user pods would be exposed to and required to change? Actually, I agree. have we officially deprecated the critical pod annotation? We should ideally add the correct tolerations to pods with the critical pod annotation to ensure backwards compatibility.
Bobby commented that the system pods have already been corrected. I also noted that the critical pod annotation is alpha.
I mean doing an HA rolling upgrade from 1.11 to 1.12.

Let's say we have 3 masters, A, B, C, all on 1.11. is upgraded to 1.12, B and C are still on 1.11. leader election could pick scheduler A as the leader, controller-manager leader election could pick controller-manager B as the leader. Now we have version skew between master components.
That feature (critical pods) was an alpha feature and we have already replaced it. The dilemma is that I am not so sure if critical pods were the only kind of pods with a "hack" for tolerating node conditions. Let me dig a bit more.
One version of skew, absolutely. comment made me concerned two versions of skew support were being claimed/proposed
I looked the scheduler code and the DaemonSet controller code to find if adding "CheckNodeCondition" could cause issues in presence of TaintNodeByCondition and I didn't find anything particularly concerning. So, I created . Let's see if it breaks tests.
that will break schedule daemonset by scheduler. and registerwithtaints of kubelet is not worked in upgrade.
I suspected that it would conflict with pods which now have tolerations and the scheduler should ignore node conditions for, but I thought it would be limited to critical pods. Thanks for pointing it out. I closed the PR. As an alternative solution, I suggested changing the order of upgrading server components in our upgrade process, so that the controller manager is upgraded before the scheduler, but I am not sure how easy it is to implement. Do you have any alternative fix in mind?
unschedulable is taint by controller & kubelet; taints node with unschedulable to avoid race condition between controller and scheduler, it dependents on of kubelet to taint when start up; but does not work in upgrade case as mentioned above (we only update status when upgrading). IMO, it should be fixed; it'll also fix upgrade e2e test but may not work in real scenario which upgrade master firstly. let me think about it, I'll propose one today.
The race that is important to resolve at creation time is that of a new node object appearing untainted. That race does not apply in an upgrade scenario, where the node object already exists, and can already be observed and acted upon. Relying on the condition-to-taint controller in upgrade scenarios is reasonable.

The race after kubelet start up is well known; but we did not handle startup case well. I think what we're doing is to deprecate , and replace it with unschedulable taint. We need to keep backward compatibility for it.


Question: // ensureHostsFile ensures that the given host file has an up-to-date ip, host // name, and domain name. <del> func ensureHostsFile(fileName, hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) error { content := hostsFileContent(hostIP, hostName, hostDomainName, hostAliases) return ioutil.WriteFile(fileName, content, 0644) </del> <ins> func ensureHostsFile(fileName, hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias, useHostNetwork bool) error { var hostsFileContent []byte var err error if useHostNetwork { // if Pod is using host network, read hosts file from the node's filesystem. // `etcHostsPath` references the location of the hosts file on the node. // `/etc/hosts` for *nix systems. hostsFileContent, err = nodeHostsFileContent(etcHostsPath) if err != nil { return err } } else { // if Pod is not using host network, create a managed hosts file with Pod IP and other information. hostsFileContent = managedHostsFileContent(hostIP, hostName, hostDomainName, hostAliases) } return ioutil.WriteFile(fileName, hostsFileContent, 0644) } // nodeHostsFileContent reads the content of node's hosts file. func nodeHostsFileContent(hostsFilePath string) ([]byte, error) { return ioutil.ReadFile(hostsFilePath) </ins> } <del> // hostsFileContent is the content of the managed etc hosts func hostsFileContent(hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) []byte { </del> <ins> // managedHostsFileContent generates the content of the managed etc hosts based on Pod IP and other // information. func managedHostsFileContent(hostIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) []byte { </ins> var buffer bytes.Buffer buffer.WriteString("# Kubernetes-managed hosts file.n") buffer.WriteString("127.0.0.1tlocalhostn")                      // ipv4 localhost

Currently, kubelet mounts hosts file for pods. For hostNetwork pods, the behavior is different and instead, kubelet relies on docker (or whatever container runtime) to mount the host's hosts file. We should have kubelet handle hosts file for hostNetwork pods. This would have several benefits: control over hosts file in case the container runtime does something consistent behavior between hostNetwork and non-hostNetwork pods consistent behavior between different container runtimes -managed hosts file enables features like to be used for hostNetwork pods Implementation would require: kubelet's or its caller output different content based on hostNetwork. i. If it's not hostNetwork, use existing behavior. ii. If it's using hostNetwork, read the content from the host's hosts file. hostNetwork constraint in deciding whether to mount the file Open Questions: do we treat changes to the host's hosts file? should the file's content be static between container restarts of the same Pod? or should every container restart pick up the latest content of the hosts file? the former seems better for consistency, but letter might be more in line with other features that use host resources like host volumes. condition to manage hosts file also excludes windows even though windows also the use of hosts file and the same format. seems like this should be supported also. windows-support ships, should it have validation against windows platform? the tricky bit is the client, API, and minions' kubelet may be running on different platforms, and currently, there's no precedence to platform validation. /kind feature /sig node


Question: Name:      "pod-forgiveness-invalid", Namespace: "ns", }, <del> Spec: extendPodSpecwithTolerations(validPodSpec(nil), []api.Toleration{{Key: "node.alpha.kubernetes.io/notReady", Operator: "Exists", Effect: "NoSchedule", TolerationSeconds: &[]int64{20}[0]}}), </del> <ins> Spec: extendPodSpecwithTolerations(validPodSpec(nil), []api.Toleration{{Key: "node.kubernetes.io/not-ready", Operator: "Exists", Effect: "NoSchedule", TolerationSeconds: &[]int64{20}[0]}}), </ins> }, }, "must be a valid pod seccomp profile": {

Almost all style strings use dashes, rather than javascript names. vs .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for notification, a quick fix is submitted.
/cc
The PR here isn't merged yet. Can we make that happen please? We're still in alpha, so should be ok
Should I add the v1.8 milestone to this issue?
Hope so. We are pending in , need approve from
I opened with safe changes. "not-ready" change needs to be discussed more, as it's a breaking change. I believe it's best with fixing that to the time we'll be moving to beta, as we'll need to do another breaking change, i.e. remove 'alpha' from the key.
Aren't we doing this at the 11th hour? Is this going to hit 1.8?
Strings like this that are part of the core API should be part of a design doc that is formally reviewed by an API group. We need to find ways to catch this stuff earlier. We should also have an evolving API style guide that lists stuff like this. /cc
Thanks , let me make some summary: quick fix for 1.8 is , will be merged soon. final fix is , we are collaborating to reach a final solution and it's not target 1.8
Agree with Joe, can someone open a PR to that documents that taint names should follow the naming conventions for annotations?


Question: // setup host path directory to pass information between pod restarts func setupHostPathDirectory(job *batchv1.Job) { <del> if len(job.Spec.Template.Spec.NodeName) > 0 { </del> <ins> if _, nodeNameSpecified := job.Spec.Template.Spec.NodeSelector["kubernetes.io/hostname"]; nodeNameSpecified { </ins> randomDir := "/tmp/job-e2e/" + rand.String(10) hostPathType := v1.HostPathDirectoryOrCreate job.Spec.Template.Spec.Volumes[0].VolumeSource = v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: randomDir, Type: &hostPathType}}

pull-kubernetes-e2e-kind example: Example, but does not seem to be specific to the test: "Using a pod failure policy to not count some failures towards the backoffLimit". First time noticed, but potentially for long if this is very rare. The 3rd pod for the job could not get assigned resources by kubelet to start. See kubelet logs from kind-worker2: Relevant lines: Due to hitting the limit the pod is marked as by kubelet which is unexpected by the test. No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suppose the reason is that we preassigning the instead of asking scheduler to do so. This might be related to
FYI
we should not preassign, is that required by this test? Do you have a link to those tests?
What is required is that all pods run on the same node, because we use HostPath to communicate some information between recreations of a pod, and between replicas for indexed job. All tests in this file: , which use the function are affected I think. Here is the problematic place: I suppose using a instead of setting directly might already help, because it the decision will go via scheduler which will delay the scheduling until node has enough capacity.
FYI for input on the proposed solution to use or another alternative
hmm ,I don't like why this is this way , if the test need to be serialized it must be tagged as Serial, but making this assumptions on e2e tests is not great
Yeah, I would very much want to avoid making the tests serial. I can see two options: term - replace the with . IIUC, this will already keep the pods unscheduled until there is capacity. With the timeout of 15min as we have for job it is very unlikely the pod would wait that long. Currently, it gets assigned but fails. the tests so that the state is kept on a mounted volume obtained by PVC.

This should eliminate the need for the pods to run on the same node. Since 2. is slightly more involving, it might be good to have 1. merged.
you can not assume the e2e test will have dynamic PV working, if you want to assume that you have to tag the test so jobs can filter based on the tag
Let's start with the NodeSelector
Actually, that wouldn't help, because currently the Job controller just recreates the Pod. With a node selector, it would just be scheduler that is now "busy-waiting" and we would still hit a timeout. Can the implementation be changed to save the information in a ConfigMap? It probably requires a specialized container image. Unless a ConfigMap can be mounted as readwrite?
Interesting, so I greped the logs for and only found this pod in this line: Oct 24 07:42:50 kind-worker2 kubelet[248]: I1024 07:42:50. 248 ] "Event occurred" object="job-712/pod-disruption-failure-ignore-3-mpvmm" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="OutOfpods" message="Node didn't have enough resource: pods, requested: 1, used: 110, capacity: 110" Which should not be that bad, because we have backoffLimit=6, yet the job failed with . So, maybe some other failures were incrementing the counter wrongly. I don't know at the moment.
My mistake here. The job has actually : This explains why the job failed with BackoffLimitExceeded after seeing failing with admission error by kubelet. Meaning 137 exit code, but without DisruptionTarget condition. So it was counted as failed and killed the job. So, I think in this case adding the would actually help. It would not help, if was for a prolonged amount of time to cause test timeout. However, the test timeout is 15min. Also, there is no indication that kubelet was in the time for long. A single event was enough to cause the pod fail with 137, and this single failure caused the job failure.

As for long term solution using config maps - yes, it could work, but I think it's better to do a quick fix to minimize the flakes before release.
I agree. Thanks for investigating!


Question: } func (plugin *glusterfsPlugin) ConstructVolumeSpec(volumeName, mountPath string) (*volume.Spec, error) { <del> glusterfsVolume := &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ Glusterfs: &v1.GlusterfsVolumeSource{ EndpointsName: volumeName, Path:          volumeName, }, }, } return volume.NewSpecFromVolume(glusterfsVolume), nil </del> <ins> // To reconstrcut volume spec we need endpoint where fetching endpoint from mount // string looks to be impossible, so returning error. return nil, fmt.Errorf("impossible to reconstruct glusterfs volume spec from volume mountpath") </ins> } // Glusterfs volumes represent a bare host file or directory mount of an Glusterfs export.

/kind bug What happened: Pod stays in terminating status forever What you expected to happen: POD removed. How to reproduce it (as minimally and precisely as possible): Provision container with PVC (Dynamic provisioned) using GlusterFS wait untill running and then remove POD Anything else we need to know?: Events: kubelet log Environment: Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} hardware cat /etc/os-release NAME="Container Linux by CoreOS" ID=coreos VERSION=1381.1.0 VERSIONID=1381.1.0 BUILDID=2017-04-25-2232 PRETTYNAME="Container Linux by CoreOS 1381.1.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" COREOSBOARD="amd64-usr" Linux server37 4.10.12-coreos SMP Tue Apr 25 22:08:35 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz GenuineIntel GNU/Linux Install tools: hyperkube
/sig node
+1
+1
This should be fixed by
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Actually will avoid printing out the error message "Could not construct volume information ..", but I believe that is not the root cause of the issue here.
Stale issues rot after 30d of inactivity.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I had the same issue for v1.11. Pod has state Terminating. Container is alive on node in Docker 1.13. Describe shows


Question: ns string ) f := framework.NewDefaultFramework("kubelet") <del> f.NamespacePodSecurityEnforceLevel = admissionapi.LevelBaseline </del> <ins> f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged </ins> ginkgo.BeforeEach(func() { c = f.ClientSet

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is no other failures in [x] fixed [ ] will fix this one.
/sig auth /priority important-soon /milestone v1.24
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /sig auth /priority important-soon /milestone v1.24 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func newE2eService(nodeName string, cgroupsPerQOS bool, evictionHard string, context *SharedContext) *e2eService { // Special log files that need to be collected for additional debugging. var logFiles = map[string]logFileData{ <del> "kern.log":   {[]string{"/var/log/kern.log"}, []string{"-k"}}, "docker.log": {[]string{"/var/log/docker.log", "/var/log/upstart/docker.log"}, []string{"-u", "docker"}}, </del> <ins> "kern.log":       {[]string{"/var/log/kern.log"}, []string{"-k"}}, "docker.log":     {[]string{"/var/log/docker.log", "/var/log/upstart/docker.log"}, []string{"-u", "docker"}}, "cloud-init.log": {[]string{"/var/log/cloud-init.log"}, []string{"-u", "cloud*"}}, </ins> } return &e2eService{

If node e2es fail, it is not possible to figure out if node initialization failed. We need to collect cloud init logs as well.
This needs to happen right away. There are many flakes on GCI and we don't know why the cloud init scripts failed.


Question: # do_start() { <ins> # Avoid a potential race at boot time when both monit and init.d start # the same service PIDS=$(pidof $DAEMON) for PID in ${PIDS}; do kill -9 $PID done </ins> # Return #   0 if daemon has been started #   1 if daemon was already running

During an e2e test run, on GCE, I ssh-ed into the box. After the reboot test runs, it appears that something is trying to start kubelet every 2 minutes after the reboot: It does appear that it is monit restarting the process: I do only have one copy of kubelet running, because the second kubelet fails to start:
The direct cause of monit thinking the kubelet is down is that does not match. Also, keeps changing (every two minutes?) My hypothesis is that the second kubelet writes the pid, because it gets pretty far in its boot-up. And thereafter, monit sees it as down so keeps trying to start more. And so the root cause is that first second kubelet start, which I think is down to this: My kubelet is currently listening to that url though: My guess is that kubelet is too slow to start listening to /healthz
Oh! ... or it's a simple race with init.d starting the services; monit vs kube-proxy. If monit starts kube-proxy before init.d starts kube-proxy =double start.
/cc
Ok, there is a race when monit is used to monitor services that are also started at boot time via init.d, in our ecosystem, such as kubelet, kube-proxy, and docker. Especially when a service is slow to start, monit can assume that the service is not running and possibly try to start it and raise an issue while the service is already about to start or already in its startup sequence. In this case, there is a stale pidfile which is used by monit during the service validation periodically (currently configured every 2 mins), thus run into the issue reported by here. A clean way to fix this is starting those kubernetes related services requiring monitoring by monit directly by monit. The issue is that monit is not installed by default in our gce images for now. My proposed solution is that for each dostart call in /etc/init.d/${servicedaemon}, checking if there is existing process running. If it does, kill it before calling start-stop-daemon --start Will send pr shortly.


Question: e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutil "k8s.io/kubernetes/test/utils" <ins> imageutils "k8s.io/kubernetes/test/utils/image" </ins> utilpointer "k8s.io/utils/pointer" )

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: # Use dynamic cgo linking for architectures other than amd64 for the server platforms # To install crossbuild essential for other architectures add the following repository. RUN echo "deb http://archive.ubuntu.com/ubuntu xenial main universe" > /etc/apt/sources.list.d/cgocrosscompiling.list  <del> && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5 3B4FE6ACC0B21F32  </del> <ins> && apt-key adv --no-tty --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5 3B4FE6ACC0B21F32  </ins> && apt-get update  && apt-get install -y build-essential  && for platform in ${KUBE_DYNAMIC_CROSSPLATFORMS}; do apt-get install -y crossbuild-essential-${platform}; done

The go team has released 1.11.3 with three security fixes. From their e-mail announcement: We should kick off upgrades of both the kubernetes build system, and our CI, as soon as possible. Do you have an ETA for the rules_go bump that will contain 1.11.3 toolchain support? cc:
I'm preparing releases that use Go 1.11.3 by default. Those will be on the rules_go 0.14, 0.15, and 0.16 branches. I expect to tag them tonight as soon as they get through CI.
Fantastic! Thank you
rules_go 0.14.7, 0.15.9, and 0.16.4 are now tagged and ready to go.
related commits and issue:
Reopening to track other branches, as well as the CI bump
go1.11.3 and go1.10.6 introduced a regression, and so go1.11.4 and go1.10.7 were released Friday afternoon with a fix. We'll need to bump everything again on master, and should hold off on cherry-picks for now.
This looks fixed by Is there anything else that needs to be done as well?
Still need to merge for k8s 1.13.
Closing as all branches are now up to date.


Question: return err } <del> return clientcmd.ModifyConfig(o.configAccess, *config, false) </del> <ins> if err := clientcmd.ModifyConfig(o.configAccess, *config, false); err != nil { return err } if _, err := fmt.Fprintf(out, "Property %q unset.n", o.propertyName); err != nil { return err } return nil </ins> } <del> func (o *unsetOptions) complete(cmd *cobra.Command) error { endingArgs := cmd.Flags().Args() if len(endingArgs) != 1 { return helpErrorf(cmd, "Unexpected args: %v", endingArgs) </del> <ins> func (o *unsetOptions) complete(cmd *cobra.Command, args []string) error { if len(args) != 1 { return helpErrorf(cmd, "Unexpected args: %v", args) </ins> } <del> o.propertyName = endingArgs[0] </del> <ins> o.propertyName = args[0] </ins> return nil }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5.3 Environment: Cloud provider or hardware configuration: minikube 0.17.1 How to reproduce it (as minimally and precisely as possible): config Note that there's only one context. the property of a non-existent context config Note that a context named foo was created just because I tried to unset a property. What you expected to happen: should give an informative message saying that context doesn't exist. Anything else we need to know: This is probably low-hanging fruit.
/sig cli
/assign i will take a look for this
GitHub didn't allow me to assign the following users: a, look, for, this, i, will, take. Note that only can be assigned. <detailsIn response to : /assign i will take a look for this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can reproduce this, this is a bug, have open PR fix this. thanks
/kind bug


Question: // if nothing is found, just print usage cmd.Root().Usage() } <del> } else if len(a) == 0 { // help message for help command :) cmd.Root().Usage() </del> } else { <ins> if len(args) == 0 { // help message for help command :) foundCmd = cmd } </ins> helpFunc := foundCmd.HelpFunc() helpFunc(foundCmd, args) }

PR broke command due to wrong check . Currently, it displays for every command if you do not specify additional arguments(for example will not work, but will).
please looks at this small fix


Question: ServiceAccountLookup:        s.ServiceAccountLookup, ServiceAccountTokenGetter:   serviceAccountGetter, KeystoneURL:                 s.GenericServerRunOptions.KeystoneURL, <ins> KeystoneCAFile:              s.GenericServerRunOptions.KeystoneCAFile, </ins> WebhookTokenAuthnConfigFile: s.WebhookTokenAuthnConfigFile, WebhookTokenAuthnCacheTTL:   s.WebhookTokenAuthnCacheTTL, RequestHeaderConfig:         s.GenericServerRunOptions.AuthenticationRequestHeaderConfig(),

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: } var map_IPBlock = map[string]string{ <del> "":       "DEPRECATED 1.9 - This group version of IPBlock is deprecated by networking/v1/IPBlock. IPBlock describes a particular CIDR (Ex. "192.168.1.1/24") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.", "cidr":   "CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24"", "except": "Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" Except values will be rejected if they are outside the CIDR range", </del> <ins> "":       "DEPRECATED 1.9 - This group version of IPBlock is deprecated by networking/v1/IPBlock. IPBlock describes a particular CIDR (Ex. "192.168.1.1/24","2001:db9::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.", "cidr":   "CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"", "except": "Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range", </ins> } func (IPBlock) SwaggerDoc() map[string]string {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : mentions an IPBlock with only IPv4 examples. Add a comment or example to clarify that IPv6 network blocks are (or possibly aren't) valid here. Why is this needed: Cluster operators who use IPv6 networking want to understand how to isolate different workloads inside their cluster network (and without reading source code). Logged against kubernetes/kubernetes rather than the website repo because this is generated documentation. Kubernetes already has single-stack support for IPv6 clusters KEP for dual-stack support is /sig docs /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved can you guide me on what steps are needed to publish the examples?
I via the pull request. (Although I want these examples , it's not a task I'm familiar with).


Question: fmt.Println("Skipping pre-flight checks") } <del> // TODO(phase1+) this we are missing args from the help text, there should be a way to tell cobra about it if len(args) == 0 && len(cfg.MasterAddresses) == 0 { return nil, fmt.Errorf("must specify master IP address (see --help)") } cfg.MasterAddresses = append(cfg.MasterAddresses, args...) </del> ok, err := kubeadmutil.UseGivenTokenIfValid(&cfg.Secrets) if !ok { if err != nil {

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: resources: ["namespaces"] verbs: ["get", "watch", "list"] - apiGroups: ["admissionregistration.k8s.io"] <del> resources: ["mutatingwebhookconfigurations", "validatingwebhookconfigurations"] </del> <ins> resources: ["mutatingwebhookconfigurations", "validatingwebhookconfigurations", "validatingadmissionpolicies", "validatingadmissionpolicybindings"] </ins> verbs: ["get", "watch", "list"] - apiGroups: ["flowcontrol.apiserver.k8s.io"] resources: ['prioritylevelconfigurations', 'flowschemas']

Deployed sample-apiserver into a kind cluster following the instructions in Checked wardle-server pod logs, found the following error: No such errors. Deployed sample-apiserver into a kind cluster following the instructions in <details</detailsKIND
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign Will submit a fix shortly.


Question: cacher.ready.wait() // Ensure there is some budget for slowing down processing. <del> cacher.dispatchTimeoutBudget.returnUnused(50 * time.Millisecond) </del> <ins> // When using the official `timeBudgetImpl` we were observing flakiness // due under the following conditions: // 1) the watch w1 is blocked, so we were consuming the whole budget once //    its buffer was filled in (10 items) // 2) the budget is refreshed once per second, so it basically wasn't //    happening in the test at all // 3) if the test was cpu-starved and we weren't able to consume events //    from w2 ResultCh it could have happened that its buffer was also //    filling in and given we no longer had timeBudget (consumed in (1)) //    trying to put next item was simply breaking the watch // Using fakeTimeBudget gives us always a budget to wait and have a test // pick up something from ResultCh in the meantime. cacher.dispatchTimeoutBudget = &fakeTimeBudget{} </ins> makePod := func(i int) *examplev1.Pod { return &examplev1.Pod{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: return result, err } for _, service := range services.Items { <del> name := strings.ToUpper(service.ID) + "_SERVICE_PORT" </del> <ins> name := makeEnvVariableName(service.ID) + "_SERVICE_PORT" </ins> value := strconv.Itoa(service.Port) result = append(result, api.EnvVar{Name: name, Value: value}) result = append(result, makeLinkVariables(service, machine)...)

In issue it appeared that consensus was reached that a service id could contain a hyphen, but that it would be translated to an underscore in the environment variables (this issues was the root problem, back in ). However, the replacement does not appear to have been done: hyphens are still not replaced with underscores in env-var names. (Although now underscores are blocked in service names!)
Hm, that's weird. I thought that this code was doing replacement. Did that get removed?
Thanks for looking into this lavalamp. Was your question directed at me? I don't see any string replacement code in the code you linked to, at least not for the (I do see escaping for the port in line 198)
Ah, no, I was just thinking out loud there. Maybe knows what happened?
Ah, that's an oversight. We do it for the Docker link style variables, but not the old school ones. I'll send a PR.
PR sent.


Question: func findProbe(probes []network.Probe, probe network.Probe) bool { for _, existingProbe := range probes { <del> if strings.EqualFold(*existingProbe.Name, *probe.Name) && *existingProbe.Port == *probe.Port { </del> <ins> if strings.EqualFold(to.String(existingProbe.Name), to.String(probe.Name)) && to.Int32(existingProbe.Port) == to.Int32(probe.Port) { </ins> return true } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. The calls to create the load balancer were failing with a 400 What you expected to happen: controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was. How to reproduce it (as minimally and precisely as possible): Just created a cluster with an add on for the dashboard. Anything else we need to know?: logs from controller manager: I0423 12:50:28. ] Starting range CIDR allocator I0423 12:50:28. ] Waiting for caches to sync for cidrallocator controller I0423 12:50:28. ] Starting NoExecuteTaintManager I0423 12:50:28. ] Initializing eviction metric for zone: westeurope: :0 W0423 12:50:28. ] Missing timestamp for Node aks-nodepool1--0. Assuming now as a timestamp. I0423 12:50:28. ] Controller detected that zone westeurope: :0 is now in state Normal. I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"aks-nodepool1--0", UID:"-4476-11e8-9b1c-", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node aks-nodepool1--0 event: Registered Node aks-nodepool1--0 in Controller I0423 12:50:28. ] Caches are synced for ClusterRoleAggregator controller I0423 12:50:28. ] Caches are synced for endpoint controller I0423 12:50:28. ] Caches are synced for attach detach controller E0423 12:50:28. ]

Error creating spec for volume "mssqldb", pod "feature-test3"/"mssqlserver--r5r7l": error processing PVC "feature-test3"/"mssql-data": PVC feature-test3/mssql-data has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("") I0423 12:50:28. ] Caches are synced for deployment controller I0423 12:50:28. ] Caches are synced for persistent volume controller I0423 12:50:28. ] Caches are synced for ReplicationController controller I0423 12:50:28. ] Caches are synced for TTL controller I0423 12:50:28. ] Caches are synced for cidrallocator controller E0423 12:50:28. ] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"-4476-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"193", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"feature-initialproject", Name:"identityserver-service", UID:"-4485-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"7879", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer I0423 12:50:28. ] Caches are synced for namespace controller I0423 12:50:28. ] Caches are synced for service account controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ]

Caches are synced for garbage collector controller I0423 12:50:28. ] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Caches are synced for stateful set controller I0423 12:50:28. ] Caches are synced for disruption controller I0423 12:50:28. ] Sending events to api server. I0423 12:50:29. ] selectLoadBalancer: cluster(kubernetes) service(feature-initialproject/identityserver-service) isInternal(true) - availabilitysetsnames [nodepool1-availabilitySet-] E0423 12:50:30. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:509 /usr/local/go/src/runtime/asmamd64.s:2337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d] goroutine 1338 [running]: (0x0, 0x0, 0x0) +0x111 panic(0x40b2e00, 0xa603590) +0x283 (Cloud).getServiceLoadBalancerStatus(0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1) +0x30d (Cloud).EnsureLoadBalancer(0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40) +0x2ea (ServiceController).ensureLoadBalancer(0xc420f56000,

0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6) +0xcc (ServiceController).createLoadBalancerIfNeeded(0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492) +0x20e (ServiceController).processServiceUpdate(0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0) +0xeb (ServiceController).syncService(0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0) +0x3aa (ServiceController).worker.func1(0xc420f56000) +0xd9 (ServiceController).worker(0xc420f56000) +0x2b (ServiceController).()-fm() +0x2a (0xc420a04570) +0x5e (0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0) +0xbd (0xc420a04570, 0x3b9aca00, 0xc4208fa0c0) +0x4d created by (ServiceController).Run +0x20c Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Azure, AKS OS (e.g. from /etc/os-release): Ubunutu Kernel (e.g. ): xenial Install tools:AKS Others:
not sure what sig should be labeled in on this. Seems like a bug in sig-azure and whoever owns the piece getting the nil pointer/panic
what's the k8s version?
1.9.1
- I suspect that the error is here. This can be either one of two things A bug in SDK serialization (LB does have a private ip) but it does not get serialized correctly.

A bug in our assumption that that each internal LB ip fe config will have a valid internal IP. Can you dump the model for this internal LB here? Also -- did this error eventually recover? (it could be a race condition between setting ip and getting ip)
this was actually a customer cluster and they deleted it before I got a chance to get things like the model/what error code on the 400 was. And it didn't ever recover.
according to the evente table it was an internal loadbalancer here is the link to it --https://jarvis- maybe it the Details logged Can help you further
I was able to get more details from logs, {"error":{"code":"LoadBalancerFrontendIPConfigurationCountLimitReached","message":"Basic sku load balancer /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/loadBalancers/kubernetes-internal cannot have more than 10 FrontendIPConfigurations.","details":[]}}
Thanks - So there are two errors here. One the user can not use more than 10 IPs. The fact that we panic on this. i believe the fix is to log and fail the operation. /assign
The issue has been fixed in and has been included in v1.9.2. Could you upgrade your cluster (e.g. to latest v1.9.7) and check whether it fixes the issue?
I'm closing this issue since it's been resolved. /close
I am seeing this on 1.10.x and 1.11.x clusters: 10.6 11.2
The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that /reopen
Reopening this issue. <detailsIn response to : The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Added this to v1.12 since this is a critical urgent bug.


Question: Log-Output ("using mgmt IP ${mgmt_ip} and mgmt subnet ${mgmt_subnet} for " + "CNI config") <ins> # We reserve .1 and .2 for gateways. Start the CIDR range from ".3" so that # IPAM does not allocate those IPs to pods. $cidr_range_start = ` ${env:POD_CIDR}.substring(0, ${env:POD_CIDR}.lastIndexOf('.')) + '.3' </ins> # Explanation of the CNI config values: #   CLUSTER_CIDR: the cluster CIDR from which pod CIDRs are allocated. #   POD_CIDR: the pod CIDR assigned to this node. <ins> #   CIDR_RANGE_START: start of the pod CIDR range. </ins> #   MGMT_SUBNET: the subnet on which the Windows pods + kubelet will #     communicate with the rest of the cluster without NAT (i.e. the subnet #     that VM internal IPs are allocated from).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Environment: On GCE with the win-bridge cni plugin On a node when creating a pod for the first time, running podsandbox would fail with the error message below. "failed to create the new HNSEndpoint: hnsCall failed in Win32: The object already exists. (0x1392)" Subsequent retry would succeed, and launching other pods afterwards would also, generally, be fine. I've found , but in that issue, the creation failed permanently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master (1.14, ) Cloud provider or hardware configuration: GCE OS (e.g: ): Windows 1809 Kernel (e.g. ): Install tools: Others: /cc
cc:
We've seen this before -- what's the root-cause?
So this happens only for the first time right? Could you please collect the following logs for us? On the node before creating the pod for the first time. start-birstranfer run collectlogs.ps1 then start the trace by running the following command C:kdebug repro the issue run "netsh trace stop" then do again collectlogs.ps1 and send us both before and after collectlogs.ps1 and C: cc
thank you for the detailed steps to capture the traces. I've uploaded the trace + log files at Files from both and are included. There was about a ten-minute delay between the node joining the cluster and the deployment of a pod to start the repro. The timestamp on the node just before creating the pod was Friday, March 1, 2019 6:56:06 PM. Please let us know if you have questions or need more information. Thanks for taking a look!
Thanks Logs were super useful in root causing the issue. You are using L2Bridge in host gateway mode, so u r allocating .2 ip as the Gateway Ip for this host.

I am seeing a request to create an endpoint with an IP "10.64.1.2" "{"Name":"1885009e399aa542ad60bab063262196a804252010ab401967a6575cc7ccd1bd_l2bridge","VirtualNetwork":"64736A7F-E9F0-4465-B861-7879393DDD22","Policies":[{"ExceptionList":["10.64.0.0/14","10.0.0.0/16","10.40.0.0/22"],"Type":"OutBoundNAT"},{"DestinationPrefix":"10.0.0.0/16","NeedEncap":true,"Type":"ROUTE"},{"DestinationPrefix":"10.40.0.3/32","NeedEncap":true,"Type":"ROUTE"}],"IPAddress":"10.64.1.2","DNSSuffix":"","DNSServerList":"10.0.0.10","GatewayAddress":"10.64.1.2"}" But this IP is assigned to CBR0(Gateway) and an endpoint already exists on HNS with that IP. so when a container request is coming with the same IP we fail saying "the object already exists in our Address MAP" Now the question is why are we creating a container with an IP 10.64.1.2 when that IP is allocated for the Gateway NIC on this host? u can see that in Before logs on HNS - ID IPAddress MacAddress IsRemoteEnd point -- --------- ---------- ----------- 428D83D1-6138-459F-9B77-4F8B48A6FF6B 10.64.1.2 00-15-5D-6E-A4-DD First Entry is the endpoint that already exists but u are also sending a request to create a container with the same IP So basically ur IPAM is not excluding the gateway NIC and hence we are seeing that only for the first time u create the container. Hope this helps in root causing the issue on ur end. Again thanks for the logs. Super Useful..


Question: "path/filepath" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: <del> // +build windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( dockertypes "github.com/docker/docker/api/types" dockercontainer "github.com/docker/docker/api/types/container" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" ) // applySandboxPlatformOptions applies platform specific options to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxPlatformOptions(hc *dockercontainer.HostConfig, config *runtimeapi.PodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { dnsConfig := config.GetDnsConfig() if dnsConfig == nil { return nil } // Setup DNS. hc.DNS = dnsConfig.GetServers() hc.DNSSearch = dnsConfig.GetSearches() hc.DNSOptions = dnsConfig.GetOptions() return nil } </del>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is trying to setup DNS for Windows containers, so that Pods on Windows could confirm to dns policies. However, as posted Docker is not managing the networking here for windows. So it doens't really care about any network settings. So passing it to docker shim's hostconfig also doens;t make sense here. DNS for Windows containers will be set by CNI plugins. So PR reverts . This issue tracks the process of pushing DNS configure into CNI and passed that from Kubelet to container runtime. What you expected to happen: Pods on Windows confirm to dns policies. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows
Need a linked issue & PR in CNI repo too so that capability arguments can be validated from kubelet
Fixed in by dns capabilities, Azure CNI has already supported this. /close
Closing this issue. <detailsIn response to : Fixed in by dns capabilities, Azure CNI has already supported this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // JSON, or if either patch or update calls fail; it will also return a second error // whenever creating a patch has failed func (c *CordonHelper) PatchOrReplace(clientset kubernetes.Interface, serverDryRun bool) (error, error) { <ins> return c.PatchOrReplaceWithContext(context.TODO(), clientset, serverDryRun) } // PatchOrReplaceWithContext provides the option to pass a custom context while updating // the node status func (c *CordonHelper) PatchOrReplaceWithContext(clientCtx context.Context, clientset kubernetes.Interface, serverDryRun bool) (error, error) { </ins> client := clientset.CoreV1().Nodes() oldData, err := json.Marshal(c.node)

What would you like to be : For , the method of the object should accept a user supplied context which is then used by the client to update the status of the node before cordon/uncordoning the node. Why is this needed: Currently uses a local copy of the drain/cordon code from the codebase. Instead of maintaining a local copy, we would like to incorporate the functionality using upstream imports from . In the CAPI codebase, we pass a object to all the controller Reconcile routines and sub-routines. We would like to keep this same behavior even when we move to importing the drain code from I noticed that the code at does not use a passed context but instead uses a placeholder context for the patch/update calls. Instead, we can modify the method to accept a custom context passed over from the drain helper.
/sig cli
/triage accepted /priority backlog


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "sync" "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" ) const ( namespace = "apiserver" subsystem = "encryption_config_controller" ) var ( encryptionConfigAutomaticReloadFailureTotal = metrics.NewCounter( &metrics.CounterOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "automatic_reload_failures_total", Help:           "Total number of failed automatic reloads of encryption configuration.", StabilityLevel: metrics.ALPHA, }, ) encryptionConfigAutomaticReloadSuccessTotal = metrics.NewCounter( &metrics.CounterOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "automatic_reload_success_total", Help:           "Total number of successful automatic reloads of encryption configuration.", StabilityLevel: metrics.ALPHA, }, ) encryptionConfigAutomaticReloadLastTimestampSeconds = metrics.NewGaugeVec( &metrics.GaugeOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "automatic_reload_last_timestamp_seconds", Help:           "Timestamp of the last successful or failed automatic reload of encryption configuration.", StabilityLevel: metrics.ALPHA, }, []string{"status"}, ) ) var registerMetrics sync.Once func RegisterMetrics() { registerMetrics.Do(func() { legacyregistry.MustRegister(encryptionConfigAutomaticReloadFailureTotal) legacyregistry.MustRegister(encryptionConfigAutomaticReloadSuccessTotal) legacyregistry.MustRegister(encryptionConfigAutomaticReloadLastTimestampSeconds) }) } func RecordEncryptionConfigAutomaticReloadFailure() { encryptionConfigAutomaticReloadFailureTotal.Inc() recordEncryptionConfigAutomaticReloadTimestamp("failure") } func RecordEncryptionConfigAutomaticReloadSuccess() { encryptionConfigAutomaticReloadSuccessTotal.Inc() recordEncryptionConfigAutomaticReloadTimestamp("success") } func recordEncryptionConfigAutomaticReloadTimestamp(result string) { encryptionConfigAutomaticReloadLastTimestampSeconds.WithLabelValues(result).SetToCurrentTime() } </ins>

ref - /sig auth
/triage accepted
/retitle encryption-at-rest: add success or failure metric when encryption configuration attempts to dynamically update
/assign


Question: package util import ( <del> "fmt" </del> <ins> "sort" </ins> storagev1 "k8s.io/api/storage/v1" <del> "k8s.io/apimachinery/pkg/api/errors" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" storagev1listers "k8s.io/client-go/listers/storage/v1"

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: optional APIServiceStatus status = 3; } <ins> // APIServiceCondition describes the state of an APIService at a particular point </ins> message APIServiceCondition { // Type is the type of the condition. optional string type = 1;

Which jobs are failing: Which test(s) are failing: Job Reason for failure --- --- Since when has it been failing: 2019-02-11 Anything else we need to know: started failing with commit started failing with commit cc /kind failing-test /priority important-soon /sig api-machinery


Question: pod.Spec.RestartPolicy = "Never" pod.Spec.Containers[0].Command = []string{"echo", "hi"} pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ <del> v1.ResourceCPU: resource.MustParse("1"), </del> <ins> v1.ResourceCPU: resource.MustParse("950m"), // leave a little room for other workloads </ins> } } createBatchPodWithRateControl(f, restartNeverPods, podCreationInterval)

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze


Question: "type": "integer" } }, <del> "type": "object" </del> <ins> "type": "object", "x-kubernetes-map-type": "atomic" </ins> }, "io.k8s.api.networking.v1beta1.IPAddress": { "description": "IPAddress represents a single IP of a single IP Family. The object is designed to be used by APIs that operate on IP addresses. The object is used by the Service core API for allocation of IP addresses. An IP address can be represented in different formats, to guarantee the uniqueness of the IP, the name of the object is the IP address in canonical format, four decimal digits separated by dots suppressing leading zeros for IPv4 and the representation defined by RFC 5952 for IPv6. Valid: 192.168.1.5 or 2001:db8::1 or 2001:db8:aaaa:bbbb:cccc:dddd:eeee:1 Invalid: 10.01.2.3 or 2001:db8:0:0:0::1",

I created an Ingress resource with a port number manually, and then tried to use an Operator with server-side apply to update the resource from a port number to a port name. This failed with the error message "cannot set both port name & port number". I expected the port number to be cleared as the operator would take over the ownership of the ServiceBackendPort. To simulate the behavior in the operator in a most simple example I use : ingress- ingress- This will print: It seems to me that the merge type for the struct ServiceBackendPort should to be set to "atomic" , so that the Operator updating the port should take over the whole struct, and two applications shouldn't update single fields in it. Red Hat OpenShift on AWS OS version of the client: NAME="Fedora Linux" VERSION="40 (Workstation Edition)" Linux 6.9.6-200.fc40.x8664 SMP PREEMPTDYNAMIC Fri Jun 21 15:48:21 UTC 2024 x86_64 GNU/Linux
/sig network
So, on one hand you made a problem when you used two different owners to manage one aspect of the object. On the other hand, I think you're right that it SHOULD work. Probably. is not a super common tag, and ingress way predates it, so no surprise it isn't used. I ACK the issue and reproduced it. I ran update-codegen and built a KinD cluster from HEAD.
this may be something you want to look at for DRA API - I didn't even realize this was a thing until today :)
The API uses for , which is very similar. So using it for LGTM. SSA will auto-migrate existing resources using once the change is released.
/triage accepted
"this" being "add "? I can see where this might be useful, probably on all one-of structs and perhaps on those containing a discriminator because otherwise SSA leaves fields set which should be cleared when some other owner takes over. Is this important enough to put into I don't see a big need for SSA in general and even less for SSA with different owners at the moment - this will change once we do more with status.
No, not urgent. This is the first time it has come up in my awareness ever. But now that I know about it, I could see how it applies to dra.

Thank to everyone involved here to resolve this! This will make things simpler in the Keycloak operator once it is released.
Thanks for filing a good (and easy to fix :) bug. Easy repro means we can jump on it.