Question: return &fakeManager{} } <del> // GetSecret returns a nil secret for testing </del> <ins> // NewFakeManagerWithSecrets creates a fake secret manager with the provided secrets func NewFakeManagerWithSecrets(secrets []*v1.Secret) Manager { return &fakeManager{ secrets: secrets, } } // GetSecret function returns the searched secret if it was provided during the manager initialization, otherwise, it returns an error. // If the manager was initialized without any secrets, it returns a nil secret." </ins> func (s *fakeManager) GetSecret(namespace, name string) (*v1.Secret, error) { <del> return nil, nil </del> <ins> if s.secrets == nil { return nil, nil } for _, secret := range s.secrets { if secret.Name == name { return secret, nil } } return nil, fmt.Errorf("secret %s not found", name) </ins> } // RegisterPod implements the RegisterPod method for testing purposes.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --imagePullSecrets can to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error. I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace. If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place. Add to a pod spec, and pull images from
/sig node (based on the field location)
(and/or) /sig auth
correct me if not. Pulling private images with nonexists secret, pod will turn to . However, pulling public image with nonexists secret, no error will display yet.
Similar issue with .
Yes that's correct.
I think so. Working on this later. Will also live up to your expectations, right?
But I check it already have a warnging events if secret not exist and pullImageFail: line:147
Now, only info message when pullsecret not exist
maybe direct return and not enter is better. if pullSecret is not existed
if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.
I think a warning event like could be here /good-first-issue /triage accepted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I would like work on this. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work in this. /assign
Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.
Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective. An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod
hi , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not.
To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.
gotcha, that seems more reasonable, thanks for clarifying as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed .

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.
/assign
based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right?
I think has created a above and labeled /lgtm already.
Hey I don't know anything about how to code but i am interested to contribute on kubernetes So help me to contribute
/cc
there is PR already here , just needs approval
it was just FYI =) Reviewed the PR, thank you
Thanks for the review :)
/assign
will like to work on it. /assign
Hey , would love to work on this issue
there is already a PR: do you plan to push the PR forward?
I have been busy with other things recently, if anyone is willing to continue this problem. I will close the PR.
if noone else is working on this task, I would like to take this up.
Feel free to work on this , you can refer to for some advices.
Thanks! Will refer it
/assign
/assign
/assign
Is this issue still open/active?
Hello I was working on it but not able to continue it due to some personal reasons. You can take over if you would like to. You can refer this PR if you want to.
Yes sure!!!
Can I try this if no one is available to work on it ?
Yes go on!!!
/assign
Sorry for keep opening the PR. something went wrong on my fork branch. I eventually needed to re-fork the repo so it caused the PR closed automatically
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Can I give it a go if no one is working on it?
Thanks for the offer, but it looks like was open and had an update that was waiting for review (which I didn't realize)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
is this issue still available to pick up?
/remove-lifecycle stale
Hi, is this issue still available?
Hi! It feels that this issue is stalled (although it lacks the label) and the related PRs have been abandoned. I'll work on it if that's ok /assign
As well as an event, we might like to directly report a warning to end users (suggested in ). More complexity though.
should it be part of this issue? it only mentions but as you guys (the contributors) decide. is based on which was labelled as and suggested as a starting point before being closed.


Question: <ins> // +build linux /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cm import ( "fmt" "strings" "time" "github.com/golang/glog" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" utilfeature "k8s.io/apiserver/pkg/util/feature" kubefeatures "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/events" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" ) const ( defaultNodeAllocatableCgroupName = "kubepods" ) //createNodeAllocatableCgroups creates Node Allocatable Cgroup when CgroupsPerQOS flag is specified as true func (cm *containerManagerImpl) createNodeAllocatableCgroups() error { cgroupConfig := &CgroupConfig{ Name: cm.cgroupRoot, // The default limits for cpu shares can be very low which can lead to CPU starvation for pods. ResourceParameters: getCgroupConfig(cm.capacity), } if cm.cgroupManager.Exists(cgroupConfig.Name) { return nil } if err := cm.cgroupManager.Create(cgroupConfig); err != nil { glog.Errorf("Failed to create %q cgroup", cm.cgroupRoot) return err } return nil } // enforceNodeAllocatableCgroups enforce Node Allocatable Cgroup settings. func (cm *containerManagerImpl) enforceNodeAllocatableCgroups() error { nc := cm.NodeConfig.NodeAllocatableConfig // We need to update limits on node allocatable cgroup no matter what because // default cpu shares on cgroups are low and can cause cpu starvation. nodeAllocatable := cm.capacity // Use Node Allocatable limits instead of capacity if the user requested enforcing node allocatable. if cm.CgroupsPerQOS && nc.EnforceNodeAllocatable.Has(kubetypes.NodeAllocatableEnforcementKey) { nodeAllocatable = cm.getNodeAllocatableAbsolute() } glog.V(4).Infof("Attempting to enforce Node Allocatable with config: %+v", nc) cgroupConfig := &CgroupConfig{ Name:               cm.cgroupRoot, ResourceParameters: getCgroupConfig(nodeAllocatable), } // Using ObjectReference for events as the node maybe not cached; refer to #42701 for detail. nodeRef := &v1.ObjectReference{ Kind:      "Node", Name:      cm.nodeInfo.Name, UID:       types.UID(cm.nodeInfo.Name), Namespace: "", } // If Node Allocatable is enforced on a node that has not been drained or is updated on an existing node to a lower value, // existing memory usage across pods might be higher than current Node Allocatable Memory Limits. // Pod Evictions are expected to bring down memory usage to below Node Allocatable limits. // Until evictions happen retry cgroup updates. // Update limits on non root cgroup-root to be safe since the default limits for CPU can be too low. // Check if cgroupRoot is set to a non-empty value (empty would be the root container) if len(cm.cgroupRoot) > 0 { go func() { for { err := cm.cgroupManager.Update(cgroupConfig) if err == nil { cm.recorder.Event(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated Node Allocatable limit across pods") return } message := fmt.Sprintf("Failed to update Node Allocatable Limits %q: %v", cm.cgroupRoot, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) time.Sleep(time.Minute) } }() } // Now apply kube reserved and system reserved limits if required. if nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedEnforcementKey) { glog.V(2).Infof("Enforcing System reserved on cgroup %q with limits: %+v", nc.SystemReservedCgroupName, nc.SystemReserved) if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.SystemReservedCgroupName), nc.SystemReserved); err != nil { message := fmt.Sprintf("Failed to enforce System Reserved Cgroup Limits on %q: %v", nc.SystemReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message) } cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on system reserved cgroup %v", nc.SystemReservedCgroupName) } if nc.EnforceNodeAllocatable.Has(kubetypes.KubeReservedEnforcementKey) { glog.V(2).Infof("Enforcing kube reserved on cgroup %q with limits: %+v", nc.KubeReservedCgroupName, nc.KubeReserved) if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.KubeReservedCgroupName), nc.KubeReserved); err != nil { message := fmt.Sprintf("Failed to enforce Kube Reserved Cgroup Limits on %q: %v", nc.KubeReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message) } cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on kube reserved cgroup %v", nc.KubeReservedCgroupName) } return nil } // enforceExistingCgroup updates the limits `rl` on existing cgroup `cName` using `cgroupManager` interface. func enforceExistingCgroup(cgroupManager CgroupManager, cName CgroupName, rl v1.ResourceList) error { cgroupConfig := &CgroupConfig{ Name:               cName, ResourceParameters: getCgroupConfig(rl), } if cgroupConfig.ResourceParameters == nil { return fmt.Errorf("%q cgroup is not config properly", cgroupConfig.Name) } glog.V(4).Infof("Enforcing limits on cgroup %q with %d cpu shares and %d bytes of memory", cName, cgroupConfig.ResourceParameters.CpuShares, cgroupConfig.ResourceParameters.Memory) if !cgroupManager.Exists(cgroupConfig.Name) { return fmt.Errorf("%q cgroup does not exist", cgroupConfig.Name) } if err := cgroupManager.Update(cgroupConfig); err != nil { return err } return nil } // getCgroupConfig returns a ResourceConfig object that can be used to create or update cgroups via CgroupManager interface. func getCgroupConfig(rl v1.ResourceList) *ResourceConfig { // TODO(vishh): Set CPU Quota if necessary. if rl == nil { return nil } var rc ResourceConfig if q, exists := rl[v1.ResourceMemory]; exists { // Memory is defined in bytes. val := q.Value() rc.Memory = &val } if q, exists := rl[v1.ResourceCPU]; exists { // CPU is defined in milli-cores. val := MilliCPUToShares(q.MilliValue()) rc.CpuShares = &val } if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.HugePages) { rc.HugePageLimit = HugePageLimits(rl) } return &rc } // getNodeAllocatableAbsolute returns the absolute value of Node Allocatable which is primarily useful for enforcement. // Note that not all resources that are available on the node are included in the returned list of resources. // Returns a ResourceList. func (cm *containerManagerImpl) getNodeAllocatableAbsolute() v1.ResourceList { result := make(v1.ResourceList) for k, v := range cm.capacity { value := *(v.Copy()) if cm.NodeConfig.SystemReserved != nil { value.Sub(cm.NodeConfig.SystemReserved[k]) } if cm.NodeConfig.KubeReserved != nil { value.Sub(cm.NodeConfig.KubeReserved[k]) } if value.Sign() < 0 { // Negative Allocatable resources don't make sense. value.Set(0) } result[k] = value } return result } // GetNodeAllocatableReservation returns amount of compute or storage resource that have to be reserved on this node from scheduling. func (cm *containerManagerImpl) GetNodeAllocatableReservation() v1.ResourceList { evictionReservation := hardEvictionReservation(cm.HardEvictionThresholds, cm.capacity) result := make(v1.ResourceList) for k := range cm.capacity { value := resource.NewQuantity(0, resource.DecimalSI) if cm.NodeConfig.SystemReserved != nil { value.Add(cm.NodeConfig.SystemReserved[k]) } if cm.NodeConfig.KubeReserved != nil { value.Add(cm.NodeConfig.KubeReserved[k]) } if evictionReservation != nil { value.Add(evictionReservation[k]) } if !value.IsZero() { result[k] = *value } } return result } // validateNodeAllocatable ensures that the user specified Node Allocatable Configuration doesn't reserve more than the node capacity. // Returns error if the configuration is invalid, nil otherwise. func (cm *containerManagerImpl) validateNodeAllocatable() error { var errors []string nar := cm.GetNodeAllocatableReservation() for k, v := range nar { value := cm.capacity[k].DeepCopy() value.Sub(v) if value.Sign() < 0 { errors = append(errors, fmt.Sprintf("Resource %q has an allocatable of %v, capacity of %v", k, v, value)) } } if len(errors) > 0 { return fmt.Errorf("Invalid Node Allocatable configuration. %s", strings.Join(errors, " ")) } return nil } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" "k8s.io/kubernetes/pkg/scheduler/algorithm/priorities" "k8s.io/kubernetes/pkg/scheduler/api" <ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" </ins> kubeschedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/core" "k8s.io/kubernetes/pkg/scheduler/factory"

Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". In latest scheduler componentconfig API (v1alpha1), it can be a file or a configmap: However, if you defines a scheduler config file and specify a YAML policy file: It doesn't work. It's because only JSON is supported: It'd nice to enhance it with an UniversalSerilizer so as to support both JSON and YAML. /sig scheduling /help /priority important-longterm
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". /help /priority important-longterm </details
/assign


Question: } }) }) <del> }) // construct a fakecpu so as to set it to status of Node object // otherwise if we update CPU/Memory/etc, those values will be corrected back by kubelet var fakecpu v1.ResourceName = "example.com/fakecpu" </del> <ins> ginkgo.Context("PreemptionExecutionPath", func() { // construct a fakecpu so as to set it to status of Node object // otherwise if we update CPU/Memory/etc, those values will be corrected back by kubelet var fakecpu v1.ResourceName = "example.com/fakecpu" var cs clientset.Interface var node *v1.Node var ns, nodeHostNameLabel string f := framework.NewDefaultFramework("sched-preemption-path") </ins> <del> var _ = SIGDescribe("PreemptionExecutionPath", func() { var cs clientset.Interface var node *v1.Node var ns, nodeHostNameLabel string f := framework.NewDefaultFramework("sched-preemption-path") </del> <ins> priorityPairs := make([]priorityPair, 0) </ins> <del> priorityPairs := make([]priorityPair, 0) </del> <ins> ginkgo.AfterEach(func() { // print out additional info if tests failed if ginkgo.CurrentGinkgoTestDescription().Failed { // list existing priorities priorityList, err := cs.SchedulingV1().PriorityClasses().List(context.TODO(), metav1.ListOptions{}) if err != nil { framework.Logf("Unable to list priorities: %v", err) } else { framework.Logf("List existing priorities:") for _, p := range priorityList.Items { framework.Logf("%v/%v created at %v", p.Name, p.Value, p.CreationTimestamp) } } } </ins> <del> ginkgo.AfterEach(func() { // print out additional info if tests failed if ginkgo.CurrentGinkgoTestDescription().Failed { // list existing priorities priorityList, err := cs.SchedulingV1().PriorityClasses().List(context.TODO(), metav1.ListOptions{}) </del> <ins> if node != nil { nodeCopy := node.DeepCopy() // force it to update nodeCopy.ResourceVersion = "0" delete(nodeCopy.Status.Capacity, fakecpu) _, err := cs.CoreV1().Nodes().UpdateStatus(context.TODO(), nodeCopy, metav1.UpdateOptions{}) framework.ExpectNoError(err) } for _, pair := range priorityPairs { cs.SchedulingV1().PriorityClasses().Delete(context.TODO(), pair.name, metav1.NewDeleteOptions(0)) } }) ginkgo.BeforeEach(func() { cs = f.ClientSet ns = f.Namespace.Name // find an available node ginkgo.By("Finding an available node") nodeName := GetNodeThatCanRunPod(f) framework.Logf("found a healthy node: %s", nodeName) // get the node API object var err error node, err = cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) </ins> if err != nil { <del> framework.Logf("Unable to list priorities: %v", err) } else { framework.Logf("List existing priorities:") for _, p := range priorityList.Items { framework.Logf("%v/%v created at %v", p.Name, p.Value, p.CreationTimestamp) } </del> <ins> framework.Failf("error getting node %q: %v", nodeName, err) } var ok bool nodeHostNameLabel, ok = node.GetObjectMeta().GetLabels()["kubernetes.io/hostname"] if !ok { framework.Failf("error getting kubernetes.io/hostname label on node %s", nodeName) </ins> } <del> } if node != nil { </del> <ins> // update Node API object with a fake resource </ins> nodeCopy := node.DeepCopy() // force it to update nodeCopy.ResourceVersion = "0" <del> delete(nodeCopy.Status.Capacity, fakecpu) _, err := cs.CoreV1().Nodes().UpdateStatus(context.TODO(), nodeCopy, metav1.UpdateOptions{}) </del> <ins> nodeCopy.Status.Capacity[fakecpu] = resource.MustParse("1000") node, err = cs.CoreV1().Nodes().UpdateStatus(context.TODO(), nodeCopy, metav1.UpdateOptions{}) </ins> framework.ExpectNoError(err) <del> } for _, pair := range priorityPairs { cs.SchedulingV1().PriorityClasses().Delete(context.TODO(), pair.name, metav1.NewDeleteOptions(0)) } }) ginkgo.BeforeEach(func() { cs = f.ClientSet ns = f.Namespace.Name // find an available node ginkgo.By("Finding an available node") nodeName := GetNodeThatCanRunPod(f) framework.Logf("found a healthy node: %s", nodeName) // get the node API object var err error node, err = cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { framework.Failf("error getting node %q: %v", nodeName, err) } var ok bool nodeHostNameLabel, ok = node.GetObjectMeta().GetLabels()["kubernetes.io/hostname"] if !ok { framework.Failf("error getting kubernetes.io/hostname label on node %s", nodeName) } // update Node API object with a fake resource nodeCopy := node.DeepCopy() // force it to update nodeCopy.ResourceVersion = "0" nodeCopy.Status.Capacity[fakecpu] = resource.MustParse("1000") node, err = cs.CoreV1().Nodes().UpdateStatus(context.TODO(), nodeCopy, metav1.UpdateOptions{}) framework.ExpectNoError(err) // create four PriorityClass: p1, p2, p3, p4 for i := 1; i <= 4; i++ { priorityName := fmt.Sprintf("p%d", i) priorityVal := int32(i) priorityPairs = append(priorityPairs, priorityPair{name: priorityName, value: priorityVal}) _, err := cs.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: priorityName}, Value: priorityVal}, metav1.CreateOptions{}) if err != nil { framework.Logf("Failed to create priority '%v/%v': %v", priorityName, priorityVal, err) framework.Logf("Reason: %v. Msg: %v", apierrors.ReasonForError(err), err) </del> <ins> // create four PriorityClass: p1, p2, p3, p4 for i := 1; i <= 4; i++ { priorityName := fmt.Sprintf("p%d", i) priorityVal := int32(i) priorityPairs = append(priorityPairs, priorityPair{name: priorityName, value: priorityVal}) _, err := cs.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: priorityName}, Value: priorityVal}, metav1.CreateOptions{}) if err != nil { framework.Logf("Failed to create priority '%v/%v': %v", priorityName, priorityVal, err) framework.Logf("Reason: %v. Msg: %v", apierrors.ReasonForError(err), err) } framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </ins> } <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) } }) </del> <ins> }) </ins> <del> ginkgo.It("runs ReplicaSets to verify preemption running path", func() { podNamesSeen := []int32{0, 0, 0} stopCh := make(chan struct{}) </del> <ins> ginkgo.It("runs ReplicaSets to verify preemption running path", func() { podNamesSeen := []int32{0, 0, 0} stopCh := make(chan struct{}) </ins> <del> // create a pod controller to list/watch pod events from the test framework namespace _, podController := cache.NewInformer( &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { obj, err := f.ClientSet.CoreV1().Pods(ns).List(context.TODO(), options) return runtime.Object(obj), err }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { return f.ClientSet.CoreV1().Pods(ns).Watch(context.TODO(), options) </del> <ins> // create a pod controller to list/watch pod events from the test framework namespace _, podController := cache.NewInformer( &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { obj, err := f.ClientSet.CoreV1().Pods(ns).List(context.TODO(), options) return runtime.Object(obj), err }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { return f.ClientSet.CoreV1().Pods(ns).Watch(context.TODO(), options) }, </ins> }, <del> }, &v1.Pod{}, 0, cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { if pod, ok := obj.(*v1.Pod); ok { if strings.HasPrefix(pod.Name, "rs-pod1") { atomic.AddInt32(&podNamesSeen[0], 1) } else if strings.HasPrefix(pod.Name, "rs-pod2") { atomic.AddInt32(&podNamesSeen[1], 1) } else if strings.HasPrefix(pod.Name, "rs-pod3") { atomic.AddInt32(&podNamesSeen[2], 1) </del> <ins> &v1.Pod{}, 0, cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { if pod, ok := obj.(*v1.Pod); ok { if strings.HasPrefix(pod.Name, "rs-pod1") { atomic.AddInt32(&podNamesSeen[0], 1) } else if strings.HasPrefix(pod.Name, "rs-pod2") { atomic.AddInt32(&podNamesSeen[1], 1) } else if strings.HasPrefix(pod.Name, "rs-pod3") { atomic.AddInt32(&podNamesSeen[2], 1) } </ins> } <del> } </del> <ins> }, </ins> }, <del> }, ) go podController.Run(stopCh) defer close(stopCh) // prepare three ReplicaSet rsConfs := []pauseRSConfig{ { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod1", Namespace:         ns, Labels:            map[string]string{"name": "pod1"}, PriorityClassName: "p1", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("200")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("200")}, </del> <ins> ) go podController.Run(stopCh) defer close(stopCh) // prepare three ReplicaSet rsConfs := []pauseRSConfig{ { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod1", Namespace:         ns, Labels:            map[string]string{"name": "pod1"}, PriorityClassName: "p1", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("200")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("200")}, }, </ins> }, }, <del> }, { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod2", Namespace:         ns, Labels:            map[string]string{"name": "pod2"}, PriorityClassName: "p2", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("300")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("300")}, </del> <ins> { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod2", Namespace:         ns, Labels:            map[string]string{"name": "pod2"}, PriorityClassName: "p2", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("300")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("300")}, }, </ins> }, }, <del> }, { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod3", Namespace:         ns, Labels:            map[string]string{"name": "pod3"}, PriorityClassName: "p3", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("450")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("450")}, </del> <ins> { Replicas: int32(1), PodConfig: pausePodConfig{ Name:              "pod3", Namespace:         ns, Labels:            map[string]string{"name": "pod3"}, PriorityClassName: "p3", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("450")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("450")}, }, </ins> }, }, <del> }, } // create ReplicaSet{1,2,3} so as to occupy 950/1000 fake resource for i := range rsConfs { runPauseRS(f, rsConfs[i]) } </del> <ins> } // create ReplicaSet{1,2,3} so as to occupy 950/1000 fake resource for i := range rsConfs { runPauseRS(f, rsConfs[i]) } framework.Logf("pods created so far: %v", podNamesSeen) framework.Logf("length of pods created so far: %v", len(podNamesSeen)) </ins> <del> framework.Logf("pods created so far: %v", podNamesSeen) framework.Logf("length of pods created so far: %v", len(podNamesSeen)) </del> <ins> // create a Preemptor Pod preemptorPodConf := pausePodConfig{ Name:              "pod4", Namespace:         ns, Labels:            map[string]string{"name": "pod4"}, PriorityClassName: "p4", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("500")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("500")}, }, } preemptorPod := createPod(f, preemptorPodConf) waitForPreemptingWithTimeout(f, preemptorPod, framework.PodGetTimeout) </ins> <del> // create a Preemptor Pod preemptorPodConf := pausePodConfig{ Name:              "pod4", Namespace:         ns, Labels:            map[string]string{"name": "pod4"}, PriorityClassName: "p4", NodeSelector:      map[string]string{"kubernetes.io/hostname": nodeHostNameLabel}, Resources: &v1.ResourceRequirements{ Requests: v1.ResourceList{fakecpu: resource.MustParse("500")}, Limits:   v1.ResourceList{fakecpu: resource.MustParse("500")}, }, } preemptorPod := createPod(f, preemptorPodConf) waitForPreemptingWithTimeout(f, preemptorPod, framework.PodGetTimeout) </del> <ins> framework.Logf("pods created so far: %v", podNamesSeen) </ins> <del> framework.Logf("pods created so far: %v", podNamesSeen) </del> <ins> // count pods number of ReplicaSet{1,2,3}: // - if it's more than expected replicas, it denotes its pods have been over-preempted // - if it's less than expected replicas, it denotes its pods are under-preempted // "*2" means pods of ReplicaSet{1,2} are expected to be only preempted once. expectedRSPods := []int32{1 * 2, 1 * 2, 1} err := wait.Poll(framework.Poll, framework.PollShortTimeout, func() (bool, error) { for i := 0; i < len(podNamesSeen); i++ { got := atomic.LoadInt32(&podNamesSeen[i]) if got < expectedRSPods[i] { framework.Logf("waiting for rs%d to observe %d pod creations, got %d", i+1, expectedRSPods[i], got) return false, nil } else if got > expectedRSPods[i] { return false, fmt.Errorf("rs%d had more than %d pods created: %d", i+1, expectedRSPods[i], got) } } return true, nil }) if err != nil { framework.Logf("pods created so far: %v", podNamesSeen) framework.Failf("failed pod observation expectations: %v", err) } </ins> <del> // count pods number of ReplicaSet{1,2,3}: // - if it's more than expected replicas, it denotes its pods have been over-preempted // - if it's less than expected replicas, it denotes its pods are under-preempted // "*2" means pods of ReplicaSet{1,2} are expected to be only preempted once. expectedRSPods := []int32{1 * 2, 1 * 2, 1} err := wait.Poll(framework.Poll, framework.PollShortTimeout, func() (bool, error) { </del> <ins> // If logic continues to here, we should do a final check to ensure within a time period, // the state is stable; otherwise, pods may be over-preempted. time.Sleep(5 * time.Second) </ins> for i := 0; i < len(podNamesSeen); i++ { got := atomic.LoadInt32(&podNamesSeen[i]) if got < expectedRSPods[i] { <del> framework.Logf("waiting for rs%d to observe %d pod creations, got %d", i+1, expectedRSPods[i], got) return false, nil </del> <ins> framework.Failf("pods of ReplicaSet%d have been under-preempted: expect %v pod names, but got %d", i+1, expectedRSPods[i], got) </ins> } else if got > expectedRSPods[i] { <del> return false, fmt.Errorf("rs%d had more than %d pods created: %d", i+1, expectedRSPods[i], got) </del> <ins> framework.Failf("pods of ReplicaSet%d have been over-preempted: expect %v pod names, but got %d", i+1, expectedRSPods[i], got) </ins> } } <del> return true, nil </del> }) <del> if err != nil { framework.Logf("pods created so far: %v", podNamesSeen) framework.Failf("failed pod observation expectations: %v", err) } // If logic continues to here, we should do a final check to ensure within a time period, // the state is stable; otherwise, pods may be over-preempted. time.Sleep(5 * time.Second) for i := 0; i < len(podNamesSeen); i++ { got := atomic.LoadInt32(&podNamesSeen[i]) if got < expectedRSPods[i] { framework.Failf("pods of ReplicaSet%d have been under-preempted: expect %v pod names, but got %d", i+1, expectedRSPods[i], got) } else if got > expectedRSPods[i] { framework.Failf("pods of ReplicaSet%d have been over-preempted: expect %v pod names, but got %d", i+1, expectedRSPods[i], got) } } </del> }) })

now is separately organized as top-level SIGDescribe. However, it's better folded into the existing top-level SIGDescribe, and it can be refactored as a . /sig scheduling /kind cleanup /good-first-issue
I'm new to Kubernetes and I'd like to work on this issue :)
Sure, feel free to take it.
/assign
Seems like the doesn't follow this issue. I will take it.
are you still working on it?
/assign
I will be working on this during the weekend because I'm on call this week.
As this is a good-first-issue issue. Let's leave it to new contributor.
Sure.
created a PR for this.
It seems was raised slightly earlier than your PR, sorry that we have to follow the first-come-first-serve principle. /remove-good-first-issue


Question: } logger.V(4).Info("Update endpoints", "service", klog.KObj(service), "readyEndpoints", totalReadyEps, "notreadyEndpoints", totalNotReadyEps) <ins> var updatedEndpoints *v1.Endpoints </ins> if createEndpoints { // No previous endpoints, create them _, err = e.client.CoreV1().Endpoints(service.Namespace).Create(ctx, newEndpoints, metav1.CreateOptions{}) } else { // Pre-existing <del> _, err = e.client.CoreV1().Endpoints(service.Namespace).Update(ctx, newEndpoints, metav1.UpdateOptions{}) </del> <ins> updatedEndpoints, err = e.client.CoreV1().Endpoints(service.Namespace).Update(ctx, newEndpoints, metav1.UpdateOptions{}) </ins> } if err != nil { if createEndpoints && errors.IsForbidden(err) {

This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?

Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it.

Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign


Question: {Name: "a999", Region: region, Hosts: []string{"node0", "node1", "node73"}}, }, }, <ins> { // One service has an external load balancer and one is nil: one call. services: []*api.Service{ newService("s0", "234", true), nil, }, expectedUpdateCalls: []fake_cloud.FakeUpdateBalancerCall{ {Name: "a234", Region: region, Hosts: []string{"node0", "node1", "node73"}}, }, }, </ins> } for _, item := range table { cloud := &fake_cloud.FakeCloud{}

In , I found a logic error (when I had a broken cloud implementation): If createLoadBalancerIfNeeded returns err, then cachedService will have service = nil. This causes updateLoadBalancerHosts to panic. It would be easy to skip cacheService if service=nil, but I wasn't sure whether this should then be included in servicesToRetry or not. Another problem (by inspection) is that I think there's a race condition where updateLoadBalancerHosts can run immediately after getOrCreate, before the service field has been set.
/cc
Great catches, Justin, thanks! I believe that simply not processing cached services with a nil api.Service object in the node pool reconciler is the correct approach. A nil Service object implies that the load balancer reconciler still intends to process it, and the LB reconciler always uses the most up-to-date list of nodes. I'll send a PR ASAP.


Question: } glog.Infof("pod %q container %q is unhealthy. Container will be killed and re-created.", podFullName, container.Name, live) } else { <del> podChanged = true </del> glog.Infof("pod %q container %q hash changed (%d vs %d). Container will be killed and re-created.", podFullName, container.Name, hash, expectedHash) } if err := kl.killContainer(dockerContainer); err != nil {

cAdvisor API is not accessible from the node. It is accessible from inside the cadvisor container though. This is the reason for the cadvisor e2e test to fail and the recent heapster errors reported by users. The cadvisor version itself hasn't changed recently though.
cc
There were some changes recently around port forwarding ( that may be the cause.
From the node: From the cadvisor container:
reported the same issue today. Restarting the container seems to fix it. So, its something to do with the initial setup.
Could this be due to Docker 1.5? On Feb 23, 2015 6:12 PM, "Rohit Jnagal" wrote:
test machine is running Docker 1.4.1 It looks like cadvisor container is sharing its networking with the pause container as usual, but the pause container got restarted, so the networking setup isn't working anymore. Restarting cadvisor connects it to the newer instance of the pause container and it starts working again. However docker inspect shows that cadvisor is connected to network of a different pause container: And that container died soon after it started Kubelet killed the pause container. I think this is kubelet's doing. It tried to restart both pause and cadvisor container. It kills and restarts cadvisor container which connects to existing pause container. Then, it goes ahead and kills/restarts the pause container, leaving the new cadvisor container hanging. Maybe this is related to infra pods issue reported by earlier, or maybe just an ordering issue in kubelet.
Thanks for the detailed debugging :) with your report I think I found the issue. In we check if the container has changed and print the message above. After that we the POD container. We should be doing this before the container restart as you pointed out. I'm assigning this to myself and I will fix tomorrow. Will also rename the bug and re-purpose it.
cool, I was just staring at that code. All yours :+1:
Thanks for debugging the issue. This is a regression we introduced lately.
And when the test, we all assumed it was the test failing. Sorry, :)
I fell for that until I realized that the basic networking can also fail :)


Question: configMap, err := newConfigMapWithEmptyKey(f) framework.ExpectError(err, "created configMap %q with empty key in namespace %q", configMap.Name, f.Namespace.Name) }) <ins> ginkgo.It("should patch ConfigMap successfully", func() { name := "configmap-test-" + string(uuid.NewUUID()) configMap := newConfigMap(f, name) ginkgo.By(fmt.Sprintf("Creating configMap %v/%v", f.Namespace.Name, configMap.Name)) _, err := f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap) framework.ExpectNoError(err) configMap.Data = map[string]string{ "data": "value", } ginkgo.By(fmt.Sprintf("Updating configMap %v/%v", f.Namespace.Name, configMap.Name)) _, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Update(configMap) framework.ExpectNoError(err) configMapFromUpdate, err := f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Get(name, metav1.GetOptions{}) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("Verifying update of configMap %v/%v", f.Namespace.Name, configMap.Name)) framework.ExpectEqual(configMapFromUpdate.Data, configMap.Data) }) </ins> }) func newEnvFromConfigMap(f *framework.Framework, name string) *v1.ConfigMap {

Using the list of tested stable endpoints referenced in , we can pick out the endpoints which are part of the ConfigMap kind: This is a stable core endpoint which has existing testing, potentially making for low-hanging fruit to upgrade to Conformance. /area test /area conformance /sig testing
/assign
Promoting will increase Conformance Coverage by 1 endpoint.
PR was closed because AdmissionWebHook is not yet GA. /close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : PR was closed because AdmissionWebHook is not yet GA. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test uses a beta version of the API. According to the , this is prohibited. The URL of the API which is test is hitting is .
/unassign The original PR was superseded by 's


Question: } } count++ <ins> if len(output) > 0 && !shortOutput { return cmdutil.PrintResourceInfoForCommand(cmd, info, f, out) } </ins> cmdutil.PrintSuccess(mapper, shortOutput, out, info.Mapping.Resource, info.Name, dryRun, "configured") return nil })

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: inj:  injectedResult{PreFilterStatus: int(framework.Error)}, }, }, <del> wantPreFilterResult: nil, wantStatusCode:      framework.Error, </del> <ins> wantSkippedPlugins: sets.New("skip"), wantStatusCode:     framework.Error, </ins> }, { name: "all PreFilter plugins returned skip",

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: }, expectedElements: []string{"Driver", "VolumeHandle", "ReadOnly", "VolumeAttributes"}, }, <ins> { name:   "test19", plugin: "gluster", pv: &corev1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: corev1.PersistentVolumeSpec{ PersistentVolumeSource: corev1.PersistentVolumeSource{ Glusterfs: &corev1.GlusterfsPersistentVolumeSource{ EndpointsNamespace: &foo, }, }, }, }, expectedElements:   []string{"EndpointsNamespace", "glusterfsendpointname"}, unexpectedElements: []string{"VolumeMode", "Filesystem"}, }, </ins> } for _, test := range testCases {

What happened: What you expected to happen: EndpointNamespace should be the string name of the namespace used. How to reproduce it (as minimally and precisely as possible): Create a glusterfs PV using an EndpointNamespace entry and run 'kubectl describe pv' on the resulting PV. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T17:09:13Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): Linux bionic-vm 4.15.0-58-generic -Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.15
/sig cli
/assign


Question: end # Kubernetes master <del> config.vm.define "master" do |config| customize_vm config config.vm.provision "shell", run: "always", path: "#{ENV['KUBE_TEMP']}/master-start.sh" config.vm.network "private_network", ip: "#{$master_ip}" config.vm.hostname = ENV['MASTER_NAME'] </del> <ins> config.vm.define "master" do |c| customize_vm c if ENV['KUBE_TEMP'] then script = "#{ENV['KUBE_TEMP']}/master-start.sh" c.vm.provision "shell", run: "always", path: script end c.vm.network "private_network", ip: "#{$master_ip}" c.vm.hostname = ENV['MASTER_NAME'] </ins> end # Kubernetes minion

The Vagrantfile requires that ENV['KUBE_TEMP'] is set. If it is not, any vagrant command will fail as the file cannot be parsed. The error received is: This is contrary to the suggested method of interaction in the documentation found at which says:
Looking at the scripts, this temp directory is set on script run and cleaned up on EXIT by a trap. I've wrapped a conditional around the provision statement which allows the other commands to work, but outputs a warning that the vm will not have a provisioner set. Pull req incoming.
Cc me on your PR and I will review. I have been swamped on other topics so it's possible something regressed.


Question: "key3": "value3", } annotations := map[string]string{ <del> "a1": "value1", "a2": "value2", </del> <ins> "a1":        "value1", "a2":        "value2", "multiline": "cnbna", </ins> } testCases := []struct { name           string

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: # taint/untaint # Pre-condition: node doesn't have dedicated=foo:PreferNoSchedule taint kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "" # expect no output <del> # taint can add a taint </del> <ins> # taint can add a taint (<key>=<value>:<effect>) </ins> kubectl taint node 127.0.0.1 dedicated=foo:PreferNoSchedule kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "dedicated=foo:PreferNoSchedule" # taint can remove a taint kubectl taint node 127.0.0.1 dedicated- <ins> # taint can add a taint (<key>:<effect>) kubectl taint node 127.0.0.1 dedicated:PreferNoSchedule kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "dedicated=<no value>:PreferNoSchedule" # taint can remove a taint kubectl taint node 127.0.0.1 dedicated- </ins> # Post-condition: node doesn't have dedicated=foo:PreferNoSchedule taint kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "" # expect no output

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: says: That doesn't work: It doesn't matter whether there is a node "foo" or not. It doesn't seem to contact the API at all. What you expected to happen: Examples form the help texts should work. Anything else we need to know?: Looking at , it seems to want either a or in the argument. Indeed, does work and does the right thing, i.e. set a taint with key and no value. Environment:
i believe there were updates about this at some point. a PR is welcome /remove-kind bug /kind documentation /sig cli side note: Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} always make sure the client MINOR version is equal or +1, or -1 of the server MINOR version. in this case it's +2, which is unsupported.
/assign bharaththiruveedula
Do you mind if I submmit a PR on this because I met this issue before?
I assigned the issue today itself and already started working on it. I will send the PR by end of today. But thanks for showing the interest.
Are we updating the documentation or the code now? Right now fails but works (because of the "-"), which isn't very consistent at all, so I'd suggest we actually fix kubectl itself to do what the help page says.


Question: return s } <ins> // skipDefaultPodTopologySpread returns true if the pod's TopologySpreadConstraints are specified. func skipDefaultPodTopologySpread(pod *v1.Pod) bool { return len(pod.Spec.TopologySpreadConstraints) != 0 } </ins> // Score invoked at the Score extension point. // The "score" returned in this function is the matching number of pods on the `nodeName`, // it is normalized later. func (pl *DefaultPodTopologySpread) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) { <ins> if skipDefaultPodTopologySpread(pod) { return 0, nil } </ins> c, err := state.Read(postFilterStateKey) if err != nil { return 0, framework.NewStatus(framework.Error, fmt.Sprintf("Error reading %q from cycleState: %v", postFilterStateKey, err))

Currently the scheduler by default spreads the pods of a deployment based on a hard-coded . We recently a pod object api (pod.Spec.TopologySpreadConstraints) that allows specifying spread behavior at the workload level. For pods that specify topology spread constraints, the default spreading behavior should not apply. /sig scheduling /cc
I can fix this behavior, can you assign it to me?
/assign sure, lets wait for feedback first.
sure, thanks
I agree. Having the default spreading run would just water down the score given by the constraints. Other than that, this would just be a temporary fix until is finalized


Question: function export-windows-docker-event-log() { local -r node="${1}" <del> local -r powershell_cmd="powershell.exe -Command $log=$(Get-EventLog -LogName Application -Source Docker); Set-Content '${WINDOWS_LOGS_DIR}docker.log' $log.Message" </del> <ins> local -r powershell_cmd="powershell.exe -Command "$logs=$(Get-EventLog -LogName Application -Source Docker | Select-Object -Property TimeGenerated, EntryType, Message); $logs | Out-File -FilePath '${WINDOWS_LOGS_DIR}docker.log'"" </ins> # Retry up to 3 times to allow ssh keys to be properly propagated and # stored.

/cc
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by


Question: t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:p") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a name + port number (service port 93 -> target port 80) location, _, err = redirector.ResourceLocation(ctx, "foo:93") if err != nil {

Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


Question: name: "v1.List in reverse", obj: &corev1.List{ Items: []runtime.RawExtension{ <del> {Raw: encodeOrDie(c)}, {Raw: encodeOrDie(b)}, {Raw: encodeOrDie(a)}, </del> <ins> {Object: c, Raw: encodeOrDie(c)}, {Object: b, Raw: encodeOrDie(b)}, {Object: a, Raw: encodeOrDie(a)}, </ins> }, }, sort: &corev1.List{ Items: []runtime.RawExtension{ <del> {Raw: encodeOrDie(a)}, {Raw: encodeOrDie(b)}, {Raw: encodeOrDie(c)}, </del> <ins> {Object: a, Raw: encodeOrDie(a)}, {Object: b, Raw: encodeOrDie(b)}, {Object: c, Raw: encodeOrDie(c)}, </ins> }, }, field: "{.metadata.name}",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: seems to have regressed in v1.13.0 and no longer sorts by creation timestamp. What you expected to happen: should sort by creation timestamp. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: N/A Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/assign
bisected to () /assign
opened to fix
/assign


Question: // added before `Run()`, eventually either the SharedInformer is // stopped or the client is notified of the update.  A client added // after `Run()` starts gets a startup batch of notifications of <del> // additions of the object existing in the cache at the time that </del> <ins> // additions of the objects existing in the cache at the time that </ins> // client was added; also, for every update to the SharedInformer's // local cache after that client was added, eventually either the // SharedInformer is stopped or that client is notified of that

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I discovered colleagues writing informer clients that modify objects delivered by informers. I went looking for where it says they should not do that, and I could not find it. What you expected to happen: I expected this critical restriction to be prominently documented. How to reproduce it (as minimally and precisely as possible): Try to find where it says that a client of an informer must not modify an object delivered by the informer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Documentation aside, I would suggest adding to your CI infra to make it harder for people to make this mistake.
/assign Thank you for fixing docs!


Question: Port *intstr.IntOrString } <del> // IPBlock describes a particular CIDR (Ex. "192.168.1.1/24") that is allowed to the pods // matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should // not be included within this rule. </del> <ins> // IPBlock describes a particular CIDR (Ex. "192.168.1.1/24","2001:db9::/64") that is allowed // to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs // that should not be included within this rule. </ins> type IPBlock struct { // CIDR is a string representing the IP Block <del> // Valid examples are "192.168.1.1/24" </del> <ins> // Valid examples are "192.168.1.1/24" or "2001:db9::/64" </ins> CIDR string // Except is a slice of CIDRs that should not be included within an IP Block <del> // Valid examples are "192.168.1.1/24" </del> <ins> // Valid examples are "192.168.1.1/24" or "2001:db9::/64" </ins> // Except values will be rejected if they are outside the CIDR range // +optional Except []string

<!-- Please only use this template for submitting enhancement requests --What would you like to be : mentions an IPBlock with only IPv4 examples. Add a comment or example to clarify that IPv6 network blocks are (or possibly aren't) valid here. Why is this needed: Cluster operators who use IPv6 networking want to understand how to isolate different workloads inside their cluster network (and without reading source code). Logged against kubernetes/kubernetes rather than the website repo because this is generated documentation. Kubernetes already has single-stack support for IPv6 clusters KEP for dual-stack support is /sig docs /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved can you guide me on what steps are needed to publish the examples?
I via the pull request. (Although I want these examples , it's not a task I'm familiar with).


Question: if [[ "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${MASTER_OS_DISTRIBUTION}" == "ubuntu" ]]; then MASTER_KUBELET_TEST_ARGS=" --experimental-kernel-memcg-notification=true" fi <del> APISERVER_TEST_ARGS="${APISERVER_TEST_ARGS:-} --vmodule=httplog=3 --runtime-config=extensions/v1beta1 ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT}" </del> <ins> APISERVER_TEST_ARGS="${APISERVER_TEST_ARGS:-} --vmodule=httplog=3 --runtime-config=extensions/v1beta1,scheduling.k8s.io/v1alpha1 ${TEST_CLUSTER_DELETE_COLLECTION_WORKERS} ${TEST_CLUSTER_MAX_REQUESTS_INFLIGHT}" </ins> CONTROLLER_MANAGER_TEST_ARGS="${CONTROLLER_MANAGER_TEST_ARGS:-} ${TEST_CLUSTER_RESYNC_PERIOD} ${TEST_CLUSTER_API_CONTENT_TYPE}" SCHEDULER_TEST_ARGS="${SCHEDULER_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" KUBEPROXY_TEST_ARGS="${KUBEPROXY_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pull-kubernetes-e2e-gce and pull-kubernetes-kubemark-e2e-gce both start kube-apiserver with admission and but without enabling priorityclasses and podpresets resources. We can see many error logs from
/sig test /sig api-machinery
/kind bug
/sig testing


Question: label:       "label=qux", expectedIDs: util.NewStringSet("qux"), }, { <del> field:       "Status.Phase=Failed", </del> <ins> field:       "status.phase=Failed", </ins> expectedIDs: util.NewStringSet("baz"), }, { <del> field:       "Status.Host=barhost", </del> <ins> field:       "status.host=barhost", </ins> expectedIDs: util.NewStringSet("bar"), }, { <del> field:       "Status.Host=", </del> <ins> field:       "status.host=", </ins> expectedIDs: util.NewStringSet("foo", "baz", "qux", "zot"), }, { <del> field:       "Status.Host!=", </del> <ins> field:       "status.host!=", </ins> expectedIDs: util.NewStringSet("bar"), }, }

We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.


Question: } func TestCreateContainerSecurityContextNonmutating(t *testing.T) { <del> // Create a pod with a security context that needs filling in createPod := func() *api.Pod { return &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{ SecurityContext: &api.SecurityContext{}, }}, }, } </del> <ins> untrue := false tests := []struct { security *api.SecurityContext }{ {nil}, {&api.SecurityContext{RunAsNonRoot: &untrue}}, </ins> } <del> // Create a PSP with strategies that will populate a blank security context createPSP := func() *extensions.PodSecurityPolicy { uid := types.UnixUserID(1) return &extensions.PodSecurityPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "psp-sa", Annotations: map[string]string{ seccomp.AllowedProfilesAnnotationKey: "*", seccomp.DefaultProfileAnnotationKey:  "foo", </del> <ins> for _, tc := range tests { // Create a pod with a security context that needs filling in createPod := func() *api.Pod { return &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{ SecurityContext: tc.security, }}, </ins> }, <del> }, Spec: extensions.PodSecurityPolicySpec{ DefaultAddCapabilities:   []api.Capability{"foo"}, RequiredDropCapabilities: []api.Capability{"bar"}, RunAsUser: extensions.RunAsUserStrategyOptions{ Rule:   extensions.RunAsUserStrategyMustRunAs, Ranges: []extensions.UserIDRange{{Min: uid, Max: uid}}, }, SELinux: extensions.SELinuxStrategyOptions{ Rule:           extensions.SELinuxStrategyMustRunAs, SELinuxOptions: &api.SELinuxOptions{User: "you"}, }, // these are pod mutating strategies that are tested above FSGroup: extensions.FSGroupStrategyOptions{ Rule: extensions.FSGroupStrategyRunAsAny, </del> <ins> } } // Create a PSP with strategies that will populate a blank security context createPSP := func() *extensions.PodSecurityPolicy { uid := types.UnixUserID(1) return &extensions.PodSecurityPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "psp-sa", Annotations: map[string]string{ seccomp.AllowedProfilesAnnotationKey: "*", seccomp.DefaultProfileAnnotationKey:  "foo", }, </ins> }, <del> SupplementalGroups: extensions.SupplementalGroupsStrategyOptions{ Rule: extensions.SupplementalGroupsStrategyRunAsAny, </del> <ins> Spec: extensions.PodSecurityPolicySpec{ DefaultAddCapabilities:   []api.Capability{"foo"}, RequiredDropCapabilities: []api.Capability{"bar"}, RunAsUser: extensions.RunAsUserStrategyOptions{ Rule:   extensions.RunAsUserStrategyMustRunAs, Ranges: []extensions.UserIDRange{{Min: uid, Max: uid}}, }, SELinux: extensions.SELinuxStrategyOptions{ Rule:           extensions.SELinuxStrategyMustRunAs, SELinuxOptions: &api.SELinuxOptions{User: "you"}, }, // these are pod mutating strategies that are tested above FSGroup: extensions.FSGroupStrategyOptions{ Rule: extensions.FSGroupStrategyRunAsAny, }, SupplementalGroups: extensions.SupplementalGroupsStrategyOptions{ Rule: extensions.SupplementalGroupsStrategyRunAsAny, }, // mutates the container SC by defaulting to true if container sets nil ReadOnlyRootFilesystem: true, </ins> }, <del> // mutates the container SC by defaulting to true if container sets nil ReadOnlyRootFilesystem: true, }, </del> <ins> } </ins> } <del> } pod := createPod() psp := createPSP() </del> <ins> pod := createPod() psp := createPSP() </ins> <del> provider, err := NewSimpleProvider(psp, "namespace", NewSimpleStrategyFactory()) if err != nil { t.Fatalf("unable to create provider %v", err) } sc, _, err := provider.CreateContainerSecurityContext(pod, &pod.Spec.Containers[0]) if err != nil { t.Fatalf("unable to create container security context %v", err) } </del> <ins> provider, err := NewSimpleProvider(psp, "namespace", NewSimpleStrategyFactory()) if err != nil { t.Fatalf("unable to create provider %v", err) } sc, _, err := provider.CreateContainerSecurityContext(pod, &pod.Spec.Containers[0]) if err != nil { t.Fatalf("unable to create container security context %v", err) } </ins> <del> // The generated security context should have filled in missing options, so they should differ if reflect.DeepEqual(sc, &pod.Spec.Containers[0].SecurityContext) { t.Error("expected created security context to be different than container's, but they were identical") } </del> <ins> // The generated security context should have filled in missing options, so they should differ if reflect.DeepEqual(sc, &pod.Spec.Containers[0].SecurityContext) { t.Error("expected created security context to be different than container's, but they were identical") } </ins> <del> // Creating the provider or the security context should not have mutated the psp or pod if !reflect.DeepEqual(createPod(), pod) { diffs := diff.ObjectDiff(createPod(), pod) t.Errorf("pod was mutated by CreateContainerSecurityContext. diff:n%s", diffs) } if !reflect.DeepEqual(createPSP(), psp) { t.Error("psp was mutated by CreateContainerSecurityContext") </del> <ins> // Creating the provider or the security context should not have mutated the psp or pod if !reflect.DeepEqual(createPod(), pod) { diffs := diff.ObjectDiff(createPod(), pod) t.Errorf("pod was mutated by CreateContainerSecurityContext. diff:n%s", diffs) } if !reflect.DeepEqual(createPSP(), psp) { t.Error("psp was mutated by CreateContainerSecurityContext") } </ins> } }

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): PodSecurityPolicy RunAsNonRoot Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta, 1.8.0 (HEAD) Environment: Enable PodSecurityPolicy Cloud provider or hardware configuration: gce, probably any OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: What happened: The PSP admission controller ignores the specified RunAsNonRoot, and overwrites it. What you expected to happen: the admission controller is not supposed to overwrite user specified fields in the podsecuritycontext and securitycontext. How to reproduce it (as minimally and precisely as possible): enable psp, and create a pod with [only] RunAsNonRoot=false specified in a securitycontext. Anything else we need to know:
/sig auth
there are actually 2 bugs here. fails to properly restrict this field when matching against a PSP, which is a latent bug as long as we overwrite the value anyway. overwrites the user-supplied setting for the securitycontext/podsecuritycontext. I suspect the best fix will be to use the API's deep copy function. We are also doing a couple other sketchy things: assigning the "copy" built-in as a variable, and creating an "sc" shadow variable.

I have a half fix in but we should probably fix it in a separate PR, because we may want to patch 1.6.
the effect is that PSP can overly restrict a pod that wanted to run as root, but expressed that as instead of since the workaround is easy (set in your pod if you want to run as root), I don't anticipate backporting the fix.
SGTM. I'll try and get the unit tests in after the GKE summit (on friday).


Question: minSyncPeriod: 10s syncPeriod: 60s kind: KubeProxyConfiguration <del> metricsBindAddress: 2.3.4.5:23456 </del> <ins> metricsBindAddress: "%s" </ins> mode: "iptables" oomScoreAdj: 17 portRange: "2-7"

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? is:issue is:open ipv6 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): N/A, these are unit test changes in latest master Environment: Cloud provider or hardware configuration: Kubernetes container dev env on Ubuntu 16.04.1 OS (e.g. from /etc/os-release): Ubuntu 16.04.1 Kernel (e.g. ): 4.4.0-78-generic -Ubuntu x8664 GNU/Linux Install tools: Others: What happened: Upon inspection, no IPv6 test cases found in kubernetes/cmd/kube- What you expected to happen: Expect to see some IPv6 test cases exercised for kube-proxy server code (kubernetes/cmd/kube-). Also need some code coverage for negative conditions in How to reproduce it (as minimally and precisely as possible): Run unit tests in kubernetes/cmd/kube- Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig networking
/area ipv6
/sig network


Question: } }) <del> It("It should present successfully [Conformance]", func() { </del> <ins> It("It should present successfully", func() { </ins> for _, conformImage := range conformImages { present, err := conformImage.Present() Expect(err).ShouldNot(HaveOccurred())

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: KUBE_CLIENT_PLATFORMS=( darwin/amd64 linux/amd64 <del> ) </del> <ins> ) </ins> readonly KUBE_CLIENT_PLATFORMS else KUBE_TEST_PLATFORMS=(linux/amd64)

Commits and on master via PRs and have broken MacOS 'make' again: where "MacOS" is a non-homebrew extended MacOS with old old bash. This is also reported on other older platforms like CentOS. Master needs fixed with a new commit, then the above PRs and and the new fix need cherry picked to 1.14 as 1.14.2 is currently not buildable as is cherry picked there. :bash_fire"
/sig release /priority critical-urgent
/cc is looking at this


Question: s := args[0] args = args[1:] switch { <ins> case s == "--": // "--" terminates the flags break Loop </ins> case strings.HasPrefix(s, "--") && !strings.Contains(s, "=") && !hasNoOptDefVal(s[2:], flags): // If '--flag arg' then // delete arg from args.

BUG REPORT: /kind bug What happened: Arguments to a command inside pod are treated as kubectl flags and deprecation warnings are shown for them from . What you expected to happen: Arguments to a command inside pod should not be treated as kubectl flags by . How to reproduce it: The arguments , , , passed to the command inside the pod above are treated as flags and deprecation warnings are shown for them from . Environment: Kubernetes version: 9: Cloud provider or hardware configuration: Ubuntu Xenial VM. OS:Kernel:Install tools: minikube:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is it ok if we remove this warning ?
the warning should be removed just for the arguments to a command inside pod that are passed after the , and not for the arguments to itself. The warning is still valid and needed for arguments with underscore () to itself.
This is a bug of cobra. I've sent out a PR to fix it. And bumped to the fixed version in .
Thanks for fixing this!


Question: s.CgroupRoot = "/" } <del> var reservedSystemCPUs cpuset.CPUSet if s.ReservedSystemCPUs != "" { // is it safe do use CAdvisor here ?? machineInfo, err := kubeDeps.CAdvisorInterface.MachineInfo() if err != nil { // if can't use CAdvisor here, fall back to non-explicit cpu list behavior klog.InfoS("Failed to get MachineInfo, set reservedSystemCPUs to empty") reservedSystemCPUs = cpuset.NewCPUSet() } else { var errParse error reservedSystemCPUs, errParse = cpuset.Parse(s.ReservedSystemCPUs) if errParse != nil { // invalid cpu list is provided, set reservedSystemCPUs to empty, so it won't overwrite kubeReserved/systemReserved klog.InfoS("Invalid ReservedSystemCPUs", "systemReservedCPUs", s.ReservedSystemCPUs) return errParse } reservedList := reservedSystemCPUs.ToSlice() first := reservedList[0] last := reservedList[len(reservedList)-1] if first < 0 || last >= machineInfo.NumCores { // the specified cpuset is outside of the range of what the machine has klog.InfoS("Invalid cpuset specified by --reserved-cpus") return fmt.Errorf("Invalid cpuset %q specified by --reserved-cpus", s.ReservedSystemCPUs) } } } else { reservedSystemCPUs = cpuset.NewCPUSet() </del> <ins> machineInfo, err := kubeDeps.CAdvisorInterface.MachineInfo() if err != nil { return err } reservedSystemCPUs, err := getReservedCPUs(machineInfo, s.ReservedSystemCPUs) if err != nil { return err </ins> } if reservedSystemCPUs.Size() > 0 { // at cmd option validation phase it is tested either --system-reserved-cgroup or --kube-reserved-cgroup is specified, so overwrite should be ok klog.InfoS("Option --reserved-cpus is specified, it will overwrite the cpu setting in KubeReserved and SystemReserved", "kubeReservedCPUs", s.KubeReserved, "systemReservedCPUs", s.SystemReserved)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On a node that has a topology of more than 1 numa node, If some or all of the CPUs of a numa node are turned off (mostly for testing purposes) a valid set can cause the Kubelet service to fail on: This failure has occurred on a node that has 2 numa nodes with hyperthreading enabled. What you expected to happen: When setting a valid set, the Kubelet service should pass validations and function properly even if some CPU cores are turned off but the reserved cpus are still under a valid range. How to reproduce it (as minimally and precisely as possible): minimal cluster should have 1 master and 1 worker node. worker node should have 2 Numa nodes and hyperthreading enabled(BIOS setting) Important: The CPUs divided between the numa nodes cannot be sequential in order to reproduce the issue that are under Numa node 0 only (under ) Kubelet service off all CPUs that belong to numa node 1 () Kubelet service systemctl status kubelet.service and journal logs that will present: Anything else we need to know?: The scenario of split ranges of Numa nodes might occur only with Hyperthreading enabled. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ): (real time kernel) Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a real example where the error occurred: For the following CPU layout (after turning off numa node 1 CPUs) using : will result in: CPUs 0-2 and 40-42 are all under numa node 0 and are active.

The validation fails here: In this scenario we have 40 active CPUs (all from numa node 0) but the range check will fail on the last item in reserved CPUs: 42 (last time) = 40 (NumCores)
/cc
/sig node
This is probably not just strictly related to NUMA. The same behaviour could happen when offlining CPUs on any multi-core system, for example: disabling HT on a single numa node may end up with a very similar scenario (depends on the CPU specifics). The kubelet validation is too naive; it needs to be aware (and act accordingly to) the online CPU set. The emerging hidden assumption in kubelet is all the physical cpus are always online.
I wasn't able to recreate with all cpus on numa 1 offline, On-line CPU(s) list: 0-11,24-35 Off-line CPU(s) list: 12-23,36-47 with HT enabled, --reserved-cpus=0-3, kubelet restart fine but as mentioned by this option kubelet validation didn't consider the situation where some cpus are offline.
Ok I was able to reproduce when I used --reserved-cpus=28-35.
Just a FYI, even with the validation update, if you take certain CPUs offline, you need to delete /var/lib/kubelet/cpumanagerstate before restart kubelet. This state check is done by cpumanager, see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


Question: # gcutil will print the "external-ip" column header even if no instances are found KUBE_MASTER_IP=$(gcutil listinstances --format=csv --sort=external-ip  --columns=external-ip --zone ${ZONE} --filter="name eq ${MASTER_NAME}"  <del> | tail --lines=+2 | tail -n 1) </del> <ins> | tail -n '+2' | tail -n 1) </ins> fi if [ -z "$KUBE_MASTER_IP" ]; then echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" 1>&2

It fails with The change seems to have been introduced in - Redirects error messages to stderr so users can see them. /cc
Sorry about that. Fix sent.


Question: # Also please update corresponding image for node e2e at: # https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml CVM_VERSION=${CVM_VERSION:-container-vm-v20170627} <del> GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-60-9592-84-0} </del> <ins> GCI_VERSION=${KUBE_GCI_VERSION:-cos-stable-60-9592-90-0} </ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}

The debian-based CVM image is going to be deprecated in September. We should: Change the default OS image for GCE cluster setup script. This means that all the GCE CI builds will use the new default OS image. Add deprecation warning to the cluster setup script if CVM is specified. Maintain the node e2e test coverage for CVM for 1.8. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
The CVM tests were turned off a while ago.


Question: Kind: kind, ID:   name, <del> Causes: []api.StatusCause{ { Type:    api.CauseTypeUnexpectedServerResponse, Message: serverMessage, }, }, </del> <ins> Causes:            causes, </ins> RetryAfterSeconds: retryAfterSeconds, }, Message: message,

If go-restful detects a particular method is not allowed, it will return this error: That is preventing useful methods that resthandler would generate from making it to the client. Expected response:
triggered with
/cc
/cc
Is the right fix to update the client to generate the message versus changing the server to allow the method and report the error? My bias is to improve the client and not change the server as it makes reading our swagger doc easier to not see a bunch of verb unsupported entries. Sent from my iPhone
My bias is also towards not changing the server to keep swagger spec correct. By changing the client, do you mean changing kubectl? can you please clarify how is the json useful to client? It does not contain any useful extra information. On Feb 11, 2015 7:28 PM, "Derek Carr" wrote:
Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to. Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect.
This was an unintended side effect, but I think the old behavior was an unexpected luxury? I agree from the end-user perspective, the old message was nice, and it's great that the client was politely told that a given verb was not supported on the resource, but I don't agree that a server should appear to accept a method to only report that its not allowed. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:04:41 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to.

Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect. Reply to this email directly or view it on GitHub:
If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error?
I think so, this is only going to get worse when we support specific types of verbs on specific resources. In practice, it seems fine for kubectl to give a better error specific to the method not allowed response. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:26:08 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error? Reply to this email directly or view it on GitHub:
We could consider submitting a PR to go-restful to allow us to provide an error-generation function or interface implementation.
Yeah, that seems reasonable.
Opened
Copying text from : Some errors return as json and some as plain text. This causes overhead on client side when parsing the errors since need to recheck content type for every error in order not to parse plain text as json (some libraries fail on that). The errors message format should be standardized. Examples: The text/plain errors are from go-restful. cc
+1, thanks
The status objects are always of the default API version, however, rather than the version requested, because the request object isn't available to the error handler.
Rather than: we need:
Can we define an error handler on a web service?
At the moment, it's at the container level:
David and I talked about keeping the API lookup mechanism so we could use that in the error handler (apirequestinfo). We still need it for some use cases (proxies) so it could be used here to identify the version and proposed resource if matched.


Question: failedPodsObserved += failedPodsObservedOnNode } <ins> // Remove pods assigned to not existing nodes. podsToDelete = append(podsToDelete, getPodsWithoutNode(nodeList, nodeToDaemonPods)...) </ins> // Label new pods using the hash label value of the current history when creating them if err = dsc.syncNodes(ds, podsToDelete, nodesNeedingDaemonPods, hash); err != nil { return err

What happened: DaemonSet doesn't delete pods that: weren't scheduled yet, node was deleted so pod can't be scheduled. What you expected to happen: Pod should be deleted when node is removed. How to reproduce it (as minimally and precisely as possible): I encountered this problem when doing tests with cluster with 1000 nodes and pods running. During tests I was deleting and adding nodes using GCE API without draining them. I reproduced problem on much smaller cluster but it requires stopping K8s scheduler. Scenario: cluster with 2 nodes. till fluentd-gcp pods are running on nodes. K8s scheduler. fluentd-gcp pods. till daemonset controller creates pending pods. one node. scheduler. of the fluentd-gcp pods will be pending and scheduler won't be able to schedule. Anything else we need to know?: Environment: GKE Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
/assign janetkuo
/assign
We have PodGC to clean up orphaned pods every 20s, is that enough?
I'm removing pods that have only node affinity set and don't have pod.Spec.NodeName set. These pods were never touched by scheduler before node was removed and after node was removed can't be scheduled as node is doesn't exist anymore. Such pods are pending forever and PodGC can't remove them as pod.Spec.NodeName is empty.
Yes, that's right :) let me take few time to review the PR, thanks.
Should I backport fix to 1.13 and 1.12?
+1


Question: newService("s0", "333", true), }, expectedUpdateCalls: []fake_cloud.FakeUpdateBalancerCall{ <del> {Name: lbName("s0"), Region: region, Hosts: []string{"node0", "node1", "node73"}}, </del> <ins> {Name: "a333", Region: region, Hosts: []string{"node0", "node1", "node73"}}, </ins> }, }, {

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: const ( CpuUsageMetrics        MetricKind = "cpu" <ins> PerCpuUsageMetrics     MetricKind = "percpu" </ins> MemoryUsageMetrics     MetricKind = "memory" CpuLoadMetrics         MetricKind = "cpuLoad" DiskIOMetrics          MetricKind = "diskIO"

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: <ins> package main import ( "fmt" "net/http" ) func helloFromKubernetes(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "Hello World! -- Kubernetes") } func main() { http.HandleFunc("/", helloFromKubernetes) err := http.ListenAndServe(":8080", nil) if err != nil { panic("ListenAndServe: " + err.Error()) } } </ins>

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for kubemark [x] [x] [x] [x] Recommendation, either: Stop depending on writing to files and remove flags , or Add go-runner to kubemark image and migrate to /sig scalability /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Resolved in both k/k and kubernetes-sigs/cloud-provider-azure. /close


Question: PortHeader = "port" ) <ins> // Appends the NodeAddresses to the passed-by-pointer slice, only if they do not already exist func AddToNodeAddresses(addresses *[]NodeAddress, addAddresses ...NodeAddress) { for _, add := range addAddresses { exists := false for _, existing := range *addresses { if existing.Address == add.Address && existing.Type == add.Type { exists = true break } } if !exists { *addresses = append(*addresses, add) } } } </ins>

Since we append hostips to the list on every node controller run invocation which happens at 10s intervals, this list grows pretty large pretty soon: Maybe: would still satisfy the requirements?
Fix is here I think we generally use list instead of map
This could have been P1, but I just merged a fix so I'll leave at P2. Remaining work to do: Add to validation something that limits the number of IP addresses. (and maybe checks dups?)
Yeah, I'll add validation later.