Question: <ins> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package compatibility import ( "fmt" "net/http/httptest" "reflect" "testing" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" utiltesting "k8s.io/client-go/util/testing" _ "k8s.io/kubernetes/pkg/apis/core/install" _ "k8s.io/kubernetes/pkg/scheduler/algorithmprovider/defaults" schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" latestschedulerapi "k8s.io/kubernetes/pkg/scheduler/api/latest" "k8s.io/kubernetes/pkg/scheduler/factory" ) func TestCompatibility_v1_Scheduler(t *testing.T) { // Add serialized versions of scheduler config that exercise available options to ensure compatibility between releases schedulerFiles := map[string]struct { JSON           string ExpectedPolicy schedulerapi.Policy }{ // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.0": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsPorts"}, {"name": "NoDiskConflict"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "LeastRequestedPriority",   "weight": 1}, {"name": "ServiceSpreadingPriority", "weight": 2}, {"name": "TestServiceAntiAffinity",  "weight": 3, "argument": {"serviceAntiAffinity": {"label": "zone"}}}, {"name": "TestLabelPreference",      "weight": 4, "argument": {"labelPreference": {"label": "bar", "presence":true}}} ] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsPorts"}, {Name: "NoDiskConflict"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "LeastRequestedPriority", Weight: 1}, {Name: "ServiceSpreadingPriority", Weight: 2}, {Name: "TestServiceAntiAffinity", Weight: 3, Argument: &schedulerapi.PriorityArgument{ServiceAntiAffinity: &schedulerapi.ServiceAntiAffinity{Label: "zone"}}}, {Name: "TestLabelPreference", Weight: 4, Argument: &schedulerapi.PriorityArgument{LabelPreference: &schedulerapi.LabelPreference{Label: "bar", Presence: true}}}, }, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.1": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsHostPorts"}, {"name": "PodFitsResources"}, {"name": "NoDiskConflict"}, {"name": "HostName"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "TestServiceAntiAffinity",  "weight": 3, "argument": {"serviceAntiAffinity": {"label": "zone"}}}, {"name": "TestLabelPreference",      "weight": 4, "argument": {"labelPreference": {"label": "bar", "presence":true}}} ] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsHostPorts"}, {Name: "PodFitsResources"}, {Name: "NoDiskConflict"}, {Name: "HostName"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "TestServiceAntiAffinity", Weight: 3, Argument: &schedulerapi.PriorityArgument{ServiceAntiAffinity: &schedulerapi.ServiceAntiAffinity{Label: "zone"}}}, {Name: "TestLabelPreference", Weight: 4, Argument: &schedulerapi.PriorityArgument{LabelPreference: &schedulerapi.LabelPreference{Label: "bar", Presence: true}}}, }, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.2": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "TestServiceAntiAffinity",  "weight": 3, "argument": {"serviceAntiAffinity": {"label": "zone"}}}, {"name": "TestLabelPreference",      "weight": 4, "argument": {"labelPreference": {"label": "bar", "presence":true}}} ] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "TestServiceAntiAffinity", Weight: 3, Argument: &schedulerapi.PriorityArgument{ServiceAntiAffinity: &schedulerapi.ServiceAntiAffinity{Label: "zone"}}}, {Name: "TestLabelPreference", Weight: 4, Argument: &schedulerapi.PriorityArgument{LabelPreference: &schedulerapi.LabelPreference{Label: "bar", Presence: true}}}, }, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.3": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2} ] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, }, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.4": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2} ] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, }, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.7": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "BindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.7 was missing json tags on the BindVerb field and required "BindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, }}, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.8": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "CheckNodeCondition"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "bindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "CheckNodeCondition"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.8 became case-insensitive and tolerated "bindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, }}, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.9": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "CheckNodeCondition"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "CheckVolumeBinding"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "bindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "CheckNodeCondition"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "CheckVolumeBinding"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.9 was case-insensitive and tolerated "bindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, }}, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.10": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "CheckNodePIDPressure"}, {"name": "CheckNodeCondition"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "CheckVolumeBinding"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "bindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true, "managedResources": [{"name":"example.com/foo","ignoredByScheduler":true}], "ignorable":true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "CheckNodePIDPressure"}, {Name: "CheckNodeCondition"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "CheckVolumeBinding"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.10 was case-insensitive and tolerated "bindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, ManagedResources: []schedulerapi.ExtenderManagedResource{{Name: v1.ResourceName("example.com/foo"), IgnoredByScheduler: true}}, Ignorable:        true, }}, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.11": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "CheckNodePIDPressure"}, {"name": "CheckNodeCondition"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "CheckVolumeBinding"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2}, { "name": "RequestedToCapacityRatioPriority", "weight": 2, "argument": { "requestedToCapacityRatioArguments": { "shape": [ {"utilization": 0,  "score": 0}, {"utilization": 50, "score": 7} ] } }} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "bindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true, "managedResources": [{"name":"example.com/foo","ignoredByScheduler":true}], "ignorable":true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "CheckNodePIDPressure"}, {Name: "CheckNodeCondition"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "CheckVolumeBinding"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, { Name:   "RequestedToCapacityRatioPriority", Weight: 2, Argument: &schedulerapi.PriorityArgument{ RequestedToCapacityRatioArguments: &schedulerapi.RequestedToCapacityRatioArguments{ UtilizationShape: []schedulerapi.UtilizationShapePoint{ {Utilization: 0, Score: 0}, {Utilization: 50, Score: 7}, }}, }, }, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.11 restored case-sensitivity, but allowed either "BindVerb" or "bindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, ManagedResources: []schedulerapi.ExtenderManagedResource{{Name: v1.ResourceName("example.com/foo"), IgnoredByScheduler: true}}, Ignorable:        true, }}, }, }, // Do not change this JSON after the corresponding release has been tagged. // A failure indicates backwards compatibility with the specified release was broken. "1.12": { JSON: `{ "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsHostPorts"}, {"name": "HostName"}, {"name": "NoDiskConflict"}, {"name": "NoVolumeZoneConflict"}, {"name": "PodToleratesNodeTaints"}, {"name": "CheckNodeMemoryPressure"}, {"name": "CheckNodeDiskPressure"}, {"name": "CheckNodePIDPressure"}, {"name": "CheckNodeCondition"}, {"name": "MaxEBSVolumeCount"}, {"name": "MaxGCEPDVolumeCount"}, {"name": "MaxAzureDiskVolumeCount"}, {"name": "MaxCSIVolumeCountPred"}, {"name": "MatchInterPodAffinity"}, {"name": "GeneralPredicates"}, {"name": "CheckVolumeBinding"}, {"name": "TestServiceAffinity", "argument": {"serviceAffinity" : {"labels" : ["region"]}}}, {"name": "TestLabelsPresence",  "argument": {"labelsPresence"  : {"labels" : ["foo"], "presence":true}}} ],"priorities": [ {"name": "EqualPriority",   "weight": 2}, {"name": "ImageLocalityPriority",   "weight": 2}, {"name": "LeastRequestedPriority",   "weight": 2}, {"name": "BalancedResourceAllocation",   "weight": 2}, {"name": "SelectorSpreadPriority",   "weight": 2}, {"name": "NodePreferAvoidPodsPriority",   "weight": 2}, {"name": "NodeAffinityPriority",   "weight": 2}, {"name": "TaintTolerationPriority",   "weight": 2}, {"name": "InterPodAffinityPriority",   "weight": 2}, {"name": "MostRequestedPriority",   "weight": 2}, { "name": "RequestedToCapacityRatioPriority", "weight": 2, "argument": { "requestedToCapacityRatioArguments": { "shape": [ {"utilization": 0,  "score": 0}, {"utilization": 50, "score": 7} ] } }} ],"extenders": [{ "urlPrefix":        "/prefix", "filterVerb":       "filter", "prioritizeVerb":   "prioritize", "weight":           1, "bindVerb":         "bind", "enableHttps":      true, "tlsConfig":        {"Insecure":true}, "httpTimeout":      1, "nodeCacheCapable": true, "managedResources": [{"name":"example.com/foo","ignoredByScheduler":true}], "ignorable":true }] }`, ExpectedPolicy: schedulerapi.Policy{ Predicates: []schedulerapi.PredicatePolicy{ {Name: "MatchNodeSelector"}, {Name: "PodFitsResources"}, {Name: "PodFitsHostPorts"}, {Name: "HostName"}, {Name: "NoDiskConflict"}, {Name: "NoVolumeZoneConflict"}, {Name: "PodToleratesNodeTaints"}, {Name: "CheckNodeMemoryPressure"}, {Name: "CheckNodeDiskPressure"}, {Name: "CheckNodePIDPressure"}, {Name: "CheckNodeCondition"}, {Name: "MaxEBSVolumeCount"}, {Name: "MaxGCEPDVolumeCount"}, {Name: "MaxAzureDiskVolumeCount"}, {Name: "MaxCSIVolumeCountPred"}, {Name: "MatchInterPodAffinity"}, {Name: "GeneralPredicates"}, {Name: "CheckVolumeBinding"}, {Name: "TestServiceAffinity", Argument: &schedulerapi.PredicateArgument{ServiceAffinity: &schedulerapi.ServiceAffinity{Labels: []string{"region"}}}}, {Name: "TestLabelsPresence", Argument: &schedulerapi.PredicateArgument{LabelsPresence: &schedulerapi.LabelsPresence{Labels: []string{"foo"}, Presence: true}}}, }, Priorities: []schedulerapi.PriorityPolicy{ {Name: "EqualPriority", Weight: 2}, {Name: "ImageLocalityPriority", Weight: 2}, {Name: "LeastRequestedPriority", Weight: 2}, {Name: "BalancedResourceAllocation", Weight: 2}, {Name: "SelectorSpreadPriority", Weight: 2}, {Name: "NodePreferAvoidPodsPriority", Weight: 2}, {Name: "NodeAffinityPriority", Weight: 2}, {Name: "TaintTolerationPriority", Weight: 2}, {Name: "InterPodAffinityPriority", Weight: 2}, {Name: "MostRequestedPriority", Weight: 2}, { Name:   "RequestedToCapacityRatioPriority", Weight: 2, Argument: &schedulerapi.PriorityArgument{ RequestedToCapacityRatioArguments: &schedulerapi.RequestedToCapacityRatioArguments{ UtilizationShape: []schedulerapi.UtilizationShapePoint{ {Utilization: 0, Score: 0}, {Utilization: 50, Score: 7}, }}, }, }, }, ExtenderConfigs: []schedulerapi.ExtenderConfig{{ URLPrefix:        "/prefix", FilterVerb:       "filter", PrioritizeVerb:   "prioritize", Weight:           1, BindVerb:         "bind", // 1.11 restored case-sensitivity, but allowed either "BindVerb" or "bindVerb" EnableHTTPS:      true, TLSConfig:        &restclient.TLSClientConfig{Insecure: true}, HTTPTimeout:      1, NodeCacheCapable: true, ManagedResources: []schedulerapi.ExtenderManagedResource{{Name: v1.ResourceName("example.com/foo"), IgnoredByScheduler: true}}, Ignorable:        true, }}, }, }, } registeredPredicates := sets.NewString(factory.ListRegisteredFitPredicates()...) registeredPriorities := sets.NewString(factory.ListRegisteredPriorityFunctions()...) seenPredicates := sets.NewString() seenPriorities := sets.NewString() for v, tc := range schedulerFiles { fmt.Printf("%s: Testing scheduler confign", v) policy := schedulerapi.Policy{} if err := runtime.DecodeInto(latestschedulerapi.Codec, []byte(tc.JSON), &policy); err != nil { t.Errorf("%s: Error decoding: %v", v, err) continue } for _, predicate := range policy.Predicates { seenPredicates.Insert(predicate.Name) } for _, priority := range policy.Priorities { seenPriorities.Insert(priority.Name) } if !reflect.DeepEqual(policy, tc.ExpectedPolicy) { t.Errorf("%s: Expected:nt%#vnGot:nt%#v", v, tc.ExpectedPolicy, policy) } handler := utiltesting.FakeHandler{ StatusCode:   500, ResponseBody: "", T:            t, } server := httptest.NewServer(&handler) defer server.Close() client := clientset.NewForConfigOrDie(&restclient.Config{Host: server.URL, ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) informerFactory := informers.NewSharedInformerFactory(client, 0) if _, err := factory.NewConfigFactory(&factory.ConfigFactoryArgs{ SchedulerName:                  "some-scheduler-name", Client:                         client, NodeInformer:                   informerFactory.Core().V1().Nodes(), PodInformer:                    informerFactory.Core().V1().Pods(), PvInformer:                     informerFactory.Core().V1().PersistentVolumes(), PvcInformer:                    informerFactory.Core().V1().PersistentVolumeClaims(), ReplicationControllerInformer:  informerFactory.Core().V1().ReplicationControllers(), ReplicaSetInformer:             informerFactory.Apps().V1().ReplicaSets(), StatefulSetInformer:            informerFactory.Apps().V1().StatefulSets(), ServiceInformer:                informerFactory.Core().V1().Services(), PdbInformer:                    informerFactory.Policy().V1beta1().PodDisruptionBudgets(), StorageClassInformer:           informerFactory.Storage().V1().StorageClasses(), HardPodAffinitySymmetricWeight: v1.DefaultHardPodAffinitySymmetricWeight, DisablePreemption:              false, PercentageOfNodesToScore:       schedulerapi.DefaultPercentageOfNodesToScore, }).CreateFromConfig(policy); err != nil { t.Errorf("%s: Error constructing: %v", v, err) continue } } if !seenPredicates.HasAll(registeredPredicates.List()...) { t.Errorf("Registered predicates are missing from compatibility test (add to test stanza for version currently in development): %#v", registeredPredicates.Difference(seenPredicates).List()) } if !seenPriorities.HasAll(registeredPriorities.List()...) { t.Errorf("Registered priorities are missing from compatibility test (add to test stanza for version currently in development): %#v", registeredPriorities.Difference(seenPriorities).List()) } } </ins>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: If you contributed in phase 1, please give others a chance before claiming this issue. You can also help out by reviewing the phase 2 PRs. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /sig scheduling /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to take it and thanks
Hi! I am contributer and I would like to take it.
while you're working on do you mind letting take a crack at this one?
OK, go head
, thanks!
/lifecycle active for
Thanks for taking care of this one! :smiley:
You are wellcome


Question: klog.V(4).InfoS("Updating PersistentVolume: binding to claim failed", "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc), "err", err) return err } klog.V(4).InfoS("Updating PersistentVolume: bound to claim", "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc)) // Save updated object from apiserver for later checking. binding.pv = newPV

When scheduling pod, we encountered the following event , then the scheduler will backoff to reschedule again, but this add more waiting time. From the event message, we can see it is the PVC update conflict leads to the , so need to add retry logic on PVC updating.### Need to add retry logic on PVC updating, to avoid the backoff, to improve the scheduling efficiency. a PVC and then create Pod; scheduing; the PVC annotation or label before the Pod scheduled success; will encounter this issue. No response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage note that this is not a supported client/server skew: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.4-tke.23", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-03-15T09:25:31Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "testing" internal "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/unversioned" </ins> api "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/runtime" )

I'd like to be able to sort by a timestamp, e.g. but that fails with an exception: I appreciate that the root of the problem is probably outside of kubernetes, yet I hope you'd agree that sorting by a timestamp is very useful functionality.
Agree with sorting by timestamp.. Just make it clear, is of type: while only these types are allowed to be sorted:


Question: Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoSchedule, } <ins> unreachableTaint := &v1.Taint{ Key:    algorithm.TaintNodeUnreachable, Effect: v1.TaintEffectNoSchedule, } </ins> tests := []struct { Name           string

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!


Question: } bFoundDisk := false for i, disk := range disks { <del> if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && *disk.Vhd.URI == diskURI) || (disk.ManagedDisk != nil && diskURI != "" && *disk.ManagedDisk.ID == diskURI) { </del> <ins> if disk.Lun != nil && (disk.Name != nil && diskName != "" && strings.EqualFold(*disk.Name, diskName)) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && strings.EqualFold(*disk.Vhd.URI, diskURI)) || (disk.ManagedDisk != nil && diskURI != "" && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { </ins> // found the disk klog.V(2).Infof("azureDisk - detach disk: name %q uri %q", diskName, diskURI) disks = append(disks[:i], disks[i+1:]...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: return nil } <ins> func parseHealthProbeProtocolAndPath(service *v1.Service) (string, string) { var protocol, path string if v, ok := service.Annotations[ServiceAnnotationLoadBalancerHealthProbeProtocol]; ok { protocol = v } else { return protocol, path } // ignore the request path if using TCP if strings.EqualFold(protocol, string(network.ProbeProtocolHTTP)) || strings.EqualFold(protocol, string(network.ProbeProtocolHTTPS)) { if v, ok := service.Annotations[ServiceAnnotationLoadBalancerHealthProbeRequestPath]; ok { path = v } } return protocol, path } </ins> func (az *Cloud) reconcileLoadBalancerRule( service *v1.Service, wantLb bool,

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign


Question: if [[ -n "${WHAT-}" ]]; then for pkg in ${WHAT} do <del> record_command run_${pkg}_tests </del> <ins> # running of kubeadm is captured in hack/make-targets/test-cmd.sh if [[ "${pkg}" != "kubeadm" ]]; then record_command run_${pkg}_tests fi </ins> done cleanup_tests return

this PR removed the script which can be used to run the integration tests. the following command is claimed to be a replacement: but instead it runs unrelated tests too. expecting the above command to only show the following output (run only kubeadm tests): /assign
GitHub didn't allow me to assign the following users: runyontr. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : this PR removed the script which can be used to run the integration tests. but instead it runs unrelated tests too. /assign </details
/sig cluster-lifecycle
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Hey! Gentle reminder that code freeze is this Friday. Since this is part of the 1.14 milestone, is it going to be resolved soon?
hopefully in the next couple of days. PR is here:


Question: Stdin bool `json:"stdin,omitempty" protobuf:"varint,1,opt,name=stdin"` // Redirect the standard output stream of the pod for this call. <del> // Defaults to true. </del> // +optional Stdout bool `json:"stdout,omitempty" protobuf:"varint,2,opt,name=stdout"` // Redirect the standard error stream of the pod for this call. <del> // Defaults to true. </del> // +optional Stderr bool `json:"stderr,omitempty" protobuf:"varint,3,opt,name=stderr"`

As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation


Question: return } // TODO: record long-running request differently? The long-running check func does not necessarily match the one of the aggregated apiserver <del> endpointmetrics.Record(req, info, aggregatorComponent, "", code, 0, 0) </del> <ins> endpointmetrics.RecordRequestTermination(req, info, aggregatorComponent, code) </ins> } func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: const tty = false <ins> Logf("ExecWithOptions: Clientset creation") </ins> req := f.ClientSet.CoreV1().RESTClient().Post(). Resource("pods"). Name(options.PodName).

We should fine grained debug statements to ExecWithOptions in the framework class , so that in tests where theres alot of execs where hangs can happen i.e. on windows, we can see where tests are getting clogged up. Mostly i want this for , but i suppose its a generic improvement
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: if !ok { return nil, nil, fmt.Errorf("invalid object type") } <del> return labels.Set{}, fields.Set{ </del> <ins> l := event.Labels if l == nil { l = labels.Set{} } return l, fields.Set{ "metadata.name":                  event.Name, </ins> "involvedObject.kind":            event.InvolvedObject.Kind, "involvedObject.namespace":       event.InvolvedObject.Namespace, "involvedObject.name":            event.InvolvedObject.Name,

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: &Job{}, &JobList{}, &v1.ListOptions{}, <ins> &v1.DeleteOptions{}, </ins> ) versionedwatch.AddToGroupVersion(scheme, SchemeGroupVersion) }

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: // SchemeGroupVersion is group version used to register these objects var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: runtime.APIVersionInternal} <del> // Unversioned is group version for unversioned API objects // TODO: this should be v1 probably var Unversioned = schema.GroupVersion{Group: "", Version: "v1"} </del> // ParameterCodec handles versioning of objects that are converted to query parameters. var ParameterCodec = runtime.NewParameterCodec(Scheme)

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: // DEPRECATED will be replace DefaultClientConfig = DirectClientConfig{*clientcmdapi.NewConfig(), "", &ConfigOverrides{ ClusterDefaults: ClusterDefaults, <del> }, nil, NewDefaultClientConfigLoadingRules()} </del> <ins> }, nil, NewDefaultClientConfigLoadingRules(), promptedCredentials{}} </ins> ) // getDefaultServer returns a default setting for DefaultClientConfig

When I want to use basic-auth, kubectl will display the password I entered: I think it is more sensible to hide the password?
And it seems that kubectl request for username/password twice...
I'll take this. There is a library here: That appears to do the trick, but we need some refactoring so that it can read from an arbitrary reader to fit our usecase.
This was fixed by but I didn't close this issue. Closing it now.


Question: original := fixtures.NewNoxuInstance("default", "foo") unstructured.SetNestedField(original.UnstructuredContent(), int64(42), "metadata", "unknown") <ins> unstructured.SetNestedField(original.UnstructuredContent(), nil, "metadata", "generation") </ins> unstructured.SetNestedField(original.UnstructuredContent(), map[string]interface{}{"foo": int64(42), "bar": "abc"}, "metadata", "annotations") unstructured.SetNestedField(original.UnstructuredContent(), map[string]interface{}{"invalid": "x y"}, "metadata", "labels") unstructured.SetNestedField(original.UnstructuredContent(), int64(42), "embedded", "metadata", "unknown")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: api-endpoint = ${GCE_API_ENDPOINT} EOF fi <del> if [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then </del> <ins> if [[ -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" ]]; then </ins> use_cloud_config="true" cat <<EOF >>/etc/gce.conf token-url = ${TOKEN_URL} token-body = ${TOKEN_BODY} <ins> EOF fi if [[ -n "${PROJECT_ID:-}" ]]; then use_cloud_config="true" cat <<EOF >>/etc/gce.conf </ins> project-id = ${PROJECT_ID} <del> network-name = ${NODE_NETWORK} </del> EOF <del> if [[ -n "${NETWORK_PROJECT_ID:-}" ]]; then cat <<EOF >>/etc/gce.conf </del> <ins> fi if [[ -n "${NETWORK_PROJECT_ID:-}" ]]; then use_cloud_config="true" cat <<EOF >>/etc/gce.conf </ins> network-project-id = ${NETWORK_PROJECT_ID} EOF <del> fi if [[ -n "${NODE_SUBNETWORK:-}" ]]; then cat <<EOF >>/etc/gce.conf </del> <ins> fi if [[ -n "${NODE_NETWORK:-}" ]]; then use_cloud_config="true" cat <<EOF >>/etc/gce.conf network-name = ${NODE_NETWORK} EOF fi if [[ -n "${NODE_SUBNETWORK:-}" ]]; then use_cloud_config="true" cat <<EOF >>/etc/gce.conf </ins> subnetwork-name = ${NODE_SUBNETWORK} EOF <del> fi </del> fi if [[ -n "${NODE_INSTANCE_PREFIX:-}" ]]; then use_cloud_config="true"

The "Services should be able to create an internal type load balancer [Slow]" e2e test is failing on our 2k-node gce clusters - https://k8s- with the following error: We're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them. cc
Seems similar to Assigning to you as you seem to have the context already :) Also, is this expected to be fixed in 1.8 timeframe?
doesn't set the necessary variables for the GCE cloud provider. Will throw you a PR.
Cool - thanks a lot!
Ref
I see the last test on 09/09 failed, but I can't see the build log due to exceeding memory. Any other way I can see the log, /reopen
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
nevermind - went directly to raw log.
Friendly ping. This e2e is still failing (https://k8s-) and it's part of release-blocking suite.
/close


Question: // kep: https://kep.k8s.io/2876 // alpha: v1.23 // beta: v1.25 <ins> // stable: v1.29 </ins> // // Enables expression validation for Custom Resource CustomResourceValidationExpressions featuregate.Feature = "CustomResourceValidationExpressions"

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc


Question: return fmt.Errorf("couldn't get key for object %#v: %v", ds, err) } <ins> // If the DaemonSet is being deleted (either by foreground deletion or // orphan deletion), we cannot be sure if the DaemonSet history objects // it owned still exist -- those history objects can either be deleted // or orphaned. Garbage collector doesn't guarantee that it will delete // DaemonSet pods before deleting DaemonSet history objects, because // DaemonSet history doesn't own DaemonSet pods. We cannot reliably // calculate the status of a DaemonSet being deleted. Therefore, return // here without updating status for the DaemonSet being deleted. if ds.DeletionTimestamp != nil { return nil } </ins> // Construct histories of the DaemonSet, and get the hash of current history cur, old, err := dsc.constructHistory(ds) if err != nil {

/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: cadvisormetrics.NetworkUsageMetrics:     struct{}{}, cadvisormetrics.AcceleratorUsageMetrics: struct{}{}, cadvisormetrics.AppMetrics:              struct{}{}, <ins> cadvisormetrics.ProcessMetrics:          struct{}{}, </ins> } if usingLegacyStats { includedMetrics[cadvisormetrics.DiskUsageMetrics] = struct{}{}

What happened: containersockets, containerthreads, containermaxthreads metrics were as part of Kubernetes 1.16 release. However, I only see 0 for these metrics. What you expected to happen: These metrics should give non-zero numbers. How to reproduce it (as minimally and precisely as possible): New cluster using Kubernetes 1.16.2 + Prometheus 2.11.1. Scraped containersockets, containerthreads, containermaxthreads metrics always return 0 Anything else we need to know?: Environment: Kubernetes version (use ): 1.16.2 Cloud provider or hardware configuration: Azure, cluster provisioned using aks-engine OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: aks-engine Network plugin and version (if this is a network-related bug): Others: /sig api-machinery /sig node
/cc /remove-sig api-machinery
It seems like this fix has only been merged into 1.17+. Is it possible to backport this fix into 1.16 as well? (I still see this issue in 1.16.9)
I am also seeing this issue in 1.6.9. Any possibility of backporting this fix to 1.16?
Also seeing this in 1.16.9, is there anything preventing this change from being backported?
/cc
Though we meant to enable process metrics in 1.16, we accidentally didn't. I consider the collection of new metrics a feature, and don't think it should be backported. It will likely cause some additional resource usage on the node, which isn't appropriate for a patch release.


Question: }, }, }, <ins> expectedElements:   []string{"EndpointsNamespace", "<unset>"}, </ins> unexpectedElements: []string{"VolumeMode", "Filesystem"}, }, {

What happened: What you expected to happen: EndpointNamespace should be the string name of the namespace used. How to reproduce it (as minimally and precisely as possible): Create a glusterfs PV using an EndpointNamespace entry and run 'kubectl describe pv' on the resulting PV. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T17:09:13Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): Linux bionic-vm 4.15.0-58-generic -Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.15
/sig cli
/assign


Question: utilruntime.Must(flowcontrolv1beta1.AddToScheme(scheme)) utilruntime.Must(flowcontrolv1beta2.AddToScheme(scheme)) utilruntime.Must(flowcontrolv1beta3.AddToScheme(scheme)) <del> // TODO(#112512): This controls serialization order, for 1.26, we can //  set the serialization version to v1beta2 because vN-1 understands that //  level. In 1.27, we should set the serialization version to v1beta3. utilruntime.Must(scheme.SetVersionPriority(flowcontrolv1beta2.SchemeGroupVersion, flowcontrolv1beta3.SchemeGroupVersion, </del> <ins> utilruntime.Must(scheme.SetVersionPriority(flowcontrolv1beta3.SchemeGroupVersion, flowcontrolv1beta2.SchemeGroupVersion, </ins> flowcontrolv1beta1.SchemeGroupVersion, flowcontrolv1alpha1.SchemeGroupVersion)) }

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: // FakeCache is used for testing type FakeCache struct { AssumeFunc func(*v1.Pod) <ins> ForgetFunc func(*v1.Pod) </ins> } func (f *FakeCache) AssumePod(pod *v1.Pod) error {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: klog.Error(err) return nil } <ins> if hist.Histogram == nil { klog.Errorf("metric %q is not a Histogram metric", metric) return nil } </ins> if err := hist.Validate(); err != nil { klog.Error(err) return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: schedulerperf now is not able to support metrics other than "Histogram", if other type of "metrics" is configured in test, it will throw below exception, What you expected to happen: should show helpful error message instead of . How to reproduce it (as minimally and precisely as possible): Define testcase with in the config which includes a , for example. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
/assign