Question: migrated_packages+=("$KUBE_ROOT/$line") done < <(cat "${migrated_packages_file}") <ins> echo "Running structured logging static check on migrated packages" </ins> ret=0 GOOS=linux    logcheck "${migrated_packages[@]}" || ret=$? GOOS=windows  logcheck "${migrated_packages[@]}" || ret=$? if [ $ret -eq 0 ]; then <del> echo "Structured logging static check is passed :)." </del> <ins> echo "Structured logging static check passed on migrated packages :)" </ins> else echo "Please fix above failures. You can locally test via:" echo "hack/verify-structured-logging.sh" fi <ins> # Ignore migrated packages as they are already tested # Trim the trailing /... from given packages ignore_packages=$(grep -oE '^[a-zA-Z0-9/]+[^/.]' < "${KUBE_ROOT}"/hack/.structured_logging) all_packages=() # shellcheck disable=SC2010 for i in $(ls -d ./*/ | grep -v 'staging' | grep -v 'vendor' | grep -v 'hack') do all_packages+=("$(go list ./"$i"/... 2> /dev/null | sed 's/k8s.io/kubernetes///g')") all_packages+=(" ") done # We exclude vendor/ except vendor/k8s.io # This is because vendor/k8s.io is symlinked to staging # and needs to be checked all_packages+=("vendor/k8s.io") # Packages to test = all_packages - ignored_packages packages=() while IFS='' read -r line; do if [ -z "$line" ]; then continue; fi packages+=("$KUBE_ROOT/$line/...") done < <(echo "${all_packages[@]}" | tr " " "n" | grep -v "$ignore_packages") echo -e "nRunning structured logging static check on all other packages" GOOS=linux    logcheck -allow-unstructured "${packages[@]}" || ret=$? GOOS=windows  logcheck -allow-unstructured "${packages[@]}" || ret=$? if [ $ret -eq 0 ]; then echo "Structured logging static check passed on all packages :)" else echo "Please fix above failures. You can locally test via:" echo "hack/verify-structured-logging.sh" fi </ins> exit $ret

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: "k8s.io/kubernetes/pkg/scheduler/framework" "k8s.io/kubernetes/pkg/scheduler/framework/parallelize" "k8s.io/kubernetes/pkg/scheduler/metrics" <ins> "k8s.io/kubernetes/pkg/util/slice" </ins> ) const (

Scheduler's default plugin returns an invalid score that is not in the range of [0, 100], resulting in no Pods being scheduled: After investigation, I found that there was a bug in the code that built the framework when the scheduler started. When using the plugins declared in the regular extension point to override the plugins declared in the MultiPoint, the traversed variable will be modified in the following loop: This may cause a same scoring plugin (e.g. ) to be loaded multiple times: Further, this results in a node being processed multiple times by a plugin's (e.g. 's) method, resulting in an invalid score that is not in the range of [0, 100]: If the scheduler's MultiPoint feature is used (either through the default plugins or through manual configuration), and the configuration (e.g. the scoring weight) of a certain plugin is overridden in a regular extension point, the plugin will only be loaded once in this extension point, and no illegal scores will be returned. An example KubeSchedulerConfiguration: No response K8s v1.24.15 or higher None <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Can I work on this?
this is being worked on. see
I see, Is there any other task that you are aware of and I can work on? Thanks.


Question: return resTags.Items } <ins> // GetNodeTags gets k8s node tag from one of the nodes func GetNodeTags(c clientset.Interface, cloudConfig CloudConfig) []string { nodes := GetReadySchedulableNodesOrDie(c) if len(nodes.Items) == 0 { Logf("GetNodeTags: Found 0 node.") return []string{} } return GetInstanceTags(cloudConfig, nodes.Items[0].Name).Items } </ins> // GetInstancePrefix returns the INSTANCE_PREFIX env we set for e2e cluster. // From cluster/gce/config-test.sh, master name is set up using below format: // MASTER_NAME="${INSTANCE_PREFIX}-master"

1 tests failed, 3 jobs failed, 73 builds failed. Failure stats cover 1 day time range '26 Jun 2017 23:52 UTC' to '27 Jun 2017 23:52 UTC'. <details<summaryRationale for assignments:</summary </details<br
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
This was broke by . I'm working on a fix for this. /assign


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package stats import ( "reflect" "testing" "time" "github.com/Microsoft/hcsshim" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" testingclock "k8s.io/utils/clock/testing" ) type fakeNetworkStatsProvider struct { containers []containerStats } type containerStats struct { container hcsshim.ContainerProperties hcsStats  []hcsshim.NetworkStats } func (s fakeNetworkStatsProvider) GetHNSEndpointStats(endpointName string) (*hcsshim.HNSEndpointStats, error) { eps := hcsshim.HNSEndpointStats{} for _, c := range s.containers { for _, stat := range c.hcsStats { if endpointName == stat.InstanceId { eps = hcsshim.HNSEndpointStats{ EndpointID:      stat.EndpointId, BytesSent:       stat.BytesSent, BytesReceived:   stat.BytesReceived, PacketsReceived: stat.PacketsReceived, PacketsSent:     stat.PacketsSent, } } } } return &eps, nil } func (s fakeNetworkStatsProvider) HNSListEndpointRequest() ([]hcsshim.HNSEndpoint, error) { uniqueEndpoints := map[string]*hcsshim.HNSEndpoint{} for _, c := range s.containers { for _, stat := range c.hcsStats { e, found := uniqueEndpoints[stat.EndpointId] if found { // add the container e.SharedContainers = append(e.SharedContainers, c.container.ID) continue } uniqueEndpoints[stat.EndpointId] = &hcsshim.HNSEndpoint{ Name:             stat.EndpointId, Id:               stat.EndpointId, SharedContainers: []string{c.container.ID}, } } } eps := []hcsshim.HNSEndpoint{} for _, ep := range uniqueEndpoints { eps = append(eps, *ep) } return eps, nil } func Test_criStatsProvider_listContainerNetworkStats(t *testing.T) { fakeClock := testingclock.NewFakeClock(time.Time{}) tests := []struct { name    string fields  fakeNetworkStatsProvider want    map[string]*statsapi.NetworkStats wantErr bool }{ { name: "basic example", fields: fakeNetworkStatsProvider{ containers: []containerStats{ { container: hcsshim.ContainerProperties{ ID: "c1", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 1, BytesSent:     10, EndpointId:    "test", InstanceId:    "test", }, }, }, { container: hcsshim.ContainerProperties{ ID: "c2", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 2, BytesSent:     20, EndpointId:    "test2", InstanceId:    "test2", }, }, }, }, }, want: map[string]*statsapi.NetworkStats{ "c1": { Time: v1.NewTime(fakeClock.Now()), InterfaceStats: statsapi.InterfaceStats{ Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, }, }, "c2": { Time: v1.Time{}, InterfaceStats: statsapi.InterfaceStats{ Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, }, }, }, wantErr: false, }, { name: "multiple containers same endpoint", fields: fakeNetworkStatsProvider{ containers: []containerStats{ { container: hcsshim.ContainerProperties{ ID: "c1", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 1, BytesSent:     10, EndpointId:    "test", InstanceId:    "test", }, }, }, { container: hcsshim.ContainerProperties{ ID: "c2", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 2, BytesSent:     20, EndpointId:    "test2", InstanceId:    "test2", }, }, }, { container: hcsshim.ContainerProperties{ ID: "c3", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 3, BytesSent:     30, EndpointId:    "test2", InstanceId:    "test3", }, }, }, }, }, want: map[string]*statsapi.NetworkStats{ "c1": { Time: v1.NewTime(fakeClock.Now()), InterfaceStats: statsapi.InterfaceStats{ Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, }, }, "c2": { Time: v1.Time{}, InterfaceStats: statsapi.InterfaceStats{ Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, }, }, "c3": { Time: v1.Time{}, InterfaceStats: statsapi.InterfaceStats{ Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, }, }, }, wantErr: false, }, { name: "multiple stats instances of same interface only picks up first", fields: fakeNetworkStatsProvider{ containers: []containerStats{ { container: hcsshim.ContainerProperties{ ID: "c1", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 1, BytesSent:     10, EndpointId:    "test", InstanceId:    "test", }, { BytesReceived: 3, BytesSent:     30, EndpointId:    "test", InstanceId:    "test3", }, }, }, { container: hcsshim.ContainerProperties{ ID: "c2", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 2, BytesSent:     20, EndpointId:    "test2", InstanceId:    "test2", }, }, }, }, }, want: map[string]*statsapi.NetworkStats{ "c1": { Time: v1.NewTime(fakeClock.Now()), InterfaceStats: statsapi.InterfaceStats{ Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, }, }, "c2": { Time: v1.Time{}, InterfaceStats: statsapi.InterfaceStats{ Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, }, }, }, wantErr: false, }, { name: "multiple endpoints per container", fields: fakeNetworkStatsProvider{ containers: []containerStats{ { container: hcsshim.ContainerProperties{ ID: "c1", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 1, BytesSent:     10, EndpointId:    "test", InstanceId:    "test", }, { BytesReceived: 3, BytesSent:     30, EndpointId:    "test3", InstanceId:    "test3", }, }, }, { container: hcsshim.ContainerProperties{ ID: "c2", }, hcsStats: []hcsshim.NetworkStats{ { BytesReceived: 2, BytesSent:     20, EndpointId:    "test2", InstanceId:    "test2", }, }, }, }, }, want: map[string]*statsapi.NetworkStats{ "c1": { Time: v1.NewTime(fakeClock.Now()), InterfaceStats: statsapi.InterfaceStats{ Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test", RxBytes: toP(1), TxBytes: toP(10), }, { Name:    "test3", RxBytes: toP(3), TxBytes: toP(30), }, }, }, "c2": { Time: v1.Time{}, InterfaceStats: statsapi.InterfaceStats{ Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, Interfaces: []statsapi.InterfaceStats{ { Name:    "test2", RxBytes: toP(2), TxBytes: toP(20), }, }, }, }, wantErr: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { p := &criStatsProvider{ windowsNetworkStatsProvider: fakeNetworkStatsProvider{ containers: tt.fields.containers, }, clock: fakeClock, } got, err := p.listContainerNetworkStats() if (err != nil) != tt.wantErr { t.Errorf("listContainerNetworkStats() error = %v, wantErr %v", err, tt.wantErr) return } if !reflect.DeepEqual(got, tt.want) { t.Errorf("listContainerNetworkStats() got = %v, want %v", got, tt.want) } }) } } func toP(i uint64) *uint64 { return &i } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When curling the endpoint for stats we don't see network stats. Network stats to be included when calling the summary. This works with dockershim for windows. This happens because Windows containerd uses hcsv2 to create containers and we query for the networks stats using hcsv1: This will be address eventually with move to CRI only metrics KEP: but is still several releases away. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: } } <del> return f, nil </del> <ins> // Cache metric streams for prescore and score plugins. for i, pl := range f.preScorePlugins { f.preScorePlugins[i] = &instrumentedPreScorePlugin{ PreScorePlugin: f.preScorePlugins[i], metric:         metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.PreScore, f.profileName), } } for i, pl := range f.scorePlugins { f.scorePlugins[i] = &instrumentedScorePlugin{ ScorePlugin: f.scorePlugins[i], metric:      metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.Score, f.profileName), } } </ins> } func (f *frameworkImpl) SetPodNominator(n framework.PodNominator) {

metrics to see more details related to scheduler's decisions in Filter phase In some cases (e.g. new feature is in scheduler and want to see if it works or not), we may want to see which filter plugin affected Pod's scheduling results and it would be nice if we can observe that in metrics. Ref: /cc /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So, your imagined metrics is like counter that has the label (i.e. one counter for one plugin) the count represents how many times did plugin have an impact on the scheduling of the pod. Am I correct? (I'm not sure if I am understanding your imagined metric correctly.)
I would also add an label that can be or . We probably need some common implementation between this and
do you have time to work on this in 1.26?
Yes, I'm gonna take time to work on this (+ ) in the next release. Let me try. /assign
We should probably extend this to Score as well
/retitle metrics to see which plugins affect to scheduler's decisions in Filter/Score phase
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale After supporting Skip in Filter and Score.
memo: is the issue to support Skip in Score.
/cc Will work on part after
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
It's time! /assign


Question: "context" "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" corelisters "k8s.io/client-go/listers/core/v1" "k8s.io/klog" <del> "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/migration" </del> framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" schedulerlisters "k8s.io/kubernetes/pkg/scheduler/listers" "k8s.io/kubernetes/pkg/scheduler/nodeinfo"

Part of See example /sig scheduling /kind cleanup /help /priority important-soon
/assign /remove-help


Question: statusCode:         http.StatusForbidden, reqPath:            "/apis", reason:             "Forbidden", <del> message:            `forbidden: User "" cannot get path "/apis": Everything is forbidden.`, </del> <ins> message:            `forbidden: User "unprivileged" cannot get path "/apis": Everything is forbidden.`, </ins> }, { name:               "401",

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: } // ConstructFirewallForIngress returns the expected GCE firewall rule for the ingress resource <del> func (j *IngressTestJig) ConstructFirewallForIngress(gceController *GCEIngressController, nodeTag string) *compute.Firewall { </del> <ins> func (j *IngressTestJig) ConstructFirewallForIngress(gceController *GCEIngressController, nodeTags []string) *compute.Firewall { </ins> nodePorts := j.GetIngressNodePorts() fw := compute.Firewall{} fw.Name = gceController.GetFirewallRuleName() fw.SourceRanges = gcecloud.LoadBalancerSrcRanges() <del> fw.TargetTags = []string{nodeTag} </del> <ins> fw.TargetTags = nodeTags </ins> fw.Allowed = []*compute.FirewallAllowed{ { IPProtocol: "tcp",

1 tests failed, 3 jobs failed, 73 builds failed. Failure stats cover 1 day time range '26 Jun 2017 23:52 UTC' to '27 Jun 2017 23:52 UTC'. <details<summaryRationale for assignments:</summary </details<br
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
This was broke by . I'm working on a fix for this. /assign


Question: Args: []string{ "/bin/sh", "-c", <del> fmt.Sprintf("for i in $(seq 1 5); do wget -T 8 %s.%s:%d -O - && exit 0 || sleep 1; done; exit 1", </del> <ins> fmt.Sprintf("for i in $(seq 1 5); do curl -s -m 8 %s.%s:%d && exit 0 || sleep 1; done; exit 1", </ins> targetService.Name, targetService.Namespace, targetPort), }, },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: Some commands used in tests are Linux specific and do not exist or do not behave the same on Windows nodes. This can cause those tests to fail on Windows nodes. Some of the tests are: [sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [NodeConformance] [sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [NodeConformance] [sig-network] Networking Granular Checks: Services [Slow] should function for node-Service: http [sig-network] Networking Granular Checks: Services [Slow] should function for node-Service: udp [sig-network] Networking Granular Checks: Services [Slow] should update nodePort: http [Slow] [sig-network] Networking Granular Checks: Services [Slow] should update nodePort: udp [Slow] [sig-network] NetworkPolicy NetworkPolicy between server and client should support a 'default-deny' policy [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on NamespaceSelector [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on Ports [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce multiple, stacked policies with overlapping podSelectors [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should support allow-all policy [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should allow ingress access on one named port [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should allow egress access on one named port [Feature:NetworkPolicy] What you expected to happen: The commands used by those tests should be platform agnostic, so they can pass using Windows Nodes as well. How to reproduce it (as minimally and precisely as possible): Run the mentioned tests using Windows Nodes. Anything else we need to know?

: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): Windows Server 1803 compute nodes Kernel (e.g. ): N/A Install tools: N/A Others:


Question: import ( "encoding/json" "testing" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" </del> ) <ins> type FakeJSONBase struct { ID string } type FakePod struct { FakeJSONBase `json:",inline" yaml:",inline"` Labels       map[string]string Int          int Str          string } </ins> func TestMakeJSONString(t *testing.T) { <del> pod := api.Pod{ JSONBase: api.JSONBase{ID: "foo"}, </del> <ins> pod := FakePod{ FakeJSONBase: FakeJSONBase{ID: "foo"}, </ins> Labels: map[string]string{ "foo": "bar", "baz": "blah", }, <ins> Int: -6, Str: "a string", </ins> } body := MakeJSONString(pod)

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig cluster-lifecycle /cc /priority critical-urgent /milestone v1.19
it seems like the cluster being brought up does not have the API (new in 1.19 and newly tested in conformance) enabled
Digging into this further now
/assign
So... The issue boils down to the way we are pulling in the k8s artifacts and the conformance test binary for this test: The installed kubernetes version is pulled in through the version marker, which is currently referencing an older build (from July 1st before the eventsv1 commit was merged). The test binary is built from the current state of master While we could target a different version marker file to get the version, I believe this issue should be resolved without any further intervention with once it is finished and merged.
This should be fixed in
Opened accompanying PRs for other Cluster API variants: capz: capa:
Thanks


Question: // ServeHTTP processes watch requests. func (h *WatchHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) { ctx := api.NewContext() <ins> namespace := req.URL.Query().Get("namespace") if len(namespace) > 0 { ctx = api.WithNamespace(ctx, namespace) } </ins> parts := splitPath(req.URL.Path) if len(parts) < 1 || req.Method != "GET" { notFound(w, req)

is always using the cross namespace context. This needs to be fixed to support scoped watches.


Question: } return subsets, readyEps, notReadyEps } <ins> func shouldPodBeInEndpoints(pod *v1.Pod) bool { switch pod.Spec.RestartPolicy { case v1.RestartPolicyNever: return pod.Status.Phase != v1.PodFailed && pod.Status.Phase != v1.PodSucceeded case v1.RestartPolicyOnFailure: return pod.Status.Phase != v1.PodSucceeded default: return true } } </ins>

If you create a job in which pods are crashing and a service spawning those pods, this may result in generating extremely huge Endpoints object (which will cause different performance issues). [Based on situation from one of our customers cluster]. After 30m from creating the objects below in my 3 node-cluster I have an endpoint which has almost 1MB of size: Example objects which are enough to cause this : :
- I think that we want to have "failed pods" that have "restart policy" "Always" to be part of "NotReadyAddresses". But do we want to keep pod that are in Failed state and have restart policy "Never"? I think it doesn't make sense. And this would be a quick and easy fix to this particular problem...
Can you update the issue title to "Failed pods with restart policy=Never should not be in the Endpoints.NotReadyAddresses"
done Do you agree that this is a reasonable request?
Geniunely interested: in what case do you want a service and a job? cc:
I can imagine a map-reduce-like thing that serves status from each worker, and uses a headless Service for discovery.
Yes - this is the example I've heard during discussions on last Kubecon.
Well - I don't know if it was map-reduce-like-thing, but it was about headless Service for discovery (and also reporting status from workers).
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We should also not put such pods in the Endpoints.NotReadyAddresses: Succeeded pods with restart policy=Never Succeeded pods with restart policy=OnFailure
- thanks for working on it. I commented on the PR.


Question: shouldError: false, }, { <ins> name:              "cloud provider is external and nodeIP unspecified", nodeIP:            netutils.ParseIPSloppy("::"), nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, expectedAddresses: []v1.NodeAddress{}, shouldError:       false, }, { name:              "cloud provider is external and no nodeIP", nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, expectedAddresses: []v1.NodeAddress{}, shouldError:       false, }, { </ins> name: "cloud doesn't report hostname, no override, detected hostname mismatch", nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"},

The kubelet logic used to get the Node addresses when using an external provider, temporary assigns an address discovered from the node , this address can be overriden later by the external cloud provider This was implemented in to improve the bootstrapping problems with some external cloud providers The side effect of this behavior is that if the external cloud provider sets a different address/es, then the pods with will be showing the "first" addresses as their , since we don't mutate those IPs once set to avoid problems I think that we should revert the existing logic for external cloud providers, since is causing a temporary and inconsistent state and it seem to address mainly a problem with kubeadm but kubeadm folks seems to not need this functionality Announced mailing list /kind bug /sig network /sig node
/assign /cc
So we don't want the external provider to update the IP of the node, right? But I think even if it updates the address, for pods with hostnetwork, it still gets the first address from nodeStatus as podIPs, why is it inconsistent?
the contrary, we don't want to shortcut the address assignment, if cloud provider is external the node should wait for the cloud provider to assign the address. Let me put an example, a node with IPs 10.0.0.1 and 2001:db2::3 and cloud-provider external that sets the IPv6 address first. Right, now, the Node will use prefer IPv4 when it starts and assign the IP 10.0.0.1 to the node, hostNetwork pods will get that IP. and later the cloud provider will override the IPs and sets 2001:db2::3 as the address, so new hostNetwork pods will use the IPv6 address but old hostNetwork pods will have 10.0.0.1
I see, Now I think it makes sense, but we need to confirm that kubeadm really doesn't need it anymore.
^^^
I don't know the history of why it works the way it currently does, but I agree that it's weird and seems likely to lead to problems, so if it turns out it's not necessary, then let's change it.
This is why we taint nodes initially and only remove the taint once cloud provider has initialized the node.

So this is reallly an issue for pods tolerating this taint
i don't understand what is being asked exactly, but if the Q is again whether kubeadm supports self-hosted control plane components in a DS, then no. kubeadm does support the --cloud-privider=external on paper, but we don't have tests. i don't think we want to maintain them either. do you have context?
For current behavior, the workaround would be if I understand correctly. I don't know if there are any cloud provider installers that depend on this behavior. I prefer to fix the weird behavior to avoid potential problems. Or a problem detector has to cover this to check if a hostNet pod is not using the correct node IP.
yeah, but I think that is fair to give those pods the responsibility to deal with the known behavior (node will not have addresses until cloud provider external initializes) , than make the kubelet have two different addresses during its lifetime (that we discussed extensively and ) We discarded this behavior because most components, apps and features like downward API depend on the existing behavior of not changing the original IP For the temporal status without addresses we can just set the node as notReady
NetworkUnavailable might be more appropriate, but overall makes sense I think
is there a reported issue or bug that is reproducible based on what you describe in the isuse description? Logically what you said makes sense but I'm wondering how common the problem actually is
Ah, is it just or are there others?
Oh -- is not even related, because they are not setting --cloud-provider=external
yeah, I opened this issue because when I tried to bring up an IPv6 only cluster with kops in GCE it failed and one of the problems was that the hostNetwork pods were using IPv4 as pod ips ... is not likely anybody hits this with IPv4 or dual stack, since cloud-provider and kubelet will agree most of the times, but as soon that you use IPv6 only or IPv6 primary you ends with this problem
Isn't this a configuration issue on the kubelet and not with external cloud providers then? You should configure kubelet to bootstrap initially with an IPv6 address and not IPv4.

Still does not work for dual-stack, or if there are multiple addresses on the node, or multiple interfaces .... I don't see the reason why kubelet should configure the node addresses in the first place just to wait for the cloud-provider to override that ... the pods bootstrap problem is another thing, kubelet should not be ready if there are no addresses in the node , but current behavior is compensating the problem of having an external source for the address allocation, that is clearly not generalizable
yeah, I opened this issue because when I tried to bring up an IPv6 only cluster with kops in GCE it failed and one of the problems was that the hostNetwork pods were using IPv4 as pod ips ... is not likely anybody hits this with IPv4 or dual stack, since cloud-provider and kubelet will agree most of the times, but as soon that you use IPv6 only or IPv6 primary you ends with this problem For cloud-provider-vsphere, what we did to workaround is to identify the preferred ip address to get, which interface, which rank, then in bootstrap time we get it and pass it to kubelet flag . Now support dual stack address with external cloud-provider. Without the kubelet starting node with some IP, if there is issue in CPI code or in the config, then it would be hard for us to debug CPI since getting the log will getting error that node doesn't have prefered ip. Might need to get the container log from containerd directly by ssh into the node. You mentioned this is for GCE, I remember brought it up in here that now the is also if CCM has route controller enable, which doesn't have clear ownership between CMM and CNI which should reconcile it. And according to the CCM logic , in GCE case it will managed the in CCM, and probably will mark it as false once the route is ready, then node could possibly becomes ready. So maybe that's why in your case the node becomes ready even though the node doesn't have the address you preferred.

I found it trying with GCE, but that is not important, I can hit this with Openshift or Rancher or Kind ... my goal is to define a general behavior of the kubelet that is consistent and easier to think about, without breaking anyone of course, but current implementation is a footgun and seems tailor made... do we enforce all cloud providers to pass the --node-ip flag to kubelet to match their IPs? I will put this for the next sig-network agenda so we can go deeper and talk more about it
Another example of this problem in Azure and CAPG
Yeah, the NetworkUnavailable condition/taint is not extensible or reusable. Any change to its current semantics will not work correctly in some environments.
Ohk we ran into similar issue today, We didn't pass --node-ips to kubelet, and we are setting ipv6 as primary ipfamily, but kubelet chose ipv4 address at the beginning, so all hostNetwork pod has ipv4 address. Now even after CPI has fixed the node ip to put ipv6 address as primary address, all following pods still has ipv4 addresses regardless. Example: Node addresses:
so , does it mean you agree we need to fix this as in
I'm in favor of removing the behavior to bootstrap initial addresses as long as we are confident it is not breaking some exitsing behavior that some provider is depending on. I still find the scenario in a little odd, those pods should not be running on the node if the CCM has no initialized them yet. I wonder if the actual problem here is that node addresses are only updated after the first initialization: Which means pods can be admitted between the time time the cloud provider taint is removed and when addresses are updated.
so , does it mean you agree we need to fix this as in At least from cloud-provider-vsphere it shouldn't depend on this behavior to bootstrap. It's just that it would be harder for people to debug the CPI issue or issues in other component that comes before CPI like KCM/kube-apiserver/etcd in the future if something goes wrong before CPI could initialize the node. Example issue: I still find the scenario in a little odd, those pods should not be running on the node if the CCM has no initialized them yet. Yeah this is weird. For pod like apiserver pod has ipv4 address, which make sense since it comes before CPI.

Kubelet could choose the wrong IP at the beginning now CPI already removes the taints and updated the node with correct IP address, also the node has correct podCIDRs, with ipv6 address at the front. Still if I create a new pod on that node, it will be assigned with a ipv4 address as primary ip. For example that test pod was created long after the CPI initialized the node. And we only experience issue with pod that get created on this node, all other pod on other node will have ipv6 address as primary address.
/triage accepted /priority important-longterm
that should not be a problem if having node addresses is a requirement of node readiness
The AWS EKS AMIs that we provide specify which avoids this issue. If that flag is not passed, then the same problem exists along with a slightly worse one that we intend to fix,
good catch is event more complex, since the ccm will not add addresses meanwhile the node is tainted


Question: return nil, fmt.Errorf("failed to create target pool %s: %v", loadBalancerName, err) } if len(hosts) <= maxTargetPoolCreateInstances { <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): created target pool", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): created target pool", loadBalancerName, serviceName) </ins> } else { <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): created initial target pool (now updating with %d hosts)", loadBalancerName, serviceName, len(hosts)-maxTargetPoolCreateInstances) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): created initial target pool (now updating with %d hosts)", loadBalancerName, serviceName, len(hosts)-maxTargetPoolCreateInstances) </ins> created := sets.NewString() for _, host := range createInstances {

Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick


Question: GetByKey(key string) (value interface{}, exists bool, err error) } <del> // DeltaType is the type of a change (addition, deletion, etc) type DeltaType string // Change type definition const ( Added   DeltaType = "Added" Updated DeltaType = "Updated" Deleted DeltaType = "Deleted" // Replaced is emitted when we encountered watch errors and had to do a // relist. We don't know if the replaced object has changed. // // NOTE: Previous versions of DeltaFIFO would use Sync for Replace events // as well. Hence, Replaced is only emitted when the option // EmitDeltaTypeReplaced is true. Replaced DeltaType = "Replaced" // Sync is for synthetic events during a periodic resync. Sync DeltaType = "Sync" ) // Delta is the type stored by a DeltaFIFO. It tells you what change // happened, and the object's state after* that change. // // [*] Unless the change is a deletion, and then you'll get the final //     state of the object before it was deleted. type Delta struct { Type   DeltaType Object interface{} } // Deltas is a list of one or more 'Delta's to an individual object. // The oldest delta is at index 0, the newest delta is the last one. type Deltas []Delta </del> // Oldest is a convenience function that returns the oldest delta, or // nil if there are no deltas. func (d Deltas) Oldest() *Delta {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While reviewing for I noticed that introduced a second type of Delta that is specific to DeltaFIFO but did not update the relevant documentation at , which was introduced by concurrent PR . What you expected to happen: I expected the documentation on DeltaFIFO to be current. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I think it would be more reader-friendly if the definitions of Deltas and related types were closer to their mentions near the top of the file rather than near the end.
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
Hi, is someone working on this? If no, can I pick it up? Thanks.
: go ahead, and thanks!
Thanks! /assign


Question: // nil NUMANodeInfo, indicating that no NUMA information is available // on this machine. This should implicitly be interpreted as having a // single NUMA node with id 0 for all CPUs. <del> nodelist, err := ioutil.ReadFile("/sys/devices/system/node/possible") </del> <ins> nodelist, err := ioutil.ReadFile("/sys/devices/system/node/online") </ins> if err != nil { return nil, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With CentOS 7.6, the kubelet systemd service may failed, because the numbers of NUMA nodes it reports is different from the actual number of nodes. the kubelet log is: Sep 25 10:26:37 node50 systemd: Started Kubernetes systemd probe. Sep 25 10:26:37 node50 kubelet: I0925 10:26:37. ] Version: v1.16.0 Sep 25 10:26:37 node50 kubelet: I0925 10:26:37. ] No cloud provider specified. Sep 25 10:26:37 node50 kubelet: I0925 10:26:37. ] Client rotation is on, will bootstrap in background Sep 25 10:26:37 node50 kubelet: I0925 10:26:37. ] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-". Sep 25 10:26:37 node50 kubelet: I0925 10:26:37. ] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / Sep 25 10:26:37 node50 kubelet: F0925 10:26:37. ] failed to run Kubelet: open /sys/devices/system/node/node2/cpulist: no such file or directory Sep 25 10:26:37 node50 systemd: kubelet.service: main process exited, code=exited, status=255/n/a Sep 25 10:26:37 node50 systemd: Unit kubelet.service entered failed state.

What you expected to happen: But this node has only 2 NUMA nodes, node0 and node1 [root ~]# lscpu Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6240Y CPU @ 2.60GHz Stepping: 7 CPU MHz: 1000.073 CPU max MHz: 3900.0000 CPU min MHz: 1000.0000 BogoMIPS: 5200.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-17,36-53 NUMA node1 CPU(s): 18-35,54-71 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 ...... How to reproduce it (as minimally and precisely as possible): With Intel Xeon CLX-SP 2 sockets CPUs, in CentOS 7.6, sometimes the sysfs /sys/devices/system/node/possible reports more than actual CPU counts. And the topology in cpumanager read this file, if it reads 4 CPUs, but there are only 2 CPUs, then kubelet service will failed. [root ~]# cat /sys/devices/system/node/possible 0-3 [root ~]# cat /sys/devices/system/node/online 0-1 Anything else we need to know?

: Suggest func GetNUMANodeInfo() read /sys/devices/system/node/online instead of /sys/devices/system/node/possible Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 2x socket Intel Xeon Gold 6240Y OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Kernel (e.g. ): [root ~]# uname -a Linux node50 3.10.0-957.el7.x8664 SMP Thu Nov 8 23:39:32 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node


Question: // GroupsClaim, if specified, causes the OIDCAuthenticator to try to populate the user's // groups with a ID Token field. If the GrouppClaim field is present in a ID Token the value <del> // must be a list of strings. </del> <ins> // must be a string or list of strings. </ins> GroupsClaim string }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, already discussed on sig-auth What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): confirmed issue on sig-auth channel Is this a BUG REPORT or FEATURE REQUEST? (choose one): Depends on interpretation <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-08-02T00:54:53Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: CoreOS Vagrant Image OS (e.g. from /etc/os-release): NAME=CoreOS ID=coreos VERSION=1122.0.0 VERSIONID=1122.0.0 BUILDID=2016-07-27-0739 PRETTYNAME="CoreOS 1122.0.0 (MoreOS)" ANSICOLOR="1;32" HOMEURL="" BUGREPORTURL="" Kernel (e.g.

): Linux localhost 4.7.0-coreos SMP Wed Jul 27 07:30:04 UTC 2016 x8664 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz GenuineIntel GNU/Linux Install tools: Vagrant Others: What happened: The claim named in --oidc-groups-claim MUST be list of strings, even if there is only one value. Talking to this is by design since its assumed that users will have more then one group. What you expected to happen: The OIDC authenticator should accept either a list of strings or a single string since: There is no concept of schema that marks an attribute as single value or multi value in JSON or a JWT There is no standard for representing groups in a JWT There is no standard for representing groups in OIDC An OIDC should be flexible in how it accepts data. How to reproduce it (as minimally and precisely as possible): Deploy with RBAC enabled Create a role binding for the group "admin" Generate a JWT such as the one that does NOT work below User won't be able to login Generate a JWT where the role is multi-value User will be authorized Does NOT Work: DOES Work: Anything else do we need to know:
These should be called groups, not roles, right? Those are different. And, is it normal for a JWT to contain keys which are arrays? That seems weird.
where do you see role defined? you mean the user_role claim?
Claims can be structured, yes. The name of the groups claim is arbitrary. Could easily have been "group". In my mind, this is all about enabling mapping a "groupish" claim from whatever IDP issued the JWT, even if it is a single value instead of an array.
That's the way it works currently. Yeah I'm fine with this.
Okay.


Question: if !strings.Contains(output, test.accBody) { t.Errorf("Expected response body to contain %v, got %v", test.accBody, output) } <ins> if test.notBody != "" && strings.Contains(output, test.notBody) { t.Errorf("Expected response not to contain %v, got %v", test.notBody, output) } </ins> } }() }

Grabbed tcpdump of a readiness probe - the src ip was the bridge address (10.244.1.1), but apart from this, there is no identifying information in the HTTP request that this is a Kubernetes daemon requesting readiness checks - this might help QoS/priority assignment by the container application.
IIUC, you mean we should add a better when kubernetes daemon request a readiness check to identify this request comes from kubernetes daemon?
Yes, we should override the default net/http user-agent with our own. Better usability for users and we have no future upgrade hazards if users start hardcoding and expecting this default User-Agent string in their analytic tools.
Any news about this one? It would be nice! I've this little hack in my nginx configuration (in order to not be spamed by health check in logs) :)
This can probably be closed now that probes have support.


Question: // verify workers that are not terminated stay open even if config no longer // sees them podWorkers.SyncKnownPods(nil) <del> if len(podWorkers.podUpdates) != 2 { </del> <ins> drainAllWorkers(podWorkers) if len(podWorkers.podUpdates) != 0 { </ins> t.Errorf("Incorrect number of open channels %v", len(podWorkers.podUpdates)) } if len(podWorkers.podSyncStatuses) != 2 {

Corresponds to merge of /kind failing-test /sig node /assign /milestone v1.27 /priority critical-urgent === RUN TestSyncKnownPods WARNING: DATA RACE Write at 0x00c000906f60 by goroutine 1395: runtime.mapdeletefaststr() +0x0 (podWorkers).cleanupPodUpdates() +0x299 (podWorkers).completeTerminated() +0x1f4 (podWorkers).podWorkerLoop() +0x18b6 (podWorkers).UpdatePod.func1() +0x175 Previous read at 0x00c000906f60 by goroutine 1380: () +0x1056 testing.tRunner() +0x216 testing.(T).Run.func1() +0x47 Goroutine 1395 (running) created at: (podWorkers).UpdatePod() +0x5131 () +0xc4 testing.tRunner() +0x216 testing.(*T).Run.func1() +0x47
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is crazy hard to reproduce locally - I also don't think I saw it one time during development. It's definitely a race caused by my change, but I don't remember hitting it (I would have been hypersensitive to this test). That's concerning.
Rank 2nd for : " TestSyncKnownPods": 15,


Question: func TestSingleWatch(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> ns := "blargh" deleteAllEtcdKeys()

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: package util import ( <ins> "context" </ins> "fmt" <ins> "github.com/google/go-cmp/cmp" </ins> "testing" "time"

I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted


Question: } func (h *hostpathCSIDriver) GetDynamicProvisionStorageClass(fsType string) *storagev1.StorageClass { <del> provisioner := h.driverInfo.Name + h.driverInfo.Framework.UniqueName </del> <ins> provisioner := GetUniqueDriverName(h) </ins> parameters := map[string]string{} ns := h.driverInfo.Framework.Namespace.Name suffix := fmt.Sprintf("%s-sc", provisioner)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: }) <del> /* Testname: Service, NodePort, same port different protocols Description: Create a service of type NodePort listening on port 53 for two protocols TCP and UDP. Service creation MUST be successful by assigning a ClusterIP and two unique nodePorts for each protocol, making service reachable on every node's IP and nodePort. TODO: Test Service reachability, good to include this check in Conformance perspective. */ ginkgo.It("should use same NodePort with same port but different protocols", func() { serviceName := "nodeports" ns := f.Namespace.Name t := e2eservice.NewServerTest(cs, ns, serviceName) defer func() { defer ginkgo.GinkgoRecover() errs := t.Cleanup() if len(errs) != 0 { framework.Failf("errors in cleanup: %v", errs) } }() ginkgo.By("creating service " + serviceName + " with same NodePort but different protocols in namespace " + ns) service := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      t.ServiceName, Namespace: t.Namespace, }, Spec: v1.ServiceSpec{ Selector: t.Labels, Type:     v1.ServiceTypeNodePort, Ports: []v1.ServicePort{ { Name:     "tcp-port", Port:     53, Protocol: v1.ProtocolTCP, }, { Name:     "udp-port", Port:     53, Protocol: v1.ProtocolUDP, }, }, }, } result, err := t.CreateService(service) framework.ExpectNoError(err, "failed to create service: %s in namespace: %s", serviceName, ns) if len(result.Spec.Ports) != 2 { framework.Failf("got unexpected len(Spec.Ports) for new service: %v", result) } if result.Spec.Ports[0].NodePort != result.Spec.Ports[1].NodePort { framework.Failf("should use same NodePort for new service: %v", result) } }) </del> ginkgo.It("should prevent NodePort collisions", func() { // TODO: use the ServiceTestJig here baseName := "nodeport-collision-"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Following are two e2e testing service functioning over same port but different protocol, but do not test service reachability. be able to update NodePorts with two same port numbers but different protocols use same NodePort with same port but different protocols Note: Both the e2e are verifying the same behavior of "service using same port but different protocols" differs in and . Primary goal: Test service reachability for these E2Es. Secondary goal: As both the above e2e are verifying same E2E behavior, it's good to have single e2e in spite of 2 differing in "updating" and "create only" nodePort service. For "create only", suite has another specific e2e . Why is this needed: E2E Improvement and make e2e eligible for Conformance. Service related e2e are getting promoted in /area conformance /sig testing /assign


Question: // and reduce the default page size to 2. storeWithCounter := &storageWithCounter{Interface: registry.Storage.Storage} registry.Storage.Storage = storeWithCounter <ins> originalDeleteCollectionPageSize := deleteCollectionPageSize </ins> deleteCollectionPageSize = 2 <ins> defer func() { deleteCollectionPageSize = originalDeleteCollectionPageSize }() </ins> numPods := 10 for i := 0; i < numPods; i++ {

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: } func (s *ContainerRuntimeOptions) AddFlags(fs *pflag.FlagSet) { <ins> dockerOnlyWarning := "This docker-specific flag only works when container-runtime is set to docker." </ins> // General settings. fs.StringVar(&s.ContainerRuntime, "container-runtime", s.ContainerRuntime, "The container runtime to use. Possible values: 'docker', 'remote', 'rkt (deprecated)'.") fs.StringVar(&s.RuntimeCgroups, "runtime-cgroups", s.RuntimeCgroups, "Optional absolute name of cgroups to create and run the runtime in.")

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When use remote container runtime, the remote can't get any PodSandboxImage config set by kubelet What you expected to happen: We want the all container runtime should know what's right PodSandboxImage How to reproduce it (as minimally and precisely as possible): in new node with into k8s cluster kubelet with options expect frakti can create sandbox container with , but get default (k8s version: v1.8.2) Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bootstrap by kubeadm v1.8.2 OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch) Kernel (e.g. ): SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x8664 GNU/Linux Install tools: kubeadm Others:
/sig node
good news! thanks!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Actually we do a no-op when the container runtime is . This is why config can not be set by flag for . In your case, the correct place to change the default sandbox image is in .


Question: */ framework.ConformanceIt("should function for intra-pod communication: http [NodeConformance]", func() { config := e2enetwork.NewCoreNetworkingTestConfig(f, false) <del> for _, endpointPod := range config.EndpointPods { config.DialFromTestContainer("http", endpointPod.Status.PodIP, e2enetwork.EndpointHTTPPort, config.MaxTries, 0, sets.NewString(endpointPod.Name)) } </del> <ins> checkNodeConnectivity(config, "http", e2enetwork.EndpointHTTPPort) </ins> }) /*

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: func (s *InsecureServingOptions) AddFlags(fs *pflag.FlagSet) { fs.IPVar(&s.BindAddress, "insecure-bind-address", s.BindAddress, ""+ "The IP address on which to serve the --insecure-port (set to 0.0.0.0 for all interfaces).") <ins> fs.MarkDeprecated("insecure-bind-address", "This flag will be removed in a future version.") </ins> fs.IntVar(&s.BindPort, "insecure-port", s.BindPort, ""+ "The port on which to serve unsecured, unauthenticated access. It is assumed "+ "that firewall rules are set up such that this port is not reachable from outside of "+ "the cluster and that port 443 on the cluster's public address is proxied to this "+ <del> "port. This is performed by nginx in the default setup. Set to zero to disable") </del> <ins> "port. This is performed by nginx in the default setup. Set to zero to disable.") fs.MarkDeprecated("insecure-port", "This flag will be removed in a future version.") </ins> } <ins> // TODO: remove it until kops stop using `--address` </ins> func (s *InsecureServingOptions) AddDeprecatedFlags(fs *pflag.FlagSet) { fs.IPVar(&s.BindAddress, "address", s.BindAddress, "DEPRECATED: see --insecure-bind-address instead.")

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: "containers": [ { "name": "kube-proxy", <del> "image": "gcr.io/google_containers/hyperkube-ARCH:VERSION", </del> <ins> "image": "REGISTRY/hyperkube-ARCH:VERSION", </ins> "command": [ "/hyperkube", "proxy",

Because of the revert on commit , when building for 3rd party registry, hyperkube is broken again.
Can we close this now? looks like the PR has merged


I am suspicious that CRD watch is not working, but was unable to find e2e test coverage to persuade myself that it is working. cf .
(in particular, I'm looking for the update tests.)
I think this was due to enabling protobuf, though it also failed in a curl for me, so something strange is happening. (Maybe we have to explicitly ask for json?)
We test CRD watch in and in , but not in . I agree we should add e2e tests as a requirement before GA, and also for the sake of defining conformance.
/area custom-resources
Are we talking about CRD or CR watch and protobuf? The later with protobuf obviously does not work, as commented.
I'm not sure "obviously" is correct - list works, for example. IIRC watch with protobuf just silently fails to do anything - it doesn't return an error. And yes I mean CR watch :-)
it should obviously not work with protobuf, but in the sense that the negotiating serializer switches to JSON if you offer it json and protobuf as alternatives. This seems to be broken.
/cc
I'd like to take this up. I'm familiarising myself with the apiextension code base, please give me some time to come up with a PR :)
We already have integration tests for this. Is an e2e test really needed?
I think we do need an e2e test for CRD watch (among other things) as a requirement for CRD going GA. The most pressing need is to define conformance (which only e2e tests can do). Currently we only test that you can create and delete a CRD, but we don't test that you can actually do anything with it: We should expand the conformance (e2e) test for CRD to include checking that the custom resource gets installed and is usable, including watches. If not for conformance, I think an e2e test for CRD watch would still be appropriate since it's an important end-user flow (a thing we expect users to do on purpose; not an edge case). For example, the apiextensions-apiserver integration test might miss problems that come from embedding into kube-apiserver. Similarly, the kubectl tests are meant to test kubectl client functionality, not to protect against regressions on the server side.

Someone might reasonably change those tests in a way that suits kubectl, without realizing that doing so could leave some server code untested at the e2e level. We also need the e2e test to make sure we honor the guarantees of the raw API, to ensure clients other than kubectl will work.
For our developer purpose I would say that e2e tests are not needed when integration tests are in-place. We can even test with a real kube-apiserver including embdded into kube-apiserver if we want that. With conformance tests might have a point. Though I wonder where the line is between e2e and features we test in integration or unit tests only. We don't test all behaviour in e2e today, by far.
Since this has the help-wanted label, adding some useful info for new contributors to get started: About CRDs: If you are new to the Kubernetes API in general, I'd recommend reading the first. CustomResourceDefinition (CRD) is a way to extend the Kubernetes API. You can read more about it here: If you'd like to do some additional reading about this, I had written something up during the time I did GSoC (shamless plug): About testing: There are lots of integration tests for CRDs and you can get a general idea about testing for CRDs and apiextensions-apiserver in general. There is for CRDs. You can use this and other e2e tests to understand how to write them. contains useful information about how you can run tests. gives general info about e2e tests and mentions tests about writing good e2e tests. Also, you can ping me on Slack (I am there) if you have further questions or need help! :)


Question: logOption.Apply() // To help debugging, immediately log version klog.Infof("Version: %+v", version.Get()) <del> if err := initForOS(s.KubeletFlags.WindowsService); err != nil { </del> <ins> if err := initForOS(s.KubeletFlags.WindowsService, s.KubeletFlags.WindowsPriorityClass); err != nil { </ins> return fmt.Errorf("failed OS init: %v", err) } if err := run(ctx, s, kubeDeps, featureGate); err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: result = IsErrorRetriable(fmt.Errorf("Retriable: true")) assert.Equal(t, true, result) } <ins> func TestHasErrorCode(t *testing.T) { // false case result := HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: 408")) assert.False(t, result) // true case 404 result = HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: %d", http.StatusNotFound)) assert.True(t, result) // true case 403 result = HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: %d", http.StatusForbidden)) assert.True(t, result) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: return sha256.New() }, } <del> legacyregistry.MustRegister(encryptionConfigAutomaticReloadFailureTotal) legacyregistry.MustRegister(encryptionConfigAutomaticReloadSuccessTotal) </del> <ins> legacyregistry.MustRegister(encryptionConfigAutomaticReloadsTotal) legacyregistry.MustRegister(deprecatedEncryptionConfigAutomaticReloadFailureTotal) legacyregistry.MustRegister(deprecatedEncryptionConfigAutomaticReloadSuccessTotal) </ins> legacyregistry.MustRegister(encryptionConfigAutomaticReloadLastTimestampSeconds) }) } func RecordEncryptionConfigAutomaticReloadFailure(apiServerID string) { apiServerIDHash := getHash(apiServerID) <del> encryptionConfigAutomaticReloadFailureTotal.WithLabelValues(apiServerIDHash).Inc() </del> <ins> encryptionConfigAutomaticReloadsTotal.WithLabelValues("failure", apiServerIDHash).Inc() deprecatedEncryptionConfigAutomaticReloadFailureTotal.WithLabelValues(apiServerIDHash).Inc() </ins> recordEncryptionConfigAutomaticReloadTimestamp("failure", apiServerIDHash) } func RecordEncryptionConfigAutomaticReloadSuccess(apiServerID string) { apiServerIDHash := getHash(apiServerID) <del> encryptionConfigAutomaticReloadSuccessTotal.WithLabelValues(apiServerIDHash).Inc() </del> <ins> encryptionConfigAutomaticReloadsTotal.WithLabelValues("success", apiServerIDHash).Inc() deprecatedEncryptionConfigAutomaticReloadSuccessTotal.WithLabelValues(apiServerIDHash).Inc() </ins> recordEncryptionConfigAutomaticReloadTimestamp("success", apiServerIDHash) }

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?