Question: klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s, %s)", nodeResourceGroup, nodeName, diskName, diskURI) rerr := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM, "detach_disk") if rerr != nil { <del> return rerr.Error() </del> <ins> klog.Errorf("azureDisk - detach disk(%s, %s) on rg(%s) vm(%s) failed, err: %v", diskName, diskURI, nodeResourceGroup, nodeName, rerr) if rerr.HTTPStatusCode == http.StatusNotFound { klog.Errorf("azureDisk - begin to filterNonExistingDisks(%s, %s) on rg(%s) vm(%s)", diskName, diskURI, nodeResourceGroup, nodeName) disks := ss.filterNonExistingDisks(ctx, *newVM.VirtualMachineScaleSetVMProperties.StorageProfile.DataDisks) newVM.VirtualMachineScaleSetVMProperties.StorageProfile.DataDisks = &disks if rerr = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM, "detach_disk"); rerr != nil { return rerr.Error() } } </ins> } <ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s, %s) succeeded", nodeResourceGroup, nodeName, diskName, diskURI) </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: check disk status before disk azure disk There is little possibility that attach disk would cost 10min, and disk could be attached, deleted in same time, that would cause race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: return fmt.Errorf("unable to load server certificate: %v", err) } c.Cert = &tlsCert <ins> } else if s.ServerCert.GeneratedCert != nil { c.Cert = s.ServerCert.GeneratedCert </ins> } if len(s.CipherSuites) != 0 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: # verify master has all required daemons echo "Validating master" local machine="master" <del> local -a required_daemon=("salt-master" "salt-minion" "kube-apiserver" "nginx" "kube-controller-manager" "kube-scheduler") </del> <ins> local -a required_daemon=("salt-master" "salt-minion" "kube-apiserver" "nginx" "kube-controller-manager" "kube-scheduler" "kubelet") </ins> local validated="1" until [[ "$validated" == "0" ]]; do validated="0"

Self-assigning as I work out a reliable fix.
cc
Running into a few issues. On GCE, did you guys block somewhere to wait for etcd to be running before reporting success on kube-up?
No, except there is a race between addson services, dns and etcd container. We introduced a retry logic on those services.
I assume pulling the image is just so fast that you don't run into e2e errors, it looks like you guys block on wait-for-minions that uses glcloud, so I guess when minions are reported in etcd, then you know you are good...


Question: type ResourceQuotaScope string const ( <del> // Match all pod objects where spec.activeDeadlineSeconds </del> <ins> // Match all pod objects where spec.activeDeadlineSeconds >=0 </ins> ResourceQuotaScopeTerminating ResourceQuotaScope = "Terminating" <del> // Match all pod objects where !spec.activeDeadlineSeconds </del> <ins> // Match all pod objects where spec.activeDeadlineSeconds is nil </ins> ResourceQuotaScopeNotTerminating ResourceQuotaScope = "NotTerminating" // Match all pod objects that have best effort quality of service ResourceQuotaScopeBestEffort ResourceQuotaScope = "BestEffort"

The following comments are incomplete. For example, comments for "ResourceQuotaScopeTerminating" should be: Ref: /good-first-issue
/sig scheduling
/assign
Per ResourceQuota is within scope of api-machinery. /sig api-machinery /remove-sig scheduling
/assign
/unassign rakeshreddybandi
/remove-good-first-issue
Hi!! I'd like to help #investigate this problem I'm a beginner, so I'll make my best. Can you please help me point me in the right direction? What's the output today vs. how does the output should look like?
Oh I get it, the comments on top of the lines are not completely accurate on what the message displays, right? I think I got it
Yeah - But I've already opened a PR for this! - See the linked PR So this is already done, just waiting on final merge.
Great Dean, thanks for that!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: include  ../../hack/make-rules/Makefile.manifest REGISTRY ?= gcr.io/kubernetes-e2e-test-images <del> GOARM=7 </del> <ins> GOARM ?= 7 </ins> QEMUVERSION=v2.9.1 GOLANG_VERSION=1.12.6 export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: login system root, and run below command from "test/images" dir, Will fail with below error message, ./image- line 91: : command not found This is cause by the empty string like "" cannot be understood as a shell command. What you expected to happen: image should be built successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ):all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: cd $ $ make ../image- bin webhook ../image- line 145: GOARM: unbound variable Makefile:23: recipe for target 'bin' failed make: [bin] Error 1 What you expected to happen: The binary or image should be built successfully. How to reproduce it (as minimally and precisely as possible): see my desc above, type "make" will see the error message. Anything else we need to know?: Environment*: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): Ubuntu Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I'm pretty sure this is supposed to be built via from which does default this environment variable.
, thanks for the information! I think the harness of the shell scripts could bring us a little more, GOARM should only been set when the ARCH is ARM Make a binary is possible where a Makefile is presented
That seems reasonable, but we should probably avoid computing this in many places, currently the GOARM=${GOARM:-7} is going to always be 7 when building in the currently supported way. We should change that if we're going to compute it elsewhere. cc-ing for test/images fun ...


Question: } } <ins> func BenchmarkNoCELExpressionFeatureOff(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{} b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, false) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, false) }) } func BenchmarkNoCELExpressionFeatureOn(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{} b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithOneCELExpressions(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithOneCELExpressionsFalse(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice2'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithTwoCELExpressions(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice'", }, { Expression: "request.uid == '1'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithTwoCELExpressionsFalse(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice'", }, { Expression: "request.uid == '2'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithManyCELExpressions(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice'", }, { Expression: "request.uid == '1'", }, { Expression: "('group1' in request.groups)", }, { Expression: "('key1' in request.extra)", }, { Expression: "!('key2' in request.extra)", }, { Expression: "('a' in request.extra['key1'])", }, { Expression: "!('z' in request.extra['key1'])", }, { Expression: "has(request.resourceAttributes) && request.resourceAttributes.namespace == 'kittensandponies'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func BenchmarkWithManyCELExpressionsFalse(b *testing.B) { expressions := []apiserver.WebhookMatchCondition{ { Expression: "request.user == 'alice'", }, { Expression: "request.uid == '1'", }, { Expression: "('group1' in request.groups)", }, { Expression: "('key1' in request.extra)", }, { Expression: "!('key2' in request.extra)", }, { Expression: "('a' in request.extra['key1'])", }, { Expression: "!('z' in request.extra['key1'])", }, { Expression: "has(request.resourceAttributes) && request.resourceAttributes.namespace == 'kittensandponies1'", }, } b.Run("compile", func(b *testing.B) { benchmarkNewWebhookAuthorizer(b, expressions, true) }) b.Run("authorize", func(b *testing.B) { benchmarkWebhookAuthorize(b, expressions, true) }) } func benchmarkNewWebhookAuthorizer(b *testing.B, expressions []apiserver.WebhookMatchCondition, featureEnabled bool) { service := new(mockV1Service) service.statusCode = 200 service.Allow() s, err := NewV1TestServer(service, serverCert, serverKey, caCert) if err != nil { b.Fatal(err) } defer s.Close() defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, features.StructuredAuthorizationConfiguration, featureEnabled)() b.ResetTimer() for i := 0; i < b.N; i++ { // Create an authorizer with or without expressions to compile _, err := newV1Authorizer(s.URL, clientCert, clientKey, caCert, 0, noopAuthorizerMetrics(), expressions) if err != nil { b.Fatal(err) } } b.StopTimer() } func benchmarkWebhookAuthorize(b *testing.B, expressions []apiserver.WebhookMatchCondition, featureEnabled bool) { attr := authorizer.AttributesRecord{ User: &user.DefaultInfo{ Name:   "alice", UID:    "1", Groups: []string{"group1", "group2"}, Extra:  map[string][]string{"key1": {"a", "b", "c"}}, }, ResourceRequest: true, Namespace:       "kittensandponies", Verb:            "get", } service := new(mockV1Service) service.statusCode = 200 service.Allow() s, err := NewV1TestServer(service, serverCert, serverKey, caCert) if err != nil { b.Fatal(err) } defer s.Close() defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, features.StructuredAuthorizationConfiguration, featureEnabled)() // Create an authorizer with or without expressions to compile wh, err := newV1Authorizer(s.URL, clientCert, clientKey, caCert, 0, noopAuthorizerMetrics(), expressions) if err != nil { b.Fatal(err) } b.ResetTimer() for i := 0; i < b.N; i++ { // Call authorize may or may not require cel evaluations _, _, err = wh.Authorize(context.Background(), attr) if err != nil { b.Fatal(err) } } b.StopTimer() } </ins> // TestV1WebhookMatchConditions verifies cel expressions are compiled and evaluated correctly func TestV1WebhookMatchConditions(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.StructuredAuthorizationConfiguration, true)()

In a follow-up (ideally before test freeze), it would also be good to add a benchmark around compilation and evaluation in a follow-up so we can quantify the cost of putting match expressions of a few different levels of complexity in a webhook and make sure our evaluation is as cheap / fast as we think it is. Originally posted by in
/triage accepted /sig auth


Question: nodeStorageClient.Nodes(), podRegistry, ) <del> go util.Forever(func() { podCache.UpdateAllContainers() }, m.cacheTimeout) </del> <ins> if c.SyncPodStatus { go util.Forever(func() { podCache.UpdateAllContainers() }, m.cacheTimeout) } </ins> go util.Forever(func() { podCache.GarbageCollectPodStatus() }, time.Minute*30) // TODO: refactor podCache to sit on top of podStorage via status calls

Split off as a smaller more specific work item. Apiserver should have a flag that makes it not attempt to GET from kubelets. This would be used in conjunction with . After we have converted the top N Getting-started-guides to the new style, we can rip out the kubelet client from the apiserver altogether.
This is tightly related to so I'll take it.


Question: dynamicEndpoint = pvSpec.Glusterfs.EndpointsName } glog.V(3).Infof("glusterfs: dynamic namespace and endpoint : [%v/%v]", dynamicNamespace, dynamicEndpoint) <del> err = d.plugin.deleteEndpointService(dynamicNamespace, dynamicEndpoint) </del> <ins> err = d.deleteEndpointService(dynamicNamespace, dynamicEndpoint) </ins> if err != nil { glog.Errorf("glusterfs: error when deleting endpoint/service :%v", err) } else {

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: "disruptive.go", "driveroperations.go", "ephemeral.go", <ins> "fsgroupchangepolicy.go", </ins> "multivolume.go", "provisioning.go", "snapshottable.go",

We need to add e2e that verifies volume permissions in pod with different values of fsGroupChangePolicy. /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: "os" "regexp" <del> "k8s.io/mount-utils" </del> <ins> "github.com/opencontainers/selinux/go-selinux" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/kubernetes/pkg/kubelet/config" </ins> "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util" "k8s.io/kubernetes/pkg/volume/util/hostutil" "k8s.io/kubernetes/pkg/volume/util/recyclerclient" "k8s.io/kubernetes/pkg/volume/validation" <ins> "k8s.io/mount-utils" </ins> ) // ProbeVolumePlugins is the primary entrypoint for volume plugins.

crio-e2e tests The following tests fail when run on a node with SELinux enabled: Probably since these tests were . No response The host path volumes created under /tmp/hostpathpv don't have the right SELinux label set for access by containers started during the test. No response_ /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk= github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= <del> github.com/google/cadvisor v0.38.5 h1:XOvqjL2+xMEuDORcLMv77NystZvQB7YgGxcKpRul/vE= github.com/google/cadvisor v0.38.5/go.mod h1:1OFB9sOOMkBdUBGCO/1SArawTnDscgMzTodacVDe8mA= </del> <ins> github.com/google/cadvisor v0.38.6 h1:5vu8NaOqsBKF6wOxBLeDPD7hcmxfg/4I5NZGYXK7gIo= github.com/google/cadvisor v0.38.6/go.mod h1:1OFB9sOOMkBdUBGCO/1SArawTnDscgMzTodacVDe8mA= </ins> github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC.

Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?

No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes.

We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.


Question: destroy-network } <del> function find-release-tars { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" exit 1 fi } </del> # The kubernetes binaries are pushed to a host directory which is exposed to the VM function upload-server-tars { tar -x -C "$POOL_PATH/kubernetes" -f "$SERVER_BINARY_TAR" kubernetes

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: "testing" "golang.org/x/net/context" <ins> "k8s.io/kubernetes/pkg/cloudprovider" </ins> "k8s.io/kubernetes/pkg/util/rand" )

ExternalID/InstanceID do not return 'cloudprovider.InstanceNotFound' error when the VM is not found. This can occur when a node VM is deleted entirely. In this case InstanceID method would throw an nil pointer exception here: since returned error here: is not being handled, vm would be nil. vm object is then dereferenced in getVirtualMachineManagedObjectReference func. In ExternalID method, the error is being handled by just passing it up. Node controller then prints out error and continues: e.g. ExternalID/InstanceID should handle the 'nod found' error being returned from vCenter and return 'cloudprovider.InstanceNotFound'.


Question: exit 1 fi kube::log::status "Checking kubectl version" <del> export KUBERNETES_MASTER=http://127.0.0.1:${API_PORT} </del> kubectl version # Generate a random namespace name, based on the current time (to make

What happened: I am using a command like below in GitHub actions: I get the following error: What you expected to happen: Since I am using flag, it should work without requiring a valid kubeconfig file. How to reproduce it (as minimally and precisely as possible): Just delete your local kubeconfig file and try the above command. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign /kind regression
This likely breaks some scripted usage of kubectl
This is related to /assign
Or
Yes, sorry I mentioned you in this issue. Because my first thought is this brings and which will check the config file existence but in fact that's not the root cause. Actually the root cause This checks config has the server or not. And with this , this function will return error when there's no local config file. I think there might be two solutions? the similar logic like to every , , ..... when or is set. Seems not a good idea. reverse this to have the default server which is better than ? By the way, is equals ? Since This works only for .
The plan I agreed on with Jordan is that I will revert since that seems to be the main culprit behind all these problems. I'll add a test ensuring this works as required. This will allow us to backport the fix to 1.18. For 1.19 I'll put together a proper fix supporting and flags.


Question: return nil }, framework.StatefulPodTimeout, 2*time.Second).Should(BeNil()) }) <del> /* Comment it for now until scale sub-resource is finalized in ref:pull/53679 for scale-sub resource specific comment. </del> It("should have a working scale subresource", func() { By("Creating statefulset " + ssName + " in namespace " + ns) ss := framework.NewStatefulSet(ssName, ns, headlessSvcName, 1, nil, nil, labels)

What would you like to be : e2e test suites for: GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale Why is this needed: PR has already been merged, but the following tests are still commented out. Therefore, scale sub-resource of statefulsets is not tested and test should be . /sig testing


Question: package plugin import ( <ins> "context" "errors" "net" </ins> "sync" <ins> "time" "google.golang.org/grpc" "google.golang.org/grpc/connectivity" "google.golang.org/grpc/credentials/insecure" </ins> utilversion "k8s.io/apimachinery/pkg/util/version" <ins> "k8s.io/klog/v2" </ins> ) // Plugin is a description of a DRA Plugin, defined by an endpoint // and the highest DRA version supported. type Plugin struct { <ins> sync.RWMutex conn                    *grpc.ClientConn </ins> endpoint                string highestSupportedVersion *utilversion.Version } <ins> func (p *Plugin) getOrCreateGRPCConn() (*grpc.ClientConn, error) { p.Lock() defer p.Unlock() if p.conn != nil { return p.conn, nil } network := "unix" klog.V(4).InfoS(log("creating new gRPC connection"), "protocol", network, "endpoint", p.endpoint) conn, err := grpc.Dial( p.endpoint, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target) }), ) if err != nil { return nil, err } ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() if ok := conn.WaitForStateChange(ctx, connectivity.Connecting); !ok { return nil, errors.New("timed out waiting for gRPC connection to be ready") } p.conn = conn return p.conn, nil } </ins> // PluginsStore holds a list of DRA Plugins. type PluginsStore struct { sync.RWMutex

See comment for details:
/sig node
/area kubelet
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/triage accepted


Question: "context" "encoding/json" "fmt" <ins> "strings" </ins> "time" appsv1 "k8s.io/api/apps/v1"

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // We can use the same metadata producer for all nodes. meta := metadataProducer(pod, nodeNameToInfo) <ins> var equivCacheInfo *equivalenceClassInfo if ecache != nil { // getEquivalenceClassInfo will return immediately if no equivalence pod found equivCacheInfo = ecache.getEquivalenceClassInfo(pod) } </ins> checkNode := func(i int) { nodeName := nodes[i].Name <del> fits, failedPredicates, err := podFitsOnNode(pod, meta, nodeNameToInfo[nodeName], predicateFuncs, ecache, schedulingQueue, alwaysCheckAllPredicates) </del> <ins> fits, failedPredicates, err := podFitsOnNode( pod, meta, nodeNameToInfo[nodeName], predicateFuncs, ecache, schedulingQueue, alwaysCheckAllPredicates, equivCacheInfo, ) </ins> if err != nil { predicateResultLock.Lock() errs[err.Error()]++

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: } return "", err } <del> func formatIfNotFormatted(disk string, fstype string, exec mount.Exec) { notFormatted, err := diskLooksUnformatted(disk, exec) if err == nil && notFormatted { args := []string{disk} // Disk is unformatted so format it. // Use 'ext4' as the default if len(fstype) == 0 { fstype = "ext4" } if fstype == "ext4" || fstype == "ext3" { args = []string{"-E", "lazy_itable_init=0,lazy_journal_init=0", "-F", disk} } glog.Infof("azureDisk - Disk %q appears to be unformatted, attempting to format as type: %q with options: %v", disk, fstype, args) _, err := exec.Run("mkfs."+fstype, args...) if err == nil { // the disk has been formatted successfully try to mount it again. glog.Infof("azureDisk - Disk successfully formatted with 'mkfs.%s %v'", fstype, args) } else { glog.Warningf("azureDisk - Error formatting volume with 'mkfs.%s %v': %v", fstype, args, err) } } else { if err != nil { glog.Warningf("azureDisk - Failed to check if the disk %s formatted with error %s, will attach anyway", disk, err) } else { glog.Infof("azureDisk - Disk %s already formatted, will not format", disk) } } } func diskLooksUnformatted(disk string, exec mount.Exec) (bool, error) { args := []string{"-nd", "-o", "FSTYPE", disk} glog.V(4).Infof("Attempting to determine if disk %q is formatted using lsblk with args: (%v)", disk, args) dataOut, err := exec.Run("lsblk", args...) if err != nil { glog.Errorf("Could not determine if disk %q is formatted (%v)", disk, err) return false, err } output := libstrings.TrimSpace(string(dataOut)) return output == "", nil } </del>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When use an existing azure disk(also called ) in pod, if that disk has partitions, the disk will be formatted in the pod mounting. What you expected to happen: How to reproduce it (as minimally and precisely as possible): create a disk with source from an os disk, e.g. that disk is already formatted before pod could access it Anything else we need to know?: Root cause: current diskLooksUnformatted func is using command to check whether disk is formatted or not while common k8s code() is using to detect disk format, could detect multiple partitions so if we use common k8s code(), following error will be returned for mulitple partition disks, which is expected: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
cc
iirc we don't support partitions on Azure volumes. There was a specific hang long ago. in all cases i think should be safe to based check now via the underlaying api
thanks for the info, I am writing this PR, and also testing carefully. And this issue depends on , could you review it first? Thanks.


Question: } } <ins> func TestTimeout(t *testing.T) { sendResponse := make(chan struct{}, 1) writeErrors := make(chan error, 1) timeout := make(chan time.Time, 1) resp := "test response" timeoutResp := "test timeout" ts := httptest.NewServer(TimeoutHandler(http.HandlerFunc( func(w http.ResponseWriter, r *http.Request) { <-sendResponse _, err := w.Write([]byte(resp)) writeErrors <- err }), func(*http.Request) (<-chan time.Time, string) { return timeout, timeoutResp })) defer ts.Close() // No timeouts sendResponse <- struct{}{} res, err := http.Get(ts.URL) if err != nil { t.Error(err) } if res.StatusCode != http.StatusOK { t.Errorf("got res.StatusCode %d; expected %d", res.StatusCode, http.StatusOK) } body, _ := ioutil.ReadAll(res.Body) if string(body) != resp { t.Errorf("got body %q; expected %q", string(body), resp) } if err := <-writeErrors; err != nil { t.Errorf("got unexpected Write error on first request: %v", err) } // Times out timeout <- time.Time{} res, err = http.Get(ts.URL) if err != nil { t.Error(err) } if res.StatusCode != http.StatusGatewayTimeout { t.Errorf("got res.StatusCode %d; expected %d", res.StatusCode, http.StatusServiceUnavailable) } body, _ = ioutil.ReadAll(res.Body) if string(body) != timeoutResp { t.Errorf("got body %q; expected %q", string(body), timeoutResp) } // Now try to send a response sendResponse <- struct{}{} if err := <-writeErrors; err != http.ErrHandlerTimeout { t.Errorf("got Write error of %v; expected %v", err, http.ErrHandlerTimeout) } } </ins> func TestGetAPIRequestInfo(t *testing.T) { successCases := []struct { method              string

I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.

I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.


Question: - --memory={{base_eventer_memory}} - --extra-memory={{eventer_memory_per_node}}Ki - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=eventer - --poll-period=300000 - --estimator=exponential

I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue?

This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Question: ensure-install-dir set-kube-env create-salt-pillar <ins> download-release </ins> run-salt echo "== kube-push node config done ==" fi

I made several attempts to update the apiserver (i.e. on the master) using and but they failed to update the binary. I noticed the same PID on the master for and a long uptime. If I tore the cluster down and rebuilt it then I could see the change I made to the apiserver.
I'm actively working on it.
I noticed exact the same issue today by update my kubelet binary.
Push is broken, sorry. I thought I tested it, but the binaries aren't getting bonked appropriately.
kubelet binary push actually worked before? That path looked dubious at best. On Mar 12, 2015 3:56 PM, "Dawn Chen" wrote:
I would appreciate external testing. I have done some sniff testing, but push is a really broad use-case. , Satnam Singh wrote:
reopen this since the fix is reversed.

On GCE, when I run to push new binaries to e2e cluster, the binaries are not pushed.
I lost a lot of time due to this bug today. I was debugging failing e2e tests, and I was using this command often. I took me some time before I realized it doesn't work and I need to do each time.
Can you be specific? I verified apiserver push at least, and I manually inspected kubelet, but it's fidgety to test push. On Mar 12, 2015 8:16 AM, "Satnam Singh" wrote:
The scenario: compiled the old version and installed it with -v -build -up; run some e2e tests, cluster with the old version was running; changed source code of some files in pkg/apiserver or pkg/registry/pod; compiled the new version and was trying to install it with -v -build -push; run some e2e tests, hoping the new version of apiserver was deployed. However, the old version was still running during step 5. It seemed that step 4. compiled the new version, but didn't deploy it. To deploy the new version I needed to run -up.
I did tests to verify that I was no longer seeing version mismatches after a push (which is a good sign that a push succeeded in bonking the apiserver), but I'll see if I can do more rigorous testing soon. To be honest, the master push hasn't changed much - the node deployment was the part that changed much more radically with the Salt changes. So I'm not exactly sure what you're seeing here. , Jerzy Szczepkowski < :


Question: // ControllerRevisionNamespaceLister. type ControllerRevisionNamespaceListerExpansion interface{} <del> // DaemonSetListerExpansion allows custom methods to be added to // DaemonSetLister. type DaemonSetListerExpansion interface{} // DaemonSetNamespaceListerExpansion allows custom methods to be added to // DaemonSetNamespaceLister. type DaemonSetNamespaceListerExpansion interface{} // DeploymentListerExpansion allows custom methods to be added to // DeploymentLister. type DeploymentListerExpansion interface{} // DeploymentNamespaceListerExpansion allows custom methods to be added to // DeploymentNamespaceLister. type DeploymentNamespaceListerExpansion interface{} // ReplicaSetListerExpansion allows custom methods to be added to // ReplicaSetLister. type ReplicaSetListerExpansion interface{} // ReplicaSetNamespaceListerExpansion allows custom methods to be added to // ReplicaSetNamespaceLister. type ReplicaSetNamespaceListerExpansion interface{} </del> // ScaleListerExpansion allows custom methods to be added to // ScaleLister. type ScaleListerExpansion interface{}

As a prerequisite of moving the , , , and controllers to use the apps/v1beta2 APIs, we need to add list expansions for , , , and to .
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps


Question: cache.ResourceEventHandlerFuncs{ // MaxPDVolumeCountPredicate: since it relies on the counts of PV. AddFunc:    c.onPvAdd, <ins> UpdateFunc: c.onPvUpdate, </ins> DeleteFunc: c.onPvDelete, }, )

What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate).

is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.


Question: # KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,experimental/v1alpha1"} KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,experimental/v1alpha1"} <ins> # Give integration tests longer to run KUBE_TIMEOUT=${KUBE_TIMEOUT:--timeout 240s} </ins> KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY:-"-1"} LOG_LEVEL=${LOG_LEVEL:-2}

The first line logged from the integration test is the last line from the service account token test is then the test panics with: I think the tests are just running too close the the timeout. On successful runs, the tests take ~115 seconds, which is ridiculously close to the threshold. For example: see
ok, do you want to send a PR to increase the timeout to 3 or 4 minutes?
Sure, I can do that tonight

cc
Has this test failed more than once, or did the two-minute timeout just happen to occur in the middle of it?
I think 2 minutes is not sufficient for the integration tests. See comment at


Question: Message: "CustomResource deletion is in progress", }) crd, err = c.crdClient.CustomResourceDefinitions().UpdateStatus(crd) <ins> if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } </ins> if err != nil { return err }

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: } __EOF__ <ins> # Ensure the API server has recognized and started serving the associated CR API local tries=5 for i in $(seq 1 $tries); do local output output=$(kubectl "${kube_flags[@]:?}" api-resources --api-group mygroup.example.com -oname) if kube::test::if_has_string "$output" resources.mygroup.example.com; then break fi echo "${i}: Waiting for CR API to be available" sleep "$i" done </ins> # Dry-run create the CR kubectl "${kube_flags[@]:?}" apply --dry-run=server -f hack/testdata/CRD/resource.yaml "${kube_flags[@]:?}" # Make sure that the CR doesn't exist

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig cli /cc /milestone v1.20 /prority important-soon
after creating the CRD, before applying the CR, kubectl should wait until includes :
/assign
/assign is this issue related to ?
no, this is simply kubectl trying to apply a CR before the server has made the API available at all
Got it! and I have filed to address this issue.
Still happening:
still happening: looks like the root cause is not related to CRD: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: /reopen
Reopened this issue. <detailsIn response to : still happening: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, the actual error is later: Line 177 succeeds, and the next line fails:


Question: apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v17 </del> <ins> name: kube-dns-v17.1 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: replicas: __PILLAR__DNS__REPLICAS__ selector: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: containers:

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features" </ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"

The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm


Question: klog.ErrorS(err, "Error listing addresses binded to dummy interface") } // nodeAddressSet All addresses *except* those on the dummy interface <del> nodeAddressSet, err := proxier.netlinkHandle.GetAllLocalAddresses() </del> <ins> nodeAddressSet, err := proxier.netlinkHandle.GetAllLocalAddressesExcept(defaultDummyDevice) </ins> if err != nil { klog.ErrorS(err, "Error listing node addresses") } <del> nodeAddressSet = nodeAddressSet.Difference(alreadyBoundAddrs) </del> hasNodePort := false for _, svc := range proxier.svcPortMap {

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: name = "cri_tools_%s" % arch, downloaded_file_path = "cri_tools.tgz", sha256 = sha, <del> urls = mirror("https://github.com/kubernetes-incubator/cri-tools/releases/download/v%s/crictl-v%s-linux-%s.tar.gz" % (CRI_TOOLS_VERSION, CRI_TOOLS_VERSION, arch)), </del> <ins> urls = mirror("https://github.com/kubernetes-incubator/cri-tools/releases/download/v%s/crictl-v%s-%s.tar.gz" % (CRI_TOOLS_VERSION, CRI_TOOLS_VERSION, arch)), </ins> ) # Use go get -u github.com/estesp/manifest-tool to find these values

Right now cri-tools only include tarballs for linux /cc /cc
/sig release
/area release-eng /milestone v1.16 /help /priority important-soon
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /area release-eng /milestone v1.16 /help /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello! I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 2 days from now). Do we still target this issue to be fixed for 1.16?
No, nobody is working on this for 1.16. cc
/milestone clear
anything we can do in release team to move this forward?
I'm working on this one now if nobody else minds. :) /assign


Question: Status: api.ConditionFalse, }, }, <ins> { name: "matching pdbs with no disruptions allowed, pod running, pod healthy, empty selector, pod not deleted by honoring the PDB", pdbs: []runtime.Object{&policyv1.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policyv1.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{}}, Status: policyv1.PodDisruptionBudgetStatus{ DisruptionsAllowed: 0, CurrentHealthy:     3, DesiredHealthy:     3, }, }}, eviction:            &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "t11", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectError:         "Cannot evict pod as it would violate the pod's disruption budget.: TooManyRequests: The disruption budget foo needs 3 healthy pods and has 3 currently", podName:             "t11", expectedDeleteCount: 0, podTerminating:      false, podPhase:            api.PodRunning, prc: &api.PodCondition{ Type:   api.PodReady, Status: api.ConditionTrue, }, }, </ins> } for _, unhealthyPodEvictionPolicy := range []*policyv1.UnhealthyPodEvictionPolicyType{unhealthyPolicyPtr(policyv1.AlwaysAllow), nil, unhealthyPolicyPtr(policyv1.IfHealthyBudget)} {

with an empty selector selects all the pods as expected. But, during the /, the pdb is not being honored correctly i.e. all the pods are getting evicted. This issue is only observed with the empty selector and works as expected if a valid label selector is provided that selects all the pods in a namespace. When an empty selector is provided in the 's spec, the created needs to be honored by not evicting all the pods during the api call Create a simple busybox deployment with 3 replicas on a local cluster (or any cluster)Create a pdb object with an selectorCheck the status of the created pdb object that has the observed to be 3 (which is expected)Call the api to see the pdb not being honored and all the pods being evictedDelete the above pdb object and create a new one with the label selectorObserve the expected status of the pdb object same as the old one with the set to 3Call the api to observe the pdb being honored as expected No response <details</details<detailslocal cluster - N/A </details<details</details<details</details<detailsCRI-O </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: }, "namespaceSelector": { "$ref": "v1.LabelSelector", <del> "description": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is other cluster scoped resource, it is not subjected to the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything." </del> <ins> "description": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything." </ins> } } },

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: "strings" "time" <ins> "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/api/core/v1" </del> "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/intstr"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: dns test failure when using kube-dns with dnsmasq 2.79 dnsmasq 2.79 introduced a change to respond to all norecurse queries with ServFail. This is to prevent cache snooping where an adversary can figure out if a particular hostname has been looked up or not. We need to change the test to drop that flag, likely the intention was to not send the query upstream(outside of kubedns), but the flag will restrict the query to just dnsmasq. One instance of the test failure: https://k8s- What you expected to happen: The test should be fixed to drop the no recursion flag. How to reproduce it (as minimally and precisely as possible): Run kube-dns with dnsmasq 2.79. dig +norecurse <any hostnameAnything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig networking
/sig network
/sig network


Question: for i := 0; i < active; i++ { go func(ix int) { defer wait.Done() <del> if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name); err != nil { </del> <ins> if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name, &job); err != nil { </ins> defer util.HandleError(err) } }(i)

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc