Question: config.Plugin{Name: pl.name}) } profile := config.KubeSchedulerProfile{Plugins: cfgPls} <del> f, err := newFrameworkWithQueueSortAndBind(registry, profile, WithRunAllFilters(tt.runAllFilters)) </del> <ins> f, err := newFrameworkWithQueueSortAndBind(registry, profile) </ins> if err != nil { t.Fatalf("fail to create framework: %s", err) }

Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers.

For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


Question: timeoutPermit       bool waitAndRejectPermit bool waitAndAllowPermit  bool <del> allowPermit         bool </del> cancelled           bool <ins> waitingPod          string rejectingPod        string allowingPod         string </ins> fh                  framework.FrameworkHandle }

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: for i := 0; i < workers; i++ { go util.Until(e.worker, time.Second, stopCh) } <ins> go func() { defer util.HandleCrash() time.Sleep(5 * time.Minute) // give time for our cache to fill e.checkLeftoverEndpoints() }() </ins> <-stopCh e.queue.ShutDown() }

So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.


Question: kc.ImageGCPolicy, kc.DiskSpacePolicy, kc.Cloud, <ins> kc.AutoDetectCloudProvider, </ins> kc.NodeLabels, kc.NodeStatusUpdateFrequency, kc.OSInterface,

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: "k8s.io/api/core/v1" apimachineryconfig "k8s.io/apimachinery/pkg/apis/config" <ins> v1meta "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/serializer" "k8s.io/apimachinery/pkg/runtime/serializer/json"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Implement the feature described here: Why is this needed: In a cluster that has a service mesh a lot of the work being done by kube-proxy is redundant and wasted. Specifically, services that are only reached via other services in the mesh will never use the service abstraction implemented by kube-proxy in iptables (or ipvs). By informing the kube-proxy of this, we can lighten the work it is doing and the burden on its proxy backend. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig network


Question: <ins> // +build !providerless </ins> /* Copyright 2016 The Kubernetes Authors.

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: printLabelsMultiline(w, "Labels", namespace.Labels) printAnnotationsMultiline(w, "Annotations", namespace.Annotations) w.Write(LEVEL_0, "Status:t%sn", string(namespace.Status.Phase)) <ins> if len(namespace.Status.Conditions) > 0 { w.Write(LEVEL_0, "Conditions:n") w.Write(LEVEL_1, "TypetStatustLastTransitionTimetReasontMessagen") w.Write(LEVEL_1, "----t------t------------------t------t-------n") for _, c := range namespace.Status.Conditions { w.Write(LEVEL_1, "%vt%vt%st%vt%vn", c.Type, c.Status, c.LastTransitionTime.Time.Format(time.RFC1123Z), c.Reason, c.Message) } } </ins> if resourceQuotaList != nil { w.Write(LEVEL_0, "n") DescribeResourceQuotas(resourceQuotaList, w) } if limitRangeList != nil { w.Write(LEVEL_0, "n") DescribeLimitRanges(limitRangeList, w) } return nil }) }

should output Conditions. For example: Including Conditions in the output is useful. In this example, a Condition can explain why a namespace deletion has not completed. While the output below includes Conditions, I believe the output of would be easier to read.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign


Question: os.Unsetenv(pair[0]) } } <ins> t.Log("Saved environment: ", savedEnv) os.Setenv("HTTP_PROXY", "http://proxy.example.com:3128") os.Setenv("NO_PROXY", "example.com,10.0.0.0/8,2001:db8::/48") // Check if we can reliably execute tests: // ProxyFromEnvironment caches the *_proxy environment variables and // if ProxyFromEnvironment already executed before our test with empty // HTTP_PROXY it will make these tests return false positive failures req, err := http.NewRequest("GET", "http://host.fake.tld/", nil) if err != nil { t.Fatalf("unexpected err: %v", err) } proxy, err := http.ProxyFromEnvironment(req) if err != nil { t.Fatalf("unexpected err: %v", err) } if proxy == nil { t.Skip("test skipped as ProxyFromEnvironment already initialized in environment without defined HTTP proxy") } t.Log("http.ProxyFromEnvironment is usable, continue executing test") </ins> return savedEnv }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When setting an HTTPPROXY with an IPv6 address, the is no HTTPProxy warning. What you expected to happen: There should be a warning that a proxy is being used. How to reproduce it (as minimally and precisely as possible): Use Kubeadm with IPv6 addresses, and IPv6 proxy in the environment. Such as: NOPROXY=2001:420:2c50:2021::/64 Anything else we need to know?: Environment: Kubernetes version (use ): 9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/area ipv6


Question: "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" <ins> . "github.com/onsi/gomega" </ins> ) func setDesiredConfiguration(initialConfig *kubeletconfig.KubeletConfiguration) {

Node Allocatable setup test has been flaking recently. Example flake: https://k8s- /sig node cc
The kubelet logs look fine. After speaking with we are querying the node api object immediately after the kubelet configuration is updated. At this point, the kubelet may not have updated its node status, meaning we see the old node status rather than the new, correct one.


Question: definitions MUST be published for custom resource definitions. release: v1.16 file: test/e2e/apimachinery/crd_publish_openapi.go <ins> - testname: Event, delete a collection codename: '[sig-api-machinery] Events should delete a collection of events [Conformance]' description: A set of events is created with a label selector which MUST be found when listed. The set of events is deleted and MUST NOT show up when listed by its label selector. release: v1.19 file: test/e2e/apimachinery/events.go </ins> - testname: Event resource lifecycle codename: '[sig-api-machinery] Events should ensure that an event can be fetched, patched, deleted, and listed [Conformance]'

[x] Original approval issue: [x] Two weeks soak end date: 6 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Events with a static label within a Namespace. that all 3 Events with the label in the Namespaces are created. the set of Namespaced Events with a label via DeleteCollection. that all Events with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen PR Merge and soaking
Reopened this issue. <detailsIn response to : /reopen PR Merge and soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: service:           getResourceGroupTestService("service4", "random-rg", "1.2.3.4", 80), expectCreateError: true, }, <ins> { desc:              "annotated service with different resourceGroup shouldn't be created but should be deleted successfully", service:           getResourceGroupTestService("service5", "random-rg", "", 80), expectCreateError: true, wrongRGAtDelete:   true, }, </ins> } ctrl := gomock.NewController(t)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: }, tests: []controllerTest{ { <del> name:            "15-4 - pvc storage class is not assigned retroactively if claim is already bound but annotations are missing", </del> <ins> name:            "15-5 - pvc storage class is assigned retroactively if there is a default", </ins> initialVolumes:  novolumes, expectedVolumes: novolumes, <del> initialClaims:   newClaimArray("claim15-4", "uid15-4", "1Gi", "test", v1.ClaimBound, &classCopper), expectedClaims:  newClaimArray("claim15-4", "uid15-4", "1Gi", "test", v1.ClaimPending, &classCopper), </del> <ins> initialClaims:   newClaimArray("claim15-5", "uid15-5", "1Gi", "", v1.ClaimPending, nil), expectedClaims:  newClaimArray("claim15-5", "uid15-5", "1Gi", "", v1.ClaimPending, &classGold), </ins> expectedEvents:  noevents, errors:          noerrors, test:            testSyncClaim,

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: // Start the controller manager HTTP server if c.SecureServing != nil { unsecuredMux := genericcontrollermanager.NewBaseHandler(&c.ComponentConfig.Generic.Debugging, healthzHandler) <ins> if utilfeature.DefaultFeatureGate.Enabled(features.ComponentSLIs) { slis.SLIMetricsWithReset{}.Install(unsecuredMux) } </ins> handler := genericcontrollermanager.BuildHandlerChain(unsecuredMux, &c.Authorization, &c.Authentication) // TODO: handle stoppedCh and listenerStoppedCh returned by c.SecureServing.Serve if _, _, err := c.SecureServing.Serve(handler, 0, stopCh); err != nil {

As per, , wire up metrics/slis for ccm
/assign /sig instrumentation /sig cloud-provider
GitHub didn't allow me to assign the following users: Richabanker. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig instrumentation /sig cloud-provider Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
/remove-triage accepted
/triage accepted


Question: "strings" "time" <ins> "gopkg.in/gcfg.v1" </ins> "k8s.io/kubernetes/pkg/api" apiservice "k8s.io/kubernetes/pkg/api/service" "k8s.io/kubernetes/pkg/api/unversioned"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} What happened: Dynamic provisioned volumes in different zones with master failed to deleted What you expected to happen: Since the volume can be dynamic provisioned when the zones differs from master, then it should be deleted after delete the PVC which created them. How to reproduce it (as minimally and precisely as possible): up an all-in-one cluster. The instance is created in zone us-central1-a. several storageClass. One in zone us-central1-a. Another in zone us-central1-b. several PVC. pvc1 using storageClass1(zone us-central1-a), pvc2 using storageClass2(zone us-central1-b). PV/volume were provisioned in proper size/mode/zone. PVCs. with volume in zone us-central1-b failed with error can not find the volume.

Anything else do we need to know: Name: pvc--70d1-11e6-b17e- Labels: failure- failure- Status: Failed Claim: default/pvc5 Reclaim Policy: Delete Access Modes: RWO Capacity: 4Gi Message: Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" Source: Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine) PDName: kubernetes-dynamic-pvc--70d1-11e6-b17e- FSType: Partition: 0 ReadOnly: false Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message 11m 11m 1 {persistentvolume-controller } Warning VolumeFailedDelete Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" creationTimestamp: '2016-09-01T22:49:26.439-07:00' description: '{"":"pvc--70d1-11e6-b17e-","":"pvc5","":"default"}' id: '' kind: compute#disk name: kubernetes-dynamic-pvc--70d1-11e6-b17e- selfLink: sizeGb: '4' status: READY type: zone:
Kubernetes can delete GCE PDs only in zones it manages. I.e. either in the same zone as the master or, when multizone is enabled in GCE config, in all zones in the same region. This should not be allowed in the first place. Kubernetes should not create disks outside zones it manages and report an error instead. So... please use following , it will be allow creation/deletion of disks in all zones in us-central1 region: And I'll fix Kubernetes to report an error when creating a disk in all other zones.


Question: func getOperationName( volumeName v1.UniqueVolumeName, podName types.UniquePodName) string { podNameStr := "" <del> if podName != emptyUniquePodName { </del> <ins> if podName != EmptyUniquePodName { </ins> podNameStr = fmt.Sprintf(" (%q)", podName) }

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.


Question: gmsaWebhookDeployScriptURL = "https://raw.githubusercontent.com/kubernetes-sigs/windows-gmsa/master/admission-webhook/deploy/deploy-gmsa-webhook.sh" ) <del> var _ = SIGDescribe("[Feature:Windows] GMSA Full [Slow]", func() { </del> <ins> var _ = SIGDescribe("[Feature:Windows] GMSA Full [Serial] [Slow]", func() { </ins> f := framework.NewDefaultFramework("gmsa-full-test-windows") ginkgo.Describe("GMSA support", func() {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-aks-engine-azure-master-staging-windows Which test(s) are flaking: There is a race condition in the one of the tests and they are run in parallel so if varies, some of the recent ones are: Testgrid link: https://k8s- Reason for failure: gmsa webhooks is not responding: The gmsa test creates the webhook and then removes it when the test is complete: causing failures for any running tests Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig windows /assign
thanks for helping track this down
The current plan is to move the deployment of the webhook out of the individual test and make sure it is installed at cluster provisioning time. do you see any issues? It doesn't look like the gce tests run the tests so this should not affect the tests runs there but in the future if would need to be set up in test scripts. fyi -
hey yes, deploying the webhook at cluster deployment definitely works. It was scoped to the test to reduce blast radius when things weren't super stable.
One alternative to creating the webhook at provisioning time ( I still thing this is the cleaner version of the fix ), we can just mark GMSA tests [SERIAL], thus we won't have a problem with this race.
I looked into this some more and found we have three options: Install the webhook at provisioning time. This solved the problem above but causes the test to fail because the webhook looks for a gmsa CRD and . One option is to update the test to add a crd in the case the webhook is installed Update the gMSA webhook to allow requests that do not have the name set but do have the set. This would allow the "GMSA Kubelet" test to pass because it would not be rejected by the webhook. Mark the tests as which would move the test out of the parallel jobs and allow the test to still function. I am leaning towards the third option as the first option of modifying the "kubelet GMSA" test would be adding extra steps (installing crd and RBAC roles) that have nothing to do with the test.

The second option seems like it could be a fair change to the webhook but I don't know enough about the rest of the webhook logic and design and would be a change in behavior of the webhook. The third option is simpler but would require changes to the serial tests to enable the domain controller. The benefit would be that the tests that run in parallel wouldn't require domains controller to be set up which would speed up provision time and after I ran an experimental test run indicated that it might bring the test runs under 1hr (currently 1.4hrs). It would add some additional time to the serial tests which are already pretty slow (3+ hrs). do you have any thoughts?


Question: } func getFakeCSINodeLister(csiNode *storagev1.CSINode) fakeframework.CSINodeLister { <ins> csiNodeLister := fakeframework.CSINodeLister{} </ins> if csiNode != nil { <del> return fakeframework.CSINodeLister(*csiNode) </del> <ins> csiNodeLister = append(csiNodeLister, *csiNode.DeepCopy()) </ins> } <del> return fakeframework.CSINodeLister{} </del> <ins> return csiNodeLister </ins> } func getNodeWithPodAndVolumeLimits(limitSource string, pods []*v1.Pod, limit int64, driverNames ...string) (*framework.NodeInfo, *storagev1.CSINode) {

handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.


Question: return utilerrors.NewAggregate(errs) } <del> if strings.HasSuffix(gvk.Kind, "List") { </del> <ins> if (gvk == schema.GroupVersionKind{Version: "v1", Kind: "List"}) { </ins> return utilerrors.NewAggregate(v.validateList(obj)) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I've created a CRD with the name . There is no validation specified on the CRD object. Creating the CRD itself succeeds, but attempting to create CRD resources with of that type fails with a validation error. What you expected to happen: I expect to be able to create CRD resources of type without a validation error. How to reproduce it (as minimally and precisely as possible): A normal working CRD example, : Applying this succeeds as expected: Simply changing the name of the CRD kind to breaks however, : CRD creates, but the actual resource fails validation: If you supply the object does create correctly. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Digitalocean VM cluster OS (e.g. from /etc/os-release): Server (Ubuntu 14.04.5) Client (Ubuntu 16.04.4) Kernel (e.g. ): Server (4.4.0-111-generic) Client (4.13.0-38-generic) Install tools: Others:
/sig api-machinery /area custom-resources
is a special suffix for name. When you create a object using , which will be render as instead of plain . That's why error occurs due to missing item list.
This sounds like a bug in , not a thing that should be "fixed" by restricting names.
I think the magic suffix is somewhat unexpected, but in the case that it doesn't change -- I would expect a warning or error when creating a CRD with that name. As for this specific behavior, I would consider it a bug -- there's no indication that the kubectl validation is against a different than the one explicitly specified in the resource, and confusingly disabling validation causes the operation to succeed normally (creating the explicitly defined resource , regardless of magic suffix).
cc
cc
PTAL . It is a kubectl bug.
Thanks! Fix works for me


Question: } if test.statusCode != 0 { if resp.StatusCode != test.statusCode { <del> t.Errorf("%d: unexpected response: %#v", resp) </del> <ins> t.Errorf("%d: unexpected response: %#v", i, resp) </ins> } continue } if resp.StatusCode != http.StatusOK { <del> t.Errorf("%d: unexpected response: %#v", resp) </del> <ins> t.Errorf("%d: unexpected response: %#v", i, resp) </ins> } var itemOut metav1alpha1.Table <del> if _, err = extractBody(resp, &itemOut); err != nil { </del> <ins> body, err := extractBody(resp, &itemOut) if err != nil { </ins> t.Fatal(err) } if !reflect.DeepEqual(test.expected, &itemOut) { <ins> t.Log(body) </ins> t.Errorf("%d: did not match: %s", i, diff.ObjectReflectDiff(test.expected, &itemOut)) } }

Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.


Question: } function kube::release::gcs::stage_and_hash() { <ins> kube::build::ensure_tar </ins> # Split the args into srcs... and dst local -r args=( "$@" ) local -r split=$((${#args[@]}-1)) # Split point for src/dst args

Since I've been having troubles with installation, I've digged a bit and found out that the permissions inside Kubernetes tarballs changed between 0.16.0 and 0.17.0. I've downloaded both tarballs, extracted them into separate directories and then extracted the Kubernetes server tarballs. This is what I get: I suppose, there has been a concious decision to restrict the world's access to those binaries. However, this breaks the installation routine mentioned above and probably makes life a bit more difficult in several other places. I'd be thankful for a correction if this wasn't intended or a pointer to the appropriate commit it this is legit -- I'd correct the installation.
assigning this to you to decide who the eventual assignee should be :)
Any news on this front? 0.20.1 is still restricted.
Apologies for the issue. It wasn't intentional -- likely an artifact of build scripts being run by different people? I'll check out 0.20.2 as I build it to make sure it has the right perms.
Apologies, but 0.20.2 didn't include fixed permissions. I now understand the issue, : "make release" builds a tarball using the permissions on files in your local git tree. I had "umask 027" set when I cloned, so those in-the-repo files had no world-readable perms. Setting "umask 022" before running "make release" is not enough -- it has to be part of creating the local tree when cloning. I'll update instructions and the next release (20.3 or 21.0) should fix this issue. And longer term, we need to put a bit more effort into releasing from a known-good environment and not rely on a developer's workstation's idiosyncrasies -- sorry this bit you.
I can't repro Trying again a different way, but I don't think the umask before clone should do anything (in particular, umasks aren't "sticky" on files, they're sticky to processes.)
And let's pretend on my personal fork isn't woefully out of date (who uses master on their personal fork? I should really just delete that that thing.) Comment updated.
Ah, I see the problem now.

works fine for the server tar, but the full tar also packages examples, etc., and for that, tar tends to just pick up the permissions of the git tree. Got it.
I'll take this. I'm going to write a script to build official bits, and this is easy enough to fix along the way.
Seems not fixed as of Kubernetes 1.0.7
sorry about that. is doing the 1.1.1 release and has been improving release scripts -- is 1.1.1 good re: permissions?
was not built by .
Reassigning to as release-oncall. Quinton, I can mentor or advise as necessary.
And yeah, is fine: There's no real bug here. There was a process issue that occurred with 1.0.7. We can rebuild the binaries if we need to.
FYI, leaving this issue closed, as per 's comment above. If anyone disagrees, please re-open this issue.


Question: // Populate global DefaultFeatureGate with value from TestContext.FeatureGates. // This way, statically-linked components see the same feature gate config as the test context. utilconfig.DefaultFeatureGate.Set(framework.TestContext.FeatureGates) <del> e := newE2EService() </del> <ins> e := newE2EServices() </ins> if err := e.run(); err != nil { glog.Fatalf("Failed to run e2e services: %v", err) } } <del> // Ports of different e2e services. </del> const ( <del> kubeletPort         = "10250" kubeletReadOnlyPort = "10255" ) // Health check urls of different e2e services. var ( kubeletHealthCheckURL = getEndpoint(kubeletReadOnlyPort) + "/healthz" ) // getEndpoint generates endpoint url from service port. func getEndpoint(port string) string { return "http://127.0.0.1:" + port } func getHealthCheckURLs() []string { return []string{ getEtcdHealthCheckURL(), getAPIServerHealthCheckURL(), kubeletHealthCheckURL, } } // e2eService manages e2e services in current process. type e2eService struct { services []*server rmDirs   []string logFiles map[string]logFileData // statically linked e2e services etcdServer   *EtcdServer apiServer    *APIServer nsController *NamespaceController } // logFileData holds data about logfiles to fetch with a journalctl command or // symlink from a node's file system. type logFileData struct { files             []string journalctlCommand []string } const ( // This is consistent with the level used in a cluster e2e test. </del> <ins> // services.log is the combined log of all services servicesLogFile = "services.log" // LOG_VERBOSITY_LEVEL is consistent with the level used in a cluster e2e test. </ins> LOG_VERBOSITY_LEVEL = "4" ) <del> func newE2EService() *e2eService { // Special log files that need to be collected for additional debugging. var logFiles = map[string]logFileData{ "kern.log":       {[]string{"/var/log/kern.log"}, []string{"-k"}}, "docker.log":     {[]string{"/var/log/docker.log", "/var/log/upstart/docker.log"}, []string{"-u", "docker"}}, "cloud-init.log": {[]string{"/var/log/cloud-init.log"}, []string{"-u", "cloud*"}}, } return &e2eService{logFiles: logFiles} } // terminationSignals are signals that cause the program to exit in the // supported platforms (linux, darwin, windows). var terminationSignals = []os.Signal{syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT} // run starts all e2e services and wait for the termination signal. Once receives the // termination signal, it will stop the e2e services gracefully. func (es *e2eService) run() error { defer es.stop() if err := es.start(); err != nil { return err } // Wait until receiving a termination signal. sig := make(chan os.Signal, 1) signal.Notify(sig, terminationSignals...) <-sig return nil } // start starts the tests embedded services or returns an error. func (es *e2eService) start() error { glog.Info("Starting e2e services...") err := es.startEtcd() if err != nil { return err } err = es.startApiServer() if err != nil { return err } s, err := es.startKubelet() if err != nil { return err } es.services = append(es.services, s) err = es.startNamespaceController() if err != nil { return nil } glog.Info("E2E services started.") return nil } // getLogFiles gets logs of interest either via journalctl or by creating sym // links. Since we scp files from the remote directory, symlinks will be // treated as normal files and file contents will be copied over. func (es *e2eService) getLogFiles() { // Nothing to do if report dir is not specified. if framework.TestContext.ReportDir == "" { return } glog.Info("Fetching log files...") journaldFound := isJournaldAvailable() for targetFileName, logFileData := range es.logFiles { targetLink := path.Join(framework.TestContext.ReportDir, targetFileName) if journaldFound { // Skip log files that do not have an equivalent in journald-based machines. if len(logFileData.journalctlCommand) == 0 { continue } glog.Infof("Get log file %q with journalctl command %v.", targetFileName, logFileData.journalctlCommand) out, err := exec.Command("sudo", append([]string{"journalctl"}, logFileData.journalctlCommand...)...).CombinedOutput() if err != nil { glog.Errorf("failed to get %q from journald: %v, %v", targetFileName, string(out), err) } else { if err = ioutil.WriteFile(targetLink, out, 0644); err != nil { glog.Errorf("failed to write logs to %q: %v", targetLink, err) } } continue } for _, file := range logFileData.files { if _, err := os.Stat(file); err != nil { // Expected file not found on this distro. continue } if err := copyLogFile(file, targetLink); err != nil { glog.Error(err) } else { break } } } } // stop stops the embedded e2e services. func (es *e2eService) stop() { glog.Info("Stopping e2e services...") es.getLogFiles() // TODO(random-liu): Use a loop to stop all services after introducing // service interface. glog.Info("Stopping namespace controller") if es.nsController != nil { if err := es.nsController.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.nsController.Name(), err) } } glog.Info("Stopping API server") if es.apiServer != nil { if err := es.apiServer.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.apiServer.Name(), err) } } for _, s := range es.services { glog.Info("Stopping service %q", s.name) if err := s.kill(); err != nil { glog.Errorf("Failed to stop %v: %v", s.name, err) } } glog.Info("Stopping etcd") if es.etcdServer != nil { if err := es.etcdServer.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.etcdServer.Name(), err) } } for _, d := range es.rmDirs { glog.Info("Deleting directory %v", d) err := os.RemoveAll(d) if err != nil { glog.Errorf("Failed to delete directory %s.n%v", d, err) } } glog.Info("E2E services stopped.") } // startEtcd starts the embedded etcd instance or returns an error. func (es *e2eService) startEtcd() error { glog.Info("Starting etcd") dataDir, err := ioutil.TempDir("", "node-e2e") </del> <ins> // startInternalServices starts the internal services in a separate process. func (e *E2EServices) startInternalServices() (*server, error) { testBin, err := osext.Executable() </ins> if err != nil { <del> return err </del> <ins> return nil, fmt.Errorf("can't get current binary: %v", err) </ins> } <del> // Mark the dataDir as directories to remove. es.rmDirs = append(es.rmDirs, dataDir) es.etcdServer = NewEtcd(dataDir) return es.etcdServer.Start() } // startApiServer starts the embedded API server or returns an error. func (es *e2eService) startApiServer() error { glog.Info("Starting API server") es.apiServer = NewAPIServer() return es.apiServer.Start() </del> <ins> startCmd := exec.Command("sudo", testBin, // TODO(mtaufen): Flags e.g. that target the TestContext need to be manually forwarded to the //                test binary when we start it up in run-services mode. This is not ideal. //                Very unintuitive because it prevents any flags NOT manually forwarded here //                from being set via TEST_ARGS when running tests from the command line. "--run-services-mode", "--server-start-timeout", serverStartTimeout.String(), "--feature-gates", framework.TestContext.FeatureGates, "--logtostderr", "--vmodule=*="+LOG_VERBOSITY_LEVEL, ) server := newServer("services", startCmd, nil, nil, getServicesHealthCheckURLs(), servicesLogFile, e.monitorParent, false) return server, server.start() </ins> } <del> // startNamespaceController starts the embedded namespace controller or returns an error. func (es *e2eService) startNamespaceController() error { glog.Info("Starting namespace controller") es.nsController = NewNamespaceController() return es.nsController.Start() } </del> <ins> const ( // Ports of different e2e services. kubeletPort         = "10250" kubeletReadOnlyPort = "10255" // Health check url of kubelet kubeletHealthCheckURL = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" ) </ins> // startKubelet starts the Kubelet in a separate process or returns an error // if the Kubelet fails to start. <del> func (es *e2eService) startKubelet() (*server, error) { </del> <ins> func (e *E2EServices) startKubelet() (*server, error) { </ins> glog.Info("Starting kubelet") var killCommand, restartCommand *exec.Cmd cmdArgs := []string{}

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: "testing" "time" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"

Kubernetes version (use ): Kubernetes HEAD Environment: Cloud provider or hardware configuration: linux OS (e.g. from /etc/os-release): Kernel (e.g. ):NAME=Fedora25 What happened: apiserver panic What you expected to happen: No panic How to reproduce it (as minimally and precisely as possible): Testcase is going to be sent as a PR
Fixed by


Question: // RegistryList holds public and private image registries type RegistryList struct { GcAuthenticatedRegistry string `yaml:"gcAuthenticatedRegistry"` <del> DockerLibraryRegistry   string `yaml:"dockerLibraryRegistry"` DockerGluster           string `yaml:"dockerGluster"` </del> E2eRegistry             string `yaml:"e2eRegistry"` E2eVolumeRegistry       string `yaml:"e2eVolumeRegistry"` PromoterE2eRegistry     string `yaml:"promoterE2eRegistry"`

Part of umbrella to migrate kubernetes e2e test images/registries to community-owned infrastructure: The following images are used by 1.20 e2e tests We should migrate them to community owned infrastructure (ref: for deciding how to implement a mirror registry) We should consider doing this for release branches as well, which may use different images/tags. So these wouldn't necessarily be cherry-picks Two possible approaches: change the registry in source (I would prefer eliminating the extra "docker library" registry, and consolidate to a single e2e test image registry) start running tests with a config that swaps out for some other registry /sig testing /sig release /area release-eng /wg k8s-infra /milestone v1.21
/kind cleanup
/triage accepted /help cc: Give a shout if any of you are interested in working on this!
I'm in! /assign
/remove-help
cc
/cc
I've had 2 PRs opened for this, before it was decited that the test jobs would use the registry mirror configuration. Has there been any consensus on the approach? If so, I can un-abandon these PRs if necessary:
ooh thanks for sharing I don't think there has been any consensus, though I personally think your work on kubernetes/kubernetes is relevant, especially that you've covered everything except gluster and redis.
We already have the redis image here [1], we'll just have to trigger a job for it afterwards. Indeed, I was missing the gluster image, I only considered the image for it.
so I've been thinking about this for a bit and I'm a bit concerned with the visibility of this setup to test authors -- I'm not sure if this is straightforward / intuitive enough for them to follow. alternatively, are there other lighter (in terms of maintenance) solutions that we can pursue?
Hm, what do you mean by the visibility of this setup of test authors? If I'm not mistaken, the current goal is to not rely so heavily on dockerhub for testing anymore. For this purpose we are mirroring these images into our own registries, and then we are going to update to use our own registry instead of the dockerhub one. Test authors shouldn't be impacted by this, IMO. From their point of view, it's the same image, from a different source.

And in terms of maintenance... hm, techically, if someone would want to use an newer image, say , so it would be consistent.
yes, I think we're on the same page on this yeah what I'm wondering is: are there ways we make it even less significant work on the first step of modifying and files? if that step could disappear altogether, especially in scenario of adding new images, that would be great.
I don't really see how. Other images are doing something similar too. For example, the pause image also has its BASEIMAGEs and VERSION in its Makefile. Not all images are built exactly the same, or have the same base image, or are built for the same arch types. Some of them are from scratch, others are from an alpine image, others are from a different alpine image, others are from a debian image... and then we also have windows base images. So, having those BASEIMAGE files addresses this issue. Plus, it also allows us to control what platforms we build for and what we test for. For example, the s390x support wasn't there from the start, it was afterwards. Another example, we want to add the base images at some point. Anyways, modifying the BASEIMAGE files is not very common, but when it is, there's a good reason behind it, so it can't really be optional. Plus, adding / modifying the BASEIMAGE and VERSION files is not significant work. For example, mirroring the 7 images in this PR [1] probably took me ~10 minutes or so. From what I can tell, you want some fast way to just say: "mirror me this image entirely onto our registry". I don't know if we have something like that, and if we do, it'll probably also mirror images with arch types we don't need, and most likely won't have images for platforms we actually need (for example, the ` images on dockerhub does not have any Windows images, but we need them). The test images building process for the test images was not designed for mirroring images, but it can easily do it and we can use it to do so. In addition to that, we can even extend those manifest lists with Windows support [2]. [1] [2]
oh right, apologies for the confusion.

I was referring to having a Dockerfile as a whole, but your explanation is very helpful for me to gain more context about image usage in our infrastructure -- thank you for elaborating I agree the what isn't significant, but the where worries me a bit. I guess in some ways I kind of wish that there is way for this work to simply exist in without touching k/k proper, though now I'm not too sure if that would be overall less complex than this approach. yeah, I was trying to find more solutions towards this direction though I haven't found anything significant which aligns with this topic.
/priority important-soon
In order to complete this, we are going to need these 2 trivial PRs: Retrigger image builder jobs for some images: Small fix for Windows httpd images:
seems like much of the work has been done by Claudiu or I do not have access to finish -- I'll unassign myself to keep this aligned with the state of things /unassign
/assign who has done a lot of the work for this


Question: t.Fatalf("Incorrect number of instances returned") } <del> if instances[0] != "kubernetes-minion-1" { </del> <ins> // no DNS in vagrant cluster, so we return IP as hostname expectedInstanceHost := "10.245.2.2" expectedInstanceIP := "10.245.2.2" if instances[0] != expectedInstanceHost { </ins> t.Fatalf("Invalid instance returned") }

In the guestbook example, the redis slaves are instructed to connect to the redis-master using the proxy provided by the minion on which they reside. The minion is designated by its hostname "kubernetes-minion-N". The container cannot resolve this name to an IP address and so slave connections fail. 1) create vagrant cluster 2) create redis-master pod 3) create redis-master service 4) create redis-slave replicationController Verify redis-slave pod locations: list pods ID Image(s) Host Labels Status redis-master-2 dockerfile/redis kubernetes-minion-1/10.245.2.2 name=redis-master Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-1/10.245.2.2 name=redisslave,replicationController=redisSlaveController Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-3/10.245.2.4 name=redisslave,replicationController=redisSlaveController Running Log onto a minion containing a slave (note: vagrant hostnames are minion-N, not kubernetes-minion-N) Find the slave container docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES brendanburns/redis-slave:latest /bin/sh -c About an hour ago Up About an hour k8s--slave.---3839-11e4-a922- docker logs | tail [7] 09 Sep 17:27:08.428  Connecting to MASTER kubernetes-minion-1: [7] 09 Sep 17:27:08.430 # Unable to connect to MASTER: No such file or directory [7] 09 Sep 17:27:09.435  Connecting to MASTER kubernetes-minion-1: sudo nsenter -m -u -n -i -p -t $(docker inspect --format '{{ .State.Pid }}' ) /bin/bash [ root ]$ ping kubernetes-minion-1 ping: unknown host kubernetes-minion-1 [ root ]$ cat /etc/hosts 244.1.4 -3839-11e4-a922- 127.0.0.

1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters [ root ]$ cat nameserver 10.0.2.3 cat /proc/1/environ HOME=/rootPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=-3839-11e4-a922-0800279696e1REDISMASTERSERVICEPORT=10000REDISMASTERPORT=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCP=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCPPROTO=tcpREDISMASTERPORT6379TCPPORT=10000REDISMASTERPORT6379TCPADDR=kubernetes-minion-1SERVICE_HOST=kubernetes-
Tim - are you the right guy to handle this? I think we need to resolve the name of the service host and only give IPs to the minion in env variables. We can't assume the containers have the same DNS set up as the host system.
To be clear -- this example is for the checked in Vagrant stuff? If that is the case would be the guy? I'm trying to catch up on IRC discussions from earlier today.
I can look at this, but there are two possible solutions. The first solution is in the vagrant cluster (or any other local vm setup) we could run a dns name server on the kubernetes-master that can resolve each minion by host name. We would then pass a -dns option to the docker daemon to specify that name server to use in each container. The second solution is that we pass IP address instead of host name and make no assumption or override on the containers specified name server. Is there any obvious negative to this solution? Has much thought been done on the expected Nameserver used by containers spawned by our service? - if you can weigh in, I would be happy to resolve using either approach. My preference is solution two.
So, the vagrant case is a little special as there is no sane DNS set up by default. Another suggestion (3) would be to revert the stuff and just devolve to using ip addresses directly. I would do (3) before (2) before (1).

But I'm cool with any of them. As for the nameserver of the containers that are spawned -- docker currently just takes the of the host and copies that into the container. It doesn't do that for . that is the core of the issue here. If/when we do offer some level of split horizon DNS (I'd love to) we can start exposing an extra layer of DNS on top of whatever is configured for the host. That is part of the "ip per service portal" stuff that is being talked about a lot right now.
If I do (3) there is a piece of code in pod storage that mangles my hostname where I return 10.245.x.x and it asks the cloud provider to resolve the IP address for "10". Since other providers appear to also be returning IP as host, I will look to change this so kubernetes makes no modifications to hostnames that it's given. Look for PR tomorrow.
you're right I'd missed that point. This really only applies to the Vagrant cluster and so a fix to the vagrant setup is sufficient. I also agree with your priorities. Injecting a line into /etc/hosts (if that's reasonable) is the simplest solution. Either reverting or allowing a switch to pass the minion public IP rather than hostname is next. Adding a DNS service to the vagrant master and forcing DNS resolution to the master host is most complicated and least preferable. Thanks both of you for helping me think this through better. Mark
I don't grok what the difference between 2 and 3 is? I sent a PR to docker a while back () to insert arbitrary lines to /etc/hosts. I still think it is a reasonable thing to want to do.
- difference between (2) and (3) is that in the past, the vagrant cloud provider was returning the IP address as the literal hostname for the minion. I had changed this after noticing that it had some ugly side-effects in the CLI, where a call to list pods would report pod ip addresses as IP / nil.

The issue (which I now believe is a bug that I want to change independent of what we do here) is that the code at this line mangles the hostname returned by the cloudprovider: This makes using the IP address as a hostname (10.245.2.2) to have the ugly side-effect of the code then asking the cloudprovider for the IP address of hostname "10", and then the cloudprovider having no idea how to proceed. Independent of what is done in this issue, I want to submit a PR tomorrow to stop that code from manipulating the reported hostname that was provided by the cloudprovider before asking that same cloudprovider to return the IP address using a potentially different input value. so with that backlog (3) means revert vagrant cloudprovider to use IPAddress as hostname, which means when we pass in what we believe is hostname in this environment, it will just work because it is in-fact the IPAddress. (2) means do not pass in hostname but pass in IPAddress by changing more core k8s code. all that being said, there is an option (4) that I am intrigued by, - after following your PR through its final conclusion, it looks like as of docker 1.2.0 you can change the /etc/hosts file of a running container. this introduces an option where the kubelet injects the /etc/hosts file of minion into the running container so it has the /etc/hosts values of the minion. I would need to play with that support a little more to know pros/cons more. I think longer term, we need a plan on what nameserver our containers are expecting to reference, and if they are inherently different in some way. I have run into a number of times the issue where doing things like running docker in docker causes the nested container to default to the nameserver 8.8.8.8, which happens to be blocked on the Red Hat network ;-) For now to unblock I will probably pursue (3) and fix the hostname bug referenced here at the same time. It does make the CLI uglier when working with this environment, but not a huge deal, at least it will function.
Thanks both for the complete explanation of what's happening and for the suggested fixes.

I think in the long run using hostnames in SERVICEHOST and ensuriing either through IP injection into container /etc/hosts or by providing DNS are the best options. I'm not sure if there might still be cases where resolution of SERVICEHOST would still be best done by presenting the IP address or not.


Question: ginkgo.By("by waiting for hugepages resource to become available on the local node") gomega.Eventually(func() string { return pollResourceAsString(f, "hugepages-2Mi") <del> }, 30*time.Second, framework.Poll).Should(gomega.Equal("100Mi")) </del> <ins> }, 30*time.Second, framework.Poll).Should(gomega.Equal("10Mi")) </ins> }) runHugePagesTests(f)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: -tests Since when has it been failing: Unsure Testgrid link: Reason for failure: Unable to pre allocate 50x2MiB huge pages, probably due to memory pressure caused by kernel level caching. If requesting an substantial amount of huge page memory (like the tests), it is recommended to to it as early after boot as possible, or by using the kernel cmd line. Eg. this one: Anything else we need to know: will fix this issue, but in awaiting for that, we can just decrease the page count in the tests.
/sig node /cc derekwaynecarr
I think the reason of railure should be because a single hugepage of x86 is 2MiB as and a single page is 4KiB, not 2KiB.
Yes, is correct. Thanks for catching the typo :smile:
Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. This is probably because the kernel is caching a lot of stuff, making it hard for the kernel to fix it. Ill do some testing with dropping the caches (no user facing stuff will happen, only kernel level) before trying to pre allocate huge pages. /reopen
Reopened this issue. <detailsIn response to : Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }() }, 240.0) <ins> It("should be able to create a functioning external load balancer", func() { serviceName := "external-lb-test" ns := api.NamespaceDefault labels := map[string]string{ "key0": "value0", } service := &api.Service{ ObjectMeta: api.ObjectMeta{ Name: serviceName, }, Spec: api.ServiceSpec{ Port:                       80, Selector:                   labels, TargetPort:                 util.NewIntOrStringFromInt(80), CreateExternalLoadBalancer: true, }, } By("cleaning up previous service " + serviceName + " from namespace " + ns) c.Services(ns).Delete(serviceName) By("creating service " + serviceName + " with external load balancer in namespace " + ns) result, err := c.Services(ns).Create(service) Expect(err).NotTo(HaveOccurred()) defer func(ns, serviceName string) { // clean up when we're done By("deleting service " + serviceName + " in namespace " + ns) err := c.Services(ns).Delete(serviceName) Expect(err).NotTo(HaveOccurred()) }(ns, serviceName) if len(result.Spec.PublicIPs) != 1 { Failf("got unexpected number (%d) of public IPs for externally load balanced service: %v", result.Spec.PublicIPs, result) } ip := result.Spec.PublicIPs[0] port := result.Spec.Port pod := &api.Pod{ TypeMeta: api.TypeMeta{ Kind:       "Pod", APIVersion: "v1beta1", }, ObjectMeta: api.ObjectMeta{ Name:   "elb-test-" + string(util.NewUUID()), Labels: labels, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "webserver", Image: "kubernetes/test-webserver", }, }, }, } By("creating pod to be part of service " + serviceName) podClient := c.Pods(api.NamespaceDefault) defer func() { By("deleting pod " + pod.Name) defer GinkgoRecover() podClient.Delete(pod.Name) }() if _, err := podClient.Create(pod); err != nil { Failf("Failed to create pod %s: %v", pod.Name, err) } expectNoError(waitForPodRunning(c, pod.Name)) By("hitting the pod through the service's external load balancer") var resp *http.Response for t := time.Now(); time.Since(t) < 4*time.Minute; time.Sleep(5 * time.Second) { resp, err = http.Get(fmt.Sprintf("http://%s:%d", ip, port)) if err == nil { break } } Expect(err).NotTo(HaveOccurred()) defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) Expect(err).NotTo(HaveOccurred()) if resp.StatusCode != 200 { Failf("received non-success return status %q trying to access pod through load balancer; got body: %s", resp.Status, string(body)) } if !strings.Contains(string(body), "test-webserver") { Failf("received response body without expected substring 'test-webserver': %s", string(body)) } }) </ins> It("should correctly serve identically named services in different namespaces on different external IP addresses", func() { serviceNames := []string{"services-namespace-test0"} // Could add more here, but then it takes longer. namespaces := []string{"namespace0", "namespace1"}   // As above.

Follow up from
Bumping this up to P1 given that 0.12.2 wasn't able to create a service with an external load balancer and we didn't notice before cutting the release.
No, to be clearer, 0.12.2 wasn't able to create a service with ELB on GKE. That was on our side entirely.
But it doesn't invalidate the need for this.
The example is using ELB.
are you actively working on this? If not, I'd be up for doing it today.
Maybe not actively but I know what to, however I won't be able to finish it today, so if you think you can do it today please move forward.
We run the e2e tests on GKE, so if we had a test for creating an ELB we would have known that the 0.12.2 release wouldn't work on GKE. I don't see how your point applies. The e2e tests are in place so that we don't have to test each feature manually to verify that it is working, and we only found out that this feature was broken on GKE after we pushed 0.12.2, hence the need for an e2e test.
I think and you are in fact in agreement
right now, k8petstore tests the external load balancer by curling down the amount of transactions ingested from the REST API. im playing with porting it as a e2e test now. The code in the shell script is a little raw, but you can see that it specifies a PUBLIC_IP and then curls down from it at the end ().
that sounds great. What I'd really like to see here though (and what is going to do early next week) is a very simple test that isolates creating a service with an ELB to verify that creating a load balancer works in isolation from a complex test of other parts of the system. This will give us a really clear signal when the test starts to fail instead of needing to debug lots of moving parts when we see a test failure.


Question: timeout  time.Duration protocol string udpData  string <ins> sctpData string </ins> ) func init() { CmdConnect.Flags().DurationVar(&timeout, "timeout", time.Duration(0), "Maximum time before returning an error") CmdConnect.Flags().StringVar(&protocol, "protocol", "tcp", "The protocol to use to perform the connection, can be tcp, udp or sctp") CmdConnect.Flags().StringVar(&udpData, "udp-data", "hostname", "The UDP payload send to the server") <ins> CmdConnect.Flags().StringVar(&sctpData, "sctp-data", "hostname", "The SCTP payload send to the server") </ins> } func main(cmd *cobra.Command, args []string) {

How to repro: an agnhost pod listening on sctp into the pod and use the connect command to probe against the same pod The stack trace that points to and this is from the vendored library , /kind bug /sig network /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Run an agnhost pod listening on sctpExec into the pod and use the connect command to probe against the same pod The stack trace that points to /sig network /help </details
/assign
/triage accepted
here we are immediately closing the connection after successful connect leading to EOF at server end (it is expecting data) causing the failure. We can tweak this to connect send some data just like UDP and then exit? Works locally writing "hostname" to the connection. [EDIT] For me it sometimes failed with EOF, sometime with nil pointer exception at . This reads sock opts via syscall and we immediately close the connection .. kind of a race between the client closing connection and server reading fds?
the panic seems a bug in that project, no? Also SCTP, has a establishment handshake, so the data should not be relevant in this case
returns fd and socket address, for is used to implement (always read from struct field). The SCTP implementation discards the returned socket address by call. , and is always a new syscall (). In our case the client immediately closes the connection, there can be chances where SCTPGETPEERADDRS will fail and will return nil in these cases. I think this is reasonable for their project. We call and panic is raised by calling on nil object. We can easily avoid this by not null check.

Also, I strongly believe that the client handling should be done in a separate go-routine instead of the main loop to avoid errors like this crashing the pod. [EDIT] The client in our case needs to send some data after connection, or else the server will throw EOF and we currently have assertions that use for errors.
panicing does not seem very reasonable It looks to me that the problem is that the SCTPConn struct needs to check if the connection is still valid before trying to access the file descriptor , see the std library implementation and then, all methods implement this check to me it is a bug on the implementation
The implementation is not causing any panic, it simply returns nil. Panic is caused when we try to call on nil object.
oh, sorry, my bad I should read your comment with more detail, is well explained there I recognize I have zero idea of how sctp works internally, maybe we can just be more resilient on our tests and code and detect if is nil and avoid getting into the complexity of this protocol ... sending some random data also sounds good, we do this for udp IIRC
are you able to send a PR to add those improvements to the test image?


Question: return nil } <ins> // emitEventForUnboundDelayBindingClaim generates informative event for claim // if it's in delay binding mode and not bound yet. func (ctrl *PersistentVolumeController) emitEventForUnboundDelayBindingClaim(claim *v1.PersistentVolumeClaim) error { reason := events.WaitForFirstConsumer message := "waiting for first consumer to be created before binding" podNames, err := ctrl.findNonScheduledPodsByPVC(claim) if err != nil { return err } if len(podNames) > 0 { reason = events.WaitForPodScheduled if len(podNames) > 1 { // Although only one pod is taken into account in // volume scheduling, more than one pods can reference // the PVC at the same time. We can't know which pod is // used in scheduling, all pods are included. message = fmt.Sprintf("waiting for pods %s to be scheduled", strings.Join(podNames, ",")) } else { message = fmt.Sprintf("waiting for pod %s to be scheduled", podNames[0]) } } ctrl.eventRecorder.Event(claim, v1.EventTypeNormal, reason, message) return nil } </ins> // syncUnboundClaim is the main controller method to decide what to do with an // unbound claim. func (ctrl *PersistentVolumeController) syncUnboundClaim(claim *v1.PersistentVolumeClaim) error {

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: b2.create(ctx, pod2claim1, pod2) framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(ctx, f.ClientSet, pod2), "start pod 2") <del> // Allow allocation of pod1 claim2, claim3 to proceed. It should fail now </del> <ins> // Allow allocation of second claim in pod1 to proceed. It should fail now </ins> // and the other node must be used instead, after deallocating // the first claim. ginkgo.By("move first pod to other node")

What happened: Shortly after was merged, the "multiple nodes reallocation works" E2E DRA test started to fail randomly. Please provide links to example occurrences, if any: /sig node /kind bug needs to stay green. No response master n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" <del> scheduling "k8s.io/api/scheduling/v1alpha1" </del> <ins> scheduling "k8s.io/api/scheduling/v1beta1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" )

/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling


Question: assert.Equal(t, ns1.Name, createdNamespace2.Name) // assert.Contains(t, createdNamespace2.Annotations, "A") <ins> ns1.DeletionTimestamp = &unversioned.Time{Time: time.Now()} namespaceWatch.Modify(&ns1) assert.Equal(t, ns1.Name, GetStringFromChan(nsDeleteChan)) assert.Equal(t, "all", GetStringFromChan(rsDeleteChan)) assert.Equal(t, "all", GetStringFromChan(serviceDeleteChan)) assert.Equal(t, "all", GetStringFromChan(secretDeleteChan)) </ins> close(stop) } <ins> func RegisterDeleteCollection(client *core.Fake, resource string) chan string { deleteChan := make(chan string, 100) client.AddReactor("delete-collection", resource, func(action core.Action) (bool, runtime.Object, error) { deleteChan <- "all" return true, nil, nil }) return deleteChan } func RegisterDelete(client *core.Fake, resource string) chan string { deleteChan := make(chan string, 100) client.AddReactor("delete", resource, func(action core.Action) (bool, runtime.Object, error) { deleteAction := action.(core.DeleteAction) deleteChan <- deleteAction.GetName() return true, nil, nil }) return deleteChan } func GetStringFromChan(c chan string) string { select { case str := <-c: return str case <-time.After(5 * time.Second): return "" } } </ins> func GetNamespaceFromChan(c chan runtime.Object) *api_v1.Namespace { namespace := GetObjectFromChan(c).(*api_v1.Namespace) return namespace

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: --dry-run[=false]: If true, only print the object that would be sent, without sending it. --env=[]: Environment variables to set in the container --expose[=false]: If true, a public, external service is created for the container(s) which are run <del> --generator="": The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'. </del> <ins> --generator="": The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'.  This will happen only for cluster version at least 1.2, for olders we will fallback to 'run/v1' for --restart=Alwyas, 'run-pod/v1' for others. </ins> --hostport=-1: The host port mapping for the container port. To demonstrate a single-machine container. --image="": The image for the container to run. -l, --labels="": Labels to apply to the pod(s).

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: "k8s.io/klog/v2" <del> "k8s.io/apimachinery/pkg/types" </del> "k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" kubefeatures "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/checkpoint" <del> "k8s.io/kubernetes/pkg/kubelet/cm/util/cdi" </del> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" )

Kubelet should stop using annotations to pass CDI device IDs to CRI runtimes. CDIDevices CRI field should be used for this purpose. This should be done when Kubelet that doesn't support CRI field reaches EOL and two major CRI runtimes support CRI field. Here is a PR comment that gives more details about this: Kubelet started using CDIDevices CRI field in . Previous version 1.27 will reach EOL .CRI-O started using it in . Previous version 1.27 will reach EOL together with Kubernetes 1.27.Containerd started using it in . Version 1.7 will reach EOL in a 6 months after Containerd 2.0 release Previous version 1.6 doesn't support CDI injections and should not be considered. So, safer option would be to wait until Containerd 1.7 reaches EOL. However it's relatively safe to do it earlier when 1.7.0 and 1.7.1 have little chances to be in real use. Containerd v1.7.0 and v1.7.1 is not used by the Linux distributions from , so it's safe to remove annotations support from the Kubelet in 1.31 in my opinion.
/sig node
/triage accepted /priority important-longterm
/assign


Question: return true, offset } <ins> // bigForIP creates a big.Int based on the provided net.IP func bigForIP(ip net.IP) *big.Int { b := ip.To4() if b == nil { b = ip.To16() } return big.NewInt(0).SetBytes(b) } // addIPOffset adds the provided integer offset to a base big.Int representing a // net.IP func addIPOffset(base *big.Int, offset int) net.IP { return net.IP(big.NewInt(0).Add(base, big.NewInt(int64(offset))).Bytes()) } </ins> // calculateIPOffset calculates the integer offset of ip from base such that // base + offset = ip. It requires ip >= base. func calculateIPOffset(base *big.Int, ip net.IP) int { <del> return int(big.NewInt(0).Sub(utilnet.BigForIP(ip), base).Int64()) </del> <ins> return int(big.NewInt(0).Sub(bigForIP(ip), base).Int64()) } // RangeSize returns the size of a range in valid addresses. func RangeSize(subnet *net.IPNet) int64 { ones, bits := subnet.Mask.Size() if bits == 32 && (bits-ones) >= 31 || bits == 128 && (bits-ones) >= 127 { return 0 } // For IPv6, the max size will be limited to 65536 // This is due to the allocator keeping track of all the // allocated IP's in a bitmap. This will keep the size of // the bitmap to 64k. if bits == 128 && (bits-ones) >= 16 { return int64(1) << uint(16) } else { return int64(1) << uint(bits-ones) } } // GetIndexedIP returns a net.IP that is subnet.IP + index in the contiguous IP space. func GetIndexedIP(subnet *net.IPNet, index int) (net.IP, error) { ip := addIPOffset(bigForIP(subnet.IP), index) if !subnet.Contains(ip) { return nil, fmt.Errorf("can't generate IP with index %d from subnet. subnet too small. subnet: %q", index, subnet) } return ip, nil </ins> }

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?