Question: framework.Failf("pod spec TerminationGracePeriodSeconds is not 1: %d", *pod.Spec.TerminationGracePeriodSeconds) } <del> time.Sleep(5 * time.Second) pod, err = podClient.Get(ctx, pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to query for pod") ginkgo.By("updating the pod to have a negative TerminationGracePeriodSeconds") pod.Spec.TerminationGracePeriodSeconds = utilpointer.Int64(-1) _, err = podClient.PodInterface.Update(ctx, pod, metav1.UpdateOptions{}) </del> <ins> // retry if the TerminationGracePeriodSeconds is overrided // see more in https://github.com/kubernetes/kubernetes/pull/115606 err = retry.RetryOnConflict(retry.DefaultBackoff, func() error { pod, err := podClient.Get(ctx, pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to query for pod") ginkgo.By("updating the pod to have a negative TerminationGracePeriodSeconds") pod.Spec.TerminationGracePeriodSeconds = utilpointer.Int64(-1) _, err = podClient.PodInterface.Update(ctx, pod, metav1.UpdateOptions{}) return err }) </ins> framework.ExpectNoError(err, "failed to update pod") pod, err = podClient.Get(ctx, pod.Name, metav1.GetOptions{})

After , /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: - apiGroups: - "" resources: <ins> - configmaps </ins> - endpoints - namespaces - secrets

BUG REPORT: /kind bug /sig auth What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops with: The kube-controller-manager is run with a with a to the default system ClusterRole , which seems to be defined at Editing the ClusterRole manually to add for was sufficient to fix the crash loop. What you expected to happen: I suspect a change between v1.11.3 and v1.12.0 release candidates altered kube-controller-manager to begin looking for configmap , but didn't update the ClusterRoles. How to reproduce it (as minimally and precisely as possible): Release candidate testing is happening in Anything else we need to know?: Environment: Kubernetes version (use kubectl version): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. uname -a): 4.14.67-coreos Install tools: Typhoon
/sig cluster-lifecycle /area kubeadm /remove-sig auth edit: sorry, got this mixed up with another 1.12/1.11 issue. is this related to the secured controller manager work you did? /remove-sig cluster-lifecycle /sig auth /sig api-machinery
Those labels are not set on the issue: <detailsIn response to : /sig cluster-lifecycle /area kubeadm /remove-sig auth /sig auth /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you please try if works to skip that lookup?
we can make this non-fatal, compare
adding that flag to the controller-manager does also work to skip the lookup, without me manually editing the ClusterRole. Although if controller-manager defaults to looking up the configmap, I'd imagine the clusterrole should default to allowing configmap get's, no?

i was just checking if there's a work around we could suggest for 1.12 :) longer term, we can lean towards whatever and decide (adding to bootstrap policy vs making things non-fatal)
This seems like it should get a Known Issue comment in the release notes for 1.12. Can somebody please draft test and drop it in the known issues bucket issue ASAP?


Question: } } } <ins> { // Add additional tags glog.V(2).Infof("Creating additional load balancer tags for %s", loadBalancerName) tags := getLoadBalancerAdditionalTags(annotations) if len(tags) > 0 { err := c.addLoadBalancerTags(loadBalancerName, tags) if err != nil { return nil, fmt.Errorf("unable to create additional load balancer tags: %v", err) } } } </ins> } // Whether the ELB was new or existing, sync attributes regardless. This accounts for things

BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


Question: // GetEndpointNodeNames returns a string set of node names on which the // endpoints of the given Service are running. func (j *TestJig) GetEndpointNodeNames() (sets.String, error) { <ins> err := j.waitForAvailableEndpoint(ServiceEndpointsTimeout) if err != nil { return nil, err } </ins> endpoints, err := j.Client.CoreV1().Endpoints(j.Namespace).Get(context.TODO(), j.Name, metav1.GetOptions{}) if err != nil { return nil, fmt.Errorf("get endpoints for service %s/%s failed (%s)", j.Namespace, j.Name, err)

Which jobs are flaking: Which test(s) are flaking: ! Testgrid link: Reason for failure: Anything else we need to know: /cc /sig network /sig scalability /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign


Question: containerTypeLabelContainer = "container" ) <del> func NewDockerSevice(client dockertools.DockerInterface) DockerLegacyService { </del> <ins> func NewDockerService(client dockertools.DockerInterface) DockerLegacyService { </ins> return &dockerService{ client: dockertools.NewInstrumentedDockerInterface(client), }

umbrella issue: kuberuntime uses to ]parse the version string of the api version]() from the runtime. When using with docker, it complains that Maybe we should just enforce that the api version returned by the runtime should be semver compatible. /cc
This is mentioned in . :) For now, we can add a after the docker apiversion to make it semver compatible.
Thanks for the reminder! We should make it clear in the API. This blocks kubelet from starting up, so we should fix it first.
I've checked the code, currently is only used in when checking the docker version compatibility. It is not exposed to the user by any means, so it should be fine to make some change ourselves, such as adding a . However, if we want to expose this to the user via in the future, it may make user confusing.


Question: } func (mounter *SafeFormatAndMount) diskLooksUnformatted(disk string) (bool, error) { <del> return true, nil </del> <ins> return true, unsupportedErr </ins> } func (mounter *Mounter) GetFileType(pathname string) (FileType, error) { <del> return FileType("fake"), errors.New("not implemented") </del> <ins> return FileType("fake"), unsupportedErr </ins> } func (mounter *Mounter) MakeDir(pathname string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } func (mounter *Mounter) MakeFile(pathname string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } func (mounter *Mounter) ExistsPath(pathname string) bool { <ins> glog.Errorf("%s", unsupportedErr) </ins> return true } func (mounter *Mounter) PrepareSafeSubpath(subPath Subpath) (newHostPath string, cleanupAction func(), err error) { <del> return subPath.Path, nil, nil </del> <ins> return subPath.Path, nil, unsupportedErr </ins> } func (mounter *Mounter) CleanSubPaths(podDir string, volumeName string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } func (mounter *Mounter) SafeMakeDir(pathname string, base string, perm os.FileMode) error { <del> return nil </del> <ins> return unsupportedErr </ins> }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Calls to on unsupported platforms do not report any error and potentially misbehave. What you expected to happen: I would expect an error to be reported, that can be consumed by the caller. How to reproduce it (as minimally and precisely as possible): for and were failing on darwin, and no error was being reported. See: & Environment: OS (e.g. from /etc/os-release): I have been testing on osx/darwin, which is not a supported platform.
/sig storage
/assign


Question: import ( "fmt" "io" <del> "sort" "strconv" </del> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/meta"

Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


Question: return err } <del> // Init tries best effort to ensure that 2 accounts standard/premium were created // to be used by shared blob disks. This to increase the speed pvc provisioning (in most of cases) func (c *BlobDiskController) init() error { if !c.shouldInit() { return nil } c.setUniqueStrings() // get accounts accounts, err := c.getAllStorageAccounts() if err != nil { return err } c.accounts = accounts if len(c.accounts) == 0 { counter := 1 for counter <= storageAccountsCountInit { accountType := storage.PremiumLRS if n := math.Mod(float64(counter), 2); n == 0 { accountType = storage.StandardLRS } // We don't really care if these calls failed // at this stage, we are trying to ensure 2 accounts (Standard/Premium) // are there ready for PVC creation // if we failed here, the accounts will be created in the process // of creating PVC // nor do we care if they were partially created, as the entire // account creation process is idempotent go func(thisNext int) { newAccountName := getAccountNameForNum(thisNext) glog.Infof("azureDisk - BlobDiskController init process  will create new storageAccount:%s type:%s", newAccountName, accountType) err := c.createStorageAccount(newAccountName, accountType, c.common.location, true) // TODO return created and error from if err != nil { glog.Infof("azureDisk - BlobDiskController init: create account %s with error:%s", newAccountName, err.Error()) } else { glog.Infof("azureDisk - BlobDiskController init: created account %s", newAccountName) } }(counter) counter = counter + 1 } } return nil } </del> //Sets unique strings to be used as accountnames && || blob containers names func (c *BlobDiskController) setUniqueStrings() { uniqueString := c.common.resourceGroup + c.common.location + c.common.subscriptionID

Is this a BUG REPORT or FEATURE REQUEST?: What happened: For unmanaged disk, now it is not working What you expected to happen: Works well How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.2.734+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-14T03:26:54Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.2.734+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-14T03:26:54Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL7 Kernel (e.g. ): Install tools: ansible Others:
/assign andyzhangx /sig azure
I cannot reproduce your issue with v1.9.0, in your storage class, an azure disk would be provisioned in storage account , it may fail because name is already occupied, but in your case it's a strage string: . So could you also paste the controller manager log, below is the way how to get it: There should be an important log like following:
I did not run containerized k8s, I deployed a cluster by ansible: I got the controller message as below: I am quit sure there is already a storage account in its resource group which is assigned in sc's parameter: I do not know why controller does not find it and wants to create a new one. But in our subscription the storage account has reached the max number and cannot be created anymore, the error is different from this.
cc
would you add in your stroage class, and try again? In your case, is not invoked due to: I am working out a PR to fix this, as a workaround, you could try storage class like following: Let me know if that works, thx
Yes. working now with this parameter :

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: is buggy. It prevents running hyperkube from other path. In fact, it shouldn't assume the code is just running in controller master at all, especially when cloud provider is running out of tree. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
This code logic is to make sure the blobController only initialize once in controller-manager other than in other component, e.g. kubelet, do you have any suggestion to use similar implementation? Current design is azure pv code could only run in controller-manager.
If the idea is to exclude other components, I prefer a black list, not a broken white a list as here.
there are multiple issues with this code: It shouldn't bootstrap any storage accounts at all. This is really poor user experience. Not all users want to/will use azure disk. These storage accounts are yet created without any explicit documentation or prior warning.Even with these storage accounts, it doesn't help accelerate PV provisioning.This is the only place to init . If controller master is launched from a different path (which is usually the case), then never gets initialized, and will cause a panic when being used.
thanks for the check, put the storage accounts bootstrap in could be more appropiate: that is only when user select kind and would like to create blob disk PVC, then those accounts would be created and it would only initialize before first PVC is created.
SGTM, looking forward to your PR and 1.7/1.8 cherry picks
update: code complete, I need to do some test before sending PR:

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The azuredd plugin will always provision two instances of storage account with different type during init. There could be cases when: may only want to use one certain type of storage account, then the other type of storage account would be not in use. chooses to use managed disk, both of the storage account would be not in use. There is a limit for storage account numbers per subscription, and the storage account should be created with a on-demand policy. Log: Code: What you expected to happen: The storage accounts be created on-demand. How to reproduce it (as minimally and precisely as possible): Setup a kubernetes v1.7.2 cluster on Azure, create a dynamic pvc. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: Azure
/sig azure
Creating a storage account is a lengthy process (~ 1.5 mins), these 2 accounts are created upfront to ensure that PVCs will be quickly provisioned into existing storage accounts. Since ACS & acs-engine currently supports mixed (managed and blob) disks we won't be able to tell if they user will or won't use blob disks. Maybe once acs- moves away from blob disks we can disable them. They have no impact on users since users get billed only for data inside storage accounts not storage accounts themselves
for the "limit for storage account numbers per subscription" issue, I have another issue to track this: , will fix it in v1.9 and cherry pick to v1.8, v1.7
/assign


Question: } else { nodeAddresses = nodeAddrSet.List() for _, address := range nodeAddresses { <ins> a := net.ParseIP(address) if a.IsLoopback() { continue } </ins> if utilproxy.IsZeroCIDR(address) { nodeIPs, err = proxier.ipGetter.NodeIPs() if err != nil {

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: --repo-root="${KUBE_ROOT}"  --node-instance-group="${NODE_INSTANCE_GROUP:-}"  --prefix="${KUBE_GCE_INSTANCE_PREFIX:-e2e}"  <del> ${KUBE_OS_DISTRIBUTION:+"--os-distro=${KUBE_OS_DISTRIBUTION}"}  </del> <ins> ${OS_DISTRIBUTION:+"--os-distro=${OS_DISTRIBUTION}"}  </ins> ${NUM_NODES:+"--num-nodes=${NUM_NODES}"}  ${E2E_CLEAN_START:+"--clean-start=true"}  ${E2E_MIN_STARTUP_PODS:+"--minStartupPods=${E2E_MIN_STARTUP_PODS}"}

We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.

at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far.

:smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:


Question: } for n := range p.visitedNamespaces { <del> if len(o.Namespace) != 0 && n != o.Namespace { continue } </del> for _, m := range namespacedRESTMappings { if err := p.prune(n, m); err != nil { return fmt.Errorf("error pruning namespaced object %v: %v", m.GroupVersionKind, err)

Not sure if this is a bug, feature requests, or a documentation issue. What happened: I am trying to apply some configs with --prune, and was surprised by the behavior. I have a config file with objects in multiple namespaces, and apply with (note: no flag). By kube context was set to a namespace that was not in the manifests. Here is what I found: If my context is set to a random namespace, nothing is pruned If its set to namespace X, only objects in namespace X are applied If I pass a flag I can control which namespace gets pruned, but that will break the apply because the file has multiple namespaces in it What you expected to happen: I expect some way to be able to apply a config file with objects in multiple namespace and having pruning across all namespaces. Seems there are a few options: all namespaces referenced in the file, prune these across all namespaces warn about why we are not pruning due to namespace mismatch How to reproduce it (as minimally and precisely as possible): Run the script from namespace default, a, and b. You will see different behavior on each run. Related? Environment: Kubernetes version (use ): 1.17.0 Others: /sig cli
I encountered this issue. I think has the bug that o.namespace gets "default" if no namespace is given. Then only a single namespace is visited with or without namespace. If my observation is correct, it changes the bahaviour drastically. In my work, pruning just stops working.


Question: if m.adapter.cgroupManagerType == libcontainerSystemd { updateSystemdCgroupInfo(libcontainerCgroupConfig, cgroupConfig.Name) } else { <del> if libcontainercgroups.IsCgroup2UnifiedMode() { libcontainerCgroupConfig.Path = m.buildCgroupUnifiedPath(cgroupConfig.Name) } else { libcontainerCgroupConfig.Path = cgroupConfig.Name.ToCgroupfs() } </del> <ins> libcontainerCgroupConfig.Path = cgroupConfig.Name.ToCgroupfs() </ins> } if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.SupportPodPidsLimit) && cgroupConfig.ResourceParameters != nil && cgroupConfig.ResourceParameters.PidsLimit != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is created on a cgroup2 host What you expected to happen: Such directory should not be created How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 with Install moby/moby with containerd/containerd and opencontainers/runc Install kubernetes-sigs/kind . This version uses as the base image. Run with Run Run to drop into the shell of the node The following directories are unexpectedly created. (No process seems to belong to these groups actually) Environment: (See above)
cc
/sig node


Question: scaleDown("rc", kindRC, rc, f) }) }) <ins> Describe("ReplicationController light", func() { It("Should scale from 1 pod to 2 pods", func() { scaleTest := &HPAScaleTest{ initPods:                    1, totalInitialCPUUsage:        150, perPodCPURequest:            200, targetCPUUtilizationPercent: 50, minPods:                     1, maxPods:                     2, firstScale:                  2, } scaleTest.run("rc-light", kindRC, rc, f) }) It("Should scale from 2 pods to 1 pod using HPA version v1", func() { scaleTest := &HPAScaleTest{ initPods:                    2, totalInitialCPUUsage:        50, perPodCPURequest:            200, targetCPUUtilizationPercent: 50, minPods:                     1, maxPods:                     2, firstScale:                  1, useV1:                       true, } scaleTest.run("rc-light", kindRC, rc, f) }) }) </ins> }) // HPAScaleTest struct is used by the scale(...) function. type HPAScaleTest struct { <del> initPods          int cpuStart          int maxCPU            int64 idealCPU          int minPods           int maxPods           int firstScale        int firstScaleStasis  time.Duration cpuBurst          int secondScale       int secondScaleStasis time.Duration </del> <ins> initPods                    int totalInitialCPUUsage        int perPodCPURequest            int64 targetCPUUtilizationPercent int minPods                     int maxPods                     int firstScale                  int firstScaleStasis            time.Duration cpuBurst                    int secondScale                 int secondScaleStasis           time.Duration useV1                       bool </ins> } // run is a method which runs an HPA lifecycle, from a starting state, to an expected // The initial state is defined by the initPods parameter. // The first state change is due to the CPU being consumed initially, which HPA responds to by changing pod counts. <del> // The second state change is due to the CPU burst parameter, which HPA again responds to. </del> <ins> // The second state change (optional) is due to the CPU burst parameter, which HPA again responds to. </ins> // TODO The use of 3 states is arbitrary, we could eventually make this test handle "n" states once this test stabilizes. func (scaleTest *HPAScaleTest) run(name, kind string, rc *ResourceConsumer, f *Framework) { <del> rc = NewDynamicResourceConsumer(name, kind, scaleTest.initPods, scaleTest.cpuStart, 0, scaleTest.maxCPU, 100, f) </del> <ins> rc = NewDynamicResourceConsumer(name, kind, scaleTest.initPods, scaleTest.totalInitialCPUUsage, 0, scaleTest.perPodCPURequest, 100, f) </ins> defer rc.CleanUp() <del> createCPUHorizontalPodAutoscaler(rc, scaleTest.idealCPU, scaleTest.minPods, scaleTest.maxPods) </del> <ins> createCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods, scaleTest.useV1) </ins> rc.WaitForReplicas(scaleTest.firstScale) <del> rc.EnsureDesiredReplicas(scaleTest.firstScale, scaleTest.firstScaleStasis) rc.ConsumeCPU(scaleTest.cpuBurst) rc.WaitForReplicas(scaleTest.secondScale) </del> <ins> if scaleTest.firstScaleStasis > 0 { rc.EnsureDesiredReplicas(scaleTest.firstScale, scaleTest.firstScaleStasis) } if scaleTest.cpuBurst > 0 && scaleTest.secondScale > 0 { rc.ConsumeCPU(scaleTest.cpuBurst) rc.WaitForReplicas(scaleTest.secondScale) } </ins> } func scaleUp(name, kind string, rc *ResourceConsumer, f *Framework) { scaleTest := &HPAScaleTest{ <del> initPods:         1, cpuStart:         250, maxCPU:           500, idealCPU:         .2 * 100, minPods:          1, maxPods:          5, firstScale:       3, firstScaleStasis: 10 * time.Minute, cpuBurst:         700, secondScale:      5, </del> <ins> initPods:                    1, totalInitialCPUUsage:        250, perPodCPURequest:            500, targetCPUUtilizationPercent: 20, minPods:                     1, maxPods:                     5, firstScale:                  3, firstScaleStasis:            10 * time.Minute, cpuBurst:                    700, secondScale:                 5, </ins> } scaleTest.run(name, kind, rc, f) } func scaleDown(name, kind string, rc *ResourceConsumer, f *Framework) { scaleTest := &HPAScaleTest{ <del> initPods:         5, cpuStart:         400, maxCPU:           500, idealCPU:         .3 * 100, minPods:          1, maxPods:          5, firstScale:       3, firstScaleStasis: 10 * time.Minute, cpuBurst:         100, secondScale:      1, </del> <ins> initPods:                    5, totalInitialCPUUsage:        400, perPodCPURequest:            500, targetCPUUtilizationPercent: 30, minPods:                     1, maxPods:                     5, firstScale:                  3, firstScaleStasis:            10 * time.Minute, cpuBurst:                    100, secondScale:                 1, </ins> } scaleTest.run(name, kind, rc, f) } <del> func createCPUHorizontalPodAutoscaler(rc *ResourceConsumer, cpu, minReplicas, maxRepl int) { </del> <ins> func createCPUHorizontalPodAutoscaler(rc *ResourceConsumer, cpu, minReplicas, maxRepl int, useV1 bool) { </ins> hpa := &extensions.HorizontalPodAutoscaler{ ObjectMeta: api.ObjectMeta{ Name:      rc.name,

lightweight test which can be run in gce suite test which verifies that version v1 works


Question: It("Pod should perfer to scheduled to nodes pod can tolerate", func() { // make the nodes have balanced cpu,mem usage ratio <del> createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) </del> <ins> err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) framework.ExpectNoError(err) </ins> //we need apply more taints on a node, because one match toleration only count 1 By("Trying to apply 10 taint on the nodes except first one.") nodeName := nodeList.Items[0].Name

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: @type kubernetes_metadata </filter> <ins> # Concatenate multi-line logs <filter **> @type concat key message multiline_end_regexp /n$/ separator "" </filter> </ins> <match **> @id elasticsearch @type elasticsearch

Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective.

I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue.

If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together?

The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


Question: ginkgo.By("attempting to create a CR with unknown metadata fields in the embedded object") metaEmbedded := fmt.Sprintf(metaPattern, testCRD.Crd.Spec.Names.Kind, testCRD.Crd.Spec.Group, testCRD.Crd.Spec.Versions[0].Name, "test-cr-embedded") <del> unknownEmbeddedMetaCR := fmt.Sprintf(embeddedCRPattern, metaEmbedded, `"unknownMetaEmbedded": "bar",`) </del> <ins> unknownEmbeddedMetaCR := fmt.Sprintf(embeddedCRPattern, metaEmbedded, `"unknownMetaEmbedded": "bar",`, ns) </ins> _, err = framework.RunKubectlInput(ns, unknownEmbeddedMetaCR, "create", "--validate=true", "-f", "-") if err == nil { framework.Failf("unexpected nil error when creating CR with unknown embedded metadata field")

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: ### Options ``` <del> --all[=false]: [-all] to select all the specified resources. </del> <ins> --all[=false]: select all resources in the namespace of the specified resource types. </ins> -f, --filename=[]: Filename, directory, or URL to file of resource(s) to be stopped. --grace-period=-1: Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. --ignore-not-found[=false]: Treat "resource not found" as a successful stop.

It actually doesn't support --all.
/assign
ohh, thanks, will make a pr to address that right now..


Question: func TestExperimentalPrefix(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> resp, err := http.Get(s.URL + "/apis/extensions/") if err != nil {

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: list = &beta case "/apis/extensions/v1beta2": list = &beta2 <ins> case "/apis/extensions/v1beta3": list = &extensionsbeta3 case "/apis/extensions/v1beta4": list = &extensionsbeta4 case "/apis/extensions/v1beta5": list = &extensionsbeta5 case "/apis/extensions/v1beta6": list = &extensionsbeta6 case "/apis/extensions/v1beta7": list = &extensionsbeta7 case "/apis/extensions/v1beta8": list = &extensionsbeta8 case "/apis/extensions/v1beta9": list = &extensionsbeta9 case "/apis/extensions/v1beta10": list = &extensionsbeta10 case "/apis/apps/v1beta1": list = &appsbeta1 case "/apis/apps/v1beta2": list = &appsbeta2 case "/apis/apps/v1beta3": list = &appsbeta3 case "/apis/apps/v1beta4": list = &appsbeta4 case "/apis/apps/v1beta5": list = &appsbeta5 case "/apis/apps/v1beta6": list = &appsbeta6 case "/apis/apps/v1beta7": list = &appsbeta7 case "/apis/apps/v1beta8": list = &appsbeta8 case "/apis/apps/v1beta9": list = &appsbeta9 case "/apis/apps/v1beta10": list = &appsbeta10 </ins> case "/api": list = &metav1.APIVersions{ Versions: []string{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: // without needing to be deleted and recreated. if firewallExists { glog.Infof("EnsureLoadBalancer(%v(%v)): updating firewall", loadBalancerName, serviceName) <del> if err := gce.updateFirewall(apiService, makeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </del> <ins> if err := gce.updateFirewall(apiService, MakeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </ins> return nil, err } glog.Infof("EnsureLoadBalancer(%v(%v)): updated firewall", loadBalancerName, serviceName) } else { glog.Infof("EnsureLoadBalancer(%v(%v)): creating firewall", loadBalancerName, serviceName) <del> if err := gce.createFirewall(apiService, makeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </del> <ins> if err := gce.createFirewall(apiService, MakeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </ins> return nil, err } glog.Infof("EnsureLoadBalancer(%v(%v)): created firewall", loadBalancerName, serviceName)

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: sched.config.Error(pod, err) sched.config.Recorder.Event(pod, v1.EventTypeWarning, "FailedScheduling", message) sched.config.PodConditionUpdater.Update(pod, &v1.PodCondition{ <del> Type:    v1.PodScheduled, Status:  v1.ConditionFalse, Reason:  reason, Message: err.Error(), </del> <ins> Type:          v1.PodScheduled, Status:        v1.ConditionFalse, LastProbeTime: metav1.Now(), Reason:        reason, Message:       err.Error(), </ins> }) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In large clusters, events that trigger rescheduling of the unschedulable pods are frequent. For example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. These events trigger rescheduling of unschedulable pods. The scheduler moves unschedulable pods to its active scheduling queue when such events are observed. There are two particularly problematic scenarios: are high-priority unschedulable pods and preemption does not help make them schedulable. For example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. Such high priority pods may go to the head of the active scheduling queue and block everything behind them. are many pending pods with the same priority, but a number of them are unschedulable. When these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by priority and these pods have the same priority, their order is unspecified. The Pop() algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a LIFO. So, pods which are tried recently, go to the head with a high probability and are retried. This starves pods behind them in the queue. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig scheduling /assign
For case 1: How about add an field to record how many times scheduler have tried to schedule it? The bigger it is, the latter we should schedule it. Then we could consider to add it as a factor for sorting pods.
Yes, we basically need a . Another approach is to . We will pursue both of these approaches.
Well differentiating optionsounds better
Agree. On the one hand we need to reduce the frequency of unnecessary rescheduling, on the other hand we need to optimize the sorting algorithm.

And the would help to reduce the frequency. But I think "differentiate between node heartbeats vs. actual node updates" is a better approach.
Yes its a better approach indeed but what if we use the recursion there
We should have all three changes to address all known scenarios efficiently. The three changes are: back off mechanism fair sorting of pods in the scheduling queue smarter processing of node updates Node updates are not the only trigger for the scheduler to retry unschedulable pods. The scheduler also retries when running pods terminate, among a couple of other events. So, without having backoff mechanism a high priority unschedulable pod could still block head of the queue in clusters with high churn, for example, in clusters where many pods terminate frequently. Back off mechanism by itself addresses the issue in all scenarios, but the other two changes make the scheduling queue smarter and more efficient.
The three prs above are quite comprehensive : ).
All of the above items are done already.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" utilerrors "k8s.io/apimachinery/pkg/util/errors" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/printers" discovery "k8s.io/client-go/discovery"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl auth can-i <verb<resourcesthe "verb" isn't check and returns yes on any resource regardless of what the verb is. What you expected to happen: I expected the cli to register an error like when you don't have enough arguments. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): client version GitVersion: 1.19.2 server version: GitVersion: 1.19.0 Cloud provider or hardware configuration: Mac OS X OS (e.g: ): n/a Kernel (e.g. ): Darwin Install tools: brew Network plugin and version (if this is a network-related bug): n/a Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
This is working as intended. The authorization layer can answer questions about arbitrary verbs, and the response to can-i mirrors the response you would get from a subjectaccessreview permission check. Getting a "yes" response to arbitrary verbs likely means you are a superuser and would be authorized for any operation.
Wouldn't a superuser only have access to the Kubernetes verbs that exist and not the arbitrary verbs that it is allowed to query that is defined here () This seems to go against the selfsubjectaccessreview which will return a set of actions that a user can perform. Arbitrary verbs seem like they shouldn't be allowed from that definition.
No, the cluster-admin superuser role allows verbs which allows any verb


Question: for (( i=0; i <${NUM_MINIONS}; i++)) do KUBE_MINION_IP_ADDRESSES[$i]="${MINION_IP_BASE}$[$i+2]" MINION_NAMES[$i]="${MINION_IP_BASE}$[$i+2]" <del> done </del>  No newline at end of file <ins> VAGRANT_MINION_NAMES[$i]="minion-$[$i+1]" done </ins>

Running vagrant from scratch gives the error: For vagrant cluster, the last minion cannot be detected. The /tmp/minions file only shows two minions, but after a while, using the same command, i.e. will get three minions. There is a race between kube-up and validate-, we should wait a while before panic.
I can look to fix this.


Question: "k8s.io/kubernetes/pkg/kubelet/util" ) <del> // maxMsgSize use 8MB as the default message size limit. </del> <ins> // maxMsgSize use 16MB as the default message size limit. </ins> // grpc library default is 4MB <del> const maxMsgSize = 1024 * 1024 * 8 </del> <ins> const maxMsgSize = 1024 * 1024 * 16 </ins> // DockerServer is the grpc server of dockershim. type DockerServer struct {

What happened: What you expected to happen: should not fail because of it's response size. How to reproduce it (as minimally and precisely as possible): N/A. Anything else we need to know?: Run batches through Argo workflow, but I don't think it's specific to the workloads. Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: Amazon Linux 2 OS (e.g: ):Kernel (e.g. ): <br<brMy question is: limit is 8 MB. limit is 16 MB. 's the difference between and ? is for Docker, and is for containerd? Can somebody help understand the difference? does have difference limit than ? I think we should give them the same limits. /cc
/sig node
+1 to make it the same. as reviewers mentioned, it's a temporary thing as the right way to fix it is to have some pagination which is not available currently
Thanks for the guidance - putting out a pr to bump the size rn.


Question: } timeout := endTime.Sub(time.Now()) <ins> errWaitTimeoutWithName := extendErrWaitTimeout(wait.ErrWaitTimeout, info) </ins> if timeout < 0 { // we're out of time <del> return gottenObj, false, wait.ErrWaitTimeout </del> <ins> return gottenObj, false, errWaitTimeoutWithName </ins> } ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), o.Timeout)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: linux/386 linux/arm linux/arm64 <del> #linux/ppc64le # temporarily disabled due to a linking error </del> <ins> linux/ppc64le </ins> darwin/amd64 darwin/386 windows/amd64

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: "reflect" "testing" <ins> "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" </ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> "k8s.io/apimachinery/pkg/runtime/schema" "sigs.k8s.io/yaml" </ins> ) func TestPrepareForUpdate(t *testing.T) {

doesn't do the same embedded resource and x-kubernetes-list-type "map" and "set" invariant validation that happens in custom resource create/update validation. Does that mean embedded objects and list-type fields under custom resource status can contain invalid data? if so, that needs to be fixed in a way that tolerates existing invalid data (ratcheting validation) Originally posted by in /sig api-machinery /kind bug /priority important-longterm
/triage accepted
I would like to collaborate for fixing the validation issue for custom resources and tolerate the existing invalid data. Could you please guide me with some code pointers or files and elaborate on how can I proceed for fixing this bug.
I haven't tested this, but something along the lines of this: we'd need tests to exercise the following scenarios on a CRD that specifies a status subresource with a list property with x-kubernetes-list-type=map and x-kubernetes-list-type=set a custom resource instance that already has duplicate items in the status property list can be updated successfully while still having duplicate items in the list (validation doesn't block an invalid → invalid status update) a custom resource instance that already has duplicate items in the status property list can be updated successfully to set unique items in the list (validation doesn't block an invalid → valid status update) a custom resource instance that does not have duplicate items in the status property list cannot be updated to set duplicate items in the list (validation blocks a valid → invalid status update)
since it's been a couple of months, i'm going to try and take a stab at this /assign
/assign
Hi k8s 1.25 Bug Triage Shadow here, just checking in to see if this is still on track for k8s 1.25? Thank you.
has stalled needing tests and a rebase
Thanks for the update.
Hello with the code freeze approaching, is this still on track for v1.25? Thanks!
not really, the status on is the same (needs tests)
/milestone clear


Question: ps.CPU = &statsapi.CPUStats{} } <del> ps.CPU.Time = cs.StartTime </del> <ins> ps.CPU.Time = cs.CPU.Time </ins> usageCoreNanoSeconds := getUint64Value(cs.CPU.UsageCoreNanoSeconds) + getUint64Value(ps.CPU.UsageCoreNanoSeconds) usageNanoCores := getUint64Value(cs.CPU.UsageNanoCores) + getUint64Value(ps.CPU.UsageNanoCores) ps.CPU.UsageCoreNanoSeconds = &usageCoreNanoSeconds

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: CPU.Time is set to cs.StartTime but ps.Memory.Time is set to timestamp when collecting the memoryusage. What you expected to happen: CPU.Time is set to cs.CPU.Time How to reproduce it (as minimally and precisely as possible): Check the code Anything else we need to know?: Environment: Kubernetes version (use ): master branch when creating this issue. Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig kubelet
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I am happy to take over this. /assign


Question: // despite different DestinationAddressPrefixes, in order to give it a chance to consolidate the two rules. func findSecurityRule(rules []network.SecurityRule, rule network.SecurityRule) bool { for _, existingRule := range rules { <del> if !strings.EqualFold(*existingRule.Name, *rule.Name) { </del> <ins> if !strings.EqualFold(to.String(existingRule.Name), to.String(rule.Name)) { </ins> continue } if existingRule.Protocol != rule.Protocol { continue } <del> if !strings.EqualFold(*existingRule.SourcePortRange, *rule.SourcePortRange) { </del> <ins> if !strings.EqualFold(to.String(existingRule.SourcePortRange), to.String(rule.SourcePortRange)) { </ins> continue } <del> if !strings.EqualFold(*existingRule.DestinationPortRange, *rule.DestinationPortRange) { </del> <ins> if !strings.EqualFold(to.String(existingRule.DestinationPortRange), to.String(rule.DestinationPortRange)) { </ins> continue } <del> if !strings.EqualFold(*existingRule.SourceAddressPrefix, *rule.SourceAddressPrefix) { </del> <ins> if !strings.EqualFold(to.String(existingRule.SourceAddressPrefix), to.String(rule.SourceAddressPrefix)) { </ins> continue } if !allowsConsolidation(existingRule) && !allowsConsolidation(rule) { <del> if !strings.EqualFold(*existingRule.DestinationAddressPrefix, *rule.DestinationAddressPrefix) { </del> <ins> if !strings.EqualFold(to.String(existingRule.DestinationAddressPrefix), to.String(rule.DestinationAddressPrefix)) { </ins> continue } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. The calls to create the load balancer were failing with a 400 What you expected to happen: controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was. How to reproduce it (as minimally and precisely as possible): Just created a cluster with an add on for the dashboard. Anything else we need to know?: logs from controller manager: I0423 12:50:28. ] Starting range CIDR allocator I0423 12:50:28. ] Waiting for caches to sync for cidrallocator controller I0423 12:50:28. ] Starting NoExecuteTaintManager I0423 12:50:28. ] Initializing eviction metric for zone: westeurope: :0 W0423 12:50:28. ] Missing timestamp for Node aks-nodepool1--0. Assuming now as a timestamp. I0423 12:50:28. ] Controller detected that zone westeurope: :0 is now in state Normal. I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"aks-nodepool1--0", UID:"-4476-11e8-9b1c-", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node aks-nodepool1--0 event: Registered Node aks-nodepool1--0 in Controller I0423 12:50:28. ] Caches are synced for ClusterRoleAggregator controller I0423 12:50:28. ] Caches are synced for endpoint controller I0423 12:50:28. ] Caches are synced for attach detach controller E0423 12:50:28. ]

Error creating spec for volume "mssqldb", pod "feature-test3"/"mssqlserver--r5r7l": error processing PVC "feature-test3"/"mssql-data": PVC feature-test3/mssql-data has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("") I0423 12:50:28. ] Caches are synced for deployment controller I0423 12:50:28. ] Caches are synced for persistent volume controller I0423 12:50:28. ] Caches are synced for ReplicationController controller I0423 12:50:28. ] Caches are synced for TTL controller I0423 12:50:28. ] Caches are synced for cidrallocator controller E0423 12:50:28. ] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"-4476-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"193", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"feature-initialproject", Name:"identityserver-service", UID:"-4485-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"7879", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer I0423 12:50:28. ] Caches are synced for namespace controller I0423 12:50:28. ] Caches are synced for service account controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ]

Caches are synced for garbage collector controller I0423 12:50:28. ] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Caches are synced for stateful set controller I0423 12:50:28. ] Caches are synced for disruption controller I0423 12:50:28. ] Sending events to api server. I0423 12:50:29. ] selectLoadBalancer: cluster(kubernetes) service(feature-initialproject/identityserver-service) isInternal(true) - availabilitysetsnames [nodepool1-availabilitySet-] E0423 12:50:30. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:509 /usr/local/go/src/runtime/asmamd64.s:2337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d] goroutine 1338 [running]: (0x0, 0x0, 0x0) +0x111 panic(0x40b2e00, 0xa603590) +0x283 (Cloud).getServiceLoadBalancerStatus(0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1) +0x30d (Cloud).EnsureLoadBalancer(0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40) +0x2ea (ServiceController).ensureLoadBalancer(0xc420f56000,

0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6) +0xcc (ServiceController).createLoadBalancerIfNeeded(0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492) +0x20e (ServiceController).processServiceUpdate(0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0) +0xeb (ServiceController).syncService(0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0) +0x3aa (ServiceController).worker.func1(0xc420f56000) +0xd9 (ServiceController).worker(0xc420f56000) +0x2b (ServiceController).()-fm() +0x2a (0xc420a04570) +0x5e (0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0) +0xbd (0xc420a04570, 0x3b9aca00, 0xc4208fa0c0) +0x4d created by (ServiceController).Run +0x20c Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Azure, AKS OS (e.g. from /etc/os-release): Ubunutu Kernel (e.g. ): xenial Install tools:AKS Others:
not sure what sig should be labeled in on this. Seems like a bug in sig-azure and whoever owns the piece getting the nil pointer/panic
what's the k8s version?
1.9.1
- I suspect that the error is here. This can be either one of two things A bug in SDK serialization (LB does have a private ip) but it does not get serialized correctly.

A bug in our assumption that that each internal LB ip fe config will have a valid internal IP. Can you dump the model for this internal LB here? Also -- did this error eventually recover? (it could be a race condition between setting ip and getting ip)
this was actually a customer cluster and they deleted it before I got a chance to get things like the model/what error code on the 400 was. And it didn't ever recover.
according to the evente table it was an internal loadbalancer here is the link to it --https://jarvis- maybe it the Details logged Can help you further
I was able to get more details from logs, {"error":{"code":"LoadBalancerFrontendIPConfigurationCountLimitReached","message":"Basic sku load balancer /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/loadBalancers/kubernetes-internal cannot have more than 10 FrontendIPConfigurations.","details":[]}}
Thanks - So there are two errors here. One the user can not use more than 10 IPs. The fact that we panic on this. i believe the fix is to log and fail the operation. /assign
The issue has been fixed in and has been included in v1.9.2. Could you upgrade your cluster (e.g. to latest v1.9.7) and check whether it fixes the issue?
I'm closing this issue since it's been resolved. /close
I am seeing this on 1.10.x and 1.11.x clusters: 10.6 11.2
The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that /reopen
Reopening this issue. <detailsIn response to : The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Added this to v1.12 since this is a critical urgent bug.


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e_node import ( "bytes" "fmt" "strings" "time" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = framework.KubeDescribe("Kubelet", func() { f := framework.NewDefaultFramework("kubelet-test") var podClient *framework.PodClient BeforeEach(func() { podClient = f.PodClient() }) Context("when scheduling a busybox command in a pod", func() { podName := "busybox-scheduling-" + string(uuid.NewUUID()) /* Release : v1.9 Testname: Kubelet, log output, default Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */ It("it should print the output to logs [NodeConformance]", func() { podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ // Don't restart the Pod since it is expected to exit RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Image:   busyboxImage, Name:    podName, Command: []string{"sh", "-c", "echo 'Hello World' ; sleep 240"}, }, }, }, }) Eventually(func() string { sinceTime := metav1.NewTime(time.Now().Add(time.Duration(-1 * time.Hour))) rc, err := podClient.GetLogs(podName, &v1.PodLogOptions{SinceTime: &sinceTime}).Stream() if err != nil { return "" } defer rc.Close() buf := new(bytes.Buffer) buf.ReadFrom(rc) return buf.String() }, time.Minute, time.Second*4).Should(Equal("Hello Worldn")) }) }) Context("when scheduling a busybox command that always fails in a pod", func() { var podName string BeforeEach(func() { podName = "bin-false" + string(uuid.NewUUID()) podClient.Create(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ // Don't restart the Pod since it is expected to exit RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Image:   busyboxImage, Name:    podName, Command: []string{"/bin/false"}, }, }, }, }) }) It("should have an error terminated reason [NodeConformance]", func() { Eventually(func() error { podData, err := podClient.Get(podName, metav1.GetOptions{}) if err != nil { return err } if len(podData.Status.ContainerStatuses) != 1 { return fmt.Errorf("expected only one container in the pod %q", podName) } contTerminatedState := podData.Status.ContainerStatuses[0].State.Terminated if contTerminatedState == nil { return fmt.Errorf("expected state to be terminated. Got pod status: %+v", podData.Status) } if contTerminatedState.Reason != "Error" { return fmt.Errorf("expected terminated state reason to be error. Got %+v", contTerminatedState) } return nil }, time.Minute, time.Second*4).Should(BeNil()) }) It("should be possible to delete [NodeConformance]", func() { err := podClient.Delete(podName, &metav1.DeleteOptions{}) Expect(err).To(BeNil(), fmt.Sprintf("Error deleting Pod %v", err)) }) }) Context("when scheduling a busybox Pod with hostAliases", func() { podName := "busybox-host-aliases" + string(uuid.NewUUID()) It("it should write entries to /etc/hosts [NodeConformance]", func() { podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ // Don't restart the Pod since it is expected to exit RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Image:   busyboxImage, Name:    podName, Command: []string{"/bin/sh", "-c", "cat /etc/hosts; sleep 6000"}, }, }, HostAliases: []v1.HostAlias{ { IP:        "123.45.67.89", Hostnames: []string{"foo", "bar"}, }, }, }, }) Eventually(func() error { rc, err := podClient.GetLogs(podName, &v1.PodLogOptions{}).Stream() defer rc.Close() if err != nil { return err } buf := new(bytes.Buffer) buf.ReadFrom(rc) hostsFileContent := buf.String() if !strings.Contains(hostsFileContent, "123.45.67.89tfoo") || !strings.Contains(hostsFileContent, "123.45.67.89tbar") { return fmt.Errorf("expected hosts file to contain entries from HostAliases. Got:n%+v", hostsFileContent) } return nil }, time.Minute, time.Second*4).Should(BeNil()) }) }) Context("when scheduling a read only busybox container", func() { podName := "busybox-readonly-fs" + string(uuid.NewUUID()) /* Release : v1.9 Testname: Kubelet, Pod with read only root file system Description: Create a Pod with security context set with ReadOnlyRootFileSystem set to true. The Pod then tries to write to the /file on the root, write operation to the root filesystem MUST fail as expected. */ It("it should not write to root filesystem [NodeConformance]", func() { isReadOnly := true podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ // Don't restart the Pod since it is expected to exit RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Image:   busyboxImage, Name:    podName, Command: []string{"/bin/sh", "-c", "echo test > /file; sleep 240"}, SecurityContext: &v1.SecurityContext{ ReadOnlyRootFilesystem: &isReadOnly, }, }, }, }, }) Eventually(func() string { rc, err := podClient.GetLogs(podName, &v1.PodLogOptions{}).Stream() if err != nil { return "" } defer rc.Close() buf := new(bytes.Buffer) buf.ReadFrom(rc) return buf.String() }, time.Minute, time.Second*4).Should(Equal("/bin/sh: can't create /file: Read-only file systemn")) }) }) }) </del>

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ### SEE ALSO * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra at 2015-05-28 22:43:52.329286408 +0000 UTC </del> <ins> ###### Auto generated by spf13/cobra at 2015-05-29 22:39:51.164275749 +0000 UTC </ins> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/kubectl_get.md?pixel)]()

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.