Question: for i, item := range table { var gotError error var gotPod *v1.Pod <ins> var gotForgetPod *v1.Pod </ins> var gotAssumedPod *v1.Pod var gotBinding *v1.Binding configurator := &FakeConfigurator{ Config: &Config{ SchedulerCache: &schedulertesting.FakeCache{ <ins> ForgetFunc: func(pod *v1.Pod) { gotForgetPod = pod }, </ins> AssumeFunc: func(pod *v1.Pod) { gotAssumedPod = pod },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { mockType.SecondClosestCommentLines = tt.tagComments <del> gotTag, gotMajor, gotMinor, err := extractKubeVersionTag(tt.tagName, mockType) </del> <ins> gotTag, gotMajor, gotMinor, err, fatalErr := safeExtractKubeVersionTag(tt.tagName, mockType) if (fatalErr != nil) != tt.wantFatal { t.Errorf("extractKubeVersionTag() fatalErr = %v, wantFatal %v", fatalErr, tt.wantFatal) return } if tt.wantFatal { return } </ins> if (err != nil) != tt.wantErr { t.Errorf("extractKubeVersionTag() error = %v, wantErr %v", err, tt.wantErr) return

The commands under will benefit of a batter unit test coverage as seen in Originally posted by in
/sig api-machinery /assign
GitHub didn't allow me to assign the following users: damiansawicki. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Hi I would like to work on this issue. Could you tell me how many test files are needed per PR under this issue?
Hi the issue is already assigned to me. Would you consider contributing to other issues to avoid duplication?
Hi do you mind if I work on this issue along with you? I would only like to write tests on limited number of files, if that's okay with you...
Hi thanks for your engagement! If you also want to work on this particular issue, we should coordinate. Are you working on something other than ?
Hey currently I'm thinking of shifting to Do let me know if you are already working on that particular directory. Also I could assist you in any of your work on this issue as I'm not preoccupied at the moment.


Question: "STANDARD_NV12_PROMO":    48, "STANDARD_NV12S_V2":      24, "STANDARD_NV12S_V3":      12, <ins> "STANDARD_NV16AS_V4":     32, </ins> "STANDARD_NV24":          64, "STANDARD_NV24_PROMO":    64, "STANDARD_NV24S_V2":      32, "STANDARD_NV24S_V3":      24, <ins> "STANDARD_NV32AS_V4":     32, </ins> "STANDARD_NV48S_V3":      32, <ins> "STANDARD_NV4AS_V4":      8, </ins> "STANDARD_NV6":           24, "STANDARD_NV6_PROMO":     24, "STANDARD_NV6S_V2":       12, <ins> "STANDARD_NV8AS_V4":      16, </ins> "STANDARD_PB6S":          12, }

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: func (cpso createPodSetsOp) patchParams(w *workload) (realOp, error) { if cpso.CountParam != "" { <del> var ok bool if cpso.Count, ok = w.Params[cpso.CountParam[1:]]; !ok { return nil, fmt.Errorf("parameter %s is undefined", cpso.CountParam) </del> <ins> var err error cpso.Count, err = w.Params.get(cpso.CountParam[1:]) if err != nil { return nil, err </ins> } } return &cpso, (&cpso).isValid(true)

Note: this issue should be viewed in the context of . If this happens, this is very likely a typo where someone used a parameter in a way they didn't mean to (for example, ). /sig scheduling /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi this is my first time contributing to kubernetes. can you provide me some guidance on how to to get started?
Hi, is this issue still valid to work on?
yes, this is still valid. I recommend starting with the and reading from there.
are you still working on this issue or should I work on it.
I would like to work on this issue if you haven't started already.
would it be ok if I took a look at this?
/assign
Hi Can you help me in why its not assigning the values to createNodes and createPods directly and using the opcode and countParam in performance-
We use a field to differentiate between different operations. The value of the field (e.g. "createNodes") determines the operation. The field defines the number of pods/nodes to create. What do you mean by "assigning the values to createNodes and createPods directly"?
Thanks for the information.

By assigning, I just meant that whether we can assign createNodes directly like ( createNodes: $initNodes ).
I don't think that's possible since there can potentially be other parameters besides the countParam. For example: Different operations could potentially have different parameters.
createNodes and createPods aren't the only operations. Other operations (e.g. churn and barrier), including future ones, may not fit that format. It's also easier to parse and more explicit this way.
Thanks , Having a doubt more, in performance- under the name:SchedulingPodAntiAffinity, ! why are we giving the same values in 5000Nodes as given in 500Nodes, or this is an error.
that sounds like a typo indeed
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello Are you still going to work on this issue?
Hi I have worked on the issue but not working for a while as was unsure about the exact problem.
, no problem if you are willing to work on this issue, shall unassign myself
/unassign
Thanks :)
/assign
/remove-lifecycle stale
hey! I am beginner and I interested in scheduling and I know basics of scheduling , so how can I start to contribute on this issue.
Sorry, PR for this issue has already created.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: "topology.kubernetes.io/zone":              "us-west-1a", "my.custom.label/foo":                      "fizz", "my.custom.label/bar":                      "foo", <ins> "app.k8snio/bar":                           "foo", "k8snio/bar":                               "foo", </ins> }, }, Spec: v1.NodeSpec{

For cloud provider nodes, there are some k8s label names are reserved. Like and The current regex is: This will match the dot . as any single character. This block adding a label name like: app.k8snio/name The regex format is incorrect, we should escape the dot in regex so that it can match the dot as a string. Can add additional label with name like: app.k8snio/name Configure AdditionalLabels in the instanceMeta to add additional label name with app.k8snio/name. No response <detailsClient Version: v1.30.1 Kustomize Version: v5.0.4-0.- Server Version: v1.30.0 </details<details</details<details</details<details</details<details</details<details</details
/sig cloud-provider
If it make sense to fix it, i would like to take it and raise a PR. /assign
Which Kubernetes component has the bug ? (eg: kubelet) Can you explain how to reproduce the problem?
It's the node controller in the shared library Though I really don't think cloud providers should be setting node labels like "", this is technically incorrect. This restriction is only applied to labels the cloud provider controller binary running this controller would attempt to add itself.
just noticed this after our SIG meeting today, cc
/triage accepted
please go ahead and raise a PR, thanks for reporting this =)
When fixing this, consider if there are any labels that cloud provider integration might legitimately set. For example: (documented as being set by cloud provider integration code). Some labels are OK to set; we could allowlist and explicitly. When rejecting some others, maybe setting a Condition on the Node ( perhaps) would help surface what's not right. Not sure. Making the simple fix is also fine by me.
OK. Thanks I have raised a PR for this: Not sure if this change looks good.


Question: var DNS952LabelErrorMsg string = fmt.Sprintf(`must be a DNS 952 label (at most %d characters, matching regex %s): e.g. "my-name"`, validation.DNS952LabelMaxLength, validation.DNS952LabelFmt) var pdPartitionErrorMsg string = InclusiveRangeErrorMsg(1, 255) var PortRangeErrorMsg string = InclusiveRangeErrorMsg(1, 65535) <ins> var IdRangeErrorMsg string = InclusiveRangeErrorMsg(0, math.MaxInt32) </ins> var PortNameErrorMsg string = fmt.Sprintf(`must be an IANA_SVC_NAME (at most 15 characters, matching regex %s, it must contain at least one letter [a-z], and hyphens cannot be adjacent to other hyphens): e.g. "http"`, validation.IdentifierNoHyphensBeginEndFmt) const totalAnnotationSizeLimitB int = 256 * (1 << 10) // 256 kB

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: "net" "github.com/golang/glog" <ins> kubeoptions "k8s.io/kubernetes/pkg/kubeapiserver/options" </ins> "k8s.io/kubernetes/pkg/registry/core/service/ipallocator" )

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: There is currently no easy way to know what Kubernetes is using as the default CIDR for service cluster IPs. What you expected to happen: should be able to tell users that "10.0.0.0/24" is used as the default. They'd better check if their deployment environment has a conflict with this assumption. They need to customize this parameter to avoid such overlaps if any. How to reproduce it (as minimally and precisely as possible): Run , from the help, the default value for --service-cluster-ip-range is not mentioned. Anything else we need to know?: No. Environment: Kubernetes version (use ): Any Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: N/A Others: N/A
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale


Question: kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/nodestatus" "k8s.io/kubernetes/pkg/kubelet/util/sliceutils" <ins> kubeletvolume "k8s.io/kubernetes/pkg/kubelet/volumemanager" </ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" taintutil "k8s.io/kubernetes/pkg/util/taints" "k8s.io/kubernetes/pkg/version"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 2019-01-15 (when they were ) Testgrid link: Reason for failure: They all seem to be failing with: The failures seem to start with .
/sig storage
Summoning and This seems to have been failing since it was in master-blocking.
This is the same as
Thanks closing this one. /close
Closing this issue. <detailsIn response to : Thanks closing this one. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I took a look at the latest test failures and I think it is a different issue than
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at one test failure here: Sequence of events: stopped, pod deleted: started again: volume to dsw started and failed because it's not in the node's volumesinuse list. This repeats for a while. also started, but we skipped cleaning up because the volume is in dsw so we want to let volume manager gracefully cleanup don't remove volume from dsw because actual state has not been updated? So it seems like there's a problem with populating the node's volumesinuse after a kubelet restart.
cc
/assign will continue investigating.

i will try repro-ing at without different commits to try to figure out which pr introduced the regression
/priority important-soon
I ran these tests against a cluster with the NodeLease feature disabled, and the tests pass. So I think the issue is related to the new NodeLease feature enabled a month ago: cc
I discussed with VolumesInUse is updated periodically in the Node.Status. So we need to make sure this flow is working well with the NodeLease feature.
and I walked through the code. The NodeLease feature introduced an early exit to the node status update function, so our theory is we need to add a call to . I will make the change and test it out


Question: }, nil } <del> func getThresholdEvents(eventfd int, eventCh chan<- int) { </del> <ins> func getThresholdEvents(eventfd int, eventCh chan<- struct{}, stopCh <-chan struct{}) { </ins> for { buf := make([]byte, 8) _, err := syscall.Read(eventfd, buf) if err != nil { return } <del> eventCh <- 0 </del> <ins> select { case eventCh <- struct{}{}: case <-stopCh: return } </ins> } } func (n *memcgThresholdNotifier) Start(stopCh <-chan struct{}) { <del> eventCh := make(chan int, 1) go getThresholdEvents(n.eventfd, eventCh) </del> <ins> eventCh := make(chan struct{}) go getThresholdEvents(n.eventfd, eventCh, stopCh) </ins> for { select { case <-stopCh:

close it may make kubelet crash, though the possibility is small. keep it open is just ok. And to avoid getThresholdEvents blocked for ever, stopCh should be passed in it.
Thanks for the PR!


Question: } else if line[0] == ':' && len(line) > 1 { // We assume that the <line> contains space - chain lines have 3 fields, // space delimited. If there is no space, this line will panic. <del> chain := Chain(line[1:bytes.Index(line, spaceBytes)]) </del> <ins> spaceIndex := bytes.Index(line, spaceBytes) if spaceIndex == -1 { panic(fmt.Sprintf("Unexpected chain line in iptables-save output: %v", string(line))) } chain := Chain(line[1:spaceIndex]) </ins> chainsMap[chain] = line } }

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved


Question: cmdArgs = append(cmdArgs, "--kubeconfig", kubeconfigPath, "--root-dir", KubeletRootDirectory, <del> "--docker-disable-shared-pid=false", </del> "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", "--allow-privileged", "true", )

Once the API for is finalized, add a node e2e test. /kind feature /assign /cc /sig node
We plan on adding this for 1.10


Question: framework.ExpectNoError(err, "failed to get pod %s", pod.Name) ginkgo.By("Reading file content from the nginx-container") <del> resultString, err = framework.LookForStringInFile(f.Namespace.Name, pod.Name, busyBoxMainContainerName, busyBoxMainVolumeFilePath, message, 30*time.Second) framework.ExpectNoError(err, "failed to match expected string %s with %s", message, resultString) </del> <ins> result := f.ExecShellInContainer(pod.Name, busyBoxMainContainerName, fmt.Sprintf("cat %s", busyBoxMainVolumeFilePath)) framework.ExpectEqual(result, message, "failed to match expected string %s with %s", message, resultString) </ins> }) })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes [It] pod should support shared volumes between containers [Conformance] Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know:
please don't CC sig-testing on every test failure :( we own the infrastructure etc. the sig-storage owns these tests.


Question: - /run.sh resources: limits: <del> cpu: "0.5" image: gcr.io/google_containers/cassandra:v4 </del> <ins> cpu: "0.1" image: gcr.io/google_containers/cassandra:v5 </ins> name: cassandra ports: - name: cql

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: # Kubernetes <del> [![GoDoc](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes?status.png)](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes) [![Travis](https://travis-ci.org/GoogleCloudPlatform/kubernetes.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/kubernetes) </del> <ins> [![GoDoc](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes?status.png)](https://godoc.org/github.com/GoogleCloudPlatform/kubernetes) [![Travis](https://travis-ci.org/GoogleCloudPlatform/kubernetes.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/kubernetes) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/kubernetes/badge.svg)](https://coveralls.io/r/GoogleCloudPlatform/kubernetes) </ins> Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.

Right now, we execute unit tests in Travis with , and you can see the results on each line: but we don't have an easy place to go browse our code and figure out what's covered and what isn't. Is there some awesome way to do this? At the very least, can we get the actual line-by-line coverage results integrated into our CI build process so we can poke at these somehow?
cc
I'm willing to take a look into this to at least see if something is possible.
Looking into using Coveralls; there are a few other teams at Google already using this. Integration with Travis and Go are supported with the goveralls package:
Unit test coverage can now be viewed at - both master push builds and PRs will be shown there; additionally, Coveralls will update the status and post comments on PRs with coverage stats. (If we find these too spammy, we can disable either or both.) The top-level also has a badge/link to the coverage dashbaord.


Question: <ins> apiVersion: v1 kind: ReplicationController metadata: name: dns-backend labels: name: dns-backend spec: replicas: 1 selector: name: dns-backend template: metadata: labels: name: dns-backend spec: containers: - name: dns-backend image: k8s.gcr.io/example-dns-backend:v1 ports: - name: backend-port containerPort: 8000 </ins>

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: cadvisorapiv1 "github.com/google/cadvisor/info/v1" cadvisorapiv2 "github.com/google/cadvisor/info/v2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/klog/v2" statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" "k8s.io/kubernetes/pkg/kubelet/cadvisor" <ins> "k8s.io/kubernetes/pkg/kubelet/server/stats" </ins> ) // defaultNetworkInterfaceName is used for collectng network stats.

I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes.

I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ).

The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet?

and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes.

It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is .

However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.


Question: "format": "byte" }, "service": { <del> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnIf there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error.", </del> <ins> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnPort 443 will be used if it is open, otherwise it is an error.", </ins> "$ref": "#/definitions/io.k8s.api.admissionregistration.v1beta1.ServiceReference" }, "url": {

What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object


Question: - watch: - pkg: docker-io - pkg: openvswitch {% endif %}

Which jobs are failing: capa-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-aws-make-conformance-v1alpha3-k8s-ci-artifacts) Which test(s) are failing: [] Pods should get a host IP [NodeConformance] [Conformance] Since when has it been failing: 06-06 17:23 PDT Testgrid link: Reason for failure: <*errors.errorString | 0xc0000daf10: { s: "pod ran to completion", } pod ran to completion occurred skipped 89 lines unfold_more Jun 6 10:07:34.676: INFO: Latency metrics for node ip-10-0-0- Jun 6 10:07:34.676: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "pods-7990" for this suite. Anything else we need to know: /sig testing /cc /priority critical-urgent /milestone v1.19
/sig scheduling observing a timeout Jun 10 05:42:34.202: INFO: found a healthy node: test1-md-0-4qcf5 [It] runs ReplicaSets to verify preemption running path [Conformance] Jun 10 05:42:42.675: INFO: pods created so far: [1 1 1] Jun 10 05:42:42.675: INFO: length of pods created so far: 3 Jun 10 05:44:42.765: FAIL: pod sched-preemption-path-3834/pod4 failed to preempt other pods Unexpected error: <*errors.errorString | 0xc00011bdd0: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace (0xc000e48f20, 0xc001dbb800, 0x1bf08eb000) +0x16b skipped 318 lines unfold_more Jun 10 05:44:44.847: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "sched-preemption-4796" for this suite.

[AfterEach] [sig-scheduling] SchedulerPreemption [Serial]
/remove-sig scheduling I observing failure since yesterday as the cluster failed to get created using the CloudFormation template Attempting to create CloudFormation stack cluster-api-provider-aws-sigs-k8s-io Error: failed to create AWS CloudFormation stack: ResourceNotReady: failed waiting for successful resource stateError: failed to create AWS CloudFormation stack: ResourceNotReady: failed waiting for successful resource state
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig remove scheduling Error: failed to create AWS CloudFormation stack: ResourceNotReady: failed waiting for successful resource stateError: failed to create AWS CloudFormation stack: ResourceNotReady: failed waiting for successful resource state` Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig scheduling
/assign
/remove-sig testing SIG testing doesn't own CAPA bringup failures
/sig cluster-lifecycle
It is expected that will provide a fix /cc
/assign
/lifecycle active


Question: } }) } <ins> } func TestKonnectivityClientMetrics(t *testing.T) { testcases := []struct { name    string metrics []string trigger func() want    string }{ { name:    "stream packets", metrics: []string{"konnectivity_network_proxy_client_stream_packets_total"}, trigger: func() { clientmetrics.Metrics.ObservePacket(ccmetrics.SegmentFromClient, client.PacketType_DIAL_REQ) }, want: ` # HELP konnectivity_network_proxy_client_stream_packets_total Count of packets processed, by segment and packet type (example: from_client, DIAL_REQ) # TYPE konnectivity_network_proxy_client_stream_packets_total counter konnectivity_network_proxy_client_stream_packets_total{packet_type="DIAL_REQ",segment="from_client"} 1 `, }, { name:    "stream errors", metrics: []string{"konnectivity_network_proxy_client_stream_errors_total"}, trigger: func() { clientmetrics.Metrics.ObserveStreamError(ccmetrics.SegmentToClient, errors.New("example"), client.PacketType_DIAL_RSP) }, want: ` # HELP konnectivity_network_proxy_client_stream_errors_total Count of gRPC stream errors, by segment, grpc Code, packet type. (example: from_agent, Code.Unavailable, DIAL_RSP) # TYPE konnectivity_network_proxy_client_stream_errors_total counter konnectivity_network_proxy_client_stream_errors_total{code="Unknown",packet_type="DIAL_RSP",segment="to_client"} 1 `, }, { name:    "dial failure", metrics: []string{"konnectivity_network_proxy_client_dial_failure_total"}, trigger: func() { clientmetrics.Metrics.ObserveDialFailure(clientmetrics.DialFailureTimeout) }, want: ` # HELP konnectivity_network_proxy_client_dial_failure_total Number of dial failures observed, by reason (example: remote endpoint error) # TYPE konnectivity_network_proxy_client_dial_failure_total counter konnectivity_network_proxy_client_dial_failure_total{reason="timeout"} 1 `, }, { name:    "client connections", metrics: []string{"konnectivity_network_proxy_client_client_connections"}, trigger: func() { clientmetrics.Metrics.GetClientConnectionsMetric().WithLabelValues("dialing").Inc() }, want: ` # HELP konnectivity_network_proxy_client_client_connections Number of open client connections, by status (Example: dialing) # TYPE konnectivity_network_proxy_client_client_connections gauge konnectivity_network_proxy_client_client_connections{status="dialing"} 1 `, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { tc.trigger() if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, strings.NewReader(tc.want), tc.metrics...); err != nil { t.Errorf("GatherAndCompare error: %v", err) } }) } </ins> }

At (originally ) component-base metrics exposed Registerer API for raw prometheus registration, and I took dependency in The resulting does not seem to emit the new metrics (example: ). The metrics handler should emit the registered metrics. Falling back to does work. So there is some material difference with . 24, 1.25, 1.26 GKE No response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign jkh52


Question: RUN clean-install bash ADD kube-addons.sh /opt/ <del> ADD namespace.yaml /opt/ </del> ADD kubectl /usr/local/bin/ CMD ["/opt/kube-addons.sh"]

What happened: Add-on manager uses kubectl apply to create the kube-system namespace at start-up. This is redundant because the kube-apiserver creates the namespace in NewBootstrapController. This is a problem because if someone uses apply to make changes to the kube-system namespace, the add-on manager will wipe out these changes. What you expected to happen: Kube-APIServer has the sole responsibility of creating the kube-system namespace. How to reproduce it (as minimally and precisely as possible): kubectl apply a label to the kube-system namespace. restart the add-on manager. Note the labels have been removed. Anything else we need to know?: /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
the contention script wipes out namespace ^^
/cc
i guess it is intentional to have the addon manager to ensure kube-system is not modified, it tries to avoid the case that some cluster admin accidentally delete kube-system that result in a non-functional cluster.
The add-on manager will only perform the check when it restarts. The API Server will also perform this check when it restarts. I think we should only have 1 system responsible for this and I believe that system should be the API server.
trying to have my first contribute here, so is it proper to have the kube-system existence checked before applying the yaml file to create the kube-system namespace once again?
The Kube API Server will check for the existence of the kube-system namespace and create it if absent. It does all of this using client-go requests. The Add-On manager is the only system creating the kube-system by applying (a private) yaml file. I believe the correct fix here is to remove the Add-On managers attempts to create the kube-system namespace entirely. This is especially important if a user writes a CI/CD controller to apply their desired state to the kube-system namespace. I would argue the CI/CD is correct but the add-on manager will randomly wipe out the users changes. The API Server on the other hand will not.
, thanks for your help, so i will remove that line and pull a request later
Trying to get my first PR here. Is the issue still open ?
The issue is still open. You may want to check on though as was trying to fix the issue with that PR.


Question: // Construct the resulting typed, unversioned object. objToUpdate := p.restPatcher.New() if err := runtime.DecodeInto(p.codec, patchedObjJS, objToUpdate); err != nil { <del> return nil, err </del> <ins> return nil, errors.NewInvalid(schema.GroupKind{}, "", field.ErrorList{ field.Invalid(field.NewPath("patch"), string(patchedObjJS), err.Error()), }) </ins> } if p.fieldManager != nil {

/kind bug /sig api-machinery /cc What happened: patching (SMP) any object's annotation using invalid patch content will result in an (500) internal-server error. expecting a (400) bad-request error in this case. patch body: received error: the error is thrown from converter instead of the patcher, so it seems that patcher considers the body as valid then the converter complained. Anything else we need to know?: when my grandpa was 65 years old he started running one mile a day to keep fit. he's 70 now and we have no idea where he is.
I just tried the same patch with "MergePatchType", and it also fails with 500.
I can't make it fail with JSONPatchType, I get a 422
I suspect the error should read "the server rejected your request due to an error in our request"


Question: } ``` <del> Once you have that pod file, you can create the redis pod in your Kubernetes cluster using the `cloudcfg` cli: </del> <ins> Once you have that pod file, you can create the redis pod in your Kubernetes cluster using the `cloudcfg` CLI: </ins> ```shell <del> cluster/cloudcfg.sh -c redis-master.json create /pods </del> <ins> $ cluster/cloudcfg.sh -c examples/guestbook/redis-master.json create /pods </ins> ``` Once that's up you can list the pods in the cluster, to verify that the master is running:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: --project "${PROJECT}"  --region "${REGION}"  --quiet  <del> "${MASTER_NAME}-ip" || true </del> <ins> "${MASTER_NAME}-ip" </ins> export CONTEXT="${PROJECT}_${INSTANCE_PREFIX}" clear-kubeconfig <ins> set -e </ins> } # Update a kubernetes cluster with latest source

'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".


Question: // IsConflict determines if the err is an error which indicates the provided update conflicts. // It supports wrapped errors. func IsConflict(err error) bool { <del> return ReasonForError(err) == metav1.StatusReasonConflict </del> <ins> reason, code := reasonAndCodeForError(err) if reason == metav1.StatusReasonConflict { return true } if _, ok := knownReasons[reason]; !ok && code == http.StatusConflict { return true } return false </ins> } // IsInvalid determines if the err is an error which indicates the provided resource is not valid. // It supports wrapped errors. func IsInvalid(err error) bool { <del> return ReasonForError(err) == metav1.StatusReasonInvalid </del> <ins> reason, code := reasonAndCodeForError(err) if reason == metav1.StatusReasonInvalid { return true } if _, ok := knownReasons[reason]; !ok && code == http.StatusUnprocessableEntity { return true } return false </ins> } // IsGone is true if the error indicates the requested resource is no longer available. // It supports wrapped errors. func IsGone(err error) bool { <del> return ReasonForError(err) == metav1.StatusReasonGone </del> <ins> reason, code := reasonAndCodeForError(err) if reason == metav1.StatusReasonGone { return true } if _, ok := knownReasons[reason]; !ok && code == http.StatusGone { return true } return false </ins> } // IsResourceExpired is true if the error indicates the resource has expired and the current action is

Checked a method error using when the error was caused by a validating webhook that returns a 403 response. In this particular case, the error was generated from a validating webhook implemented with controller-runtime that returns controller-runtime's response. to return true. returned false Many (but not all) of the other family of functions in suffer the same kind of problem. I have changes ready to submit in a PR if we can get agreement that this is a problem and that it should be fixed in the functions that ought to check the of an erorr value that is or wraps an Kubernetes version (use ): 19.3 for the server
/sig api-machinery
are you going to send a PR for this? The issue description looks reasonable. /triage accepted
Thanks, PR is submitted.
/assign


Question: <ins> apiVersion: v1beta3 kind: Service metadata: name: openshift spec: ports: - port: 8443 name: openshift targetPort: 8443 selector: name: openshift createExternalLoadBalancer: true </ins>

This PR was prematurely merged: Need to update the PR based on the actual steps that were needed to get what was demonstrated at the public Kubernetes meet-up last week.
Any chance you can get rid of the hostPort from the example?
- depending on the Kubernetes provider, yes. For Vagrant based provider, I think I still need the hostPort because I do not have routes. For the GCE provider, I should be able to get rid of it because I have routes. In both providers, there were a number of setup steps that did not make the merged PR. For those that are tracking this issue (which I know folks have pinged me on separately) needs to be configured to allow privelaged containers needs to be configured to allow pushing to an insecure registry (this is point in time) Ideally, I could use a ReplicationController in the example to load balance OpenShift, but until we can provide a pattern to run etcd as a pod, I am unable to do this and need to use the embedded etcd we end up running.
Note: I will resolve this issue in 2 phases. Vagrant (which was demonstrated) to document using GCE (which has some additional steps on firewall rules)
Turns out OpenShift Origin is too secure now ;-) This is blocked until OpenShift Origin allows you to mint certs on the client, and then supply the certs on startup as a secret to the pod. This feature is coming soon. /cc
I now have all the required pieces to get this to work. I hope to get a PR submitted to fix this issue by end of this week.


Question: errNotFound = errors.NewNotFound(gvr.GroupResource(), "pod") assert.EqualError(t, err, errNotFound.Error()) } <ins> func Test_resourceCovers(t *testing.T) { type args struct { resource string action   Action } tests := []struct { name string args args want bool }{ { args: args{ resource: "*", action:   ActionImpl{}, }, want: true, }, { args: args{ resource: "serviceaccounts", action:   ActionImpl{}, }, want: false, }, { args: args{ resource: "serviceaccounts", action: ActionImpl{ Resource: schema.GroupVersionResource{ Resource: "serviceaccounts", }, }, }, want: true, }, { args: args{ resource: "serviceaccounts/token", action: ActionImpl{ Resource: schema.GroupVersionResource{}, }, }, want: false, }, { args: args{ resource: "serviceaccounts/token", action: ActionImpl{ Resource: schema.GroupVersionResource{ Resource: "serviceaccounts", }, }, }, want: false, }, { args: args{ resource: "serviceaccounts/token", action: ActionImpl{ Resource:    schema.GroupVersionResource{}, Subresource: "token", }, }, want: false, }, { args: args{ resource: "serviceaccounts/token", action: ActionImpl{ Resource: schema.GroupVersionResource{ Resource: "serviceaccounts", }, Subresource: "token", }, }, want: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := resourceCovers(tt.args.resource, tt.args.action); got != tt.want { t.Errorf("resourceCovers() = %v, want %v", got, tt.want) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Can't match "serviceaccounts/token", can only match "serviceaccounts" Can match "serviceaccounts/token"
/kind feature
/sig testing
/assign
/triage accepted


Question: } type effectiveVersion struct { <ins> // When true, BinaryVersion() returns the current binary version useDefaultBuildBinaryVersion atomic.Bool // Holds the last binary version stored in Set() </ins> binaryVersion atomic.Pointer[version.Version] // If the emulationVersion is set by the users, it could only contain major and minor versions. // In tests, emulationVersion could be the same as the binary version, or set directly,

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: fi IFS="$OLD_IFS" <del> IFS="," __debug "RET=${RET[@]} len=${#RET[@]}" </del> <ins> IFS="," __kubectl_debug "RET=${RET[@]} len=${#RET[@]}" </ins> for w in ${RET[@]}; do if [[ ! "${w}" = "${cur}"* ]]; then

/kind bug I wasn't able to find issue opened for ZSH case, and aware about similar, fixed, case with bash in 1.10.0-b1 bash completion is working fine
/sig cli
can you verify this and label for v1.10 milestone and note if it will be considered a release blocker
I cannot reproduce this on my local machine (zsh 5.3.1 x86_64-apple-darwin16.3.0). Please produce a docker container (with Dockerfile) that shows this issue.
And another important question for the release: can you try with the 1.9 kubectl? Is that working in your setup?
little bit more info: i'm running this on CentOS 7.4 1708 x64 and zsh from official repo same issue happenes with 1.10.0-b3: 3
From sig-cli Slack: Kazuki Suda [01:22] It seems that have to be renamed to due to changes contained (edited)
You can reproduce this with the below:
was renamed to at , but has been used yet in completions generated by . So we have to replace to .
issue still exists with v10.0.0-b4
10.0-beta.4 does not contain the fix, yet.
Because the fix is not cherry-picked to release-1.10 yet.


Question: // ProgressReporter is a ginkgo reporter which tracks the total number of tests to be run/passed/failed/skipped. // As new tests are completed it updates the values and prints them to stdout and optionally, sends the updates // to the configured URL. <del> // TODO: Number of test specs is not available now, we can add it back when this is fixed in the Ginkgo V2. // pls see: https://github.com/kubernetes/kubernetes/issues/109744 </del> <ins> // One known limitation of the ProgressReporter it this reporter will not consolidate the reporter from each sub-process // if the tests are run in parallel. // As what's observed the reporter sent before test suite is started will be assembled correctly but each process will report // on its own after each test or suite are completed. // Here is a sample report that 5 testcases are executed totally, and run in parallel with 3 procs, // 3 of them are failed and other 2 passed. // {"msg":"","total":5,"completed":0,"skipped":0,"failed":0} // {"msg":"Test Suite starting","total":5,"completed":0,"skipped":0,"failed":0} // {"msg":"FAILED [sig-node] NoExecuteTaintManager Single Pod doesn't evict pod with tolerations from tainted nodes","total":0,"completed":0,"skipped":1332,"failed":1,"failures":["[sig-node] NoExecuteTaintManager..."]} // {"msg":"FAILED [sig-node] NoExecuteTaintManager Single Pod evicts pods from tainted nodes","total":5,"completed":0,"skipped":2524,"failed":1,"failures":["[sig-node] NoExecuteTaintManager Single Pod evicts pods from tainted nodes"]} // {"msg":"PASSED [sig-node] NoExecuteTaintManager Single Pod removing taint cancels eviction [Disruptive] [Conformance]","total":0,"completed":1,"skipped":1181,"failed":0} // {"msg":"Test Suite completed","total":0,"completed":1,"skipped":2592,"failed":0} // {"msg":"PASSED [sig-node] NoExecuteTaintManager Single Pod eventually evict pod with finite tolerations from tainted nodes","total":0,"completed":1,"skipped":1399,"failed":1,"failures":["[sig-node] NoExecuteTaintManager..."]} // {"msg":"Test Suite completed","total":0,"completed":1,"skipped":1399,"failed":1,"failures":["[sig-node] NoExecuteTaintManager Single Pod doesn't evict pod with tolerations from tainted nodes"]} // {"msg":"FAILED [sig-node] NoExecuteTaintManager Single Pod pods evicted from tainted nodes...","total":5,"completed":0,"skipped":3076,"failed":2,"failures":["[sig-node] NoExecuteTaintManager...","[sig-node] NoExecuteTaintManager..."]} // {"msg":"Test Suite completed","total":5,"completed":0,"skipped":3076,"failed":2,"failures":["[sig-node] NoExecuteTaintManager Single Pod evicts pods from tainted nodes","[sig-node] NoExecuteTaintManager..."]} </ins> type ProgressReporter struct { LastMsg string `json:"msg"` <ins> TestsTotal     int `json:"total"` </ins> TestsCompleted int `json:"completed"` TestsSkipped   int `json:"skipped"` TestsFailed    int `json:"failed"`

Our team uses sonobuoy to run the conformormance tests on a cluster and displays the results to the users. We have been using conformance image 1.21.x until now and it sends back progress data to sonobuoy pod and our service reads it from the sonobuoy pod. The progress data included the total number of tests to be run and the completed tests. For 1.21 conformance we were receiving this data properly but when switching to 1.25.x conformance we are getting the total tests as zero. This is causing issues in our UI. Not sure what version of conformance image introduced this but we were using 1.21 and started seeing this issue after moving to 1.25.x. The conformance image versions in between 1.21 -1.25 are not CVE free so we cannot use them. Total number of tests should be non-zero for conformance image 1.25.x. Ideally i think it should come back as 360. provision a 1.25 kubernetes kind (or anything else) cluster Install sonobuoy Run conformance scan on the cluster Check the annotation on the sonobuoy pod . This is set by sonobuoy based on the data it gets back from the e2e conformance pod. The total tests comes as zero I just found it on 1.25.x conformance. Not really sure when this was introduced. Need to make sure back dated versiosn also have this fix in case they have this issue. <details25 cluster with 1.25.x conformance with any sonobuoy version </details<detailsANY </details<details</details<details</details<details</details<detailsConformacne e2e plugin </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
FYI
Yeah, it seems that it gets the info here I vaguely remember something about a progress custom reporter that was by sonobuoy and we didn't know what was used for
You remember correctly: It looks like that TODO didn't get addressed? I thought it had been. the other thing that was unclear regarding sonobuoy was whether it reads these test statistics from stderr or gets them via HTTP. We removed the stderr output (I think in 1.26) but kept (). This seems to be backed by (sonobuoy sets the parameter), but it would be good to get definite confirmation. Also, does Sonobuoy run tests in parallel? If it does, then these statistics are probably useless or at least will need considerable post-processing.
We depend on for this.
/assign I will take a look as has been fixed and closed
/reopen The fix went in, but let's keep the issue open for a while longer until we really know that it now works as expected. /assign
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
/remove-lifecycle rotten I discussed this further with (see also ) and while a final decision is pending, I am inclined to remove the progress reporting feature because I suspect that it is conceptually broken (to be confirmed) and lacks in-tree support (unit tests, owner). Before I do that, can you clarify: Are you running tests in parallel and does progress reporting work properly, i.e. does it go from 0 to 100%? When you said, "causing issues in our UI", what UI is that?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Ping Please see
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package auth import ( "fmt" "time" "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) const ( NodesGroup     = "system:nodes" NodeNamePrefix = "system:node:" ) var _ = SIGDescribe("[Feature:NodeAuthorizer]", func() { f := framework.NewDefaultFramework("node-authz") // client that will impersonate a node var c clientset.Interface var ns string var asUser string var defaultSaSecret string var nodeName string BeforeEach(func() { ns = f.Namespace.Name nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(len(nodeList.Items)).NotTo(Equal(0)) nodeName = nodeList.Items[0].Name asUser = NodeNamePrefix + nodeName sa, err := f.ClientSet.CoreV1().ServiceAccounts(ns).Get("default", metav1.GetOptions{}) Expect(len(sa.Secrets)).NotTo(Equal(0)) Expect(err).NotTo(HaveOccurred()) defaultSaSecret = sa.Secrets[0].Name By("Creating a kubernetes client that impersonates a node") config, err := framework.LoadConfig() Expect(err).NotTo(HaveOccurred()) config.Impersonate = restclient.ImpersonationConfig{ UserName: asUser, Groups:   []string{NodesGroup}, } c, err = clientset.NewForConfig(config) Expect(err).NotTo(HaveOccurred()) }) It("Getting a non-existent secret should exit with the Forbidden error, not a NotFound error", func() { _, err := c.CoreV1().Secrets(ns).Get("foo", metav1.GetOptions{}) Expect(apierrors.IsForbidden(err)).Should(Equal(true)) }) It("Getting an existent secret should exit with the Forbidden error", func() { _, err := c.CoreV1().Secrets(ns).Get(defaultSaSecret, metav1.GetOptions{}) Expect(apierrors.IsForbidden(err)).Should(Equal(true)) }) It("Getting a secret for a workload the node has access to should succeed", func() { By("Create a secret for testing") secret := &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ Namespace: ns, Name:      "node-auth-secret", }, Data: map[string][]byte{ "data": []byte("keep it secret"), }, } _, err := f.ClientSet.CoreV1().Secrets(ns).Create(secret) Expect(err).NotTo(HaveOccurred()) By("Node should not get the secret") _, err = c.CoreV1().Secrets(ns).Get(secret.Name, metav1.GetOptions{}) Expect(apierrors.IsForbidden(err)).Should(Equal(true)) By("Create a pod that use the secret") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pause", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pause", Image: framework.GetPauseImageName(f.ClientSet), }, }, NodeName: nodeName, Volumes: []v1.Volume{ { Name: "node-auth-secret", VolumeSource: v1.VolumeSource{ Secret: &v1.SecretVolumeSource{ SecretName: secret.Name, }, }, }, }, }, } _, err = f.ClientSet.CoreV1().Pods(ns).Create(pod) Expect(err).NotTo(HaveOccurred()) By("The node should able to access the secret") err = wait.Poll(framework.Poll, 1*time.Minute, func() (bool, error) { _, err = c.CoreV1().Secrets(ns).Get(secret.Name, metav1.GetOptions{}) if err != nil { framework.Logf("Failed to get secret %v, err: %v", secret.Name, err) return false, nil } return true, nil }) Expect(err).NotTo(HaveOccurred()) }) It("A node shouldn't be able to create an other node", func() { node := &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, TypeMeta: metav1.TypeMeta{ Kind:       "Node", APIVersion: "v1", }, } By(fmt.Sprintf("Create node foo by user: %v", asUser)) _, err := c.CoreV1().Nodes().Create(node) Expect(apierrors.IsForbidden(err)).Should(Equal(true)) }) It("A node shouldn't be able to delete an other node", func() { By(fmt.Sprintf("Create node foo by user: %v", asUser)) err := c.CoreV1().Nodes().Delete("foo", &metav1.DeleteOptions{}) Expect(apierrors.IsForbidden(err)).Should(Equal(true)) }) }) </ins>

Is this a BUG REPORT or FEATURE REQUEST? (choose one): TESTING REQUEST Spawned from: It would be nice with some e2e tests for the Node Authorizer so consumers of it (kubeadm, GCE, GKE, etc.) could easily make sure it's enabled and working, just by editing when running e2e's. I wrote a quick bash script of what it could do: The bash script above is in no way comprehensive enough, but on the other hand it took ~5mins to write :) For now (1.7), we might use something like this bash script in our kubeadm e2e CI job to make sure the Node Authorizer/Admission plugin is enabled correctly. cc
was assigned for triaging this... Does such an e2e test already exist somewhere? said that he could take this on if not...
no negative e2e test exists (make sure things the Node authorizer disallows fail) negative integration tests exist at negative e2e tests for this will be tricky, since you'll need node credentials and a specific authz and admission config.
See the above bash script, I was able to do it purely from kubectl by impersonating the group and node user... If we feature-gate this e2e test, we can run it for the deployments that actually have unique node credentials
cc this maybe could be a good task for you... :)
Cool, thanks for point me this.
You basically only have to implement these commands as an e2e test initially (by impersonating a node in the cluster). Then when that's done, we can check if there are more than these four cases that needs to be verified. This should go under a separate and be enabled in deployments (like kubeadm) where nodes have unique credentials.
I didn't find a way to simulate a node credential, do you have any good idea ? after reading the integration testing code, I fell it's enough the guarantee the quality, WDYT?


Question: Command:         []string{"testCommand"}, WorkingDir:      "testWorkingDir", SecurityContext: &v1.SecurityContext{ <del> RunAsNonRoot: &RunAsNonRoot, </del> <ins> RunAsNonRoot: &runAsNonRootTrue, </ins> RunAsUser:    &runAsUser, }, },

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubelet securitycontext Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta/1.8.0 (HEAD) Environment: Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: docker runtime (rkt seems to have a different code path, but I haven't checked it) What happened: containers could not run as root when "RunAsNonRoot=false" What you expected to happen: containers could run as root How to reproduce it (as minimally and precisely as possible): add a securitycontext to a container, don't set privileged, do set RunAsNonRoot, use an image that runs as root. The PodSecurityPolicy admission controller will always set a securitycontext if it's not specified, and cause this to be hit more often. Anything else we need to know: The bug is in the checks here: I have a fix in but that going in depends on an extension being granted. We may want to fix the bug regardless.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
I'll send a fix shortly.


Question: return nil, err } <del> Logf("Controller %s: Found %d pods out of %d", name, len(pods.Items), replicas) </del> <ins> Logf("Pod name %s: Found %d pods out of %d", name, len(pods.Items), replicas) </ins> if len(pods.Items) == replicas { return pods, nil } } <del> return nil, fmt.Errorf("controller %s: Gave up waiting for %d pods to come up", name, replicas) </del> <ins> return nil, fmt.Errorf("Pod name %s: Gave up waiting for %d pods to come up", name, replicas) </ins> } func waitForPodsRunning(c *client.Client, pods *api.PodList) []error {

Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of


Question: } dv.SetBytes(data) } else { <del> dv.Set(reflect.Zero(dt)) </del> <ins> dv.Set(reflect.MakeSlice(dt, 0, 0)) </ins> } return nil }

I have created the following secret in my cluster. When I add labels to my secret and try to apply it in the cluster I get the following error This also happens when I use or using the go client. Not sure why, but works. I expect that the secret can be updated with the new labels since the is not actually changed. Steps explained above. No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
/remove-sig cli /sig apps


Question: } m := master.New(config) <del> // We serve on 3 ports.  See docs/reaching_the_api.md </del> <ins> // We serve on 3 ports.  See docs/accessing_the_api.md </ins> roLocation := "" if s.ReadOnlyPort != 0 { <del> roLocation = net.JoinHostPort(config.PublicAddress.String(), strconv.Itoa(config.ReadOnlyPort)) </del> <ins> roLocation = net.JoinHostPort(config.PublicAddress.String(), strconv.Itoa(s.ReadOnlyPort)) </ins> } secureLocation := "" if s.SecurePort != 0 { secureLocation = net.JoinHostPort(config.PublicAddress.String(), strconv.Itoa(s.SecurePort)) } <del> rwLocation := net.JoinHostPort(s.Address.String(), strconv.Itoa(int(s.Port))) </del> <ins> wideOpenLocation := net.JoinHostPort(s.Address.String(), strconv.Itoa(s.WideOpenPort)) </ins> // See the flag commentary to understand our assumptions when opening the read-only and read-write ports.

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: return fmt.Errorf("failed to try resolving symlinks in path %q: %v", path, err) } path = evaluated <del> f, err := os.Open(path) </del> <ins> f, err := openFileShareDelete(path) </ins> if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) }

The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows


Question: }, }, { <del> name: "custom extra-args", </del> <ins> name: "custom extra-args for v1.12.0-beta.2", </ins> cfg: &kubeadmapi.ClusterConfiguration{ Networking:                 kubeadmapi.Networking{PodSubnet: "10.0.1.15/16"}, ControllerManagerExtraArgs: map[string]string{"node-cidr-mask-size": "20"}, CertificatesDir:            testCertsDir, <del> KubernetesVersion:          "v1.7.0", </del> <ins> KubernetesVersion:          "v1.12.0-beta.2", </ins> }, expected: []string{ "kube-controller-manager",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: port80 := int32(80) port443 := int32(443) proto := protocol("tcp") <ins> ip1234 := "1.2.3.4" </ins> portMappings := []*PortMapping{ { &proto, &port80, &port80, <ins> ip1234, </ins> }, { &proto, &port443, &port443, <ins> ip1234, </ins> }, } checkpoint1 := newFakeCheckpointV1("check1", portMappings, true)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm using a proxy-pod to forward both and privileged ports to an service using the following configuration: However, when I start this service, it somehow highjacks the ports for other host IPs as well. I have a legacy nginx-proxy running on the same server, inside docker bound on specific non-conflicting IPs, but as soon as I start the pod above, it breaks SSL termination of the existing nginx proxy (outputing the "Fake ingress certificate"). Existing docker-compose: Name Command State Ports
nginxproxy nginx -g daemon off; Up 5.135.123.160:443-443/tcp,5.135.123.162:443-443/tcp, 5.135.123.160:80-80/tcp,5.135.123.162:80-80/tcp nginxproxydockergen /bin/sh -c /usr/local/bin/ ... Up nginxproxyletsencrypt /bin/bash /app/entrypoint. ... Up [...] Server certificate: subject: start date: Jun 11 21:55:37 2018 GMT expire date: Sep 9 21:55:37 2018 GMT subjectAltName: host "" matched cert's "" issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 SSL certificate verify ok.

iptables -t nat -nvL
I've flushed/dropped all iptables then reboot and again to make sure I had a clean state. Here is the iptables snapshots along the setup: Can confirm that SSL breaks for other IPs as soon as I start the pod proxy (last step). Looks like the culprit is . Destination should be and not . : Does work if I manually patch the IPtables rules: Which gives: Is there a way to automate this kind of patching to make sure the rules are not reverted by kubernetes? Thanks for the kind help :)
Thx for the reply about the iptable rules. According to the describe from u, I guess the cni plugin u used dont use the pod's data clearly. Maybe in the cni flannel,The flannel just change the rule by the hostport. And it think all the ip packets in the host must do the dnat and snat. I think flannel dont catch the hostip in the pod's yaml.
Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin
GitHub didn't allow me to assign the following users: liucimin. Note that only and repo collaborators can be assigned. <detailsIn response to : Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
any update on this?

Did you ever find out the root cause? /remove-lifecycle rotten
Yes,I have push a pr in k8s.
v1.11.5 also the problem. Has Version 1.12 been fixed? thanks.
The pr is not merge yet. So the problem is also in v1.12 and v1.13


Question: fi } <del> find-release-tars() { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" RELEASE_DIR="${KUBE_ROOT}/server/" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" RELEASE_DIR="${KUBE_ROOT}/_output/release-tars/" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" exit 1 fi } </del> rackspace-set-vars() { CLOUDFILES_CONTAINER="kubernetes-releases-${OS_USERNAME}"

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)