Question: //modify labels pod.Labels["key3"] = "value3" <ins> pod.ResourceVersion = "" // to force update </ins> _, err = f.Client.Pods(f.Namespace.Name).Update(pod) Expect(err).NotTo(HaveOccurred())

In parallel suite:
Can you recreate this? It could be a race or a buffer not flushing.
I'm having a look
Anoter example of the failure: - awesome thanks!
unluckily I cannot seet this page. Thanks anyway
to delegate? (If you don't think it's a node bug, maybe can find an owner?)
Here's our latest conformance log from master with the labels/annotations both failing (mesos/docker cluster): Downward API volume should update labels on modification [Conformance] Downward API volume should update annotations on modification [Conformance]
These failures can be reproduced on master with a single command:
thanks for the logs. I'm investigating the issue using (never worked with so I'm not sure if it's the same or not) Running only one instance of like this --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true --"*on.modification" --kubeconfig="$" basically your command (I guess) I never had a failure. Running the same command in a loop it's different and I managed to reproduce the issue, most of the time the timeout looks to aggressive 15s (while in most of the tests the timeout is something like 5 minutes). since it's seems that this is blocking for several person I'd like to raise the timout (at least to 30seconds). I'm not saying that this is the fix (the old joke about ) but at least it should reduce the problem on busy system.
It looks to me that everytime the fails happens I ran into something like: the same is valid for logs. I tried to in the code (maybe I'm blind) but I couldn't find... Any ideas?
- those lines are coming from this metric: the "kubelet_" at the beginning is "subsystem" in prometheus metrics, that's why you probably didn'g find it
Actually it's printed here:
thanks thanks
We should verify if increasing the timeout: fixed this issue.
Still fails on the mesos/docker cluster on master:
I have been informed that the mesos executor wasn't actually handling annotation/label updates. So these tests were failing legitimately on the mesos/docker cluster.

This will hopefully be fixed by
don't know if makes sense for you but kubelet broke this and we add to fix
Another failure:
:( Sorry I didn't see this... Ok. Having a look again (but today I'm pretty busy)
is not enough. We need also to increase timeout in image which generates the logs sorry for this I didn't realize it when we talked about this thanks for the file!
submitted to increase timout in running image
- great - thanks for looking into it!
I'm reopening it, so that we can monitor if this really fixed the problem. Will close it if we don't observe more errors for a couple of days.
There weren't any failures of this test within last 3 days - closing! Thanks
Saw this flake again in a GKE run:
same test but another issue. we got this: . While the previous issue was a too short timeout (container exited too early for slow envs). would you like to open another issue for this?
Working on this to propose a fix
submitted please have a look
I saw you xref this issue from . I cannot see the relations (and I could be wrong). Mind explain? Let me know if I need to Thanks
- the test you are trying to fix failed in the presubmit of that PR: I didn't review but it's not merged yet, so we don't know if that helps (hopefully yes :))
I think will help on these kind of issues (retrying on the update). thanks for the explanation.
I encountered a failure of this test for a different reason, I can create a separate issue if you think it's unrelated: link:
In the current implementation is performed without retry. implement a retry mechanism similar to
thanks. FYI, Actually we can set the Pod.ResourceVersion="", that will force an update, it would be cleaner than using a Poll.
I'm going to update the PR with what you're suggesting. Thanks


Question: "k8s.io/kubernetes/pkg/api/legacyscheme" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/capabilities" <del> "k8s.io/kubernetes/pkg/client/chaosclient" </del> "k8s.io/kubernetes/pkg/cloudprovider" "k8s.io/kubernetes/pkg/credentialprovider" "k8s.io/kubernetes/pkg/features"

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: if err != nil { return false, err } <del> if len(pods.Items) == 0 { </del> <ins> // Pods may be created by overlapping deployments right after this deployment is deleted, ignore them if len(overlapWith) == 0 && len(pods.Items) == 0 { </ins> return true, nil <ins> } else if len(overlapWith) != 0 { noOverlapPods := []api.Pod{} for _, pod := range pods.Items { if !strings.HasPrefix(pod.Name, overlapWith) { noOverlapPods = append(noOverlapPods, pod) } } if len(noOverlapPods) == 0 { return true, nil } </ins> } return false, nil }); err != nil {

https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
Looking
Flakes happened when stopping overlapping deployments at clean up step.
You probably need to move the check before the deletionTimestamp check. Just make sure that observedGeneration is updated for the first case.
Because the deployments are overlapping, when deleting one deployment and determining if its RSes are deleted, there might be a race when the other deployment creates an RS
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: // we don't have an entry and we do want one (2B,2C) case apierrors.IsNotFound(err) && desired != nil: _, err := c.apiServiceClient.APIServices().Create(desired) <ins> if apierrors.IsAlreadyExists(err) { // created in the meantime, we'll get called again return nil } </ins> return err // we aren't trying to manage this APIService (3A,3B,3C)

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: - deleteCoreV1Node - deleteCoreV1PersistentVolume - deleteStorageV1CollectionCSINode <del> - deleteStorageV1CollectionStorageClass </del> - deleteStorageV1CollectionVolumeAttachment - deleteStorageV1CSINode <del> - deleteStorageV1StorageClass </del> - deleteStorageV1VolumeAttachment - getInternalApiserverAPIGroup - getResourceAPIGroup - listCoreV1PersistentVolumeClaimForAllNamespaces - listStorageV1CSINode <del> - listStorageV1StorageClass </del> - listStorageV1VolumeAttachment - patchCoreV1NamespacedPersistentVolumeClaim - patchCoreV1NamespacedPersistentVolumeClaimStatus - patchCoreV1PersistentVolume - patchCoreV1PersistentVolumeStatus - patchStorageV1CSINode <del> - patchStorageV1StorageClass </del> - patchStorageV1VolumeAttachment - patchStorageV1VolumeAttachmentStatus - readCoreV1NamespacedPersistentVolumeClaim

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "strings" "testing" "k8s.io/client-go/util/flowcontrol" ) func TestRegisterMetricAndTrackRateLimiterUsage(t *testing.T) { testCases := []struct { ownerName   string rateLimiter flowcontrol.RateLimiter err         string }{ { ownerName:   "owner_name", rateLimiter: flowcontrol.NewTokenBucketRateLimiter(1, 1), err:         "", }, { ownerName:   "invalid-owner-name", rateLimiter: flowcontrol.NewTokenBucketRateLimiter(1, 1), err:         "error registering rate limiter usage metric", }, } for i, tc := range testCases { e := RegisterMetricAndTrackRateLimiterUsage(tc.ownerName, tc.rateLimiter) if e != nil { if tc.err == "" { t.Errorf("[%d] unexpected error: %v", i, e) } else if !strings.Contains(e.Error(), tc.err) { t.Errorf("[%d] expected an error containing %q: %v", i, tc.err, e) } } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Shows that prometheus.MustRegister will panic on things like dashes in the metric name. There are currently 138 calls to prometheus.MustRegister in the Kubernetes code base almost exclusively from controllers. We should audit the code to ensure that panic is the desirable response to a bad metric name for these calls vs. something like prometheus.Register which will instead return an error. What you expected to happen: See above How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: No Environment: All Kubernetes version (use ): 1.9 Cloud provider or hardware configuration**: All OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I did an analysis on this issue, and this is what I found: of the calls use a static metric name (hardcoded at the method invocation via a string and/or constant); is only a single case where the metric name is constructed dynamically (from a parameter), at the file (and happens to be where panicked); My proposal is: a. For static metric names (1), leave them as they are; b. For dynamic metric names (2), where the caller might not know that a parameter will be used to construct a metric name: Update the file to use instead of , and return the error if the metric cannot be registered (already submitted PR ); A quick search from the method invocation (from the above file) results in 17 invocations, mostly controllers, but although returns an error, no one of those 17 results is catching the error. So, we need to update all invocations to catch the error (I plan to submit new PRs to address this).

/sig instrumentation
We can trace all the task for update metrics.RegisterMetricAndTrackRateLimiterUsage invocations, and I am very happy to help fix it. [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [ ] [ ] [x] [x] [x]
Please go ahead with the ones not yet updated if you want.
OK, thanks!
now, we catch the error and when there return an error, we return in the function . But in the test file we use function for many times and Then there will be a lot of error in the test file, such as: then the test will always go fail. I think we should distinguish the error and when the error is Duplicate registration we dhould ignore it and return the normal controller.
Can you please provide an example where this is happening? I ran the tests on several controllers and haven't found that error. Ignoring this error doesn't seem a good idea IMO, if there're some tests that fail, we should investigate why are failing and fix them appropriately.
Sorry, I thought you were talking about the unit tests, didn't see the integration tests problem. I've submitted to fix this and ran the integration tests to check that creating multiple controllers don't fail anymore.
actually in some cases the unit tests is also failed such as endpoint controllerreplicaset controller. because different set up before test. So we should take a action to resolve it.


Question: } } <ins> // SetStacktracePredicate sets a custom stacktrace predicate for the // logger associated with the given request context. func SetStacktracePredicate(ctx context.Context, pred StacktracePred) { if rl := respLoggerFromContext(ctx); rl != nil { rl.StacktraceWhen(pred) } } </ins> // Log is intended to be called once at the end of your request handler, via defer func (rl *respLogger) Log() { latency := time.Since(rl.startTime)

panics with error when health probes time out. The following stack trace shows this error (from an apiserver that uses : The is thrown by net/http because attempts to call after the request had timed out (handler had completed) here is the code site that triggers the panic (on master): Is there any historical context why we use the original by calling . If we use the decorated object then the panic will be prevented: uses the underlying object only when the request handler has not completed yet. No panic hard to reproduce, maybe we should add a unit test to reproduce the issue.
/sig api-machinery cc
/assign
GitHub didn't allow me to assign the following users: benluddy. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


Question: type StatFunc func(string) (os.FileInfo, error) type GlobFunc func(string) ([]string, error) <del> func waitForPathToExist(devicePath string, maxRetries int, deviceInterface string) bool { </del> <ins> func waitForPathToExist(devicePath string, maxRetries int, deviceTransport string) bool { </ins> // This makes unit testing a lot easier <del> return waitForPathToExistInternal(devicePath, maxRetries, deviceInterface, os.Stat, filepath.Glob) </del> <ins> return waitForPathToExistInternal(devicePath, maxRetries, deviceTransport, os.Stat, filepath.Glob) </ins> } <del> func waitForPathToExistInternal(devicePath string, maxRetries int, deviceInterface string, osStat StatFunc, filepathGlob GlobFunc) bool { </del> <ins> func waitForPathToExistInternal(devicePath string, maxRetries int, deviceTransport string, osStat StatFunc, filepathGlob GlobFunc) bool { </ins> for i := 0; i < maxRetries; i++ { var err error <del> if deviceInterface == "default" { </del> <ins> if deviceTransport == "tcp" { </ins> _, err = osStat(devicePath) } else { fpath, _ := filepathGlob(devicePath)

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: $kubectl config set-context queen-anne-context --cluster=pig-cluster --user=black-user --namespace=saw-ns $kubectl config set-context federal-context --cluster=horse-cluster --user=green-user --namespace=chisel-ns $kubectl config use-context federal-context <del> ``` </del>  No newline at end of file <ins> ``` </ins>

After installing nginx on the kubernetes master (currently non-k8s docker container) and forwarding external http requests to the kubernetes-ro service through nginx with simple htpasswd auth, I tested it (to use within CI tool), but it failed to pass nginx. Is it because we are required to use https when querying the apiserver using kubectl? Or because the -a option is used for a different auth ? How would you solve this issue ? Thank you in advance!
I tried to run kubectl proxy in one shell and query kubectl get in the other, without success: Shell 1 Shell 2
Looks like the isn't forwarding authorization as expected? Can you take a look please?
takes a full qualified URL. If you pass the plain ip string it will do a get over http, which the apiserver correctly rejects. Replace with and it should work as expected. Note that if you set up a kubeconfig file you can avoid some of this pain. Run for help/commands to set up a kubeconfig.
Please reopen if the above does not work for you.
I understood that I cannot use kubectl with remote server through http, so I modified my nginx to support ssl. I still have a problem with the or parameter. Is the kubernetes_auth supposed to have a different format maybe? I do not have the option to reopen the ticket. thanks for the kubectl config tip. But it's only a v0.9+ feature, so I'll try it when I upgrade later from v0.8.1
after testing , I came up with that kubeconfig: Still getting the same error:
That config looks fine. message is usually a json unmarshalling error. I wonder if there might be something wrong with your .kubernetes_auth file? It should look like
Sorry, forgot to come back here after solving the issue. Correct, the problem was with the kubernetesauth file. The keys must be capitalized, and I was trying This naming convention should be better documented, or more useful error should be raised. Neither the Json standard (ECMA) nor the JSON-RPC specs are requiring capitalized keys. Renaming this issue to "kubernetesauth missing documentation"
Thanks Adrien. I think the right thing to do here is raise an error if the kubernetes_auth file exists but doesn't have the required keys.
JSON RPC states that identifiers must be case-sensitive.


Question: ip = v.IP } if ip.IsLoopback() || ip.IsLinkLocalUnicast() { <del> break </del> <ins> continue </ins> } successTest := test{ nodeIP:   ip.String(),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: UT TestValidateNodeIPParam does not really tests . The first IP and interface are always 127.0.0.1 and lo and will break the loop. So will never be appended into . What you expected to happen: gets tested correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/sig node


Question: LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, <ins> { Type:               v1.NodePIDPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, </ins> }, Capacity: v1.ResourceList{ v1.ResourceName(v1.ResourceCPU):    resource.MustParse("10"),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: when the node lost connection with master , using 'kubectl describe node' find the NodePIDPressure is false , no switch to unknown. What you expected to happen: PIDPressure Unknown Fri, 14 Sep 2018 05:13:09 +0800 Fri, 14 Sep 2018 05:13:53 +0800 NodeStatusUnknown Kubelet stopped posting node status. How to reproduce it (as minimally and precisely as possible): stop kubelet, wait the node Notready check the node status 'kubectl describe node' Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node


Question: "k8s.io/kubernetes/pkg/volume" ) <ins> type TB interface { Errorf(format string, args ...any) } </ins> type FakePod struct { Pod       *kubecontainer.Pod NetnsPath string

We have many "test only" packages through out the project, containing fixtures, helpers, utilities, etc... These test libraries are not intended for use in production binaries. They often lack rigorous review and sufficient testing for use in production binaries. We should make sure they don't accidentally get pulled in. Strawman proposal, test only packages must import "testing". Blank import is sufficient: Then write a presubmit that fails if we find symbol in the release binary. This approach is trivial and fairly effective although a more sophisticated approach could print a dep chain.
/assign
I hope I will be able to contribute once this gets marked as .
/triage accepted
I think that will be enough a new script or a check in the building scripts /cc since he has experience from the providerless building effort
I got what we want to achieve here but I'm little confused about: If I execute: after adding the import then only it is giving - what do we want to achieve by executing this command exactly? I got what it is doing. I've opened a thread in slack as well:
Hello! Is there anyone working on this issue at the moment? If not and the issue it's still relevant, I would like to work on it if possible. Thank you.
I opened a PR in the meantime, if it's not needed anymore, I can close it.


Question: } for _, tc := range tests { <ins> tc := tc </ins> t.Run(tc.desc, func(t *testing.T) { fakeSystemBus := &fakeSystemDBus{} bus := DBusCon{

tc.shutdownActive is captured by 'func' literals in 'go' statements(Line177), it may cause a data race. I use t.Run(..., ){...}() or t.Run() {...}(tc) to avoid it . I don't know whether it is accurate, so I 'm looking forward to your reply. Data Race Nil No response <detailsNil </details<detailsNil </details<details</details<details</details<details</details<details</details
/sig node
Target file:
/area test /priority backlog
have you observed any test failures or this is found by code analysis?
Yes, it is found by code analysis. In terms of the source code, the scenario can happen.
/assign
/remove-kind bug /kind failing-test since it is not a product bug, but the test issue
Although each test case calls a goroutine, tests run serially and block on the goroutine return, so it doesn't look like it could cause an issue. However, to be safe we can create a local copy of inside the loop. I'll send a PR briefly.
Created to address this.
/triage accepted /priority important-longterm


Question: classListerSynced  cache.InformerSynced podLister          corelisters.PodLister podListerSynced    cache.InformerSynced <ins> podIndexer         cache.Indexer </ins> NodeLister         corelisters.NodeLister NodeListerSynced   cache.InformerSynced

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: <ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Builds an official release based on a git tag, with instructions for # how to proceed after the bits are built. set -o errexit set -o nounset set -o pipefail # Get the md5 (duplicated from common.sh, but don't want to pull in # all of common.sh here) function md5() { if which md5 >/dev/null 2>&1; then md5 -q "$1" else md5sum "$1" | awk '{ print $1 }' fi } # Get the sha1 (duplicated from common.sh, but don't want to pull in # all of common.sh here) function sha1() { if which shasum >/dev/null 2>&1; then shasum -a1 "$1" | awk '{ print $1 }' else sha1sum "$1" | awk '{ print $1 }' fi } declare -r KUBE_GITHUB="https://github.com/GoogleCloudPlatform/kubernetes.git" declare -r KUBE_RELEASE_VERSION=${1-} declare -r KUBE_RELEASE_UMASK=${KUBE_RELEASE_UMASK:-022} VERSION_REGEX="^v(0|[1-9][0-9]*).(0|[1-9][0-9]*).(0|[1-9][0-9]*)$" [[ ${KUBE_RELEASE_VERSION} =~ ${VERSION_REGEX} ]] || { echo "!!! You must specify the version you are releasing in the form of '${VERSION_REGEX}'" >&2 exit 1 } declare -r KUBE_BUILD_DIR="/tmp/kubernetes-release-${KUBE_RELEASE_VERSION}-$(date +%s)" # Set the default umask for the release. This ensures consistency # across our release builds. umask "${KUBE_RELEASE_UMASK}" echo "Cloning ${KUBE_GITHUB} at ${KUBE_RELEASE_VERSION}." echo echo "NOTE: Ignore the deatched HEAD warning you're about to get. We want that." echo git clone ${KUBE_GITHUB} -b "${KUBE_RELEASE_VERSION}" "${KUBE_BUILD_DIR}" # !!! REMINDER !!! # # Past this point, you are dealing with a different release. Don't # assume you're executing code from the same repo as this script is # running in. This needs to be a version agnostic build. echo echo "Cloned, building release." echo cd "${KUBE_BUILD_DIR}" export KUBE_RELEASE_RUN_TESTS=n export KUBE_SKIP_CONFIRMATIONS=y make release if ${KUBE_BUILD_DIR}/cluster/kubectl.sh version | grep Client | grep dirty; then echo "!!! Tag at invalid point, or something else is bad. Build is dirty. Don't push this build." >&2 exit 1 fi ln -s ${KUBE_BUILD_DIR}/_output/release-tars/kubernetes.tar.gz ${KUBE_BUILD_DIR} MD5=$(md5 "${KUBE_BUILD_DIR}/kubernetes.tar.gz") SHA1=$(sha1 "${KUBE_BUILD_DIR}/kubernetes.tar.gz") echo "" echo "Success! You must now do the following: (you may want to cut" echo "  and paste these instructions elsewhere, step 1 can be spammy)" echo "" echo "  1) (cd ${KUBE_BUILD_DIR}; build/push-official-release.sh ${KUBE_RELEASE_VERSION})" echo "  2) Go to https://github.com/GoogleCloudPlatform/kubernetes/releases" echo "     and create a new 'Release ${KUBE_RELEASE_VERSION} Candidate' release" echo "     with the ${KUBE_RELEASE_VERSION} tag. Mark it as a pre-release." echo "  3) Upload the ${KUBE_BUILD_DIR}/kubernetes.tar.gz to GitHub" echo "  4) Use this template for the release:" echo "" echo "## [Documentation](http://releases.k8s.io/${KUBE_RELEASE_VERSION}/docs)" echo "## [Examples](http://releases.k8s.io/${KUBE_RELEASE_VERSION}/examples)" echo "## Changes since <last release> (last PR <last PR>)" echo "" echo "<release notes>" echo "" echo "binary | hash alg | hash" echo "------ | -------- | ----" echo "`kubernetes.tar.gz` | md5 | `${MD5}`" echo "`kubernetes.tar.gz` | sha1 | `${SHA1}`" echo "" echo "     We'll fill in the release notes in the next stage." echo "  5) Ensure all the binaries are in place on GitHub and GCS before cleaning." echo "  6) (cd ${KUBE_BUILD_DIR}; make clean; cd -; rm -rf ${KUBE_BUILD_DIR})" echo "" </ins>

Since I've been having troubles with installation, I've digged a bit and found out that the permissions inside Kubernetes tarballs changed between 0.16.0 and 0.17.0. I've downloaded both tarballs, extracted them into separate directories and then extracted the Kubernetes server tarballs. This is what I get: I suppose, there has been a concious decision to restrict the world's access to those binaries. However, this breaks the installation routine mentioned above and probably makes life a bit more difficult in several other places. I'd be thankful for a correction if this wasn't intended or a pointer to the appropriate commit it this is legit -- I'd correct the installation.
assigning this to you to decide who the eventual assignee should be :)
Any news on this front? 0.20.1 is still restricted.
Apologies for the issue. It wasn't intentional -- likely an artifact of build scripts being run by different people? I'll check out 0.20.2 as I build it to make sure it has the right perms.
Apologies, but 0.20.2 didn't include fixed permissions. I now understand the issue, : "make release" builds a tarball using the permissions on files in your local git tree. I had "umask 027" set when I cloned, so those in-the-repo files had no world-readable perms. Setting "umask 022" before running "make release" is not enough -- it has to be part of creating the local tree when cloning. I'll update instructions and the next release (20.3 or 21.0) should fix this issue. And longer term, we need to put a bit more effort into releasing from a known-good environment and not rely on a developer's workstation's idiosyncrasies -- sorry this bit you.
I can't repro Trying again a different way, but I don't think the umask before clone should do anything (in particular, umasks aren't "sticky" on files, they're sticky to processes.)
And let's pretend on my personal fork isn't woefully out of date (who uses master on their personal fork? I should really just delete that that thing.) Comment updated.
Ah, I see the problem now.

works fine for the server tar, but the full tar also packages examples, etc., and for that, tar tends to just pick up the permissions of the git tree. Got it.
I'll take this. I'm going to write a script to build official bits, and this is easy enough to fix along the way.
Seems not fixed as of Kubernetes 1.0.7
sorry about that. is doing the 1.1.1 release and has been improving release scripts -- is 1.1.1 good re: permissions?
was not built by .
Reassigning to as release-oncall. Quinton, I can mentor or advise as necessary.
And yeah, is fine: There's no real bug here. There was a process issue that occurred with 1.0.7. We can rebuild the binaries if we need to.
FYI, leaving this issue closed, as per 's comment above. If anyone disagrees, please re-open this issue.


Question: clientConfig = DefaultClientConfig(flags) } <del> clients := &clientCache{ clients: make(map[string]*client.Client), loader:  clientConfig, } </del> <ins> clients := NewClientCache(clientConfig) </ins> return &Factory{ clients:    clients,

This seems like a bug to me No output, and command exits success... wat. Have to explicitly pass to see expected output. If the client version is greater the server version, it should be making requests based on the server's newest accepted apiversion.
cc


Question: fs.Int32Var(&f.NodeStatusMaxImages, "node-status-max-images", f.NodeStatusMaxImages, "<Warning: Alpha feature> The maximum number of images to report in Node.Status.Images. If -1 is specified, no cap will be applied.") // DEPRECATED FLAGS <del> fs.BoolVar(&f.Containerized, "containerized", f.Containerized, "Running kubelet in a container.") fs.MarkDeprecated("containerized", "This feature will be removed in a later release.") </del> fs.StringVar(&f.BootstrapKubeconfig, "experimental-bootstrap-kubeconfig", f.BootstrapKubeconfig, "") fs.MarkDeprecated("experimental-bootstrap-kubeconfig", "Use --bootstrap-kubeconfig") fs.DurationVar(&f.MinimumGCAge.Duration, "minimum-container-ttl-duration", f.MinimumGCAge.Duration, "Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'")

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }, []string{"requestKind"}, ) <ins> requestTerminationsTotal = compbasemetrics.NewCounterVec( &compbasemetrics.CounterOpts{ Name:           "apiserver_request_terminations_total", Help:           "Number of requests which apiserver terminated in self-defense.", StabilityLevel: compbasemetrics.ALPHA, }, []string{"verb", "group", "version", "resource", "subresource", "scope", "component", "code"}, ) </ins> kubectlExeRegexp = regexp.MustCompile(`^.*((?i:kubectl.exe))`) metrics = []resettableCollector{

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: var defaultContainerName = "" var storageAccountNamePrefix = "" var storageAccountNameMatch = "" <del> var initFlag int64 </del> var accountsLock = &sync.Mutex{} func newBlobDiskController(common *controllerCommon) (*BlobDiskController, error) { c := BlobDiskController{common: common} <del> err := c.init() </del> <ins> c.setUniqueStrings() // get accounts accounts, err := c.getAllStorageAccounts() </ins> if err != nil { <del> return nil, err </del> <ins> glog.Errorf("azureDisk - getAllStorageAccounts error: %v", err) c.accounts = make(map[string]*storageAccountState) return &c, nil </ins> } <ins> c.accounts = accounts </ins> return &c, nil }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: For unmanaged disk, now it is not working What you expected to happen: Works well How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.2.734+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-14T03:26:54Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.2.734+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-14T03:26:54Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL7 Kernel (e.g. ): Install tools: ansible Others:
/assign andyzhangx /sig azure
I cannot reproduce your issue with v1.9.0, in your storage class, an azure disk would be provisioned in storage account , it may fail because name is already occupied, but in your case it's a strage string: . So could you also paste the controller manager log, below is the way how to get it: There should be an important log like following:
I did not run containerized k8s, I deployed a cluster by ansible: I got the controller message as below: I am quit sure there is already a storage account in its resource group which is assigned in sc's parameter: I do not know why controller does not find it and wants to create a new one. But in our subscription the storage account has reached the max number and cannot be created anymore, the error is different from this.
cc
would you add in your stroage class, and try again? In your case, is not invoked due to: I am working out a PR to fix this, as a workaround, you could try storage class like following: Let me know if that works, thx
Yes. working now with this parameter :

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: is buggy. It prevents running hyperkube from other path. In fact, it shouldn't assume the code is just running in controller master at all, especially when cloud provider is running out of tree. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
This code logic is to make sure the blobController only initialize once in controller-manager other than in other component, e.g. kubelet, do you have any suggestion to use similar implementation? Current design is azure pv code could only run in controller-manager.
If the idea is to exclude other components, I prefer a black list, not a broken white a list as here.
there are multiple issues with this code: It shouldn't bootstrap any storage accounts at all. This is really poor user experience. Not all users want to/will use azure disk. These storage accounts are yet created without any explicit documentation or prior warning.Even with these storage accounts, it doesn't help accelerate PV provisioning.This is the only place to init . If controller master is launched from a different path (which is usually the case), then never gets initialized, and will cause a panic when being used.
thanks for the check, put the storage accounts bootstrap in could be more appropiate: that is only when user select kind and would like to create blob disk PVC, then those accounts would be created and it would only initialize before first PVC is created.
SGTM, looking forward to your PR and 1.7/1.8 cherry picks
update: code complete, I need to do some test before sending PR:

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The azuredd plugin will always provision two instances of storage account with different type during init. There could be cases when: may only want to use one certain type of storage account, then the other type of storage account would be not in use. chooses to use managed disk, both of the storage account would be not in use. There is a limit for storage account numbers per subscription, and the storage account should be created with a on-demand policy. Log: Code: What you expected to happen: The storage accounts be created on-demand. How to reproduce it (as minimally and precisely as possible): Setup a kubernetes v1.7.2 cluster on Azure, create a dynamic pvc. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: Azure
/sig azure
Creating a storage account is a lengthy process (~ 1.5 mins), these 2 accounts are created upfront to ensure that PVCs will be quickly provisioned into existing storage accounts. Since ACS & acs-engine currently supports mixed (managed and blob) disks we won't be able to tell if they user will or won't use blob disks. Maybe once acs- moves away from blob disks we can disable them. They have no impact on users since users get billed only for data inside storage accounts not storage accounts themselves
for the "limit for storage account numbers per subscription" issue, I have another issue to track this: , will fix it in v1.9 and cherry pick to v1.8, v1.7
/assign


Question: kubeDNSProbeSRV = "SRV" kubeDNSProbeA   = "A" <del> coreDNSVersion  = "1.0.0" </del> <ins> coreDNSVersion  = "1.0.1" </ins> ) // GetDNSVersion returns the right kube-dns version for a specific k8s version func GetDNSVersion(kubeVersion *version.Version, dns string) string { // v1.8.0+ uses kube-dns 1.14.5 // v1.9.0+ uses kube-dns 1.14.7 <del> // v1.9.0+ uses CoreDNS  1.0.0 </del> <ins> // v1.9.0+ uses CoreDNS  1.0.1 </ins> // In the future when the version is bumped at HEAD; add conditional logic to return the right versions // Also, the version might be bumped for different k8s releases on the same branch

Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: When CoreDNS is deployed by default using feature-gates of kubeadm, it does not log anything when processing a DNS query. That is due to change of log option of CoreDNS in latest version. that version was bumped up in kubeadm, but the manifest was not adapted to the new log option. initial change of 'log stdout' in CoreDNS : version of CoreDNS 1.0.0 bumped into Kubeadm : NOTE: I guess we should also changed manifests for kube-up : see The change in manifest expected is to remove the option "stdout" in the line "log ...". As it is now interpreted as a scope of domain to log. Invalid manifest: Environment: Kubernetes version (use ): 1.9 and MAIN Cloud provider or hardware configuration: any Install tools: kubeadm, kube-up the configmap for coredns kubectl get configmaps -n kube-system coredns -o yaml kubectl replace -f - the coredns deployment kubectl scale deployment --replicas=0 -n kube-system coredns kubectl scale deployment --replicas=1 -n kube-system coredns
/cc /sig network
NOTE: only log is impacted. CoreDNS working as expected.
/cc
I have a question : the initial manifest is done for the image 0.9.10 of CoreDNS. Bumping the version of CoreDNS was done by changing the version in file (see ) which means that the image's version is STILL 0.9.10 in the manifests ... and the corefile config is correct against the version of the manifest. QUESTION: when modifying the , is there a way to upgrade the manifest by program ? I guess at the time the new version number is applied on the manifest ...
thanks for reporting this! I send PR to update the Corefile. I don't think of a way so far. I guess we can't unless there is a way to get the Corefile from some place of upstream (CoreDNS).


Question: numNodes, err := e2enode.TotalRegistered(c) framework.ExpectNoError(err) originalNodeCount = int32(numNodes) <del> common.NewRCByName(c, ns, name, originalNodeCount, nil) </del> <ins> common.NewRCByName(c, ns, name, originalNodeCount, nil, nil) </ins> err = e2epod.VerifyPods(c, ns, name, true, originalNodeCount) framework.ExpectNoError(err)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes Since when has it been failing: These tests have been failing since 6/28 Failures coincide with the following changes: Testgrid link: Reason for failure: The failure messages look as follows: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes /cc /milestone v1.16 /priority critical-urgent /kind failing-test /cc
/sig cluster-lifecycle
For more context, the code for the failing tests is here The error for both comes through  - some Pod is not responding. It is failing to find pods prefixed with "my-hostname-add-node". Within the logs we can see this: So the Pods are stuck in a Pending state. The Pods are supposed to be created by a ReplicationController in , which itself uses this method from the e2e framework . However, after being in a Pending state (after the "failed to wait for pods responding: timed out waiting for the condition" error is registered), they do seem to reach a Ready state
/assign for triage.
I think agnhost needs to be passed , but did not do so in /assign sig-testing
GitHub didn't allow me to assign the following users: sig-testing. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I think agnhost needs to be passed , but did not do so in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopen to monitor job /reopen
Reopened this issue.

<detailsIn response to : Reopen to monitor job /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for working on the PR to solve this. indeed solved the issue reported here. /close
Closing this issue. <detailsIn response to : Thank you for working on the PR to solve this. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Compile migrate migrate_tmp_dir=$(shell mktemp -d);  docker run --rm --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes$(DOCKER_VOL_OPTS) -v $${migrate_tmp_dir}:/build$(DOCKER_VOL_OPTS) -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION)  <del> /bin/bash -c "CGO_ENABLED=0 go build -o /build/migrate k8s.io/kubernetes/cluster/images/etcd/migrate";  </del> <ins> /bin/bash -c "CGO_ENABLED=0 GO111MODULE=off go build -o /build/migrate k8s.io/kubernetes/cluster/images/etcd/migrate";  </ins> $(BIN_INSTALL) $${migrate_tmp_dir}/migrate $(TEMP_DIR);  docker run --rm --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes$(DOCKER_VOL_OPTS) -v $${migrate_tmp_dir}:/build$(DOCKER_VOL_OPTS) -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION)  <del> /bin/bash -c "CGO_ENABLED=0 go build -o /build/cp k8s.io/kubernetes/cluster/images/etcd/cp";  </del> <ins> /bin/bash -c "CGO_ENABLED=0 GO111MODULE=off go build -o /build/cp k8s.io/kubernetes/cluster/images/etcd/cp";  </ins> $(BIN_INSTALL) $${migrate_tmp_dir}/cp $(TEMP_DIR); ifeq ($(ARCH),amd64)

Right now the golang version in the Makefile is set to GOLANGVERSION?=1.12.17. Build fails with " undefined: errors.Is", presumably because errors.Is didn't exist in golang 1.12.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can take a look.
/triage accepted
According to , we should use 1.16. /kind bug /area release-eng
I send a fix. It would be good to make sure this doesn't break in the future. Question to How we can include building cloud/image as part of presubmits?
Somehow I thought maybe we'd switched to an ectd image built from the etcd project and needed to eventually drop this one? If not this is rather concerning. We don't have a great presubmit to shove this in, I would suggest a runifchanged presubmit to let reviewers of this image know if it builds w/o making it a required job on all PRs (since also changes should be scoped to a directory and infrequent).
For example the conformance image works this way


Question: "sync" "time" <ins> libcontainercgroups "github.com/opencontainers/runc/libcontainer/cgroups" </ins> "golang.org/x/sys/unix" "k8s.io/klog/v2" )

Start kubelet on a cgroupv2 image with kubelet config option or the flag enabled. Kubelet will periodically log something like: This is because the kernel memcg notifications relies on kubelet's which uses evenfd on file which is cgroupv1 specific and not supported on cgroupv2. Kubelet will disable kernel memcg notifications on cgroupv2 if they are unsupported and consider long term support for more cgroupv2 "compatible" ways to monitor for memory pressure. The closest analogous cgroupv2 concept is using inotify notifications on file and or combination with PSI pressure. This requires to be set on pod cgroup which requires the MemoryQOS feature to be enabled Start kubelet on a cgroupv2 image with enabled.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /cc


Question: "encoding/json" "errors" "fmt" <del> "regexp" </del> "strconv" "strings" "sync"

When a pod is marked as deleted while kubelet is down / being restarted, newly started kubelet does not clean up CSI filesystem volumes of the pod. Newly started kubelet tries to reconstruct the volume using CSI's function. This part looks working, CSI volume plugin loads its json file. But then VolumeManager checks if the volume is still mounted in directory. There are two issues: does not require volumes to be presented as mounts. They can be just directories with files on them. This will be case of the most of in-line volumes. Kubelet checks the right directory given by
/sig storage related to cc
Maybe a stupid idea... Reconciler sync has found a directory in . Reconciler has no way how to check that CSI volume (or any other volume) is really mounted / published there. Depending on DSW: The volume is in DSW -from comment here it seems that the SetUp (=NodePublish) will be called on the volume: The volume is not in DSW and it will be to ASW, so it's unmounted / unpublished in the next sync: In both cases, we can expect that the volume plugin / CSI driver is idempotent and SetUp() won't do anything if the volume is already set up / TearDown() will not do anything if the volume has been torn down already. Are our volume plugins really idempotent? IMO they are. We perhaps don't need to check for presence of mount point anywhere, just presence of directory should be enough to reconstruct the volume and set it up / tear it down. Adding to the loop.
Filed , PTAL
I'm curious how the reconstruction e2e tests passed
https://k8s-*force
They start kubelet with a pod already deleted. If the pod still exists, volume reconstruction can see it in DSW and does nothing when fails: [Reconstruction fails because it checks for a mount in a wrong directory.] In this case, kubelet waits for the volume to appear in VolumesInUse before calling . The pod is deleted before that and kubelet then just deletes DSW entry and the volume is never cleaned up.
They pass because their container does not handle SIGTERM and it takes 30 seconds to kill them. Kubelet has enough time to the volume during reconciliation.

It turns out that I tested with broken version of and reconstruction is broken only for ephemeral volumes that don't mount.
So the conclusion is that we only need to fix for ephemeral volumes? Thanks!
Yes, sorry again about the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle rotten
/lifecycle frozen
From this issue surfaced through e2e tests using the hostPath CSI driver. The cause is (which is used in for reconstruction) - it doesn't play well with volumes that bind mount from a directory instead of a device.
To work around this bug here, volume life cycle checking was disabled for all tests, not just the subpath test: This check is useful and should be enabled again.
Hey everyone! I arrived at this conversation by investigating on issue (seems to me like a duplicate of ). I'm wondering if there's something I can help with? I'm still a bit new to the codebase, though. I was able to reproduce the problem by using the following ginkgo focus command in the e2e framework: And do see that if we manually add the --check-volume-lifecycle=false to the statefulset/csi-hostpathplugin tests will pass correctly. I also see that during the process, something in the cleanup function doesn't correctly delete the pv's after the tests are done, so they remain alive if we execute a kubectl get pv command.
I think I found a bit of more info on the issue probably could be of some help. I think that the volume is not being correclty marked as unmounted inside in GenerateUnmountVolumeFunc. I'm not sure why but I'll continue to investigate on it. I think this error is coming from DeletePodFromVolume in
I a couple of extra debug lines to that lines in the file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn't find it in the asw.attachedVolumes struct, like so: Probably we should search for the name of the attached volume differently for ephemeral volumes?
/assign
There are two problems: 1) The IsLikelyNotMountPoint() call that , which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with "... is not mounted", the PVC never never gets to the list of attached volumes, and that's why we end up hitting this error: 2) CSI inline volumes hit a different error (in addition to the first problem). The call path looks like this (starting in ): Inline volumes do not have a PV spec, so it fails . We should be calling instead.

There is an ongoing that changes to utilize syscall to detect mount point by using . By using , bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize in . Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.


Question: glog.Errorf("No ip address for node %v", node.Name) } else { address := api.NodeAddress{Type: api.NodeLegacyHostIP, Address: addrs[0].String()} <del> node.Status.Addresses = append(node.Status.Addresses, address) </del> <ins> api.AddToNodeAddresses(&node.Status.Addresses, address) </ins> } } }

Since we append hostips to the list on every node controller run invocation which happens at 10s intervals, this list grows pretty large pretty soon: Maybe: would still satisfy the requirements?
Fix is here I think we generally use list instead of map
This could have been P1, but I just merged a fix so I'll leave at P2. Remaining work to do: Add to validation something that limits the number of IP addresses. (and maybe checks dups?)
Yeah, I'll add validation later.


Question: return nil } <ins> // checkPodActive checks if the given pod is still in activePods list func (m *ManagerImpl) checkPodActive(pod *v1.Pod) bool { activePods := m.activePods() for _, activePod := range activePods { if activePod.UID == pod.UID { return true } } return false } </ins> // GetDeviceRunContainerOptions checks whether we have cached containerDevices // for the passed-in <pod, container> and returns its DeviceRunContainerOptions // for the found one. An empty struct is returned in case no cached state is found.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Pod worker panic when trying to re-allocating the device plugin resource during the node restart. The issue is potentially caused by an un-matching pod id. The map will be later used in an , which caused panic. ( func by itself is not thread-safe) Stack trace: DeviceManager Allocate() should be able to handle such race safely. Customer reported issue after upgrading to 1.18+. This is the new behavior introduced since 1.18+ from . Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/assign looking into the issue
I think your PR is related to this issue.
/unassign seems pretty relatable to PR cc
/triage accepted /priority critical-urgent
/remove-priority critical-urgent I don't think this is a release blocker at this point given the history reading through ; we can fix this in a cherry-pick.
Hi all, We are the GCP/GKE customer from which this issue originates. I can provide more information: it happened multiple times on our production cluster since we upgraded to 1.18 it happens in 2 scenarios: when GCP has a "host maintenance" event on our VMs with GPU (they don't support VM live migrations when GPU are involved), it forces a VM reboot: when multiple pods on that k8s node request a GPU prior to the reboot, we sometimes get the kubelet panic on node restart. when GKE upgrades our node-pools: new nodes (fresh VMs) get multiple scheduled pods requesting a GPU, sometimes leading to the kubelet panic. =this happens outside of our control The end result is a Pod scheduled on a node, but with no container ever created: our production workload is effectively broken until we manually delete the stuck pods. More details about our device usage: we use a of the each node has one GPU, and our device plugin declares N devices per real GPU; nvidia/cuda supports multi processing so it effectively shares the GPU among N pods/containers.

(our workload is memory-gpu bound, and uses <1/Nth of the gpu memory each, so no conflict) it worked great for almost 3 years now, with N=8
/cc
Now that we have confirmation that it occurs in live cluster I will take a closer look later this week.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" clientset "k8s.io/client-go/kubernetes" <del> apiv1pod "k8s.io/kubernetes/pkg/api/v1/pod" </del> <ins> "k8s.io/kubectl/pkg/util/podutils" </ins> "k8s.io/kubernetes/test/e2e/framework" e2ekubectl "k8s.io/kubernetes/test/e2e/framework/kubectl" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"

Several test/e2e/framework/* packages have as their only dependency on k/k. The following functions are shared that way: IsPodAvailable returns true if a pod is available; false otherwise. IsPodReady returns true if a pod is ready; false otherwise. VisitContainers invokes the visitor function with a pointer to every container spec in the given pod spec with type set in mask. If visitor returns false, visiting is short-circuited. VisitContainers returns true if visiting completes, false if visiting was short-circuited. AllFeatureEnabledContainers returns a ContainerType mask which includes all container types except for the ones guarded by feature gate. The feature gate check in AllFeatureEnabledContainers is problematic. E2E tests don't know what the feature gates in the cluster are. The rest of the functions look like something that could be moved into == Having the functions in makes it clearer that the checks only depend on aspects of the public API. Reduces dependencies of e2e/framework on k/k, in case that we want to move it into staging. Prevents accidentally using checks that only work in-tree (like feature gates).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/assign


Question: hc.NetworkMode = dockercontainer.NetworkMode(sandboxNSMode) hc.IpcMode = dockercontainer.IpcMode(sandboxNSMode) hc.UTSMode = "" <del> hc.PidMode = "" </del> if hostNetwork { hc.UTSMode = namespaceModeHost

Kubernetes version (use ): 1, 1.6.0 (both run into this issue) Environment: Cloud provider or hardware configuration: VirtualBox 5.1.18 VM OS: 16.04.2 LTS (Xenial Xerus) Kernel: Install tools: kubeadm What happened: A pod scheduled using hostPID wasn't using the host's PID namespace. I've investigated this issue together with What you expected to happen: I expected in the Docker container's config. Instead, the scheduled pod's container had How to reproduce it: Use this config to get a pod running a Docker container: Use on the running container and check whether PidMode is set to host or not. Anything else we need to know*: This was working properly on Kubernetes 1.5.3.
doesn't look like you've specified Docker version, it may be crucial here.
I've tested this with Docker 17.03.1-ce and can confirm that it's the case for pods.
It was Docker 1.12.6 from what I know. I'll check again to be sure it was 1.12.6 and update the post.
Is this with the CRI? If you disable the CRI do you get the desired outcome?
I don't know if that's enabled by default. I've noticed that hostPID doesn't work as it should with Docker. I'm not familiar with CRI.
I'll test with .
With CRI disabled it doesn't work on Docker 17.03, simply because it doesn't like the new versioning scheme. it'd be really great if you can try setting in your , e.g. by modifying (will need to do , and after that you either reinitialise the cluster or kill all the pods).
Seems a bug of kuberuntime, will file a PR to fix this.
Filed for this.
Reoppened this issue to ensure we cherrypick the fix to the 1.6 branch
has been merged to the release-1.6 branch. From what has been discussed so far, this will make it to Kubernetes 1.6.2. I'll close this issue now.
Can anyone help how this issue is fixed? I tried with pidMode: host but it is failing. I am using i.11 version.
I found it --hostPID: true


Question: readonly master_ssh_supported_providers="gce aws kubemark" readonly node_ssh_supported_providers="gce gke aws" <del> readonly master_logfiles="kube-apiserver kube-scheduler kube-controller-manager etcd glbc cluster-autoscaler" </del> <ins> readonly master_logfiles="kube-apiserver kube-scheduler rescheduler kube-controller-manager etcd glbc cluster-autoscaler" </ins> readonly node_logfiles="kube-proxy" readonly aws_logfiles="cloud-init-output" readonly gce_logfiles="startupscript"

We need to have collected by: fluentd jenkins
could you please take a look?
I this is actually two separate problems. Exporting as artifact from master is easy. Exporting logs to the logging backend will require fluentd configuration changes =images version changes, which are blocked by and


Question: var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, svcProto) <del> } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIP().String(), endpointIP, svcProto) </del> <ins> if err != nil { klog.Errorf("Failed to delete nodeport-related %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) } </ins> } <ins> err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIP().String(), endpointIP, svcProto) </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Performing a on a UDP-based service using nodePort leads to traffic from another pod to service being interrupted. Flushing manually the conntrack entry is enabling the traffic to flow again. What you expected to happen: Traffic to continue flowing after rollout. How to reproduce it (as minimally and precisely as possible): The following using some basic python UDP code is exhibiting the issue when the rollout command is executed. The issue is happening often (2 times over 3 I would say). Anything else we need to know?: I notice the k8s doing the flushing is using a conntrack parameter that seems to be ineffective in that particular usecase: However, the following altered command line works: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-26T14:23:04Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: baremetal OS (e.g: ): NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 20.04.1 LTS" VERSIONID="20.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=focal UBUNTUCODENAME=focal Kernel (e.g.

): Linux falcon 5.4.0-52-generic -Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm v.1.19.3 Network plugin and version (if this is a network-related bug): calico , deployed using calico-typha operator - no network policy or endpoint objects deployed. Others: Bug originally found on a kops-based cluster with 1.17.9 with SIP based traffic on port 5060, we checked on a more recent baremetal 1.19 cluster deployment.
/sig network
Proxy mode is iptables btw.
/cc
so , the whole logic to delete conntrack entries in kube-proxy happens here The code that you link has an interesting comment that seems to link to an old issue with, the same problem you are reporting??? I think that the problem is here if is a nodePort it only deletes the NAT entries that are directed to the NodePort, but as we can see in your example, we are not deleting the entries directed to the ClusterIP, that as you demonstrated it seems to work. I think that removing the in that part of the code, so we ALWAYS delete the cluster IP -endpoint conntrack entries will solve the problem are you interested in take on this and fix it, and correct me if I'm wrong?
tentative PR
/assign /triage accepted


Question: } // set namespace options and supplemental groups. <del> podSc := pod.Spec.SecurityContext if podSc == nil { return synthesized } </del> synthesized.NamespaceOptions = &runtimeapi.NamespaceOption{ HostNetwork: pod.Spec.HostNetwork, HostIpc:     pod.Spec.HostIPC, HostPid:     pod.Spec.HostPID, } <del> if podSc.FSGroup != nil { synthesized.SupplementalGroups = append(synthesized.SupplementalGroups, *podSc.FSGroup) </del> <ins> podSc := pod.Spec.SecurityContext if podSc != nil { if podSc.FSGroup != nil { synthesized.SupplementalGroups = append(synthesized.SupplementalGroups, *podSc.FSGroup) } if podSc.SupplementalGroups != nil { synthesized.SupplementalGroups = append(synthesized.SupplementalGroups, podSc.SupplementalGroups...) } </ins> } if groups := m.runtimeHelper.GetExtraSupplementalGroupsForPod(pod); len(groups) > 0 { synthesized.SupplementalGroups = append(synthesized.SupplementalGroups, groups...) } <del> if podSc.SupplementalGroups != nil { synthesized.SupplementalGroups = append(synthesized.SupplementalGroups, podSc.SupplementalGroups...) } </del> return synthesized }

Kubernetes version (use ): 1, 1.6.0 (both run into this issue) Environment: Cloud provider or hardware configuration: VirtualBox 5.1.18 VM OS: 16.04.2 LTS (Xenial Xerus) Kernel: Install tools: kubeadm What happened: A pod scheduled using hostPID wasn't using the host's PID namespace. I've investigated this issue together with What you expected to happen: I expected in the Docker container's config. Instead, the scheduled pod's container had How to reproduce it: Use this config to get a pod running a Docker container: Use on the running container and check whether PidMode is set to host or not. Anything else we need to know*: This was working properly on Kubernetes 1.5.3.
doesn't look like you've specified Docker version, it may be crucial here.
I've tested this with Docker 17.03.1-ce and can confirm that it's the case for pods.
It was Docker 1.12.6 from what I know. I'll check again to be sure it was 1.12.6 and update the post.
Is this with the CRI? If you disable the CRI do you get the desired outcome?
I don't know if that's enabled by default. I've noticed that hostPID doesn't work as it should with Docker. I'm not familiar with CRI.
I'll test with .
With CRI disabled it doesn't work on Docker 17.03, simply because it doesn't like the new versioning scheme. it'd be really great if you can try setting in your , e.g. by modifying (will need to do , and after that you either reinitialise the cluster or kill all the pods).
Seems a bug of kuberuntime, will file a PR to fix this.
Filed for this.
Reoppened this issue to ensure we cherrypick the fix to the 1.6 branch
has been merged to the release-1.6 branch. From what has been discussed so far, this will make it to Kubernetes 1.6.2. I'll close this issue now.
Can anyone help how this issue is fixed? I tried with pidMode: host but it is failing. I am using i.11 version.
I found it --hostPID: true


Question: ) // NewProxyServer returns a new ProxyServer. <del> func NewProxyServer(config *proxyconfigapi.KubeProxyConfiguration, cleanupAndExit bool, scheme *runtime.Scheme, master string) (*ProxyServer, error) { </del> <ins> func NewProxyServer(o *Options) (*ProxyServer, error) { return newProxyServer(o.config, o.CleanupAndExit, o.scheme, o.master) } func newProxyServer(config *proxyconfigapi.KubeProxyConfiguration, cleanupAndExit bool, scheme *runtime.Scheme, master string) (*ProxyServer, error) { </ins> if config == nil { return nil, errors.New("config is required") }

https://k8s- https://k8s- cc: /kind bug /priority failing-test
This is at least twice in a week we've had cross break due to a public method signature change not being brought to all platforms () We should do:
I'm lost - the PR in question adds an argument and fixes the call-site - what is broken?
Ahh, I see.
How did that PR get through, if it is so obviously broken on Cross?
Cross build doesn't run 100% of the time as a pre-submit (because it takes so long), and it can't block. It's something sig-testing is tracking, but there isn't a quick fix yet.
Looks like was skipped for that PR: "pull-kubernetes-cross Skipped" not sure why yet.
Cross runs when a file matching: changes, (which needs to be fixed, opening a test-infra issue to expand the regex), but also some files could have build tags without the file name matching () Additionally cross is not blocking (), the underlying bug should be fixed now though so once we get the fix in to k/k we can make it blocking when it does run.
I find has already sent a PR. Thanks!


Question: deps = [ "//pkg/api:go_default_library", "//pkg/apis/componentconfig:go_default_library", <ins> "//pkg/apis/componentconfig/v1alpha1:go_default_library", </ins> "//pkg/util:go_default_library", <ins> "//pkg/util/configz:go_default_library", </ins> "//pkg/util/iptables:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/diff:go_default_library", ], )

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? is:issue is:open ipv6 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): N/A, these are unit test changes in latest master Environment: Cloud provider or hardware configuration: Kubernetes container dev env on Ubuntu 16.04.1 OS (e.g. from /etc/os-release): Ubuntu 16.04.1 Kernel (e.g. ): 4.4.0-78-generic -Ubuntu x8664 GNU/Linux Install tools: Others: What happened: Upon inspection, no IPv6 test cases found in kubernetes/cmd/kube- What you expected to happen: Expect to see some IPv6 test cases exercised for kube-proxy server code (kubernetes/cmd/kube-). Also need some code coverage for negative conditions in How to reproduce it (as minimally and precisely as possible): Run unit tests in kubernetes/cmd/kube- Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig networking
/area ipv6
/sig network


Question: r = append(r, reporters.NewJUnitReporter(path.Join(framework.TestContext.ReportDir, fmt.Sprintf("junit_%v%02d.xml", framework.TestContext.ReportPrefix, config.GinkgoConfig.ParallelNode)))) } } <del> klog.Infof("Starting e2e run %q on Ginkgo node %d", framework.RunID, config.GinkgoConfig.ParallelNode) </del> <ins> // Stream the progress to stdout and optionally a URL accepting progress updates. r = append(r, e2ereporters.NewProgressReporter(framework.TestContext.ProgressReportURL)) klog.Infof("Starting e2e run %q on Ginkgo node %d", framework.RunID, config.GinkgoConfig.ParallelNode) </ins> ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }

What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func (c *gitRepoVolumeUnmounter) TearDownAt(dir string) error { // Wrap EmptyDir, let it do the teardown. <del> wrapped, err := c.plugin.host.NewWrapperUnmounter(c.volName, wrappedVolumeSpec, c.podUID) </del> <ins> wrapped, err := c.plugin.host.NewWrapperUnmounter(c.volName, wrappedVolumeSpec(), c.podUID) </ins> if err != nil { return err }

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question: return duration } <ins> // contextForChannel derives a child context from a parent channel. // // The derived context's Done channel is closed when the returned cancel function // is called or when the parent channel is closed, whichever happens first. // // Note the caller must *always* call the CancelFunc, otherwise resources may be leaked. func contextForChannel(parentCh <-chan struct{}) (context.Context, context.CancelFunc) { ctx, cancel := context.WithCancel(context.Background()) go func() { select { case <-parentCh: cancel() case <-ctx.Done(): } }() return ctx, cancel } </ins> // ExponentialBackoff repeats a condition check with exponential backoff. // // It checks the condition up to Steps times, increasing the wait by multiplying

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--Related to , there remains a leak in which does not ensure the channel it passes to is closed. This causes goroutines to leak under common scenarios, such as when calling . /cc What happened: Each call to leaks an active goroutine (in this case spawned by via . The goroutine will not end until the passed to is closed - however this requirement can and should be handled internally. Often callers will pass a context with a longer lifespan. What you expected to happen: should ensure that the polling goroutine is ended after its condition is met. This can be accomplished by creating a new channel with a child cancellation relationship to the original, and passing that on instead. This child channel can be closed once returns. How to reproduce it (as minimally and precisely as possible): The minimal reproduction would be to call with a true condition, and not close the passed . However, a more complete reproduction involving a can be seen . Anything else we need to know?: This leak was discovered in practice and caused a steady leak of memory and cpu in a project I work on. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
fyi Oren has 2 options to fix this. He'll be pasting sample code shortly and we can decide which approach we prefer.
Hi everyone! As promised, I've solved this two similar ways, and wanted to get your temperature on them before submitting the PR. I can then create a child context and pass its done channel to : This ensures that poller's goroutine will be cleaned up either when stopCh is closed or when the condition is met, whichever comes first. If there is aversion to using contexts here, we can mimic the subset of functionality from context that we need (safe to cancel from multiple goroutines + and a parent-child cancellation relationship): I lean towards option 1 and contexts - their semantics are familiar and result in less code on our end. Let me know what you think!

I would prefer using contexts - we actually should in the long term have context variants of all waits, so it’s the right direction to start using them more here
+1 to option 1
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten