Question: os.Setenv(KubeMaxPDVols, previousValue) } } <ins> func TestCheckNodeUnschedulablePredicate(t *testing.T) { testCases := []struct { name string pod  *v1.Pod node *v1.Node fit  bool }{ { name: "Does not schedule pod to unschedulable node (node.Spec.Unschedulable==true)", pod:  &v1.Pod{}, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: true, }, }, fit: false, }, { name: "Schedule pod to normal node", pod:  &v1.Pod{}, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: false, }, }, fit: true, }, { name: "Schedule pod with toleration to unschedulable node (node.Spec.Unschedulable==true)", pod: &v1.Pod{ Spec: v1.PodSpec{ Tolerations: []v1.Toleration{ { Key:    algorithm.TaintNodeUnschedulable, Effect: v1.TaintEffectNoSchedule, }, }, }, }, node: &v1.Node{ Spec: v1.NodeSpec{ Unschedulable: true, }, }, fit: true, }, } for _, test := range testCases { nodeInfo := schedulercache.NewNodeInfo() nodeInfo.SetNode(test.node) fit, _, err := CheckNodeUnschedulablePredicate(test.pod, nil, nodeInfo) if err != nil { t.Fatalf("Failed to check node unschedulable: %v", err) } if fit != test.fit { t.Errorf("Unexpected fit: expected %v, got %v", test.fit, fit) } } } </ins>

https://k8s- This test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all. /kind bug /priority failing-test /sig storage /assign /cc /milestone v1.12
looks like some permissions issue after upgrade /assign
has triaged this to an understanding that the failures happen because the pod is scheduled on the master and getting logs through the method they're using is disabled on master nodes. This makes me think this is similar to and we have non-release-blocking issues in test cases not being quite robust enough now in the face of node taints.
It is interesting that in some places this test is always failing, some it is always succeeding, and even in one it flakes but with a high success rate: https://k8s-
Adding on to comment: The pod that we are creating does not have any special "master schedulable" annotations or tolerations, so from my understanding this should still be not scheduled to the master for any reason. The Pod Spec in question: Potentially relevant scheduler logs.
The fix mentioned in is only handling DaemonSets. I don't think it will fix this issue.
Do we need to get sig-scheduling involved to see why the master is showing up as a choice for this pod?
/sig scheduling Can you triage this for 1.12 release blocking?
It seems that this pvc-tester- pod is only scheduled to master in every run which caused the test to fail consistently.
However in this test suite, it seems like the test passes most of the time (pod is scheduled to a node), then occasionally flakes (pod scheduled to master).: https://k8s- Whereas in this one it doesn't seem to fail at all!: https://k8s- I talked to about why the behavior might be different across different test suites but we were unable to come up with anything concrete.
Is the master node in your test environment tainted properly?
When we were doing unrelated development with DaemonSets, we noticed a similar issue where our DaemonSet pod would get scheduled on gce environment using hack/cluster-, and we noticed masters brought up using that script were not tainted
What release was the tainting master logic ?
cc we had similar issue that gpu driver installer is being scheduled on master at , can also be related?

Not sure if this is a source of truth on the cluster but the test artifacts here on a successful run show that the master node doesn't have a taint: and on a failed run ALSO no taint: But I'm not really sure where this file comes from. And whether this is the cluster before or after the upgrade
ie:
Don't think will help. It looks specific to gpu labeling
Also ^^ is GKE specific, and GKE docu says no taints are inherited to a pool from the cluster, so shouldn't be something from that direction.
Some log snippets around the area of interest:
/priority critical-urgent
I think the fix is obvious: we need to taint the master node on the gce master setup in cluster/. What I still don't understand is how did we get this far without anything else failing, and why only this one test?
Do taints persist across master upgrades, or do they need to be re-applied again?
Based on Slack discussion and the triage so far, this looks like it's a case of unfortunate cluster/* directory bit rot. We can add the taint and have time before the release still to see clear CI signal, but I would not rate this as a release blocker, especially considering node-"" appears to have been introduced in 1.6.
/assign I'm going to contribute in my daytime :)
I think there's what happened: only update status when upgrading from 1.11 to 1.13, as the node was previously registered (1.11); so is not updated, including unschedulable taint. scheduler is 1.13 now, TaintNodesByCondition is enabled by default and NodeConditionCheck is disabled by default, so it ignores controller manager is 1.13 now, it'll taint nodes according to node's configurations, e.g. . Because (including unschedulable) is not updated by kubelet when upgrading, it has to dependent on controller manager (nodelifecycle) to update it which has race condition with kube-scheduler.

There's an another issue that, we'll reject taints update to Node in noderestriction admission controller :( For the solution, I think we need to patch (at least ) and add a whitelist to noderestriction admission controller for known taints key; we can not use to start the kubelet, as it also be skipped (only update node status). Refer to the following logs for what happened: All logs are from https://k8s- Build- The taint of master; the taint will include if tainted by controller manager. scheduler-
/sig node
if no comments on the solution, i will open a pr for it.
cc on relaxing node restriction for this upgrade case Could another potential solution be to keep the previous scheduler node.unschedulable logic for one more release in order to handle this 2 version upgrade scenario?
Actually to clarify, is this both a 1.10 to 1.12 and 1.11 to 1.12 issue?
please get a PR started so we can get review happening ASAP
Thanks, for the investigation. Initially, I wanted to suggest to keep the logic of checking node condition in the scheduler along side TaintNodeByCondition for one more release, but I guess there would be conflicts. For example, a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition. So, looks like we'll have to fix the tainting logic.
can you clarify on what fixing the tainting logic would entail? If I understand your example, wouldn't the Pod object need to be updated to also tolerate the taint that the condition got converted to?
I was specifically referring to what suggested: "I think we need to patch node (at least node.Spec.Taints) and add a whitelist to noderestriction admission controller for known taints key." Yes, the pod needs to have the right toleration for the taint. We already create our critical system pods with proper tolerations for certain node taints, such as unreachable, etc.
I'm going back to feeling this is actually a release blocking upgrade issue. please clarify in more detail your concern for breakages, the possible paths for resolution, and preferred path.
I agree with you that this is release blocking. We commented at the same time.

I mentioned above that I think we need to address tainting of the nodes in kubelet.
Who's the right owner for this now? I'm assuming we've lost Klaus to the weekend in his timezone, but some of us still have a few work hours yet on Friday...
I already pinged Dawn as our Node guru, but didn't get any reply. She might be on vacation. Let me see if I can find other folks here.
cc for what we can do on the node side. cc for the security/policy related matters associated with adding taints for node condition
help please
regarding your example "a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition", could this be an API compatibility issue? Will a pod that tolerates a node condition in 1.11 fail to schedule in 1.12 because the scheduler no longer looks at node conditions and only node taints?
Pods were not able to tolerate node conditions by setting some API fields. We had some annotations, such as pod critical annotation and other hard-coded logic in DS controller, etc. that would have the same effect of tolerating conditions, but they were not in our API.
Thanks for the explanation. It looks like the critical pod annotation was alpha, so does not have a backwards compatibility requirement, and we've fixed all the known users of it.
xref: - cc Should this be handled by the node controller? I.e. if the Unschedulable bool is set on the node, add the unschedulable taint?
I don't believe the kubelet has the ability to update its taints anymore. Assuming this is the case, this seems like a version skew problem between master components. I.E. the 1.11 controller manager isn't compatible with the 1.13 scheduler because it doesn't add taints that the scheduler requires. This means that if the scheduler is upgraded before the controller manager (or before it has the chance to perform updates), we can run into issues. I think making the scheduler respect conditions in 1.13, and thus making it compatible with the 1.11 controller manager, seems like the best remedy to this.

Regardless of this issue, user pods would need to include both the critical annotation as well as the toleration in order to maintain the same behavior across releases that transition from conditions to taints.
Oh, I see that the node controller is already doing that, but there's a race condition (), sorry I missed that earlier.
Yes, translation of conditions and unschedulable field to taints is done by the controller
I wonder if we can change our upgrade flow to ensure that the controller manager is upgraded and becomes stable (adds the taints) before the scheduler is upgraded.
The kubelet has never required the ability to update taints. Was that a typo, supposed to be 1.12? Have we ever supported two version skew between those components? Under what circumstances do we upgrade to 1.13 scheduler while a 1.11 controller manager is still running? Same question about two version skew between these two components. That's more concerning… is that something all user pods would be exposed to and required to change?
+1 to suggestion in to unblock the release
What about HA master upgrade? scheduler and controller-manager each have independent leader elections, so during a HA rolling upgrade, the current leaders for scheduler and controller managers could be on different releases in either direction
I wouldn't expect an upgrade from 1.12 to 1.13 to begin until the upgrade from 1.11 to 1.12 was complete/successful
Thanks for correcting me. It wasn't a typo, but just a lack of knowledge about how multi-version upgrades are performed. In that case we need to support 1 version skew. Yeah, it should be 1.12, not 1.13. But generally, the newer scheduler should still be compatible with the state left by the previous version's controller manager, even if it isn't running anymore. That's more concerning… is that something all user pods would be exposed to and required to change? Actually, I agree. have we officially deprecated the critical pod annotation? We should ideally add the correct tolerations to pods with the critical pod annotation to ensure backwards compatibility.
Bobby commented that the system pods have already been corrected. I also noted that the critical pod annotation is alpha.
I mean doing an HA rolling upgrade from 1.11 to 1.12.

Let's say we have 3 masters, A, B, C, all on 1.11. is upgraded to 1.12, B and C are still on 1.11. leader election could pick scheduler A as the leader, controller-manager leader election could pick controller-manager B as the leader. Now we have version skew between master components.
That feature (critical pods) was an alpha feature and we have already replaced it. The dilemma is that I am not so sure if critical pods were the only kind of pods with a "hack" for tolerating node conditions. Let me dig a bit more.
One version of skew, absolutely. comment made me concerned two versions of skew support were being claimed/proposed
I looked the scheduler code and the DaemonSet controller code to find if adding "CheckNodeCondition" could cause issues in presence of TaintNodeByCondition and I didn't find anything particularly concerning. So, I created . Let's see if it breaks tests.
that will break schedule daemonset by scheduler. and registerwithtaints of kubelet is not worked in upgrade.
I suspected that it would conflict with pods which now have tolerations and the scheduler should ignore node conditions for, but I thought it would be limited to critical pods. Thanks for pointing it out. I closed the PR. As an alternative solution, I suggested changing the order of upgrading server components in our upgrade process, so that the controller manager is upgraded before the scheduler, but I am not sure how easy it is to implement. Do you have any alternative fix in mind?
unschedulable is taint by controller & kubelet; taints node with unschedulable to avoid race condition between controller and scheduler, it dependents on of kubelet to taint when start up; but does not work in upgrade case as mentioned above (we only update status when upgrading). IMO, it should be fixed; it'll also fix upgrade e2e test but may not work in real scenario which upgrade master firstly. let me think about it, I'll propose one today.
The race that is important to resolve at creation time is that of a new node object appearing untainted. That race does not apply in an upgrade scenario, where the node object already exists, and can already be observed and acted upon. Relying on the condition-to-taint controller in upgrade scenarios is reasonable.

The race after kubelet start up is well known; but we did not handle startup case well. I think what we're doing is to deprecate , and replace it with unschedulable taint. We need to keep backward compatibility for it.


Question: meta.Annotations[api.AffinityAnnotationKey] = string(affinityAnnotation) } <ins> // MasterNodeAffinity returns api.NodeSelectorRequirement to be used with SetNodeAffinity to set affinity to master node func MasterNodeAffinity() api.NodeSelectorRequirement { return api.NodeSelectorRequirement{ Key: "kubeadm.alpha.kubernetes.io/role", Operator: api.NodeSelectorOpIn, Values: []string{"master"}, } } // NativeArchitectureNodeAffinity returns api.NodeSelectorRequirement to be used with SetNodeAffinity to nodes with CPU architecture // the same as master node func NativeArchitectureNodeAffinity() api.NodeSelectorRequirement { return api.NodeSelectorRequirement{ Key: "beta.kubernetes.io/arch", Operator: api.NodeSelectorOpIn, Values: []string{runtime.GOARCH}, } } </ins> func createDummyDeployment(client *clientset.Clientset) { fmt.Println("<master/apiclient> attempting a test deployment") dummyDeployment := NewDeployment("dummy", 1, api.PodSpec{

Right now we create 3 daemonsets for in , but will only give use image names for , so this feature is broken. We should only deploy one daemon set for the native architecture, but ensure it has the right label selectors, so we don't try to run pods on nodes of different architecture. We should also make sure all addons have architecture labels, and not just . If user is building heterogeneous clusters, the will need to take about deploying addons for additional architectures, we shouldn't pre-deploy multiple things knowing that they will most likely remain unschedulable.
I have the same issue as well. I tried to create a cluster with x86 (Intel NUC) and arm (Raspberry-Pi) and I see that the kube-proxy is crashing. Here's the log entry from kube-proxy Logs from kube-proxy exec user process caused "exec format error" Attached is the kube-proxy yaml file. I see that its picking up arm image Don't know how to fix this.


Question: {"invalid: CRI rmp failure", "unix:///var/run/crio/crio.sock", []string{"k8s_p1", "k8s_p2", "k8s_p3"}, true}, {"invalid: CRI stopp failure", "unix:///var/run/crio/crio.sock", []string{"k8s_p1", "k8s_p2", "k8s_p3"}, true}, {"valid: remove containers using docker", constants.DefaultDockerCRISocket, []string{"k8s_c1", "k8s_c2", "k8s_c3"}, false}, <del> {"invalid: remove containers using docker", constants.DefaultDockerCRISocket, []string{"k8s_c1", "k8s_c2", "k8s_c3"}, true}, </del> <ins> {"invalid: docker rm failure", constants.DefaultDockerCRISocket, []string{"k8s_c1", "k8s_c2", "k8s_c3"}, true}, {"invalid: docker stop failure", constants.DefaultDockerCRISocket, []string{"k8s_c1", "k8s_c2", "k8s_c3"}, true}, </ins> } for _, tc := range cases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Run on a work node of kubernetes. Containers are killed, instead of exiting normally. Some resources are not released such as vips created by keepalived. What you expected to happen: The container exits properly so that system resources are released correctly How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: , line 139: Can I try to stop containers before removing them? Environment: Kubernetes version (use ): v1.18.5 Cloud provider or hardware configuration: OS (e.g: ): CentOS 7.4.1708 Kernel (e.g. ): 3.10.0-1127.18.2.el7.x86_64 Install tools: kubeadm
/sig cluster-lifecycle These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/area kubeadm /remove-kind bug /kind feature there is some discussion / Qs on the PR.


Question: "github.com/golang/glog" "k8s.io/apimachinery/pkg/api/errors" <del> "k8s.io/apimachinery/pkg/labels" </del> "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

For reasons that are not entirely clear to me, and as spotted by in , there are two functions for selecting nodes in includeNodeFromNodeList & getNodeConditionPredicate. I think includeNodeFromNodeList should use the logic in getNodeConditionPredicate
/sig network


Question: func (rc *reconciler) Run(stopCh <-chan struct{}) { rc.reconstructVolumes() klog.InfoS("Reconciler: start to sync state") <del> wait.Until(rc.reconcileNew, rc.loopSleepDuration, stopCh) </del> <ins> wait.Until(rc.reconcile, rc.loopSleepDuration, stopCh) </ins> } <del> func (rc *reconciler) reconcileNew() { </del> <ins> func (rc *reconciler) reconcile() { </ins> readyToUnmount := rc.readyToUnmount() if readyToUnmount { // Unmounts are triggered before mounts so that a volume that was

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: the Job MUST execute to completion. release: v1.16 file: test/e2e/apps/job.go <ins> - testname: ReplicaSet, is created, Replaced and Patched codename: '[sig-apps] ReplicaSet Replace and Patch tests [Conformance]' description: Create a ReplicaSet (RS) with a single Pod. The Pod MUST be verified that it is running. The RS MUST scale to two replicas and verify the scale count The RS MUST be patched and verify that patch succeeded. release: v1.21 file: test/e2e/apps/replica_set.go </ins> - testname: ReplicaSet, completes the scaling of a ReplicaSet subresource codename: '[sig-apps] ReplicaSet Replicaset should have a working scale subresource [Conformance]'

[x] two weeks soak end date : 8 March 2021 According to this APIsnoop query, there are still some remaining Replicaset endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, core endpoints replaceAppsV1NamespacedReplicaSet does not show up as untested as it touched by another not promoted to Conformance. Create a ReplicaSet with a static label Read the ReplicaSet info Scale the ReplicaSets to 2 Patch the ReplicaSet Read the ReplicaSet to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent --------------------------------------|
createAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet ReplicaSet lifecycle tests patchAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet ReplicaSet lifecycle tests readAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet ReplicaSet lifecycle tests replaceAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet ReplicaSet lifecycle tests (4 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance /sig apps
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: glog.Errorf("Failed while searching for datastore %+q. err %s", datastorePathObj.Datastore, err) return "", "", err } <ins> vmDiskPath = removeClusterFromVDiskPath(vmDiskPath) </ins> disk := vmDevices.CreateDisk(scsiController, ds.Reference(), vmDiskPath) unitNumber, err := getNextUnitNumber(vmDevices, scsiController) if err != nil {

With version 1.4.7 I could use a datastore that is in a cluster as a vsphere volume as . When I upgraded to 1.6.0, this same exact path does not work and throws a error. I'm struggling to find what changed, but am guessing it's difference in how the datastore is being looked up in vcenter. Has anyone had luck with this?
We have verified creating pod using static PV and dynamic provisioning, both works fine with Can you post here? cc:
Did that include using a datastore that was in a datastore cluster? That seemed to be working before because the datastore previously (before 1.5) was read out of the global config, which supported datastore path with cluster name. But post 1.5 datastore is parsed from the entry, which if you include the path to the cluster it gets you past the datastore lookup, but then when you go to add the disk it fails because that is not valid to vCenter. It's kind of this weird mixed mode, because to look up the datastore you need the path including the cluster, but to create the file you need the path without the cluster. I presume the PV is working because they use the cloud config values? I was not using a PV but putting a volumePath directly in my deployment yaml. Should I be using a PV to do this? If the deployment is deleted and re-created how do I guarantee that my pod would get the same PV claim? That was why I did not go down that path originally. I'm pretty sure I have narrowed down where in the provider this is happening, but didn't see a good path to get it to work. I have, admittantly, not tried PV/PVCs yet, due to the concern raised above, but could try those to see. The pertinent part of the deployment yaml: The cloud config:
Just confirmed I have the same issue using a PV and PVC....
Here it needs it in format: (which it gets on line 791 by parsing . But here it needs it in true format, without the datastorecluster: Therefore if you do it gets you past line 796 but fails on 802. But if you do then you don't get past line 796.
Thanks for detailed look up. This is the commit which caused this regression. We will work on the fix.

cc:
Ah yes, I see that now. What is the lifecycle of a dynamic disk? Does that disk live on after the PVC is released? After the PV is deleted? It seems like I may just not fully be exercising the power of the volume plugins as well. I would still have the issue with existing volumes as they have data that I need to maintain, but would be interesting for new volumes to look into these more.
RECLAIMPOLICY for PV created using Dynamic Provisioning is Delete PV will get deleted if you delete PVC. Disk also gets deleted from datastore.


Question: kind: ConfigMap apiVersion: v1 metadata: <del> name: fluentd-es-config-v0.1.5 </del> <ins> name: fluentd-es-config-v0.1.6 </ins> namespace: kube-system labels: addonmanager.kubernetes.io/mode: Reconcile

Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective.

I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue.

If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together?

The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


Question: } func (m *manager) reconcileState() (success []reconciledContainer, failure []reconciledContainer) { <ins> if !m.sourcesReady.AllReady() { return } </ins> success = []reconciledContainer{} failure = []reconciledContainer{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: "k8s.io/kubernetes/test/e2e/framework/providers/gce" e2eresource "k8s.io/kubernetes/test/e2e/framework/resource" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" <ins> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" </ins> imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .


Question: if err != nil { return err } <ins> if cml.cm.Annotations == nil { cml.cm.Annotations = make(map[string]string) } </ins> cml.cm.Annotations[LeaderElectionRecordAnnotationKey] = string(recordBytes) cml.cm, err = cml.Client.ConfigMaps(cml.ConfigMapMeta.Namespace).Update(cml.cm) return err

What happened: Using leaderelection with ConfigMapLock, I get a panic when I fail to renew the lease. What you expected to happen: No crash How to reproduce it (as minimally and precisely as possible): Create a leaderelection setup with ReleaseOnCancel=true Run the program, allow a lock to be acquired. Change RBAC permissions to block configmap access Eventually we fail to acquire the lock and run into the panic above
/sig api-machinery
/assign
let me know if you need help getting more info, I am not sure if its easy to reproduce or not. I can add some extra debug logging, etc
I will see if I can reproduce via unit test.
/cc
Can you check the log around the time of crash to see if there was the following ? Checking the events around the time of crash would also help: thanks
I have the log snippet at the top, there is a
I can also hit race conditions from a unit test. I can make this a new issue if preferred. and similar but slightly different:
There is some race around accessing observedRecord. We can use a Lock to protect the access. Please log a new issue. Thanks


Question: return err } <del> if _, err := az.reconcileLoadBalancer(clusterName, service, nil, false /* wantLb */); err != nil { </del> <ins> if _, err := az.reconcileLoadBalancer(clusterName, service, nil, false /* wantLb */); err != nil && !retry.HasStatusForbiddenOrIgnoredError(err) { </ins> return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/fields" </ins> "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/kubernetes/pkg/kubelet/events" </ins> "k8s.io/kubernetes/test/e2e/framework" e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: } func TestAzureTokenSourceScenarios(t *testing.T) { <del> configMode := configModeDefault </del> expiredToken := newFakeAzureToken("expired token", time.Now().Add(-time.Second)) extendedToken := newFakeAzureToken("extend token", time.Now().Add(1000*time.Second)) fakeToken := newFakeAzureToken("fake token", time.Now().Add(1000*time.Second))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl, authentication fails with the below error For some reason kubectl is adding a lot of s to the start of the SPN - this can be observed in What you expected to happen: Authentication works as in previous versions of Kubectl How to reproduce it (as minimally and precisely as possible): Create a cluster using Azure AD authentication. Authenticate from a client and wait until the token expires. Use Kubectl and have it automatically refresh the token; it will fail due to the extra 'spn:' entries. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: on-premise OS (e.g: ): macOS 10.14.6 Kernel (e.g. ): Darwin Hostname 18.7.0 Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASEX8664 x86_64 Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
/sig cli
Kubectl version 1.18.0 Kubernetes version 1.16.6
can you confirm whether your original kubeconfig already has in apiserver-id or not? in the mean time, you may try it can convert your kubeconfig from azure module to exec module and do the token thing outside kubectl
At this point if I remove the tokens and try to reauth i get the error described above. I've tried it both ways with apiserver-id having an initially and not, both lead to the same behavior in the end.
i know what's going on now. will prepare a fix. the root cause is that token's target resource (adal.Token.Resource) needs to have prefix and this prefix gets unexpectedly to as well, which gets written back to kubeconfig. So every time kubectl is run, it adds to apiserver-id
/priority important-soon
I'm seeing this with kubectl 1.17.3 and kubernetes 1.16.7.

It's unclear to me what versions are impacted and in what version it is fixed, since only version 1.18 is mentioned.
This still happens on kubectl 1.16.12 and 1.17.8
This still happens on kubectl v1.18.5
same
can you guys share your kubectl version --client and kubeconfig with me? you can find me on kubernetes slack. the slack user name is same as my github handle.
Version 1.17.6 works fine.
Since I was mistaken thinking was still causing this for me, I figured I would just drop a note for others in case it's helpful. Turns out my problem was actually due to and looks to have been resolved as of the v3.3.0 release.
This issue has now reappeared for me - although not necessarily the same root cause. AKS 1.19.0 Authentication works, but it spams frequently.


Question: --quiet } <ins> # Vars set: #   NODE_SUBNETWORK </ins> function create-subnetworks() { <ins> NODE_SUBNETWORK=$(gcloud beta compute networks subnets list  --network=${NETWORK}  --regions=${REGION}  --project=${PROJECT}  --limit=1  --format='value(name)' 2>/dev/null) if [[ -z ${NODE_SUBNETWORK:-} ]]; then echo "${color_red}Could not find subnetwork with region ${REGION}, network ${NETWORK}, and project ${PROJECT}" exit 1 fi echo "Found subnet for region ${REGION} in network ${NETWORK}: ${NODE_SUBNETWORK}" </ins> case ${ENABLE_IP_ALIASES} in true) echo "IP aliases are enabled. Creating subnetworks.";; false) echo "IP aliases are disabled." <del> if [[ "${ENABLE_BIG_CLUSTER_SUBNETS}" = "true" ]]; then </del> <ins> if [[ "${ENABLE_BIG_CLUSTER_SUBNETS}" = "true" ]]; then </ins> if [[  "${PREEXISTING_NETWORK}" != "true" ]]; then expand-default-subnetwork else

The "Services should be able to create an internal type load balancer [Slow]" e2e test is failing on our 2k-node gce clusters - https://k8s- with the following error: We're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them. cc
Seems similar to Assigning to you as you seem to have the context already :) Also, is this expected to be fixed in 1.8 timeframe?
doesn't set the necessary variables for the GCE cloud provider. Will throw you a PR.
Cool - thanks a lot!
Ref
I see the last test on 09/09 failed, but I can't see the build log due to exceeding memory. Any other way I can see the log, /reopen
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
nevermind - went directly to raw log.
Friendly ping. This e2e is still failing (https://k8s-) and it's part of release-blocking suite.
/close


Question: echo "OPTIONS='-b=kbr0 --selinux-enabled ${DOCKER_OPTS}'" >/etc/sysconfig/docker systemctl daemon-reload systemctl start docker <ins> systemctl restart kubelet </ins> } EOF

Self-assigning as I work out a reliable fix.
cc
Running into a few issues. On GCE, did you guys block somewhere to wait for etcd to be running before reporting success on kube-up?
No, except there is a race between addson services, dns and etcd container. We introduced a retry logic on those services.
I assume pulling the image is just so fast that you don't run into e2e errors, it looks like you guys block on wait-for-minions that uses glcloud, so I guess when minions are reported in etcd, then you know you are good...


Question: create_and_use_new_namespace kube::log::status "Testing secrets" <ins> # Ensure dry run succeeds and includes kind, apiVersion and data output_message=$(kubectl create secret generic test --from-literal=key1=value1 --dry-run -o yaml) kube::test::if_has_string "${output_message}" 'kind: Secret' kube::test::if_has_string "${output_message}" 'apiVersion: v1' kube::test::if_has_string "${output_message}" 'key1: dmFsdWUx' </ins> ### Create a new namespace # Pre-condition: the test-secrets namespace does not exist kube::test::get_object_assert 'namespaces' '{{range.items}}{{ if eq $id_field "test-secrets" }}found{{end}}{{end}}:' ':'


Question: package fake_test import ( <ins> "errors" </ins> "testing" <ins> "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/version" fakediscovery "k8s.io/client-go/discovery/fake" fakeclientset "k8s.io/client-go/kubernetes/fake" <ins> kubetesting "k8s.io/client-go/testing" </ins> ) func TestFakingServerVersion(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was trying to write a test in a service I'm working on to catch if an error returns from . When I overrode the result of the fake client with a reactor to return an error I did not get the error back. What you expected to happen: If the reactor is set to return an error, it should come back. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This only happens with the fake. Environment: Kubernetes version (use ): 1.14 Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: any Network plugin and version (if this is a network-related bug): n/a Others:
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Let's take a look at this /priority important-longterm /sig api-machinery /remove-sig testing /remove-lifecycle rotten
Is there any progress?
I submited a PR to fix this issue. PTAL .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
^^^ would it be possible to reconsider this PR? I ran into the same issue today.
this hasn't been fixed and we recently ran into this issue. Can we /reopen? Happy to bring up to date.
/reopen still relevant as above
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted
/assign


Question: BootID:                  "1b3", KernelVersion:           "3.16.0-0.bpo.4-amd64", OSImage:                 "Debian GNU/Linux 7 (wheezy)", <del> OperatingSystem:         "linux", Architecture:            "amd64", </del> <ins> OperatingSystem:         goruntime.GOOS, Architecture:            goruntime.GOARCH, </ins> ContainerRuntimeVersion: "test://1.5.0", KubeletVersion:          version.Get().String(), KubeProxyVersion:        version.Get().String(),

TestUpdateNewNodeStatus TestUpdateExistingNodeStatus TestUpdateNodeStatusWithRuntimeStateError The above tests seem to be architecture specific now (and fail on unexpected objects "linux" vs "darwin"). Per I would expect unit tests to still pass on OSX


Question: } func upgradeComponent(component string, waiter apiclient.Waiter, pathMgr StaticPodPathManager, cfg *kubeadmapi.MasterConfiguration, beforePodHash string, recoverManifests map[string]string) error { <ins> // Special treatment is required for etcd case, when rollbackOldManifests should roll back etcd // manifests only for the case when component is Etcd recoverEtcd := false if component == constants.Etcd { recoverEtcd = true } </ins> // The old manifest is here; in the /etc/kubernetes/manifests/ currentManifestPath := pathMgr.RealManifestPath(component) // The new, upgraded manifest will be written here

kubeadm etcd upgrade modifying recovery steps /assign /sig cluster-lifecycle /kind bug


Question: var badMetrics []LatencyMetric for _, metric := range metrics { if !ignoredResources.Has(metric.resource) && <del> metric.verb != "WATCHLIST" && </del> <ins> !ignoredVerbs.Has(metric.verb) && </ins> // We are only interested in 99%tile, but for logging purposes // it's useful to have all the offending percentiles. metric.quantile <= 0.99 &&

I saw the below failure in [Performance suite] should allow starting 30 pods per node. Some requests with verb:PROXY took more than 10 seconds, and are causing the test to fail. I think that proxy issues are a different test, right? (maybe testMonitoringUsingHeapsterInfluxdb?) And that this taking 10 seconds could be very reasonable, depending on what we're proxying. There is this note in If that is the case, sounds like we should either clear the metrics, or if that is hard, record them before we run our test and ignore the pre-existing ones.
Yeah we should clearly exclude proxy from those metrics. This test is still being actively worked on :)
+1 for excluding proxy metrics This should be very simple fix, because we're already exluding all "watches" from the metrics. Reassigning it to me - will fix it today.
is sent out for review.
thanks for fixing!


Question: return retErr } <ins> // ValidateHost verifies that the host and runtime is capable of enforcing AppArmor profiles. // Note, this is intentionally only check the host at kubelet startup and never re-evaluates the host // as the expectation is that the kubelet restart will be needed to enable or disable AppArmor support. </ins> func (v *validator) ValidateHost() error { return v.validateHostErr } <del> // Verify that the host and runtime is capable of enforcing AppArmor profiles. </del> <ins> // validateHost verifies that the host and runtime is capable of enforcing AppArmor profiles. </ins> func validateHost() error { // Check feature-gates if !utilfeature.DefaultFeatureGate.Enabled(features.AppArmor) {

This issue is created by this Currently AppArmor validator gets host info only . API . Would it make sense to do it dynamically every time ValidateHost is called? It would help to get the latest AppArmor status, which can be changed if AppArmor kernel module is loaded or unloaded.
/sig node
KEP xref:
/kind feature /triage accepted /sig security (adding sig security as well as they co-own a KEP)
seems like this is only one line change?
correct
/assign
How expensive is this check? The result of it is that we will stop admitting pods when it was disabled? Do we need to do anything about already running Pods? Ideally we want to update files in this PR to describe this behavior: (cc do you see any issues with this change?
- it's just doing a stat+read on (), then checking the first byte read, so overall it's pretty snappy. Probably not a super cache-friendly use case; if your node has apparmor config enabled, presumably you'd want that to kick in now_, rather than sometime soon whilst another pod might sneak in before then.
what is it in terms of time? Any delay on pod admission, especially this unnecessary will not be ideal. I don't care that much about enable AppArmor scenario - one can restart kubelet after this, should not be a big deal. Disable AppArmor scenario is harder. If it was disabled, we may want to stop admitting. But I'm not sure how immediate this should be. Also if we covering this scenario, we may need to understand how already running Pods will behave if it's disabled. In both cases - disabled and kubelet is still running. And disabled and kubelet being restarted. Should we re-admit Pods?
For me, locally, it's ~1.3ms, but that's entirely going to depend on the host its running on. If we can't handle a couple of milliseconds at pod admission, we probably shouldn't do this change at all. If we're going to admit a pod with AppArmor config, whilst AppArmor is enabled for the kubelet, I'd imagine we'd want to uphold that guarantee we're giving by admitting said pod? I'd err on the side of it being immediate, assuming we're okay with the time spent validating the host.

If we decide we want to provide this relatively strong guarantee that any running pod is aligned with the host + kubelet's config, I'd argue in favour of re-admission.
I'm trying to think of other implications. Any dependency on pod startup flow is not ideal. I cannot tell how important it is to pick up the change without the kubelet restart. I'd defer for for this. Mechanics of the change are lgtm
Yes, I agree with Sergey. Requiring Kubelet to be restarted on module loading/unloading does not seem unreasonable to me. Are there any other instance of the Kubelet handling (or not handling, for that matter) dynamic kernel modules? 3ms is not much, but it's not nothing. I personally don't see much value in this change.
Thanks for the review! closing.


Question: github.com/google/go-flow-levee v0.1.5 gotest.tools/gotestsum v1.6.4 honnef.co/go/tools v0.2.0 <del> k8s.io/klog/hack/tools v0.0.0-20210303110520-14dec3377f55 </del> <ins> k8s.io/klog/hack/tools v0.0.0-20210917071902-331d2323a192 </ins> sigs.k8s.io/zeitgeist v0.2.0 )

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: return printModelInfo(w, pointedModel, pointedModelAsProp) } <ins> func splitGroupFromResource(resource string) (string, string) { seg := strings.SplitN(resource, "/", 2) if len(seg) == 1 { return "", seg[0] } else { return seg[0], seg[1] } } </ins> func splitDotNotation(model string) (string, []string) { var fieldsPath []string dotModel := strings.Split(model, ".")

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


Question: fs.BoolVar(&s.AllowPrivileged, "allow_privileged", s.AllowPrivileged, "If true, allow privileged containers.") fs.Var(&s.PortalNet, "portal_net", "A CIDR notation IP range from which to assign portal IPs. This must not overlap with any IP ranges assigned to nodes for pods.") fs.StringVar(&s.MasterServiceNamespace, "master_service_namespace", s.MasterServiceNamespace, "The namespace from which the kubernetes master services should be injected into pods") <ins> fs.BoolVar(&s.SyncPodStatus, "sync_pod_status", s.SyncPodStatus, "If true, periodically fetch pods statuses from kubelets.") </ins> fs.Var(&s.RuntimeConfig, "runtime_config", "A set of key=value pairs that describe runtime configuration that may be passed to the apiserver.") client.BindKubeletClientConfigFlags(fs, &s.KubeletConfig) fs.StringVar(&s.ClusterName, "cluster_name", s.ClusterName, "The instance prefix for the cluster")

Split off as a smaller more specific work item. Apiserver should have a flag that makes it not attempt to GET from kubelets. This would be used in conjunction with . After we have converted the top N Getting-started-guides to the new style, we can rip out the kubelet client from the apiserver altogether.
This is tightly related to so I'll take it.


Question: } } <del> // ensure the subnet is properly configured subnet, err := az.SubnetsClient.Get(az.ResourceGroup, az.VnetName, az.SubnetName, "") if err != nil { // 404 is fatal here return err } if subnet.RouteTable != nil { if *subnet.RouteTable.ID != *routeTable.ID { return fmt.Errorf("The subnet has a route table, but it was unrecognized. Refusing to modify it. active_routetable=%q expected_routetable=%q", *subnet.RouteTable.ID, *routeTable.ID) } } else { subnet.RouteTable = &network.RouteTable{ ID: routeTable.ID, } glog.V(3).Info("create: updating subnet") _, err := az.SubnetsClient.CreateOrUpdate(az.ResourceGroup, az.VnetName, az.SubnetName, subnet, nil) if err != nil { return err } } </del> targetIP, err := az.getIPForMachine(kubeRoute.TargetNode) if err != nil { return err

Feature Request Currently the Kubernetes Azure assumes kubernetes is deployed to the same Azure Resource Group as all of the dependencies. In my case our case we're interested in deploying kubernetes to a Virtual Network that sits in a different resource group to the cluster itself, this will enable us to deploy kubernetes to a subnet in a locked down vnet. I think there might be a few ways to do this. can add a field to , and if the need arises to do the same for the NetworkSecurityGroup or RouteTable then we could do the same there. We could assume that if the isn't provided then the cluster is in the same RG as the VNET. support for Resource IDs (). I think this is a substantial change, looking at the code. I'm inclined to think the first option is best for now, as it's the path of least resistance. What do you think? When we decide what's needed I'm happy to muck in and make the changes :) Cheers,
I'm curious for thoughts on this. I laid out the three options here: The easiest one is to simply remove the check where we inspect the subnet to make sure the route table is set correctly - the subnet/vnet references aren't used anywhere else. The second easiest is to add . The hardest, but most correct is the third option of using full identifiers. I'm leaning towards the first one in the short term, and the full identifier option in the longer-run; there are some platform improvements that will tie-in nicely with supporting full identifiers, but without requiring reving/versioning the config file.
Hadn't thought of the first option, quickly knocked together the . At least I think that's all is needed anyway, but i could be missing something. I agree with you on the third option, especially about not having a breaking change. If we choose two then I'll raise a PR for the above change. But i'll hold off for now :)
cc:
Is there any chance you'd be willing to remove the subnet-routetable check instead of the second option and see if that meets your needs? I think that's what we want to go with short-term. Chances are, if people aren't configuring the route table / subnet correctly, then they likely won't find their way to the error message anyway.

(Longer term we hope to move away from needing the config file at all, similar to other clouds.)
Yeah sure, I'll knock it up this evening then tag you in the PR :)


Question: contentType:  "text/html", forwardedURI: "/proxy/minion/minion1:10250/logs/log.log", }, <ins> "full document": { input:        `<html><header></header><body><pre><a href="kubelet.log">kubelet.log</a><a href="/google.log">google.log</a></pre></body></html>`, sourceURL:    "http://myminion.com/logs/log.log", transport:    testTransport, output:       `<html><header></header><body><pre><a href="kubelet.log">kubelet.log</a><a href="http://foo.com/proxy/minion/minion1:10250/google.log">google.log</a></pre></body></html>`, contentType:  "text/html", forwardedURI: "/proxy/minion/minion1:10250/logs/log.log", }, </ins> "trailing slash": { input:        `<pre><a href="kubelet.log">kubelet.log</a><a href="/google.log/">google.log</a></pre>`, sourceURL:    "http://myminion.com/logs/log.log",

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: var ( nilRs *apps.ReplicaSet <ins> // DeploymentGroupVersionResource identifies a Deployment resource. DeploymentGroupVersionResource = schema.GroupVersionResource{Group: extensions.GroupName, Version: "v1beta1", Resource: "deployments"} </ins> ) var _ = SIGDescribe("Deployment", func() {

Which jobs are failing: ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel Which test(s) are failing: [sig-apps] Deployment deployment should support rollback Since when has it been failing: 06-22 - 17:13 PDT Testgrid link: Reason for failure: Anything else we need to know: /priority critical-urgent /kind failing-test /sig apps
I will look in test together with
/milestone v1.16
stopped serving those APIs in 1.16. The test in the release-1.15 branch should skip if the server it is testing against does not serve the deprecated extensions/v1beta1 API
closing
and did not completely resolve the failing tests here or in /reopen
Reopened this issue. <detailsIn response to : and did not completely resolve the failing tests here or in /reopen </details
this job is running 1.14-level tests against master (1.16). the test should be updated to run 1.15 against master
Possible issue here is that we may have misconfigured upgrade/downgrade jobs (upgrading/downgrading to incorrect k8s versions). For full discussion see
/assign
Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close
Closing this issue. <detailsIn response to : Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ErrorHandlers is a list of functions which will be invoked when an unreturnable // error occurs. <del> var ErrorHandlers = []func(error){logError} </del> <ins> // TODO(lavalamp): for testability, this and the below HandleError function // should be packaged up into a testable and reusable object. var ErrorHandlers = []func(error){ logError, (&rudimentaryErrorBackoff{ lastErrorTime: time.Now(), // 1ms was the number folks were able to stomach as a global rate limit. // If you need to log errors more than 1000 times a second you // should probably consider fixing your code instead. :) minPeriod: time.Millisecond, }).OnError, } </ins> // HandlerError is a method to invoke when a non-user facing piece of code cannot // return an error and needs to indicate it has been ignored. Invoking this method

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TPR. GC. OwnerReference. REST mapping. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master. v1.5.1 Environment: Cloud provider or hardware configuration: Repro in "hack/local-up-". But I have seen it in many different env. OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The controller manager is unresponsive and overloaded with error message: This error just keeps bloating up without any backoff. This seems that REST mapping didn't recognize the kind. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Generic way: Create a TPR. Create a TPR object "A" Create pods that has OwnerReference pointing to "A". Easy way: Use kubectl create -f kubectl create -f example/example-etcd- Anything else do we need to know:
well, the garbage collector still uses a static RESTMapper(), and in this case, TPRs aren't recognized at all.
Not recognizing is fine and expected. But controller manager keeps on printing out error message without backing off. Can you confirm this is unintended? Or we are missing anything here?
Right. I should have mentioned it: the controller manager is unresponsive and overloaded with error message. Updated in title and top comment.

there are two things that need fixing here: garbage collector should use a dynamic restmapper that picks up types the server knows about, not just compiled-in types a. this means, at minimum, discovering types at start-up b. it also means refreshing at some interval cases where an ownerRef references a type that is not found by the garbage collector's restmapper, it should not panic or hotloop. I'm not really sure what the behavior should be, actually should it treat owners it can't look up as if they were deleted, or ignore ownerRefs for unknown kinds?
If a user runs kubernetes and sees messages like "ignore XXX ownerRef due to unknown kind", he can't really do anything with it. It would be great to recognize the kind, even if there is a delay.
ownerref could have bad data, or it could reference a kind that is no longer part of the cluster. need to define what happens in those cases. when it is referencing a dynamic kind (either a kind contributed by an add-on server or by a thirdpartyresource), yes, it should re-discover on some interval to recognize the kind
I have the same issue when launching an etcd cluster with the coreos etcd-operator. If causes the api server to run out of disk space as it logs infinitely. I have to keep restart the controller container to reclaim disk space. From a security point of view this means a tenant of a cluster can bring the whole cluster down.
Why the target fix is in 1.6 not a patch release for 1.5?
I would not anticipate the garbage collector actually enabling support for TPRs in 1.5, but the bug that causes it to become unresponsive when it sees an ownerRef of an unknown type should be fixed
Changed the milestone to 1.5
I agree. I do not expect the gc to work with TPR either. But it should not "kill" controller manager. And it is the issue we should fix in 1.5. We can make GC work for TPR in 1.6.
For me this is currently a DOS attack
Repro instructions on master in GCE:
so you don't even need to have something with an owner ref pointing at the TPR to trigger this?

I would guess an ownerRef pointing to any arbitrary unknown kind would trigger the same behavior
the etcd operator creates etcd pods with: I suspect is correct.
Ah, the operator triggers it. OK, I see.
etcd operator does not rely on the k8s GC features (it has internal GC). But it does write owner ref. when k8s gc works for tpr, we will remove our temp internal GC. but now the owner ref causes controller manager to spam.
shall we reopen? i do not think this is fixed.
Yeah this just happened to me. Does not seem fixed.
what version are you running? This isn't in a release version yet.
"masks" the problem, not fixes it. We should close this issue when the fix lands.
I just installed with kubeadm two days ago. I believe is 1.5.2. I then tried using etcd-operator and it killed my master with out of disk space writing 300MB/s of log from GC
a fix will be in the 1.5.3 release. It was just merged and hasn't had time to make it into a release. Sure, not arguing that this issue should be closed. I do think is probably good enough for the 1.4, 1.5 branches.
I'll do it after the refactor of GC, , lands. makes gc use the rate limited workqueue, so it will prevents the hotloop. [edit] GC treats unresolveable ownerRefs as though they perpetually exist. That is, an object with an unresolvable ownerRef will not be deleted by GC. It is meant to be a safety net. For example, during a cluster upgrades, if the APIVersion of replication controllers bumps from "v1" to "v2", and assuming all the pods' ownerRefs still have "v1", if GC treats those refs as if the owners were deleted, then all the pods will be accidentally deleted.
Does GC only treat the preferred version as a legitimate ownerRef, or all known versions, or was that just an example?
I guess there is a third option as well, treat unresolveable ownerRefs as though they perpetually exist (which is different from ignoring them).
All known versions.

What I mean in the example is that assuming v1/replicatoncontroller is not supported by the API server anymore.
Actually this is the current case. GC doesn't delete an object with unresolvable ownerRef. I edited my previous comment.
40Gb per day of logs to this mess for us... So we would love a fix too.
is close to get in. It will fix the hot loops. We still need a dynamic restmapper. I'll start a fix after lands.
FWIW, here's the patch I wrote to get back to a more reasonable behavior: (then, rebuild your controller-manager or hyperkube)
Thanks changed gc to use the ratelimited queue, so the hotloop should have been fixed on the master branch. [edit] That said, we still need to make the restmapper dynamic, i'm working on a fix.
My patch was done on the branch, should anyone prefer to avoid upgrading to alpha.0 :)
BTW, is there a backport to 1.5 planned? is flagged so, should a smaller patch be needed, feel free to use this one of course.
Update: I tried to reuse to dynamically translate TPR, but it didn't seem to work because of I'll find a workaround.
what version are you on? said was landed in v1.5.3, so the problems should be masked. Is that true?
I had to overwrite hyperkube in the docker image
Ahhh it has this commit which says: Adjust global log limit to 1ms So probably not a difference a human can detect.
I see. 1 ms is still too spamming.
At least, it make the log unusable without filtering ;) I have a global feeling that Kubernetes log system could be improved but I still need to have a better of that. The global problem to solve is probably the repetition of log, something that's solved by events (or at the way kubectl shows them); so maybe I should move my practice more to events and, when I don't have the answer in the events, file an issue/PR.
the backoff removed the log flooding, right? if so, I think this can be moved to 1.7
c.f.
Thanks You are right, this is not a 1.6 blocker. makes sure the log message is only printed once per un-resolvable ownerReference.

I think the original issue of "spamming error messages" was solved. I created to track the issue of "GC should support non-core API". Closing this issue.


Question: apiVersion: v1 kind: ReplicationController metadata: <del> name: heapster-v11 </del> <ins> name: heapster-v12 </ins> namespace: kube-system labels: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> kubernetes.io/cluster-service: "true" spec: replicas: 1 selector: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> template: metadata: labels: k8s-app: heapster <del> version: v11 </del> <ins> version: v12 </ins> kubernetes.io/cluster-service: "true" spec: containers: <del> - image: gcr.io/google_containers/heapster:v0.18.4 </del> <ins> - image: gcr.io/google_containers/heapster:v0.18.5 </ins> name: heapster resources: # keep request = limit to keep this container in guaranteed class

I'll take a look at the failed builds and triage.
I went through the recent reboot failures in kubernetes-e2e-gke-ci-reboot and here is a summary. [6481, 6486, 6488] Kubelet never came back from the reboot, and stopped posting node status. Throughout the test suite, the failed node stayed silent: It's hard to dig further due to the lack of clarity. Gathering log would at least help us rule out some possible causes . [6485] Heapster seems to crash loop quite frequently, and sometimes it failed the test (due to being not ready).. cc [6487] kube-proxy pod is running but not ready before the reboot test even started: This is not really a reboot test failure since the cluster was already in a bad state. We should make sure all the pods are ready after each test (e.g. "should work after restarting kube-proxy" ran before the failed reboot test). Another thing that would help is to increase the log level of the gke cluster. The log level of a gce cluster is 4, but I think gke is at the regular 2. Due to the low log level, it's unclear when kube-proxy started failing the readiness probe.
Filed for (3).
The reboot tests basically haven't worked since 9pm last night: Interestingly, this lines up with the GKE equivalent for , but in that same window, has had no trouble. cc
Digging into logs from the runs linked.
All of the gke-ci failures in are heapster crash-looping after a reboot.
Changes to the heapster config in the last day: , . Nothing sticks out to me in those PRs.
Heapster could be in crashloop if it cannot reach influxdb. But that is a bad design, and should fix it in today's heapster with image 18. I double checked them, it is image version 18.
From , could be the num_nodes is unset, we give heapster no resource to run? If that is true, it could be point to the same version screw issue of our test infra.
Heapster v0.18.4 will crashloop if InfluxDB isn't available. However we can update HEAD to v0.19.0, which will not crash if the backend goes down.

I will send a PR to update InfluxDB version. , Dawn Chen wrote:
GKE doesn't use InfluxDB, right?
Heapster is the only kube-system pod that's not ready in the tests. If it relies on influxdb, shouldn't influxdb be not ready too?
I like theory. My salt suggestion might've been horrible for unset ? I thought I did that correctly, but maybe my Python/Jinja was bad.
Yes, GKE doesn't use influxDB. My point is that if the backend storage is not ready, heapster is in crashloop. Vishnu made some changes to make heapster can run standalone, but looks like that fix is not included in heapster 18 based on his comment.
Okay. I'm still trying to catch up. Why is heapster on GKE waiting on backend storage? And if is unset (e.g. we haven't finished running configure-), it looks like we just give the
Catching others up: is seeing nodes that are actually hung with CPU pegged on reboot, so the heapster conversation may be a red herring.
There are plenty of tests that failed because kubelet didn't come back (and post status) after the reboot. There are a few that failed only because heapster was crash looping. They may be related or not...
I spun up a GKE test cluster: and the heapster manifest looks fine, so I think I ruled out anything regarding
I think is the winner. The reboot tests are a red herring.
I'm actually just seeing heapster flap on a fresh cluster, no reboots.
Heapster crashlooping is one problem. There are other cases, however, where kubelet just stops posting status all together. If heapster crashlooping can render kubelet unresponsive, that's a bug.
is the serial console output from the node that got pegged (in ). I rebooted the node to see if it would repro, and it came back up fine. EDIT: the link to the jenkins run.
I guess the serial console you posted is from build , the very recent one. I saw a similar one and the node never successfully come back. At this moment, I ignore those failures.

The reboot failure caused by heapster crashloop is similar to what I observed 2 days ago for influxdb in gce cluster. The comment is at Can we retitle this one for (1)? Reboot test flakiness is caused by many reasons: version screw which is handled by test-infra, heapster and influxdb crashloop is handled by , kube-proxy not ready is handled by cluster team.
Ok. So we think we have the heapster flakes, versioning stuff, and kube-proxy stuff all taken care of elsewhere? I'm fine with targeting this issue at the "node doesn't come back after reboot" case. Is that what you were recommending?
Yes. 1) heapster in crashloop if backend is not ready is a known issue for a while, and fix is merged to heapster repositry for a while. We just need to upgrade our heapster after more testing (PR: ) 2) influxdb one requires more information, but we can handle them through 3) version screw is handled by test-infra through . already test/flake label for it 4) kube-proxy readiness issue is handled through
where else should I be looking for hints on why that VM didn't come back to life until I kicked it? Or did I destroy any evidence we might have had?
Fun fact: I thought this might be related the timing of when I made the ContainerVM change on the GKE side, but I made that change for all releases 1.1.4. That picks up this test suite, too, because it's at , and it's green: So this is a divergence from master to 1.1, or between projects.
That suite does not run any reboot tests...
Dammit, I thought I scanned for that. Good call. We need to fix that.
Here's the set of suites that run the reboot tests (thanks kubernetes-e2e-gce kubernetes-e2e-gce-reboot kubernetes-e2e-gke-ci kubernetes-e2e-gke-ci-reboot
Shouldn't we decided only X-reboot test suites run those reboot tests since they are disruptive?
Hm. builds 6478-6480 are all green and all after the (GKE-side) ContainerVM change, so we're looking for something else.

The container-vm change was merged at , which lines up with kubernetes-e2e-gke-ci-reboot 6481 (the beginning of the failures).
6481 is the one with the new containervm change. It includes kubernetes-build/7822, which has the change.
/ That's the OSS side change, which is irrelevant. GKE flipped at 19:45:55.
Ahhh. Fair enough.
Actually, and I just determined that the GKE-side change didn't make it out until 01:45:44, it looks like, so indeed nothing to do with ContainerVM.
I took a more careful look at the serial output pasted, and this is interesting: We're actually in the middle of a clean shutdown, trying to unmount and failing. This same stacktrace repeats every 120s.
I spun up an e2e cluster and with minimal testing have a node in the state of It's the same kernel hang on shutdown. ... and then I accidentally deleted it when I went to bisect, but ohwell. I'm sure I'll be able to get one back if I need.
Okay, so I've confirmed that at the build level at , I can spin up a GKE cluster and reboot tests fail. I'm going to peel back to and see if they pass (because that matches ). ETA: For reference, I'm able to tickle this with just:
7819 had a kernel hang on the second run as well, though slightly different signature (still aufs in the end):
For my next test, I'm peeling back to v1.1.4. We have no test coverage on reboots on the release branch of GKE right now, and I just want to guarantee this isn't some weird GCE hypervisor issue. And v1.1.4 is clean, at least for two runs worth.
This failed quickly on v1.1.5. Hrm. I'm wondering if my v1.1.4 test was a fluke, peeling back to try it there again. v1.1.5 only differs by ContainerVM (and go 1.4.3 vs 1.4.2), but all of our other timelines show that not being directly related. (Alternately, it's a lot easier on the newer kernel.)

(Still no idea why the tests aren't triggered, but is.)
was very solid. I'm now trying on GKE test. This is a GKE unique thing, because after Friday morning at 1am, anything = will get deployed using the new ContainerVM. So I'm ret-conning a way to effectively try with the new ContainerVM, just to see if it's a contributing factor. It doesn't look like it should be. One of the other commits in the stream is a heapster version bump (from / ). If there's additional flapping or something caused by that (unique to GKE), it may be causing the hang. But that timeline doesn't match the line (relevant to , etc.)
Bad news bears: failed reboot tests. I'm having trouble rectifying this information with what we know about the timeline. But it does suggest there's a problem with ContainerVM at the very least, so there might be two problems here, so I'm rolling one back on GKE and we'll see what happens.
I've manually confirmed that by reverting deployment of on GKE, I can run reboot tests at and a latest build. Using a special build of latest that will still trigger deployment of , reboot tests fail. Well, damn.
GKE has been repushed and should, in theory, go back green soon. There's a lot I don't understand about this bug, and it's effectively going to stop us shipping and until we can figure it out.
Okay, I think that reverting was not all that's necessary. We unfortunately haven't managed to collect a green on yet. That gives me some comfort, because it means that the timeline-discomfort I had previously was founded.
Heh, I lied and right right writing that we collected our first one. Woo. I suspect will follow soon.
Haven't we updated go to 1.5 on Friday as well? I don't know if we use the same version of go for GKE as we do in GKE, but in general we know it causes trouble. cc for Heapster thing.
The timeline there is off, too. The golang1.5.3 PR went in at , which is well after the trouble started.
Morning status: Looks like we made a little greener, and didn't touch , so the patient had two viruses.

Okay, this is my suspicion, but I can't quite piece together what's going on: For some reason, something in (? but that busted the Heapster config, so it's confusing me as to why), caused this. I'm basing this on: the timeline of the timeline of If you suppose that we injected a bug in build 7820 that caused about a 60-70% cointoss as to whether we regress (which is similar to the rate we're seeing now that has been rolled back on GKE), it's not unreasonable that was green for , and that's the only one you have to explain away on this theory. After that, it goes kind of red, and then at about 1am, comes in and wrecks completely as well. Pulling it back much further, to, say, build 7819, and you have to explain the entire range from 6477-6480 as green, which is a much lower probability. So my money is on build 7820. (7821 is revision-equivalent, so boring to talk about.)
Oh. It is Heapster. I spun up a 7819 cluster and a 7820 cluster, and the difference is that even though still had slightly busted Salt syntax, it was functional enough to create the replication controller, whereas on 7819, there's no Heapster replication controller at all. So I think the Heapster flapping is entirely the cause of the remaining issues, so we really need to get in ASAP.
So yesterday afternoon I was wondering why build 7820 triggered such a large regression when, say, the bump to heapster in didn't regress us at all. Build 7820 finally let the the fix in , which was actually important because GKE at HEAD wasn't deploying (until last night), it was deploying only , so the pillar wasn't configured. I'm still not exactly sure what's going on here, and I had intentionally left blank for a while during this investigation because it didn't seem like it should matter, but it's obvious that if 7820 is the culprit that it's interesting. I pushed a fix last night to GKE that adds on 1.2 clusters as well, and we've collected our first green dot. I don't know why is still flakey, though.
Er, correction to my last comment.

My change didn't make it out last night and won't be relevant until . The green is a fluke. Sad panda.
PR introduced syntax error, PR fixed the error although it introduced another syntax issue which was finally fixed in . if you are testing anything make sure you are using version with merged. I agree with that the problem is not with Heapster accessing InfluxDB. Also the same problem takes places in autoscaling suite (described here ). We started observing such problems around Jan 19th. AFAIK once the container crashes kubelet invoke docker inspect on the failing container. How about invoking also docker logs then? It would be very useful for debugging purposes.
Yes, I understand that fixed the issue fully, but at build 7820 the cluster that's created actually spins up a cluster with a replication controller .yaml with resources. At build 7819, it doesn't. The syntax in is busted, but it's apparently not so busted. Previous to , in build 7819 (and builds previous), there was no controller .yaml at all, so there was a window where GKE wasn't spinning up Heapster at all. Hence my conclusion.
The syntax in is correct from Salt point of view, although the generated file has empty line in the begging and is not accepted by Api Server, so the RC is not created.
That's inconsistent with the fact that Heapster is starting on a 7820 (GKE) cluster.
I have a new theory, which is that Heapster hasn't been running on GKE in 1.2 since , until .
It's not clear to me that upgrading heapster will solve this issue, since heapster doesn't rely on any backend on gke. Instead of asking kubelet to do that, it makes more sense for the test itself to get the logs through the apiserver. I think we should add that to the e2e test.
I spun up a cluster from just prior to and this is the set of RCs I see: So basically, when went in, Heapster was enabled for the first time in GKE (master, not the release branches) in months and we started flipping out. This is a serious test coverage issue, obviously.
I edited the above comment, but we would obviously catch this in the release branches because we have monitoring tied to Heapster related things.

We need to be better here for CI.
I made some investigation. The problem with crashlooping Heapster is due to: The issue was fixed in I'll create a patch release and bump the version.
It looks like is green. I'm hoping that , which is the first build past , will go green as well. (The cycle time on that build is waaaaay too long.)
It looks like we're good! is green now!
for node problem detector.


Question: } } } <ins> func TestStaticPodNameGenerate(t *testing.T) { testCases := []struct { nodeName  types.NodeName podName   string expected  string overwrite string shouldErr bool }{ { "node1", "static-pod1", "static-pod1-node1", "", false, }, { "Node1", "static-pod1", "static-pod1-node1", "", false, }, { "NODE1", "static-pod1", "static-pod1-node1", "static-pod1-NODE1", true, }, } for _, c := range testCases { assert.Equal(t, c.expected, generatePodName(c.podName, c.nodeName), "wrong pod name generated") pod := &core.Pod{} pod.Name = c.podName if c.overwrite != "" { pod.Name = c.overwrite } errs := validation.ValidatePod(pod) if c.shouldErr { specNameErrored := false for _, err := range errs { if err.Field == "metadata.name" { specNameErrored = true } } assert.NotEmpty(t, specNameErrored, "expecting error") } else { for _, err := range errs { if err.Field == "metadata.name" { t.Fail() } } } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a static pod for etcd as following with vsphere as cloud provider: But the hostname is using upper name, and when kubelet start the etcd static pod, it will report error as follows: I think that the vsphere should not return upper case hostname, but only lower case hostname. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area cloudprovider
Here is a quick
/sig node