Question: informerFactory.Start(wait.NeverStop) testPod := validPod("testPod", 1, api.ResourceRequirements{}) <del> err = handler.Admit(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "", admission.Update, false, nil)) </del> <ins> err = handler.Admit(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "", admission.Create, false, nil)) </ins> if err != nil { t.Fatal(err) } <del> err = handler.Validate(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "", admission.Update, false, nil)) </del> <ins> err = handler.Validate(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "", admission.Create, false, nil)) </ins> if err == nil { <del> t.Errorf("Expected an error since the pod did not specify resource limits in its update call") </del> <ins> t.Errorf("Expected an error since the pod did not specify resource limits in its create call") } err = handler.Validate(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "", admission.Update, false, nil)) if err != nil { t.Errorf("Expected not to call limitranger actions on pod updates") </ins> } err = handler.Validate(admission.NewAttributesRecord(&testPod, nil, api.Kind("Pod").WithVersion("version"), limitRange.Namespace, "testPod", api.Resource("pods").WithVersion("version"), "status", admission.Update, false, nil))

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod stuck in terminating state. What you expected to happen: Pod successfully cleaned up. How to reproduce it (as minimally and precisely as possible): up a LimitRange. (Say "limits: defaultRequest: cpu: 100m type: Container") a mutating webhook on pod which creates an additional container on the pod (Eg. ) without the limit range. a Pod with foreground deletion finalizer. that you have a Pod with 2 containers. The 1 you created which now has a limit and the istio container which does not. to delete the Pod. garbage collector tries to update the pod to remove the finalizer, however this triggers the limit range admission control to add the limit to the istio container. This then causes the API server to get the error "invalid: spec: Forbidden: pod updates may not change fields other than , , or " the pod is now stuck in terminating. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.6 /sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale PR open


Question: t.Error("Expected error, but got success") } <del> if err != nil && !strings.Contains(err.Error(), tt.expectedErr) { t.Fatalf("enforceRequirements returned unexpected error, expected: %s, got %v", tt.expectedErr, err) </del> <ins> expErr := tt.expectedErr // pre-flight check expects the user to be root, so the root and non-root should hit different errors if os.Getuid() != 0 && len(tt.expectedErrNonRoot) != 0 { expErr = tt.expectedErrNonRoot } if err != nil && !strings.Contains(err.Error(), expErr) { t.Fatalf("enforceRequirements returned unexpected error, expected: %s, got %v", expErr, err) </ins> } })

unit tests should run without root. This test should be refactored or otherwise removed from . should pass as an ordinary user. run tests as non-root
/sig cluster-lifecycle /area kubeadm
thanks for logging the issue. this was changed in so this root preflight test must be excluded.
/triage accepted link


Question: cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" ) <ins> /* WARNING: this logic is duplicated, with minor changes, in cmd/hyperkube/kubectl.go Any salient changes here will need to be manually reflected in that file. */ </ins> func Run() error { cmd := cmd.NewKubectlCommand(cmdutil.NewFactory(nil), os.Stdin, os.Stdout, os.Stderr) return cmd.Execute()

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: } else { allNames.Insert(ctr.Name) } <ins> // TODO: do not validate leading and trailing whitespace to preserve backward compatibility. // for example: https://github.com/openshift/origin/issues/14659 image = " " is special token in pod template // others may have done similar </ins> if len(ctr.Image) == 0 { allErrs = append(allErrs, field.Required(idxPath.Child("image"), "")) }

Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version (use ): kubernetes 1.5+ Environment: n/a What happened: examining events that happened in my cluster, i see an event like the following: this surprised me, as it implies a container had an imageName=" " What you expected to happen: i expected a validation failure error if a [i].imageName had leading or trailing whitespace. How to reproduce it (as minimally and precisely as possible): kubectl create -f my-pod kubectl edit pod/my-pod -o json change image to " " save Anything else we need to know: nope
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
discovered when looking at odd event messages in production cluster.


Question: "github.com/coreos/etcd/store" "github.com/coreos/etcd/wal" "github.com/coreos/etcd/wal/walpb" <ins> "github.com/coreos/go-semver/semver" </ins> "github.com/golang/glog" )

I tried to do a rollback on my own test cluster and got a panic. I will attach the data-dir tarball. The script that performs the migration goes from 2.2.1-2.3.7-3.0.14-2.3.7, and it is this last step that barfs. Script is forthcoming in a PR, but there's a snapshot at
This seems that it's not even about the real data, but about some metadata of the cluster. - any thoughts on it?
Is the data you uploaded the original data before 2.2.1-2.3.7?
yes
The rollback tool couldn't accept any v2 data existed. One way to make it work is to remove snap files:
To clarify, the panic is caused by existing v2 snap files. In the migrate step 2.3.7 -3.0.x, the script didn't remove v2 snap files.
- thanks a lot for looking into it. I will add a step to our migration script to remove the v2 data before rollback. Did it work after removing the data?
send out for review
To clarify - I verified that it fixed the problem reported (rollback works on the data provided in the issue).
The fix in the shell script is good, but the error message in the rollback tool could definitely be improved upon.
says that removing the snap file was not the right way to go.
i marked as milestone 1.5 and marking this as a non-release-blocker. hope this is correct.
Hmm... I tried re-running this on my test data, and now 2.3.7 always panics after the last step, whether I begin with backup data or not. Here is the data before running rollback: after running rollback:
I do not understand this. Does the migration tool still panic? Or the etcd 2.3.7 panics? etcd 2.3.7 should panic if you build the migration tool with 3.0 wal pkg. I believe I sent you the correct migration binary over slack.
Can we close this issue if the rollback tool stop panics? If the migrated data panics etcd 2.3, it is expected. And there is some other issues tracking it. I think we went though them a few weeks ago before thinks giving.

your slack message was lost while I was out of office, but the attached file is still there. Trying it now.
it works with the binary you shared. We still need a way to build that binary, but I will open a separate issue about that.


Question: return newFakeAPIResourceConfigSource() }, expectedAPIConfig: func() *serverstore.ResourceConfig { <del> config := newFakeAPIResourceConfigSource() config.DisableVersions(extensionsapiv1beta1.SchemeGroupVersion) return config </del> <ins> return newFakeAPIResourceConfigSource() </ins> }, <del> err: true, </del> <ins> err: false, // no error for backwards compatibility </ins> }, } for index, test := range testCases {

https://k8s- sample failure: (also affecting other CI jobs) cc /sig gcp
/assign We're interested in disabling the job as blocking entirely. We're of the opinion this shouldn't have bee allowed to block the community in the first place. I swear there's a historical issue around tightening the regex somewhere.
... wrong button sorry about that
possibly caused by erroring on invalid values that were previously silently skipped opened to downgrade that error to a warning for a release
Well, I mentioned the idea of making the job optional in
PR out to drop the runifchanged regex from pull-kubernetes-e2e-gke
Confirmed turns the test green
Can call this closed when merges
/pony hooray
Couldn't find a pony matching that query. <detailsIn response to : /pony hooray Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "Metric":    gstruct.Ignore(), "Value":     gomega.BeNumerically("==", value), "Timestamp": gstruct.Ignore(), <ins> "Histogram": gstruct.Ignore(), </ins> })) }

There's been a quite a few failures across jobs that query node metrics. The test is Last passing - First failing - appears to point to which is bump in prometheus client library. Possibly related? /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Verified with presubmit that reverting the prometheus client library results in the test passing. Opened PR for revert in
/cc
/assign
thanks for filing this issue! I think seems like a promising fix (thanks ) and there's a fair chance we won't need the revert
ack, thanks for the fix, it seems we can go with that then and hold the revert.


Question: func testCheckStop(t *testing.T, i int, w watch.Interface) { select { <del> case _, ok := <-w.ResultChan(): </del> <ins> case e, ok := <-w.ResultChan(): </ins> if ok { <del> t.Errorf("#%d: ResultChan should have been closed", i) </del> <ins> var obj string switch e.Object.(type) { case *api.Pod: obj = e.Object.(*api.Pod).Name case *unversioned.Status: obj = e.Object.(*unversioned.Status).Message } t.Errorf("#%d: ResultChan should have been closed. Event: %s. Object: %s", i, e.Type, obj) </ins> } case <-time.After(wait.ForeverTestTimeout): t.Errorf("#%d: time out after waiting 1s on ResultChan", i)

It failed in cc
I have some hypothesis. But I need to verify it. I will submit something to print more info in the test.
and others referencing this Please share links to unit test logs
We are able to reproduce this locally and confirm the failure: Submitted to fix it.
For a concrete reason why the test failed: That specific sequence is kinda non-blocking because the object is filtered. It calls Stop() really fast and cancel context. sync() returns context.Canceled


Question: requestSizeInMegabytes:   requestSizeInMegabytes, } go rc.makeConsumeCPURequests() <del> rc.ConsumeCPU(initCPU) </del> <ins> rc.ConsumeCPU(initCPUTotal) </ins> go rc.makeConsumeMemRequests() <del> rc.ConsumeMem(initMemory) </del> <ins> rc.ConsumeMem(initMemoryTotal) </ins> return rc }

Is this fixed via the merged PR?
There are two issues: one is fixed by and another is fixed by , however it requires also new release of Heapster which hasn't been done yet so that I haven't closed this issue.
related PRs ,
It looks like you're on top of this, but it's been failing pretty frequently this morning as well, all on "Horizontal pod autoscaling [Skipped][Autoscaling Suite] should scale from 5 pods to 3 pods and from 3 to 1 (scale resource: Memory)"
The issue is about cluster size autoscaling tests, which haven't failed since was merged. I'll close this issue tomorrow once there will be no more failures. Also there is another issue with Horizontal Pod Autoscaling test - tracking issue
The test is stable now so closing the issue.
This problem is occurring again:
blocking merges; raising to P0
looks like was merged to hopefully address this. Lowering to p1.
Yep, disabled the test, tracking issue and there is fix in flight
Closing as is resolved.


Question: $ kubectl get pods ``` <del> If you ssh to that machine, you can run `docker ps` to see the actual pod and `mount` to see if the Glusterfs volume is mounted. </del>  No newline at end of file <ins> If you ssh to that machine, you can run `docker ps` to see the actual pod and `mount` to see if the Glusterfs volume is mounted. </ins>

a set of endpoints to your GlusterFS cluster. Example a pod that uses the GlusterFS Volume Plugin in ReadWrite Mode (readonly=false) to mount a particular GlusterFS volume. Example the pod is running the contents of that GlusterFS Volume and see that your files exist. the pod - "kubectl delete pod glusterfs" the contents of your GlusterFS Volume and see that your files no longer exist. Environment: GlusterFS - verified with GlusterFS running on either Fedora 21 or RHEL 6.5. Kubernetes - running on Fedora 21 Note: This issue does not occur if the GlusterFS Volume Plugin is in ReadOnly Mode (readonly=true)
heh, this is why I wanted to refactor the mount/unmount/cleanup code - get it right once :)
cc
I went back to commit (the original merge for glusterfs volume) and couldn't replicate the bug. after updating to HEAD i could replicate. so something changed between first merge and HEAD causing issue.
It's caused by commit : This is called before unmounting of volumes, which are mounted somewhere into . why RemoveAll()? It was not there before your commit.
Fixed in PR
Sorry for injecting the bug. I was thinking of moving volume & pod dir cleanup into the function, but it didn't pan out, and the code was left unintentionally. Thanks for fixing it! The logic still seems strange in some cases. looks at the containers we failed to kill, and skip cleaning up volumes associated with them. But doesn't look at the those containers and would just go ahead and remove the directories for all the pods that we supposedly killed. I will file and issue later.
Is this fixed now? If so, please close. , Yu-Ju Hong wrote:


Question: clusterNameKey = "kubernetes-cluster-name" ) <del> var ( // supportedServiceTags holds a list of supported service tags on Azure. // Refer https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags for more information. supportedServiceTags = sets.NewString("VirtualNetwork", "VIRTUAL_NETWORK", "AzureLoadBalancer", "AZURE_LOADBALANCER", "Internet", "INTERNET", "AzureTrafficManager", "Storage", "Sql") ) </del> // GetLoadBalancer returns whether the specified load balancer exists, and // if so, what its status is. func (az *Cloud) GetLoadBalancer(ctx context.Context, clusterName string, service *v1.Service) (status *v1.LoadBalancerStatus, exists bool, err error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider has a limited support for service tags by annotations: But according to , there are actually much more service tags than this list. Hence, errors would happen if the service tag is not included in the above list. What you expected to happen: Azure cloud provider should support all service tags from Azure. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig azure /priority important-soon
/assign


Question: gomega.Expect(p.Status.HostIPs).Should(gomega.BeNil()) ginkgo.By("deleting the pod") <del> err := podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(30)) </del> <ins> err := podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(1)) </ins> framework.ExpectNoError(err, "failed to delete pod") })

4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


Question: ) const ( <del> criticalPodName            = "critical-pod" guaranteedPodName          = "guaranteed" burstablePodName           = "burstable" bestEffortPodName          = "best-effort" systemCriticalPriorityName = "critical-pod-test-high-priority" </del> <ins> criticalPodName   = "static-critical-pod" guaranteedPodName = "guaranteed" burstablePodName  = "burstable" bestEffortPodName = "best-effort" </ins> ) var _ = framework.KubeDescribe("CriticalPod [Serial] [Disruptive] [NodeFeature:CriticalPod]", func() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: return nil, fmt.Errorf("pod %q container %q changed request for resource %q from %d to %d", string(podUID), contName, resource, devices.Len(), required) } } <del> klog.V(3).InfoS("Need devices to allocate for pod", "deviceNumber", needed, "resourceName", resource, "podUID", string(podUID), "containerName", contName) healthyDevices, hasRegistered := m.healthyDevices[resource] // Check if resource registered with devicemanager if !hasRegistered { return nil, fmt.Errorf("can't allocate unregistered device %s", resource) } // Check if registered resource has healthy devices if healthyDevices.Len() == 0 { return nil, fmt.Errorf("can't allocate unhealthy devices %s", resource) } // Check if all the previously allocated devices are healthy if !healthyDevices.IsSuperset(devices) { return nil, fmt.Errorf("previously allocated devices are no longer healthy; can't allocate unhealthy devices %s", resource) } </del> if needed == 0 { // No change, no work. return nil, nil } <ins> klog.V(3).InfoS("Need devices to allocate for pod", "deviceNumber", needed, "resourceName", resource, "podUID", string(podUID), "containerName", contName) // Check if resource registered with devicemanager if _, ok := m.healthyDevices[resource]; !ok { return nil, fmt.Errorf("can't allocate unregistered device %s", resource) } </ins> // Declare the list of allocated devices. // This will be populated and returned below.

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: registry              = initReg() dockerLibraryRegistry = registry.DockerLibraryRegistry e2eRegistry           = registry.E2eRegistry <del> etcdRegistry          = registry.EtcdRegistry </del> gcRegistry            = registry.GcRegistry // PrivateRegistry is an image repository that requires authentication PrivateRegistry = registry.PrivateRegistry

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I see only amd64 architecture etcd is used for the which will create issues for the other architectures like ppc64le, arm etc. We need to replace this etcd image with the k8s maintained fat manifest image present in
/sig testing


Question: syncPeriod    time.Duration minSyncPeriod time.Duration // Values are CIDR's to exclude when cleaning up IPVS rules. <del> excludeCIDRs []string </del> <ins> excludeCIDRs []*net.IPNet </ins> // Set to true to set sysctls arp_ignore and arp_announce strictARP      bool iptables       utiliptables.Interface

What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping


Question: Value: "tcp://machine:8080", }, { <del> Name:  "TEST_PORT_900_TCP", </del> <ins> Name:  "TEST_PORT_8080_TCP", </ins> Value: "tcp://machine:8080", }, { <del> Name:  "TEST_PORT_900_TCP_PROTO", </del> <ins> Name:  "TEST_PORT_8080_TCP_PROTO", </ins> Value: "tcp", }, { <del> Name:  "TEST_PORT_900_TCP_PORT", </del> <ins> Name:  "TEST_PORT_8080_TCP_PORT", </ins> Value: "8080", }, { <del> Name:  "TEST_PORT_900_TCP_ADDR", </del> <ins> Name:  "TEST_PORT_8080_TCP_ADDR", </ins> Value: "machine", }, {

We have seen that on VirtualBox in particular kubelet picks first interfaces, which is usually NATed and is of no use. In particular the issue comes up with service IP, which gets mapped to endpoint on the NATed interface.
If this is still a problem, can you explain more - I don't quite understand - and suggest heuristics for better solutions?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: fi src_dir="/home/kubernetes/kube-manifests/kubernetes/gci-trusty" <del> cp "${src_dir}/abac-authz-policy.jsonl" /etc/srv/kubernetes/ </del> <ins> local -r abac_policy_json="${src_dir}/abac-authz-policy.jsonl" remove_salt_config_comments "${abac_policy_json}" sed -i -e "s@{{kube_user}}@${KUBE_USER}@g" "${abac_policy_json}" cp "${abac_policy_json}" /etc/srv/kubernetes/ </ins> src_file="${src_dir}/kube-apiserver.manifest" remove_salt_config_comments "${src_file}" # Evaluate variables

abac-authz- hardcodes "admin" instead of using the $KUBE_USER variable. This means that "admin" is the only basic-auth username that will work with that abac config.
To workaround until it is fixed, you can add a line to the master's abac-authz- file: and then restart the kube-apiserver (docker kill it, and let it restart).


Question: applyPodPresetsOnContainer(&ctr, podPresets) pod.Spec.Containers[i] = ctr } <ins> for i, iCtr := range pod.Spec.InitContainers { applyPodPresetsOnContainer(&iCtr, podPresets) pod.Spec.InitContainers[i] = iCtr } </ins> // add annotation if pod.ObjectMeta.Annotations == nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /sig cluster-lifecyle What happened: I created a Pod Preset to inject an environment variable. The Pod container gets properly injected, but the init container does not. A small sample is here: What you expected to happen: The init container would get the pod preset injected How to reproduce it (as minimally and precisely as possible): Run kubectl create -f on the following gist: Reproduced here: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-09T07:26:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: minikube 0.23.0 OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Started with:
/sig cluster-lifecyle /sig service-catalog
My use case: I have an a git init container that checkout out configuration. This needs to run before the main container (hence why it is an init container). I'd like to be able to simply annotate the pod with a label and have various environment variables (GIT_REPO, etc) supplied to the init container
Update from Slack discussion: Supporting Init containers is in the Beta proposal for PodPreset. PodPreset resource is going to be moved from core to service-catalog. We have an implementation based on Initializers, but looks like Initializers are not going to Beta soon and we might have to reimplement them using Webhooks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
It'd be nice if this worked, I suspect it would help the folks over at Knative Build implement support for passing HTTP_PROXY env vars and CA cert secrets to build steps.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a customer interested in seeing this fixed - if we submit a PR, are there objections to merging it?
cc
This is also affecting me - I have an init container that loads data from AWS Secrets and stores them in a volume that is accessed by the main container. The init container needs credentials that are passed via a podpreset
any ETA when support for initcontainers will be provided by PodPreset ?
This was fixed in 1.14.


Question: spec.Volume.Name = wrapperVolumeName } <del> b, err := vh.kubelet.newVolumeMounterFromPlugins(&spec, pod, opts) if err == nil && b == nil { return nil, errUnsupportedVolumeType } return b, nil </del> <ins> return vh.kubelet.newVolumeMounterFromPlugins(&spec, pod, opts) </ins> } <ins> // NewWrapperUnmounter attempts to create a volume unmounter // from a volume name and pod uid. // Returns a new volume Unmounter or an error. </ins> func (vh *volumeHost) NewWrapperUnmounter(volName string, spec volume.Spec, podUID types.UID) (volume.Unmounter, error) { // The name of wrapper volume is set to "wrapped_{wrapped_volume_name}" wrapperVolumeName := "wrapped_" + volName

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: return pod } <del> // IssueCommandWithResult issues command on given node and returns stdout. func (h *hostExecutor) IssueCommandWithResult(cmd string, node *v1.Node) (string, error) { </del> <ins> // Execute executes the command on the given node. If there is no error // performing the remote command execution, the stdout, stderr and exit code // are returned. // This works like ssh.SSH(...) utility. func (h *hostExecutor) Execute(cmd string, node *v1.Node) (Result, error) { result, err := h.exec(cmd, node) if codeExitErr, ok := err.(exec.CodeExitError); ok { // extract the exit code of remote command and silence the command // non-zero exit code error result.Code = codeExitErr.ExitStatus() err = nil } return result, err } func (h *hostExecutor) exec(cmd string, node *v1.Node) (Result, error) { result := Result{ Host: node.Name, Cmd:  cmd, } </ins> pod, ok := h.nodeExecPods[node.Name] if !ok { pod = h.launchNodeExecPod(node.Name) if pod == nil { <del> return "", fmt.Errorf("failed to create hostexec pod for node %q", node) </del> <ins> return result, fmt.Errorf("failed to create hostexec pod for node %q", node) </ins> } h.nodeExecPods[node.Name] = pod } args := []string{ <del> "exec", fmt.Sprintf("--namespace=%v", pod.Namespace), pod.Name, "--", </del> "nsenter", "--mount=/rootfs/proc/1/ns/mnt", "--",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : separate stdout and stderr command exit code: remove kubectl dependency: Why is this needed: to replace
/assign
/sig storage HostExec was originally written for local PV tests. If we want to replace SSH in e2e, some features must be supported and I’d like to help on this. cc


Question: return nil, nil, fmt.Errorf("could not access credential file %v - %v", pemFile, err) } <del> t := jwt.NewToken(*serviceAccount, authScope, pemBytes) token, err := t.Assert(&http.Client{}) </del> <ins> jwtConfig := &jwt.Config{ Email:      *serviceAccount, Scopes:     []string{authScope}, PrivateKey: pemBytes, TokenURL:   "https://accounts.google.com/o/oauth2/token", } token, err := jwtConfig.TokenSource(oauth2.NoContext).Token() </ins> if err != nil { <del> fmt.Printf("Invalid token: %vn", err) </del> return nil, nil, err } <del> config := &oauth.Config{ ClientId:     *clientId, ClientSecret: *clientSecret, Scope:        authScope, AuthURL:      "https://accounts.google.com/o/oauth2/auth", TokenURL:     "https://accounts.google.com/o/oauth2/token", </del> <ins> if !token.Valid() { return nil, nil, fmt.Errorf("invalid token for BigQuery oauth") </ins> } <del> transport := &oauth.Transport{ Token:  token, Config: config, </del> <ins> config := &oauth2.Config{ ClientID:     *clientId, ClientSecret: *clientSecret, Scopes:       []string{authScope}, Endpoint: oauth2.Endpoint{ AuthURL:  "https://accounts.google.com/o/oauth2/auth", TokenURL: "https://accounts.google.com/o/oauth2/token", }, </ins> } <del> client := transport.Client() </del> <ins> client := config.Client(oauth2.NoContext, token) </ins> service, err := bigquery.New(client) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted


Question: // NewFakeGCECloud constructs a fake GCE Cloud from the cluster values. func NewFakeGCECloud(vals TestClusterValues) *Cloud { <del> service, _ := compute.NewService(context.Background()) </del> <ins> service, err := compute.NewService(context.Background(), option.WithoutAuthentication()) if err != nil { panic(err) } </ins> gce := &Cloud{ region:           vals.Region, service:          service,

What happened: Unit test failed by command : What you expected to happen: How to reproduce it (as minimally and precisely as possible): Just run command . Anything else we need to know?: Happened on . I first met this issue at v1.15. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/remove-kind bug /kind failing-test /sig cloud-provider
/area provider/gcp
I would be curious to know how is passing then
The reason is that is required here: I guess all CI machinery has a . This will fail in all environments without a .
ah, fixed in
Great!


Question: . "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> storagev1 "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource"

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: (e.g. ssh keys, passwords) separately from the Pods that use them, protecting the sensitive data from proliferation by tools that process pods. <ins> * **API Overview** ([api.md](api.md)): Pointers to API documentation on various topics and explanation of Kubernetes's approaches to API changes and API versioning. </ins> * **Accessing the API** ([accessing_the_api.md](accessing_the_api.md)): Ports, IPs, proxies, and firewall rules.

Simple version is done, more complex version needs work. Done, now that readiness corresponds to Running and is tracked by Kubelet status if the readiness hook is set? Done Done Do we want to do a more generic one or a less generic one? OpenShift was looking more at specific topical plugin commands ("add-volume", "set-default-resources"), but not sure how much hand-holding users need. Getting close. Exec has landed and is a replacement for ssh Attach on start remains hard. Implemented
I/we should also file issues for outstanding items without issues, and add lines for newly filed issues (e.g., command flag for run-container). Readiness is not yet used by rollingupdate. With respect to patch on the command line: I was thinking of a generic mechanism, but I could see the case for domain-aware commands, also. I think we need to better factor the command code and library code, and generic functionality vs. domain-aware functionality. Neither should ever convert to the internal representation. Domain-aware functionality (e.g., describe) should target a specific API version. Configuration generators, such as run-container, should be provided as services, or at least as separate plugin commands.
ref
ref
ref
ref
ref
ref


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-download-manifests.service

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: prometheusURL := "http://" + l.Addr().String() + "/metrics" mux := http.NewServeMux() <del> //lint:ignore SA1019 ignore deprecated warning until we move off of global registries mux.Handle("/metrics", legacyregistry.Handler()) </del> <ins> handler := compbasemetrics.HandlerFor(gather, compbasemetrics.HandlerOpts{}) mux.Handle("/metrics", handler) </ins> server := &http.Server{ Addr:    l.Addr().String(), Handler: mux,

TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in


Question: Host:        "foo.com", PathPrepend: "/proxy/node/node1:10250", } <ins> emptyHostAndSchemeTransport := &Transport{ PathPrepend: "/proxy/node/node1:10250", } </ins> type Item struct { input        string sourceURL    string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When accessing a service (e.g., ) by using the Apiserver proxy feature and the service returns with header, such as The returned header does not preserve the path if the service sent a redirect with an absolute path. Depending on the cluster, the header written by the service itself might have been and the apiserver proxy did rewrite the host:port to match the correct values. Unfortunately, it did not rewrite the path segment. What you expected to happen: I would expect the apiserver to rewrite the path segment of the response header so that it reflects the fact that it's proxied under the path: How to reproduce it (as minimally and precisely as possible): The path was not preserved. Anything else we need to know?: To work around this issue, a service may return strictly a relative url in the header such as: The feature to rewrite headers is quite common. For instance, Apache Httpd does it thanks to: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: custom_ OS (e.g. from /etc/os-release): Kernel (e.g. ):
/sig api-machinery /sig network
/assign
Thanks for the bug report!
GitHub didn't allow me to assign the following users: roycaihw. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return } if v.Len() > 0 { <del> extra := reflect.MakeSlice(v.Type(), 0, max) </del> <ins> extra := reflect.MakeSlice(v.Type(), v.Len(), max) </ins> reflect.Copy(extra, v) v.Set(extra) } else {

has by default, and since 1.22 has the same. They seem to return wrong results in certain situations combined with / . And / also use , so they also don't work correctly. outputs partial list of pods on the node, doesn't evict some of the pods. To make it work correctly, I had to specify works correctly with any , since it's just a performance optimizing parameter. I can't provide an actual concrete example (no = default ), but instead I show how to reproduce it with a smaller I believe that the same problem occurs with create two deployments. there are 5 pods in each. with --selector=app=another-server and , it shows only 4 pods. I could confirm that the problem happens in both case of and regardless namespace, even with , or <details</details<detailsGCP / GKE Standard </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig node
I think this is indeed a bug, after some simple testing, it seems to be an bug unique to 1.24, let me fix it. /assign
/assign
I can reproduce this error in a 1.24 cluster, but not in a 1.23.6 cluster. All clusters are created via kind, and I'm guessing gke may have cherry-picked some commits or had some of their own changes to their branch. So I think this fix should at least cherry-pick to 1.23 to help other distributions fix this issue.
/remove-sig api-machinery


Question: // The agnhost's image default entrypoint / args are: "/agnhost pause" // which will print out "Paused". <del> gomega.Eventually(ctx, pollLogs, 3, framework.Poll).Should(gomega.ContainSubstring("Paused")) </del> <ins> gomega.Eventually(ctx, pollLogs, time.Minute, framework.Poll).Should(gomega.ContainSubstring("Paused")) </ins> }) /*

pull-kubernetes-node-e2e-containerd ci-cos-containerd-node-e2e ci-kubernetes-e2e-node-canary Containers should use the image defaults if command and args are blank At least 7th June: ! No response /sig node
/cc This also flake in
/triage accepted /assign
/priority backlog
couple more flakes
Opened up to see if we can resolve this.


Question: } func onlyLocalNodePorts(t *testing.T, fp *Proxier, ipt *iptablestest.FakeIPTables) { <del> shouldLBTOSVCRuleExist := len(fp.clusterCIDR) > 0 </del> svcIP := "10.20.30.41" svcPort := 80 svcNodePort := 3001

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: This is discussed on When externalTrafficPolicy: Local is used, in GKE, external traffic coming in to the Load Balancer works correctly. But traffic that originates within the GKE cluster is broken, a TCP SYN is not responded to. My use case: I run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes. But I also access the docker-registry outside kubernetes. So my hostname for it is ''. In DNS this resolves to the LB IP. Actually, all 3 services have DNS names that resolve to the same LB IP, which then goes to the nginx ingress. so , , are generally addressed as LB-IP w/ HOSTNAME+SNI. Its 'astonishing' to not be able to access this DNS name when I'm 'inside' but can when I'm 'outside'. If I enable the IP transparency (because in my wiki I want the IP of the user who made the edit), then my gitlab-runner fails(since it cannot pull from the registry). Tim Hockin has this to say: and then: What you expected to happen: My expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public IP it should work. How to reproduce it (as minimally and precisely as possible): Configure a Load Balancer, an ingress, and 2 services. Have service 2 talk to service one using a DNS name that maps to the Load Balancer. Anything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-05-10T17:23:18Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): BUILDID=.89.0 NAME="Container-Optimized OS" KERNELCOMMITID= GOOGLECRASHID=Lakitu VERSIONID=66 PRETTYNAME="Container-Optimized OS from Google" VERSION=66 GOOGLEMETRICSPRODUCTID=26 HOMEURL="" ID=cos Kernel (e.g. ): Linux gke-k8s-default-pool--d3ft 4.14.22+ SMP Thu May 10 17:54:42 PDT 2018 x8664 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux Install tools: Others:
/sig network
Agree that this is weird and feels left-out. The fix is not complicated but I need an owner to carry it through testing.
if nobody is working on it can I take it as my first issue?.
Is anybody working on this issue. Seems like ipvs might have the same problem.
/assign
I believe I'm running into this problem and I think adding as Tim mentioned would fix it. are you working on this? If not I don't mind taking a stab at it :)
Thanks, Sorry I kinda forgot this thing. I was fixed this thing just like you said But encounter some problem when testing in gce environment, which caused by route rules. Sorry I forgot the detail, can you do some test in your environment?
It seems the pull request is stalled on this. Does anyone have any input?
Being relied on by istio
Following this.
Did this fix get lost in the shuffle?

Any chance getting this merged in? This also impacts Digital Ocean's Kubernetes + LB setups out of the box.
As far as I know it hasn't
Merge On Mon, Feb 18, 2019, 7:58 PM Jarred Capellman wrote:
still affecting me on gke. i want to enable the so that I can use the spam-blocking in wordpress (which is by ext ip), but I also want to refer to other services which may be internal to the cluster via their proper name (to get the `tls etc).
Same here.
I have more or less spelled out the fix, but I don't have teh bandwidth to polish it off right now. I am looking for volunteers...
I can take this given I've run into this a while back /assign
I was only able to reproduce this when trying to reach the public IP from the host network. Reaching the public IP from inside the pod network is working (as per this ). It's possible I missed a step in trying to reproduce this but wanted to confirm with you first. Are you seeing this for traffic originating only from host network or does it also apply for pods in the cluster network?
Hi. My situation. I have a DNS name (git.MYDOMAIN). It runs on a self-hosted gitlab via nginx-ingress via LoadBalancer in GKE. Other pods of mine sometimes reference https://git.MYDOMAIN. They may happen to run in the same cluster as git.MYDOMAIN, they may run in a different cluster. When they run in the same cluster on GKE, they fail if is enabled. Specifically, since the DNS resolves to (My external Loadbalancer IP) an IP outside of GKE, it goes out, hits the LB, and then somewhere along the reverse path is lost. I have calico enabled (as per GKE managed). I am not using Istio. I am using a cluster that predates the new private IP ones. I have not tried disabling the LoadBalancer and just using node ip for the ingress. I have not tried to see if there is any difference if the Pod trying to reach my git is on the same node as it or not.

Based on the proxy rules (may be different at this point cause I'm looking at master), I don't think your traffic is even reaching the LB, it's just being dropped by the iptable rules on the host. My original question was, are the pods trying to reach the same-cluster external IP have , if yes, can you reproduce on pods not on host network?
I do not have set.
Are you still on v1.10?
this is the most recent version on gke afaik.
I opened a fix here based on the comments and some testing I did on my GKE cluster. It needs more validation though
Again, I wasn't able to reproduce this for pods that are within the cluster CIDR (but I could on the host network). The rule that allows for pods inside the cluster CIDR to still reach pods behind an LB is this: Are you able to check if a rule like ^ exists on your cluster?
Here's an example, right now (www is hosted on ingress w/ loadbalancer w/ local policy). Note: this example is from the node (not a pod).
Here's the full rules:
^ can you show me the same test from within a pod (not host network)? I have a fix for within a node but I'd like to make sure it's reproducible inside a pod before I dig deeper
Interestingly its working in a Pod on this cluster now. (?!) I guess some previous upgrade of k8s on gke has altered it? on cannot reach on on can reach. hmmm.
ok w/o the node being able to reach this i'm still blocked (since my container registry might run there, and thus kubelet cannot reach, so my gitlab CI pipeline cannot reach). but, i guess, now the pods w/o hostnetwork can, so some progress
Thanks for checking should fix the internal reachability issue from the host. Though it requires SNAT.
Hello Maybe better way is to check if and then skip to KUBE-SERVICES chain? So, request will be sent to loadbalancer and source IP will be preserved. Can you try this?
Solution described above will affect traffic from pods, it will not be . Then maybe we can replace with something to send traffic to loadbalancer?

I think this will break existing implementations, the fwchain is what routes the LB IPs to the correct service chain. Not all LB implementations (think MetalLB) depend on node port for routing
I tried to exclude node IP with from on node without local endpoints and it works! What do you think about this solution? I tested with:
I believe there's a similar rule for that already but it doesn't check source IP If you jump to FW chain, you end up in XLB chain which drops traffic if there's no local endpoints. Does your test also work on nodes where there's no local endpoints? Can you also verify that your traffic actually goes to the LB and is not being masqueraded through the service chain?
Yes, I just source IP check to this rule on node without local endpoints, so local traffic doesn't jump to FW, it doesn't drops in XLB and it goes to LB. Of course, if you do this on node with local endpoints you will not reach that endpoint (at least when using Metallb). This solution doesn't work if nodes haven't network access to LB. Your solution haven't this problem. For me it is more important then saving source IP, so now I think your solution is better. But why you add jump to MARK-MASQ before jump to SVC when each SEP chain already contains jump to MARK-MASQ? I tested your solution with:
Good point, I think might be redundant, will update the PR soon :)


Question: --terminated-pod-gc-threshold=12500: Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled. ``` <del> ###### Auto generated by spf13/cobra on 26-May-2016 </del> <ins> ###### Auto generated by spf13/cobra on 27-May-2016 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->

Immediately after this test was reenabled in we got huge numbers of parallel flakes in . This might be a coincidence. I'll make a PR to move it to flaky. If things haven't improved after it's moved then I'll revert, else I'll assign this issue to !
Do you have any logs? references , but that has nothing to do with NFS nor PersistentVolumes.
. It's pretty clear this test caused the flakes we were seeing.
do you have any logs?
It hasn't flaked since your fix went in, but we should keep the issue open until it's moved out of the flaky suite.
The fix Joe Finney developed was merged on 5/30. The signature for the problem in the log file referenced here is: 6 18:56:07.895: Expected PersistentVolume to be bound, but got nil ClaimRef: kind:"" apiVersion:""` It looks like from this dashboard the last instance of this error log signature for the test PersistentVolumes NFS test to the flaky suite. was on 5/30 right around the time the code was merged. I found the off the dashboard at https://k8s- Since then the error has not occurred with that signature since I am tempted to close this bug, but keep the the test suite as marked "Flaky". The merge bot should start flagging the new bugs for this test going forward.
If it's no longer flaky we should move it out of the flaky suite. While it's in the flaky suite it won't block merges if it breaks and mergebot won't open bugs for it. I'm worried that if we just close this issue it'll sit in the flaky suite and break and noone will notice for a long time.
This bug has not been reproduced for past 2 months, can we close it?
It looks like the test is still in the flaky suite, although with a different name. Once it's moved out of there then this can be closed.
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
Not reproduced for three months, closing.

Consider reducing the PVClaimBinderSyncPeriod from 10 minutes down to something more reasonable, so that volumes don't sit around unbound for half an hour if it fails twice. This would reduce the impact of issues like until the binder/recycler/provosioner consolidation/refactor is complete. CC
+100
do you remember what was the reason for 10 minutes?
I checked provisioner and recycler, lowering the sync period should not harm them. Still, it will impact deleter a bit. Now we try to delete a volume three times every sync-period (10 minutes), mostly because if user deletes a pod that uses a volume and deletes also the PVC bound to it, the volume is still attached to the node for some time and all deletion attempts fail. I've seen AWS detach taking around 5 minutes in some cases. So, if we lower the timer, we should also increase the number of retries. Or have a different retry timer for deletion.


Question: expected := []*runtimeapi.PodSandbox{} state := runtimeapi.PodSandboxState_SANDBOX_READY <del> var createdAt int64 = 0 </del> <ins> var createdAt int64 = fakeClock.Now().UnixNano() </ins> for i := range configs { id, err := ds.RunPodSandbox(configs[i]) assert.NoError(t, err)

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


Question: drainedNodes.Insert(info.Name) printObj(info.Object, o.Out) } else { <ins> if o.drainer.IgnoreErrors && len(o.nodeInfos) > 1 { fmt.Fprintf(o.ErrOut, "error: unable to drain node %q due to error:%s, continuing command...n", info.Name, err) continue } fmt.Fprintf(o.ErrOut, "DEPRECATED WARNING: Aborting the drain command in a list of nodes will be deprecated.n"+ "The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain.n"+ "For now, users can try such experience via: --ignore-errorsn") </ins> fmt.Fprintf(o.ErrOut, "error: unable to drain node %q, aborting command...nn", info.Name) remainingNodes := []string{} fatal = err

<!-- Please only use this template for submitting enhancement requests --What would you like to be : on multiple nodes, if one node failed, the command aborting. Without special purpose, maybe it's better to continue processing other nodes. Why is this needed: Follow the pattern of other subcommand, such as kubectl delete.
/sig cli
/triage accepted /assign
done /close
Closing this issue. <detailsIn response to : done /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would wait the patch get merged before closing :)


Question: // You HAVE TO explicitly call w.RUnlock() after this function. func (w *watchCache) waitUntilFreshAndBlock(ctx context.Context, resourceVersion uint64) error { startTime := w.clock.Now() <ins> defer func() { if resourceVersion > 0 { metrics.WatchCacheReadWait.WithContext(ctx).WithLabelValues(w.groupResource.String()).Observe(w.clock.Since(startTime).Seconds()) } }() </ins> // In case resourceVersion is 0, we accept arbitrarily stale result. // As a result, the condition in the below for loop will never be

Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: BackendIPConfigurations: &[]network.InterfaceIPConfiguration{ { Name: to.StringPtr("ip-1"), <del> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/networkInterfaces/nic"), </del> <ins> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-0/virtualMachines/0/networkInterfaces/nic"), </ins> }, }, },

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: }) } } <ins> func TestGenerateNodeDebugPod(t *testing.T) { defer func(old func(int) string) { nameSuffixFunc = old }(nameSuffixFunc) var suffixCounter int nameSuffixFunc = func(int) string { suffixCounter++ return fmt.Sprint(suffixCounter) } for _, tc := range []struct { name, nodeName string opts           *DebugOptions expected       *corev1.Pod }{ { name:     "minimum options", nodeName: "node-XXX", opts: &DebugOptions{ Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, expected: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "node-debugger-node-XXX-1", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:                     "debugger", Image:                    "busybox", ImagePullPolicy:          corev1.PullIfNotPresent, TerminationMessagePolicy: corev1.TerminationMessageReadFile, VolumeMounts: []corev1.VolumeMount{ { MountPath: "/host", Name:      "host-root", }, }, }, }, HostIPC:       true, HostNetwork:   true, HostPID:       true, NodeName:      "node-XXX", RestartPolicy: corev1.RestartPolicyNever, Volumes: []corev1.Volume{ { Name: "host-root", VolumeSource: corev1.VolumeSource{ HostPath: &corev1.HostPathVolumeSource{Path: "/"}, }, }, }, }, }, }, { name:     "debug args as container command", nodeName: "node-XXX", opts: &DebugOptions{ Args:       []string{"/bin/echo", "one", "two", "three"}, Container:  "custom-debugger", Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, expected: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "node-debugger-node-XXX-1", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:                     "custom-debugger", Command:                  []string{"/bin/echo", "one", "two", "three"}, Image:                    "busybox", ImagePullPolicy:          corev1.PullIfNotPresent, TerminationMessagePolicy: corev1.TerminationMessageReadFile, VolumeMounts: []corev1.VolumeMount{ { MountPath: "/host", Name:      "host-root", }, }, }, }, HostIPC:       true, HostNetwork:   true, HostPID:       true, NodeName:      "node-XXX", RestartPolicy: corev1.RestartPolicyNever, Volumes: []corev1.Volume{ { Name: "host-root", VolumeSource: corev1.VolumeSource{ HostPath: &corev1.HostPathVolumeSource{Path: "/"}, }, }, }, }, }, }, { name:     "debug args as container args", nodeName: "node-XXX", opts: &DebugOptions{ ArgsOnly:   true, Container:  "custom-debugger", Args:       []string{"echo", "one", "two", "three"}, Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, expected: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "node-debugger-node-XXX-1", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:                     "custom-debugger", Args:                     []string{"echo", "one", "two", "three"}, Image:                    "busybox", ImagePullPolicy:          corev1.PullIfNotPresent, TerminationMessagePolicy: corev1.TerminationMessageReadFile, VolumeMounts: []corev1.VolumeMount{ { MountPath: "/host", Name:      "host-root", }, }, }, }, HostIPC:       true, HostNetwork:   true, HostPID:       true, NodeName:      "node-XXX", RestartPolicy: corev1.RestartPolicyNever, Volumes: []corev1.Volume{ { Name: "host-root", VolumeSource: corev1.VolumeSource{ HostPath: &corev1.HostPathVolumeSource{Path: "/"}, }, }, }, }, }, }, } { t.Run(tc.name, func(t *testing.T) { tc.opts.IOStreams = genericclioptions.NewTestIOStreamsDiscard() suffixCounter = 0 pod := tc.opts.generateNodeDebugPod(tc.nodeName) if diff := cmp.Diff(tc.expected, pod); diff != "" { t.Error("unexpected diff in generated object: (-want +got):n", diff) } }) } } </ins>

This issue tracks implementing node debugging with privileged containers for , as described in the This feature allows administrators to start a privileged container on a node in host namespaces rather than having to SSH to the node. For example: /kind feature /sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: srcs = [":package-srcs"], tags = ["automanaged"], ) <ins> go_test( name = "go_default_test", srcs = ["configfiles_test.go"], library = ":go_default_library", deps = [ "//pkg/kubelet/apis/kubeletconfig:go_default_library", "//pkg/kubelet/apis/kubeletconfig/scheme:go_default_library", "//pkg/kubelet/apis/kubeletconfig/v1alpha1:go_default_library", "//pkg/kubelet/kubeletconfig/util/files:go_default_library", "//pkg/kubelet/kubeletconfig/util/filesystem:go_default_library", "//pkg/kubelet/kubeletconfig/util/test:go_default_library", "//vendor/github.com/davecgh/go-spew/spew:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", ], ) </ins>

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: }) // Checks whether the output split by line contains the required elements. <del> func checkOutput(output string, required [][]string) { </del> <ins> func checkOutputReturnError(output string, required [][]string) error { </ins> outputLines := strings.Split(output, "n") currentLine := 0 for _, requirement := range required {

https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing.

I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close


Question: "unsafe" "golang.org/x/sys/windows" <del> "k8s.io/apimachinery/pkg/api/resource" </del> ) var (

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: // trigger a new deployment framework.Logf("%02d: triggering a new rollout for deployment %q", i, deployment.Name) deployment, err = e2edeployment.UpdateDeploymentWithRetries(c, ns, deployment.Name, func(update *appsv1.Deployment) { <del> newEnv := v1.EnvVar{Name: "A", Value: fmt.Sprintf("%d", i)} </del> <ins> newEnv := v1.EnvVar{Name: fmt.Sprintf("A%d", i), Value: fmt.Sprintf("%d", i)} </ins> update.Spec.Template.Spec.Containers[0].Env = append(update.Spec.Template.Spec.Containers[0].Env, newEnv) randomScale(update, i) })

Seen in the KIND e2e test: a) maybe we shouldn't log this more than 1x per second or something? b) maybe the error could make it clearer that the problem is the user's data? c) maybe we could fix the test triggering this? /sig api-machinery /wg apply Logs that I saw this in:
The 'duplicate entries for key' error came from validatingObjectWalker#visitListItems.
/assign
Yeah, they have multiple environment variables with the same name. I don't think that should be an error though ...
/cc
At the same time, it's not really useful to have multiple times the same variable and is probably a recipe for trouble.
It's definitely user error. Spamming apiserver logs about it doesn't help things, though.
I don't want to hide it, it's really not supposed to happen :-) it's not really user-data though. The fact that the error triggers here rather than in the actual validation means that there is a discrepancy between our validation and the regular validation. That's probably a problem that we have to fix. In that specific example, we can see that Kubernetes allows multiple environment variable with the same name, while we have assumed that it's not OK. It's a very legitimate error and we have to think about it to fix it. I think the test is relevant, and people out there are doing that. We could consider doing ratcheting if we want to stop allowing multiple variables, but I don't think that's how we want to fix that problem.
For history: should fix that.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We've definitely improved that, maybe let's close it for now.
/reopen We still see this happening. Kube users get nervous, I would too. Please fix the root cause.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Name:         "vol1", VolumeSource: v1.VolumeSource{NFS: &v1.NFSVolumeSource{Server: "localhost", Path: "/somepath", ReadOnly: false}}, } <del> doTestPlugin(t, volume.NewSpecFromVolume(vol)) </del> <ins> doTestPlugin(t, volume.NewSpecFromVolume(vol), "localhost:/somepath") } func TestIPV6VolumeSource(t *testing.T) { vol := &v1.Volume{ Name:         "vol1", VolumeSource: v1.VolumeSource{NFS: &v1.NFSVolumeSource{Server: "0:0:0:0:0:0:0:1", Path: "/somepath", ReadOnly: false}}, } doTestPlugin(t, volume.NewSpecFromVolume(vol), "[0:0:0:0:0:0:0:1]:/somepath") } func TestIPV4VolumeSource(t *testing.T) { vol := &v1.Volume{ Name:         "vol1", VolumeSource: v1.VolumeSource{NFS: &v1.NFSVolumeSource{Server: "127.0.0.1", Path: "/somepath", ReadOnly: false}}, } doTestPlugin(t, volume.NewSpecFromVolume(vol), "127.0.0.1:/somepath") </ins> } func TestPluginPersistentVolume(t *testing.T) {

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: } wg.Done() }() <del> wg.Add(1) </del> } wg.Wait() <del> if len(nameToNodeInfo) != len(nodeList.Items) { </del> <ins> if len(nm.nameToNodeInfo) != len(nodeList.Items) { </ins> return errors.New("all nodes not mapped to respective vSphere") } return nil

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: fs.StringVar(&s.RktStage1Image, "rkt-stage1-image", s.RktStage1Image, "image to use as stage1. Local paths and http/https URLs are supported. If empty, the 'stage1.aci' in the same directory as '--rkt-path' will be used.") fs.MarkDeprecated("rkt-stage1-image", "Will be removed in a future version. The default stage1 image will be specified by the rkt configurations, see https://github.com/coreos/rkt/blob/master/Documentation/configuration.md for more details.") fs.BoolVar(&s.ConfigureCBR0, "configure-cbr0", s.ConfigureCBR0, "If true, kubelet will configure cbr0 based on Node.Spec.PodCIDR.") <ins> fs.MarkDeprecated("configure-cbr0", "Will be removed in a future version. Please use kubenet or other network plugins.") </ins> fs.StringVar(&s.HairpinMode, "hairpin-mode", s.HairpinMode, "How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none".") fs.BoolVar(&s.BabysitDaemons, "babysit-daemons", s.BabysitDaemons, "If true, the node has babysitter process monitoring docker and kubelet.") fs.MarkDeprecated("babysit-daemons", "Will be removed in a future version.")

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: set -o nounset set -o pipefail <del> # As of go 1.6, the vendor experiment is enabled by default. export GO15VENDOREXPERIMENT=1 #### HACK #### # Sometimes godep just can't handle things. This lets use manually put # some deps in place first, so godep won't fall over. preload-dep() { org="$1" project="$2" sha="$3" # project_dir ($4) is optional, if unset we will generate it if [[ -z ${4:-} ]]; then project_dir="${GOPATH}/src/${org}/${project}.git" else project_dir="${4}" fi echo "**HACK** preloading dep for ${org} ${project} at ${sha} into ${project_dir}" git clone "https://${org}/${project}" "${project_dir}" > /dev/null 2>&1 pushd "${project_dir}" > /dev/null git checkout "${sha}" popd > /dev/null } </del> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh"

Didn't see an issue for this. Godep recently updated its version to v80. This broke several builds that directly godep, grabbing latest. /kind bug cc Example https://k8s- Here are some examples where we grab master:
We have helpers to grab godep. We should use them everywhere.
/cc
/assign