Question: "//vendor/golang.org/x/net/websocket:go_default_library", "//vendor/k8s.io/api/autoscaling/v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", <del> "//vendor/k8s.io/api/extensions/v1beta1:go_default_library", </del> "//vendor/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/fields:go_default_library",

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: var addrs []net.IP addrs, _ = net.LookupIP(node.Name) for _, addr := range addrs { <del> if err = validateNodeIP(addr); err == nil { </del> <ins> if err = kl.nodeIPValidator(addr); err == nil { </ins> if addr.To4() != nil { ipAddr = addr break

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: } } <ins> func TestSingleJobFailedCondition(t *testing.T) { clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) manager, sharedInformerFactory := newControllerFromClient(clientset, controller.NoResyncPeriodFunc) fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady var actual *batch.Job manager.updateStatusHandler = func(ctx context.Context, job *batch.Job) (*batch.Job, error) { actual = job return job, nil } job := newJob(1, 1, 6, batch.NonIndexedCompletion) activeDeadlineSeconds := int64(10) job.Spec.ActiveDeadlineSeconds = &activeDeadlineSeconds start := metav1.Unix(metav1.Now().Time.Unix()-15, 0) job.Status.StartTime = &start job.Status.Conditions = append(job.Status.Conditions, *newCondition(batch.JobFailed, v1.ConditionFalse, "DeadlineExceeded", "Job was active longer than specified deadline")) sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job) forget, err := manager.syncJob(context.TODO(), testutil.GetKey(job, t)) if err != nil { t.Errorf("Unexpected error when syncing jobs %v", err) } if !forget { t.Errorf("Unexpected forget value. Expected %v, saw %vn", true, forget) } if len(fakePodControl.DeletePodName) != 0 { t.Errorf("Unexpected number of deletes.  Expected %d, saw %dn", 0, len(fakePodControl.DeletePodName)) } if actual == nil { t.Error("Expected job modificationn") } failedConditions := getConditionsByType(actual.Status.Conditions, batch.JobFailed) if len(failedConditions) != 1 { t.Error("Unexpected number of failed conditionsn") } if failedConditions[0].Status != v1.ConditionTrue { t.Errorf("Unexpected status for the failed condition. Expected: %v, saw %vn", v1.ConditionTrue, failedConditions[0].Status) } } </ins> func TestSyncJobComplete(t *testing.T) { clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) manager, sharedInformerFactory := newControllerFromClient(clientset, controller.NoResyncPeriodFunc)

If the user update , duplicate condition: ! Only one Failed condition Create Job and wait it marked as Failed Update the Failed condition to False No response <details</details<details</details<details</details<details</details<details</details<details</details
It's not a bug. The is a list, so it can contain many conditions with different probe timestamps. Also, we should not update the fields.
But should we avoid the duplicate Type's conditionIn my opinionthis will confuse the user. According to kubelet, it will rewrite the 'Ready' condition in pod, and take is as managed only by kubelet. Should we did this for job Failed condition? Refer:
/sig apps
It is a bug. We should only have one condition for each type. /triage accepted
/assign
/assign
/assign
I have reproduced the issue locally and intend work on the fix
/priority backlog Since the user is manually modifying the status, this is not an inherent bug in the controller. Still worth avoiding duplicates.


Question: return runChecks(checks, os.Stderr) } <del> func RunJoinNodeChecks() error { </del> <ins> func RunJoinNodeChecks(cfg *kubeadmapi.NodeConfiguration) error { </ins> // TODO: Some of these ports should come from kubeadm config eventually: checks := []PreFlightCheck{ IsRootCheck{root: true},

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: <del> #!/bin/bash # Copyright 2015 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Create generic token following GCE standard create_token() { echo $(cat /dev/urandom | base64 | tr -d "=+/" | dd bs=32 count=1 2> /dev/null) } get_tokens_from_csv() { KUBE_BEARER_TOKEN=$(awk -F, '/admin/ {print $1}' ${KUBE_TEMP}/${1}_tokens.csv) KUBELET_TOKEN=$(awk -F, '/kubelet/ {print $1}' ${KUBE_TEMP}/${1}_tokens.csv) KUBE_PROXY_TOKEN=$(awk -F, '/kube_proxy/ {print $1}' ${KUBE_TEMP}/${1}_tokens.csv) } generate_admin_token() { echo "$(create_token),admin,admin" >> ${KUBE_TEMP}/known_tokens.csv } # Creates a csv file each time called (i.e one per kubelet). generate_kubelet_tokens() { echo "$(create_token),kubelet,kubelet" > ${KUBE_TEMP}/${1}_tokens.csv echo "$(create_token),kube_proxy,kube_proxy" >> ${KUBE_TEMP}/${1}_tokens.csv } </del>

Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week.

However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


Question: echo "DBG: running $(CONVERSION_GEN) for $$pkgs";             fi;                                                               ./hack/run-in-gopath.sh $(CONVERSION_GEN)                         <del> --extra-peer-dirs $(CONVERSION_EXTRA_PEER_DIRS)               </del> <ins> --extra-peer-dirs $$(echo $(CONVERSION_EXTRA_PEER_PKGS) | sed 's/ /,/g')  --extra-dirs $$(echo $(CONVERSION_EXTRA_PKGS) | sed 's/ /,/g')  </ins> --v $(KUBE_VERBOSE)                                           --logtostderr                                                 -i "$$pkgs"

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: --bind-address="${API_BIND_ADDR}"  --insecure-bind-address="${API_HOST_IP}"  --insecure-port="${API_PORT}"  <del> --advertise-address="${API_HOST_IP}"  </del> --etcd-servers="http://${ETCD_HOST}:${ETCD_PORT}"  --service-cluster-ip-range="10.0.0.0/24"  --cloud-provider="${CLOUD_PROVIDER}"

seems to have broken the kubernetes service when using : With this patch: Without it: Tested with v 1.3.2.
Looks like we don't like explicitly don't want to allow 127.0.0.1 loopback address here -


Question: file: {{ file }} {% endfor %} <del> /etc/logrotate.d/docker: </del> <ins> /etc/logrotate.d/docker-containers: </ins> file: - managed <del> - source: salt://logrotate/docker </del> <ins> - source: salt://logrotate/docker-containers </ins> - template: jinja - user: root - group: root

We have rotation of logs from docker containers, but not for logs from docker Without rotation, the logs may fill the root partition entirely.
I can add rotation (5 files, 100MB each) if you decide it goes for v1.0 (for me it's a blocker). I'm not sure which platforms are affected. For sure GCE. cc
Sure, send a PR and I'm sure we can merge it.


Question: } } <ins> func updateFile(t *testing.T, configDir, filename string, newContent []byte) { t.Helper() // Create a temporary file tempFile, err := os.CreateTemp(configDir, "tempfile") if err != nil { t.Fatal(err) } defer tempFile.Close() // Write the new content to the temporary file _, err = tempFile.Write(newContent) if err != nil { t.Fatal(err) } // Atomically replace the original file with the temporary file err = os.Rename(tempFile.Name(), filepath.Join(configDir, filename)) if err != nil { t.Fatal(err) } } </ins> func TestKMSHealthz(t *testing.T) { encryptionConfig := ` kind: EncryptionConfiguration

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil, err } return certificates.NewCertificateController( <ins> "csrsigning", </ins> client, csrInformer, signer.handle,

There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


Question: install: - if ! go get code.google.com/p/go.tools/cmd/cover; then go get golang.org/x/tools/cmd/cover; fi <ins> - go get github.com/mattn/goveralls </ins> - ./hack/travis/install-etcd.sh - ./hack/verify-gofmt.sh - ./hack/verify-boilerplate.sh

Right now, we execute unit tests in Travis with , and you can see the results on each line: but we don't have an easy place to go browse our code and figure out what's covered and what isn't. Is there some awesome way to do this? At the very least, can we get the actual line-by-line coverage results integrated into our CI build process so we can poke at these somehow?
cc
I'm willing to take a look into this to at least see if something is possible.
Looking into using Coveralls; there are a few other teams at Google already using this. Integration with Travis and Go are supported with the goveralls package:
Unit test coverage can now be viewed at - both master push builds and PRs will be shown there; additionally, Coveralls will update the status and post comments on PRs with coverage stats. (If we find these too spammy, we can disable either or both.) The top-level also has a badge/link to the coverage dashbaord.


Question: install: - if ! go get code.google.com/p/go.tools/cmd/cover; then go get golang.org/x/tools/cmd/cover; fi <ins> - go get github.com/mattn/goveralls </ins> - ./hack/travis/install-etcd.sh - ./hack/verify-gofmt.sh - ./hack/verify-boilerplate.sh

Our test- script invokes 'go test' serially for each pkg - currently taking 2 minutes on my laptop. Running a single 'go test' for all pkgs cuts the time down to 45 seconds. Even running with -race is only 1:15. As far as I can tell, the only time we HAVE to run serially is when coverage is on. Does anyone have time to fix this?
I'd created on Friday to fix this, though it's not yet been reviewed.
I cobbled together but I'll review yours since it is probably better :) , Jeff Grafton wrote:


Question: case err != nil && len(test.err) == 0, err == nil && len(test.err) != 0, err != nil && test.err != err.Error(): <del> t.Errorf("%s unexpected error: %v", testName, err) </del> <ins> t.Errorf("%s want: %v,got: %v", testName, test.err, err) </ins> } } }

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: } // If PV contains zone related label, it may impact cached NoVolumeZoneConflict <del> for k := range pv.ObjectMeta.Labels { if k == kubeletapis.LabelZoneFailureDomain || k == kubeletapis.LabelZoneRegion { </del> <ins> for k := range pv.Labels { if isZoneRegionLabel(k) { </ins> invalidPredicates.Insert("NoVolumeZoneConflict") break }

What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate).

is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.


Question: if attr.GetSubresource() != "" { return nil } <del> // ignore all operations that are not namespaced if attr.GetNamespace() == "" { </del> <ins> // ignore all operations that are not namespaced or creation of namespaces if attr.GetNamespace() == "" || isNamespaceCreation(attr) { </ins> return nil } return a.evaluator.Evaluate(attr) } <ins> func isNamespaceCreation(attr admission.Attributes) bool { return attr.GetOperation() == admission.Create && attr.GetKind().GroupKind() == namespaceGVK } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: } } <ins> func TestSingleWatch(t *testing.T) { _, s := runAMaster(t) defer s.Close() ns := "blargh" deleteAllEtcdKeys() client := client.NewOrDie(&client.Config{Host: s.URL, Version: testapi.Version()}) mkEvent := func(i int) *api.Event { name := fmt.Sprintf("event-%v", i) return &api.Event{ ObjectMeta: api.ObjectMeta{ Namespace: ns, Name:      name, }, InvolvedObject: api.ObjectReference{ Namespace: ns, Name:      name, }, Reason: fmt.Sprintf("event %v", i), } } rv1 := "" for i := 0; i < 10; i++ { event := mkEvent(i) got, err := client.Events(ns).Create(event) if err != nil { t.Fatalf("Failed creating event %#q: %v", event, err) } if rv1 == "" { rv1 = got.ResourceVersion if rv1 == "" { t.Fatal("did not get a resource version.") } } t.Logf("Created event %#v", got.ObjectMeta) } w, err := client.Get(). Prefix("watch"). NamespaceIfScoped(ns, len(ns) > 0). Resource("events"). Name("event-9"). Param("resourceVersion", rv1). Watch() if err != nil { t.Fatalf("Failed watch: %v", err) } defer w.Stop() select { case <-time.After(5 * time.Second): t.Fatal("watch took longer than 15 seconds") case got, ok := <-w.ResultChan(): if !ok { t.Fatal("Watch channel closed unexpectedly.") } // We expect to see an ADD of event-9 and only event-9. (This // catches a bug where all the events would have been sent down // the channel.) if e, a := watch.Added, got.Type; e != a { t.Errorf("Wanted %v, got %v", e, a) } switch o := got.Object.(type) { case *api.Event: if e, a := "event-9", o.Name; e != a { t.Errorf("Wanted %v, got %v", e, a) } default: t.Fatalf("Unexpected watch event containing object %#q", got) } } } </ins> func TestMultiWatch(t *testing.T) { // Disable this test as long as it demonstrates a problem. // TODO: Reenable this test when we get #6059 resolved.

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred(), "while creating pod") <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, time.Minute) </del> <ins> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) </ins> Expect(err).ToNot(HaveOccurred(), "while waiting for pod to be running") By("Failing liveness probe")

Failing Job: These tests started flaking recently (they were consistently green for a long time): Mounted volume expand[Slow] Should verify mounted devices can be resized Volume expand [Slow] Verify if editing PVC allows resize Subpath [Volume type: gcePDPVC] should support restarting containers using file as subpath [Slow] Subpath [Volume type: gcePDPVC] should fail for new directories when readOnly specified in the volumeSource Subpath [Volume type: gcePDPVC] should fail if subpath with backstepping is outside the volume [Slow] Volumes PD should be mountable with ext4 Affected runs: https://k8s- https://k8s- https://k8s- /kind bug /sig storage /priority important-soon /milestone v1.12 /cc - Could you PTAL or delegate? Thanks.
Volume expansion tests have been fixed by
/assign can you investigate the remaining failures?
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently: https://k8s- https://k8s- /reopen
I think the latest failures are caused by gce API rate limiting: /assign
It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration
/sig scalability
/kind failing-test
/status approved-for-milestone Friendly ping. This is affecting release-blocking scalability job.
hey what needs to be done here? by whom?
Could you comment on what needs to be done here and by whom?
can you guys take a look at the API quota issue?

This is happening consistently in the scale test environment
as Code Freeze is today, this needs to be of priority . Please triage! cc
It looks like the latest run on 9/1 has passed.
Marking closed as the fixes have been merged and the test runs are now passing /close
Closing this issue. <detailsIn response to : Marking closed as the fixes have been merged and the test runs are now passing /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: e.inProgress.Delete(ns) } <ins> // getWork returns a namespace, a list of work items in that // namespace, and a shutdown boolean.  If not shutdown then the return // must eventually be followed by a call on completeWork for the // returned namespace (regardless of whether the work item list is // empty). </ins> func (e *quotaEvaluator) getWork() (string, []*admissionWaiter, bool) { uncastNS, shutdown := e.queue.Get() if shutdown {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I wrote a simple driver program () that creates ConfigMap objects as fast as it can, and ran it like this: . In the CSV file produced, two of the creates recorded an error that simply says "timeout": Here is an excerpt from the api-server's log: A few minutes later I checked that my etcd cluster is healthy; it was: I also looked at the on each host for etcd, and each reported that etcd had been running for over a day. What you expected to happen: I expected no such problem, or at least a more informative error message. How to reproduce it (as minimally and precisely as possible): That test driver is pretty minimal. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: SoftLayer bare metal machines, SSD storage on the etcd servers OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): Linux mjs-api-dal10-b 4.4.0-122-generic -Ubuntu SMP Mon Apr 23 15:34:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm driven by an Ansible playbook I wrote, based on Others:
That's weird, I'm not aware of a 10s timeout anywhere (there is a 60s one). does the etcd client have such a timeout?
FYI:
- can you try etcd without SSL/TLS? it could be ( https://k8s-*&repos=kubernetes )
Yes, I will try that, the next time it is convenient to tear everything down and start over.
OK, I tried again with etcd NOT using TLS (neither client/server nor server/server). I still got a "timeout" error. The driver at commit .
I am running kube release 1.10.3 now. I turned v to 5 in the api-server driven by the client, and ran another trial.

Here are the errors logged by the driver (with timestamps in UTC): I found this in the stderr of the api-server (the timestamp seems to be in UTC):
The etcd cluster should have no trouble with that write rate.
Around that time, the etcd server that was loaded by the kube api-server logged the following (in US CDT):
try tweaking and ?
I to the loaded api-server's command line, and still got 10s timeouts.
The doc string for seems to say it is irrelevant for create operations.
However, making the client specify a timeout of 17 sec (see ) warded off the problem --- according to my first and only trial so far.
However, my second trial was much more interesting. The api-server logged an error on create --- but the client did not! And the subsequent update and delete operations on the object in question got 404s. Here is what the driver logged about that object: Here is what the api-server logged around that time: The etcd leader logged nothing unusual around that time.
has suggested the mysterious timeout might be due to . I do indeed have the admission controller enabled. However, I have no ResourceQuota objects in any namespace.
I tried removing from the admission controller list, and that made the mysterious 10 second timeouts go away. Why is the ResourceQuota admission controller causing these 10 second timeouts? Remember, they are rare: I can do hundreds of thousands of create operations in rapid fire, and get a single digit number of timeouts, while the other create operations complete in well under 10 seconds. See for an example.
Here is another example (now with and the etcd cluster using TLS): <img width="578" alt="screen shot 2018-05-25 at 1 29 51 pm" src="https://user-"
So I slowed down from 3000 to 2000 kube writes per second, but I still got a timeout. <img width="576" alt="screen shot 2018-05-25 at 2 13 59 pm" src="https://user-"And no, that is not a really tight cluster of data points at (17:35:25.870, 10.036) --- it really is just a single data point. No other operation took anywhere near that long.

Interestingly, there is a roughly 50 ms bubble of latency around the time of the start of the operation that timed out. As you can see from the graph, there are other bubbles of latency. All much smaller than 10 sec. Here are 36 consecutive lines of driver data from around the timeout, sorted by preop time.
Looking in , I see something odd. In two places --- and in --- there is a special case for an empty list of objects. Now, it is not easy to imagine how the list could ever be empty when the namespace is in the queue --- but if this ever comes to pass then something bad happens: is called twice (first in and second in ).
So I made to test the hypothesis that the double-call of is a critical part of the pathology. I ran a few trials, and got no timeouts. Here is the longest:
wanna file a WIP PR so we can see the exact changes?
yeah, when I get a chance. Weekend now.
very nice investigation you've done here!


Question: genericregistrytest "k8s.io/apiserver/pkg/registry/generic/testing" "k8s.io/apiserver/pkg/registry/rest" etcd3testing "k8s.io/apiserver/pkg/storage/etcd3/testing" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> api "k8s.io/kubernetes/pkg/apis/core" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/registry/core/persistentvolume" "k8s.io/kubernetes/pkg/registry/registrytest" )

pkg/registry/core/persistentvolume/storage TestUpdateStatus unknown No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/priority important-soon as it failed in master blocking


Question: // TestPriorityQueue_AssignedPodAdded tests AssignedPodAdded. It checks that // when a pod with pod affinity is in unschedulablePods and another pod with a // matching label is added, the unschedulable pod is moved to activeQ. <del> func TestPriorityQueue_AssignedPodAdded(t *testing.T) { logger, ctx := ktesting.NewTestContext(t) ctx, cancel := context.WithCancel(ctx) defer cancel() affinityPod := st.MakePod().Name("afp").Namespace("ns1").UID("afp").Annotation("annot2", "val2").Priority(mediumPriority).NominatedNodeName("node1").PodAffinityExists("service", "region", st.PodAffinityWithRequiredReq).Obj() labelPod := st.MakePod().Name("lbp").Namespace(affinityPod.Namespace).Label("service", "securityscan").Node("node1").Obj() c := testingclock.NewFakeClock(time.Now()) m := makeEmptyQueueingHintMapPerProfile() m[""][AssignedPodAdd] = []*QueueingHintFunction{ </del> <ins> func TestPriorityQueue_AssignedPodAdded_(t *testing.T) { tests := []struct { name               string unschedPod         *v1.Pod unschedPlugin      string updatedAssignedPod *v1.Pod wantToRequeue      bool }{ </ins> { <del> PluginName:     "fakePlugin", QueueingHintFn: queueHintReturnQueue, </del> <ins> name:               "Pod rejected by pod affinity is requeued with matching Pod's update", unschedPod:         st.MakePod().Name("afp").Namespace("ns1").UID("afp").Annotation("annot2", "val2").PodAffinityExists("service", "region", st.PodAffinityWithRequiredReq).Obj(), unschedPlugin:      names.InterPodAffinity, updatedAssignedPod: st.MakePod().Name("lbp").Namespace("ns1").Label("service", "securityscan").Node("node1").Obj(), wantToRequeue:      true, }, { name:               "Pod rejected by pod affinity isn't requeued with unrelated Pod's update", unschedPod:         st.MakePod().Name("afp").Namespace("ns1").UID("afp").Annotation("annot2", "val2").PodAffinityExists("service", "region", st.PodAffinityWithRequiredReq).Obj(), unschedPlugin:      names.InterPodAffinity, updatedAssignedPod: st.MakePod().Name("lbp").Namespace("unrelated").Label("unrelated", "unrelated").Node("node1").Obj(), wantToRequeue:      false, }, { name:               "Pod rejected by pod topology spread is requeued with Pod's update in the same namespace", unschedPod:         st.MakePod().Name("tsp").Namespace("ns1").UID("tsp").SpreadConstraint(1, "node", v1.DoNotSchedule, nil, nil, nil, nil, nil).Obj(), unschedPlugin:      names.PodTopologySpread, updatedAssignedPod: st.MakePod().Name("lbp").Namespace("ns1").Label("service", "securityscan").Node("node1").Obj(), wantToRequeue:      true, }, { name:               "Pod rejected by pod topology spread isn't requeued with unrelated Pod's update", unschedPod:         st.MakePod().Name("afp").Namespace("ns1").UID("afp").Annotation("annot2", "val2").PodAffinityExists("service", "region", st.PodAffinityWithRequiredReq).Obj(), unschedPlugin:      names.PodTopologySpread, updatedAssignedPod: st.MakePod().Name("lbp").Namespace("unrelated").Label("unrelated", "unrelated").Node("node1").Obj(), wantToRequeue:      false, }, { name:               "Pod rejected by other plugins isn't requeued with any Pod's update", unschedPod:         st.MakePod().Name("afp").Namespace("ns1").UID("afp").Annotation("annot2", "val2").Obj(), unschedPlugin:      "fakePlugin", updatedAssignedPod: st.MakePod().Name("lbp").Namespace("unrelated").Label("unrelated", "unrelated").Node("node1").Obj(), wantToRequeue:      false, </ins> }, <del> } q := NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithQueueingHintMapPerProfile(m)) // To simulate the pod is failed in scheduling in the real world, Pop() the pod from activeQ before AddUnschedulableIfNotPresent()s below. q.activeQ.Add(q.newQueuedPodInfo(unschedulablePodInfo.Pod)) if p, err := q.Pop(logger); err != nil || p.Pod != unschedulablePodInfo.Pod { t.Errorf("Expected: %v after Pop, but got: %v", unschedulablePodInfo.Pod.Name, p.Pod.Name) } q.activeQ.Add(q.newQueuedPodInfo(affinityPod)) if p, err := q.Pop(logger); err != nil || p.Pod != affinityPod { t.Errorf("Expected: %v after Pop, but got: %v", affinityPod.Name, p.Pod.Name) } q.Add(logger, medPriorityPodInfo.Pod) err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fakePlugin"), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(affinityPod, "fakePlugin"), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) </del> } <del> // Move clock to make the unschedulable pods complete backoff. c.Step(DefaultPodInitialBackoffDuration + time.Second) // Simulate addition of an assigned pod. The pod has matching labels for // affinityPod. So, affinityPod should go to activeQ. q.AssignedPodAdded(logger, labelPod) if getUnschedulablePod(q, affinityPod) != nil { t.Error("affinityPod is still in the unschedulablePods.") } if _, exists, _ := q.activeQ.Get(newQueuedPodInfoForLookup(affinityPod)); !exists { t.Error("affinityPod is not moved to activeQ.") } // Check that the other pod is still in the unschedulablePods. if getUnschedulablePod(q, unschedulablePodInfo.Pod) == nil { t.Error("unschedulablePodInfo is not in the unschedulablePods.") </del> <ins> for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { logger, ctx := ktesting.NewTestContext(t) ctx, cancel := context.WithCancel(ctx) defer cancel() c := testingclock.NewFakeClock(time.Now()) m := makeEmptyQueueingHintMapPerProfile() m[""][AssignedPodAdd] = []*QueueingHintFunction{ { PluginName:     "fakePlugin", QueueingHintFn: queueHintReturnQueue, }, { PluginName:     names.InterPodAffinity, QueueingHintFn: queueHintReturnQueue, }, { PluginName:     names.PodTopologySpread, QueueingHintFn: queueHintReturnQueue, }, } q := NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithQueueingHintMapPerProfile(m)) // To simulate the pod is failed in scheduling in the real world, Pop() the pod from activeQ before AddUnschedulableIfNotPresent()s below. if err := q.activeQ.Add(q.newQueuedPodInfo(tt.unschedPod)); err != nil { t.Errorf("failed to add pod to activeQ: %v", err) } if p, err := q.Pop(logger); err != nil || p.Pod != tt.unschedPod { t.Errorf("Expected: %v after Pop, but got: %v", tt.unschedPod.Name, p.Pod.Name) } err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(tt.unschedPod, tt.unschedPlugin), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } // Move clock to make the unschedulable pods complete backoff. c.Step(DefaultPodInitialBackoffDuration + time.Second) q.AssignedPodAdded(logger, tt.updatedAssignedPod) if _, exists, _ := q.activeQ.Get(newQueuedPodInfoForLookup(tt.unschedPod)); exists != tt.wantToRequeue { t.Fatalf("unexpected Pod move: Pod should be requeued: %v. Pod is actually requeued: %v", tt.wantToRequeue, exists) } }) </ins> } }

/sig scheduling /kind cleanup Ref: A test helper function creates a goroutine to pop a Pod from the scheduling queue. This goroutine never stops until the scheduling queue is ed or is somehow stopped for waiting for a new Pod. Ideally, when reaching timeout, we should somehow stop from waiting for Pod to be enqueued to activeQ.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can replace this function with and it should be enough? I think we no longer need to maintain this function.
Not enough, the core of this problem is that we cannot cancel once we start . Even if we change function to whatever, we will hit this problem. So, we need:
Or, alternatively and much more easily, maybe we can just replace all with and remove . kills the entire test if reaching timeout and we could care less about leaked goroutine then.
I don't think we can do this. From the test case: The expected behavior of this case is that we receive a timeout to make sure no pods are to . So, reaching timeout is acutally a success. So, I think: option 1: using sth like this in : option2: do sth in the test code This part is where it confused me. The current code is fine as we closed schdulerQueue after called , and after I think there is no test case using this this function anymore(seeems that you refactored the test), so are we trying to solve a potential problem?
Oh, then we can just remove NextPod in the PR.
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

/sig scheduling /kind bug AssignedPodAdded(): AssignedPodUpdated(): When assigned Pods are created/updated, cluster events (Pod/Add or Pod/Update) should be delivered for all plugins that subscribe those events. But, actually pre-filters Pods to receive those events based on Pods' affinity and . - meaning only PodAffinity/NodeResourceFit receive them actually. The problem here is similar to - in in-tree plugins, PodAffinity, PodTopologySpread and NodeResourceFit subscribe Pod add and/or update events. And, Pods rejected by PodTopologySpread will never be requeued to activeQ by Pod-related events. The same happens for custom plugins registering pod add and/or update events. All Pods will get chances to get back to activeQ by Pod-related events. Run ↓ in No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_IPBlock = map[string]string{ <del> "":       "IPBlock describes a particular CIDR (Ex. "192.168.1.1/24") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.", "cidr":   "CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24"", "except": "Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" Except values will be rejected if they are outside the CIDR range", </del> <ins> "":       "IPBlock describes a particular CIDR (Ex. "192.168.1.1/24","2001:db9::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.", "cidr":   "CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"", "except": "Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range", </ins> } func (IPBlock) SwaggerDoc() map[string]string {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : mentions an IPBlock with only IPv4 examples. Add a comment or example to clarify that IPv6 network blocks are (or possibly aren't) valid here. Why is this needed: Cluster operators who use IPv6 networking want to understand how to isolate different workloads inside their cluster network (and without reading source code). Logged against kubernetes/kubernetes rather than the website repo because this is generated documentation. Kubernetes already has single-stack support for IPv6 clusters KEP for dual-stack support is /sig docs /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved can you guide me on what steps are needed to publish the examples?
I via the pull request. (Although I want these examples , it's not a task I'm familiar with).


Question: externalIPs []string } <ins> func (info *serviceInfo) setAlive(b bool) { var i int32 if b { i = 1 } atomic.StoreInt32(&info.isAliveAtomic, i) } func (info *serviceInfo) isAlive() bool { return atomic.LoadInt32(&info.isAliveAtomic) != 0 } </ins> func logTimeout(err error) bool { if e, ok := err.(net.Error); ok { if e.Timeout() {

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: } ` <ins> func Test4xxStatusCodeInvalidPatch(t *testing.T) { _, client, closeFn := setup(t) defer closeFn() obj := []byte(`{ "apiVersion": "apps/v1", "kind": "Deployment", "metadata": { "name": "deployment", "labels": {"app": "nginx"} }, "spec": { "selector": { "matchLabels": { "app": "nginx" } }, "template": { "metadata": { "labels": { "app": "nginx" } }, "spec": { "containers": [{ "name":  "nginx", "image": "nginx:latest" }] } } } }`) resp, err := client.CoreV1().RESTClient().Post(). AbsPath("/apis/apps/v1"). Namespace("default"). Resource("deployments"). Body(obj).Do().Get() if err != nil { t.Fatalf("Failed to create object: %v: %v", err, resp) } result := client.CoreV1().RESTClient().Patch(types.MergePatchType). AbsPath("/apis/apps/v1"). Namespace("default"). Resource("deployments"). Name("deployment"). Body([]byte(`{"metadata":{"annotations":{"foo":["bar"]}}}`)).Do() var statusCode int result.StatusCode(&statusCode) if statusCode != 422 { t.Fatalf("Expected status code to be 422, got %v (%#v)", statusCode, result) } result = client.CoreV1().RESTClient().Patch(types.StrategicMergePatchType). AbsPath("/apis/apps/v1"). Namespace("default"). Resource("deployments"). Name("deployment"). Body([]byte(`{"metadata":{"annotations":{"foo":["bar"]}}}`)).Do() result.StatusCode(&statusCode) if statusCode != 422 { t.Fatalf("Expected status code to be 422, got %v (%#v)", statusCode, result) } } </ins> // Tests that the apiserver returns 202 status code as expected. func Test202StatusCode(t *testing.T) { s, clientSet, closeFn := setup(t)

/kind bug /sig api-machinery /cc What happened: patching (SMP) any object's annotation using invalid patch content will result in an (500) internal-server error. expecting a (400) bad-request error in this case. patch body: received error: the error is thrown from converter instead of the patcher, so it seems that patcher considers the body as valid then the converter complained. Anything else we need to know?: when my grandpa was 65 years old he started running one mile a day to keep fit. he's 70 now and we have no idea where he is.
I just tried the same patch with "MergePatchType", and it also fails with 500.
I can't make it fail with JSONPatchType, I get a 422
I suspect the error should read "the server rejected your request due to an error in our request"


Question: "k8s.io/klog" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/apimachinery/pkg/util/wait"

What happened: Network services with heavy load will cause "connection reset" from time to time. Especially those with big payloads. When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. The packet goes back the the original pod, who doesn't recognize the packet because of the wrong source ip, end up RSTing the connection. What you expected to happen: connection not reset How to reproduce it (as minimally and precisely as possible): wrote this just to reproduce the issue: Basically run it in a cluster, and after a while you will be able to see "Connection reset by peer" on both server and client pods. Anything else we need to know?: Related issue: Environment: Kubernetes version (use ): 1.11.7 Cloud provider or hardware configuration: gke (should be general to all provider) OS (e.g: ): PRETTYNAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSIONID="9" VERSION="9 (stretch)" ID=debian HOMEURL="" SUPPORTURL="" BUGREPORTURL="" Kernel (e.g. ): Linux gke-cluster-1-11-default-pool--6nzv 4.14.89+ SMP Wed Jan 9 13:35:00 PST 2019 x86_64 GNU/Linux
/sig network
I created a small app to reproduce this issue:
I've been hitting this exact problem for packets between a Pod and the outside world. So the DROP rule has to go into the INPUT chain. Would that be a job for a CNI driver or the kube-proxy?
was put to iptables mode but not to , right? Isn't ipvs mode affected by this issue, too?
Can anyone point me out how netfilter ignore this STATE invalid packet? I checked the code in netfilter for function iptablenatdo_chain, but didn't find the code that handles the STATE=invalid packet. Thanks for help.
Invalid is connection tracking state. My understanding is: If for some reason conntrack doesn't think it's the packet is "tracked" it could be grouped into "invalid" state. The problem is once it happens, .
ipvs should be fine. do you see similar issue in ipvs?
I also encountered this problem.

Have you solved it
is this a similar issue for ipvs?
The error is a very general networking issue, which could be caused by many different reasons. This specific cause can be reproduced by the It looks like you got an answer already there :)


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package windows import ( "regexp" "strings" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/test/e2e/framework" e2elog "k8s.io/kubernetes/test/e2e/framework/log" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" ) const dnsTestPodHostName = "dns-querier-1" const dnsTestServiceName = "dns-test-service" var _ = SIGDescribe("DNS", func() { ginkgo.BeforeEach(func() { framework.SkipUnlessNodeOSDistroIs("windows") }) f := framework.NewDefaultFramework("dns") ginkgo.It("should support configurable pod DNS servers", func() { ginkgo.By("Preparing a test DNS service with injected DNS names...") testInjectedIP := "1.1.1.1" testSearchPath := "resolv.conf.local" ginkgo.By("Creating a pod with dnsPolicy=None and customized dnsConfig...") testUtilsPod := generateDNSUtilsPod() testUtilsPod.Spec.DNSPolicy = v1.DNSNone testUtilsPod.Spec.DNSConfig = &v1.PodDNSConfig{ Nameservers: []string{testInjectedIP}, Searches:    []string{testSearchPath}, } testUtilsPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(testUtilsPod) framework.ExpectNoError(err) e2elog.Logf("Created pod %v", testUtilsPod) defer func() { e2elog.Logf("Deleting pod %s...", testUtilsPod.Name) if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(testUtilsPod.Name, metav1.NewDeleteOptions(0)); err != nil { e2elog.Failf("Failed to delete pod %s: %v", testUtilsPod.Name, err) } }() framework.ExpectNoError(f.WaitForPodRunning(testUtilsPod.Name), "failed to wait for pod %s to be running", testUtilsPod.Name) ginkgo.By("Verifying customized DNS option is configured on pod...") cmd := []string{"ipconfig", "/all"} stdout, _, err := f.ExecWithOptions(framework.ExecOptions{ Command:       cmd, Namespace:     f.Namespace.Name, PodName:       testUtilsPod.Name, ContainerName: "util", CaptureStdout: true, CaptureStderr: true, }) framework.ExpectNoError(err) e2elog.Logf("ipconfig /all:n%s", stdout) dnsRegex, err := regexp.Compile(`DNS Servers[s*.]*:(s*[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})+`) if dnsRegex.MatchString(stdout) { match := dnsRegex.FindString(stdout) if !strings.Contains(match, testInjectedIP) { e2elog.Failf("customized DNS options not found in ipconfig /all, got: %s", match) } } else { e2elog.Failf("cannot find DNS server info in ipconfig /all output: n%s", stdout) } // TODO: Add more test cases for other DNSPolicies. }) }) func generateDNSUtilsPod() *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind: "Pod", }, ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-dns-utils-", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:    "util", Image:   imageutils.GetE2EImage(imageutils.Dnsutils), Command: []string{"sleep", "10000"}, }, }, }, } } </ins>

We need to add Windows-specific DNS tests to cover the following fields: Test cases needed for : Default, ClusterFirst, None Test cases needed for Test cases needed for Test cases needed for These need to run on Windows containers because the domain resolution policy is different, so different results need to be checked. Windows assumes that any name with a in it is a FQDN, not a PQDN. There is an to cover simple PQDN resolution, but they don't cover the Pod DNS configurations above. /sig windows /milestone v1.14
FYI I'm working on this. I plan to start out by porting over to Windows.
I just moved it into the in progress column. THanks!
What test cases are needed for , , and ? My thoughts were that covers and . I'm adding test cases for Default and ClusterFirst. Does cover the test cases needed for and or do I need more? Or should I just be porting over the entire file?
For the first one - yes. The main differences will be checking the output of instead of , and maybe using instead of but the flow is the same Checking second one now
Agree on second one. I think that will cover it. It may make sense to simplify it a bit because the common methods used there are all based on running bash + dig. If it works with busybox, that may be feasible. The description at matches those test cases
If at all possible, could we use native Windows cmdlets such as instead of relying on Linux cmdlets that have been ported to Windows like ? Am not 100% confident if they are going to behave consistently.
yeah, I think what David's suggesting makes sense if the test cases are forked for Windows.
Opened PR for enabling custom DNS on Azure CNI:
/milestone clear This will need to merge after 1.14 as we're waiting on CNI updates for these tests to pass. I this status note to the Windows kep
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: framework.Failf("Service HealthCheck NodePort still present") } <del> endpointNodeMap, err := jig.GetEndpointNodes() </del> <ins> epNodes, err := jig.ListNodesWithEndpoint() </ins> framework.ExpectNoError(err) <del> noEndpointNodeMap := map[string][]string{} </del> <ins> // map from name of nodes with endpoint to internal ip // it is assumed that there is only a single node with the endpoint endpointNodeMap := make(map[string]string) // map from name of nodes without endpoint to internal ip noEndpointNodeMap := make(map[string]string) for _, node := range epNodes { ips := e2enode.GetAddresses(&node, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", node.Name) } endpointNodeMap[node.Name] = ips[0] } </ins> for _, n := range nodes.Items { <del> if _, ok := endpointNodeMap[n.Name]; ok { continue </del> <ins> ips := e2enode.GetAddresses(&n, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", n.Name) } if _, ok := endpointNodeMap[n.Name]; !ok { noEndpointNodeMap[n.Name] = ips[0] </ins> } <del> noEndpointNodeMap[n.Name] = e2enode.GetAddresses(&n, v1.NodeExternalIP) </del> } <ins> framework.ExpectNotEqual(len(endpointNodeMap), 0) framework.ExpectNotEqual(len(noEndpointNodeMap), 0) </ins> svcTCPPort := int(svc.Spec.Ports[0].Port) svcNodePort := int(svc.Spec.Ports[0].NodePort) ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) path := "/clientip" <ins> dialCmd := "clientip" </ins> <del> ginkgo.By(fmt.Sprintf("endpoints present on nodes %v, absent on nodes %v", endpointNodeMap, noEndpointNodeMap)) for nodeName, nodeIPs := range noEndpointNodeMap { ginkgo.By(fmt.Sprintf("Checking %v (%v:%v%v) proxies to endpoints on another node", nodeName, nodeIPs[0], svcNodePort, path)) GetHTTPContent(nodeIPs[0], svcNodePort, e2eservice.KubeProxyLagTimeout, path) } </del> <ins> config := e2enetwork.NewNetworkingTestConfig(f, false, false) </ins> <del> for nodeName, nodeIPs := range endpointNodeMap { ginkgo.By(fmt.Sprintf("checking kube-proxy health check fails on node with endpoint (%s), public IP %s", nodeName, nodeIPs[0])) var body bytes.Buffer pollfn := func() (bool, error) { result := e2enetwork.PokeHTTP(nodeIPs[0], healthCheckNodePort, "/healthz", nil) if result.Code == 0 { </del> <ins> ginkgo.By(fmt.Sprintf("endpoints present on nodes %v, absent on nodes %v", endpointNodeMap, noEndpointNodeMap)) for nodeName, nodeIP := range noEndpointNodeMap { ginkgo.By(fmt.Sprintf("Checking %v (%v:%v/%v) proxies to endpoints on another node", nodeName, nodeIP[0], svcNodePort, dialCmd)) _, err := GetHTTPContentFromTestContainer(config, nodeIP, svcNodePort, e2eservice.KubeProxyLagTimeout, dialCmd) framework.ExpectNoError(err, "Could not reach HTTP service through %v:%v/%v after %v", nodeIP, svcNodePort, dialCmd, e2eservice.KubeProxyLagTimeout) } for nodeName, nodeIP := range endpointNodeMap { ginkgo.By(fmt.Sprintf("checking kube-proxy health check fails on node with endpoint (%s), public IP %s", nodeName, nodeIP)) var body string pollFn := func() (bool, error) { // we expect connection failure here, but not other errors resp, err := config.GetResponseFromTestContainer( "http", "healthz", nodeIP, healthCheckNodePort) if err != nil { return false, nil } if len(resp.Errors) > 0 { </ins> return true, nil } <del> body.Reset() body.Write(result.Body) </del> <ins> if len(resp.Responses) > 0 { body = resp.Responses[0] } </ins> return false, nil } <del> if pollErr := wait.PollImmediate(framework.Poll, e2eservice.TestTimeout, pollfn); pollErr != nil { </del> <ins> if pollErr := wait.PollImmediate(framework.Poll, e2eservice.TestTimeout, pollFn); pollErr != nil { </ins> framework.Failf("Kube-proxy still exposing health check on node %v:%v, after ESIPP was turned off. body %s", <del> nodeName, healthCheckNodePort, body.String()) </del> <ins> nodeName, healthCheckNodePort, body) </ins> } }

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: newStatus := calculateStatus(rc, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. <del> if err := updateReplicationControllerStatus(rm.kubeClient.Core().ReplicationControllers(rc.Namespace), *rc, newStatus); err != nil { </del> <ins> updatedRC, err := updateReplicationControllerStatus(rm.kubeClient.Core().ReplicationControllers(rc.Namespace), *rc, newStatus) if err != nil { </ins> // Multiple things could lead to this update failing.  Returning an error causes a requeue without forcing a hotloop return err } <ins> // Resync the ReplicationController after MinReadySeconds as a last line of defense to guard against clock-skew. if manageReplicasErr == nil && updatedRC.Spec.MinReadySeconds > 0 && updatedRC.Status.ReadyReplicas == *(updatedRC.Spec.Replicas) && updatedRC.Status.AvailableReplicas != *(updatedRC.Spec.Replicas) { rm.enqueueControllerAfter(updatedRC, time.Duration(updatedRC.Spec.MinReadySeconds)*time.Second) } </ins> return manageReplicasErr }

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: ResourceVersion string `json:"resourceVersion,omitempty"` // Optional. If referring to a piece of an object instead of an entire object, this string <del> // should contain a valid field access statement. For example, // if the object reference is to a container within a pod, this would take on a value like: // "desiredState.manifest.containers[2]". Such statements are valid language constructs in // both go and JavaScript. This is syntax is chosen only to have some well-defined way of </del> <ins> // should contain information to identify the sub-object. For example, if the object // reference is to a container within a pod, this would take on a value like: // "spec.containers{name}" (where "name" refers to the name of the container that triggered // the event) or if no container name is specified "spec.containers[2]" (container with // index 2 in this pod). This syntax is chosen only to have some well-defined way of </ins> // referencing a part of an object. // TODO: this design is not final and this field is subject to change in the future. FieldPath string `json:"fieldPath,omitempty"`

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004