Question: }, Spec: api.PodSpec{ RestartPolicy: rPol, <ins> Volumes: []api.Volume{ { Name: "data", VolumeSource: api.VolumeSource{ EmptyDir: &api.EmptyDirVolumeSource{}, }, }, }, </ins> Containers: []api.Container{ { Name:    "c", Image:   "gcr.io/google_containers/busybox", Command: []string{}, <ins> VolumeMounts: []api.VolumeMount{ { MountPath: "/data", Name:      "data", }, }, </ins> }, }, },

Job should run a job to completion when tasks sometimes fail and are locally restarted Expected error: <*errors.errorString | 0xc20807c990: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred
Couple of failures: from
Another failure:
Another failure:
This test was introduced by and in and . It's currently failing at a rate of about 7%. When it passes, it typically does so in 30-90 seconds. Occasionally it times out after 15 minutes. I think that there's a real bug in the system here. The timeout occurs while waiting for the Job to terminate, which it never does.
I'm going to label it as flaky until the bug has been fixed.
Another portion of flakes:
/sub
any progress on this? Is it being actively worked on?
I've been working on this today. Saw 1 e2e failure in 50 runs. Logs suggest the problem is that half of kubelet's brain thinks it started a pod, and the other half of it thinks it didn't. In fact it didn't. So the test times out waiting for the not-started pod to complete.
Update: the kubelet did start all the pods, and they all terminated, and then the job controller somehow didn't notice that they were done until 40+ minutes later.
This is interesting. There were 3 apiserver 500's around the time of the test failure, and no 500's any other time in the 8 hours the cluster was up, and many other test runs occurred. 500s:
The code that is failing is this in at line 271 the gets an error, and then decides the error is a 500.
can you say anything about the above mentioned apiserver internal error which occurs immediately on receiving a watch request?
Look in the relevant kubelet logs to see how they interpreted the error. It's a bit silly that we printed a stack trace but not the error text :( , Eric Tune wrote:
E1022 04:13:24. 3622 ] Failed to watch *api.Pod: Internal error occurred: too old resource version: 171 ()
This is happening at line 269 of Is this an unexpected condition? If not, why is it a 500?
Oh, I thought that was fixed.

It shouldn't be an internal error, you're out of the watch window and need to re-list. , Eric Tune wrote:
OK I reopened . did the kubelets involved re-list soon after getting these 500s? If so, then this is coincidence and not your problem. , Daniel Smith wrote:
In all 3 cases, they relist immediately, but then they don't re-watch for about an hour. That seems wrong, no? Relist at 3:56 and rewatch at 4:53 Relist at 4:07 and rewatch at 5:00 relist at 4:13 and rewatch at 4:55
The time reported is the end of the watch, not the beginning. The last one it seems you actually didn't get the log line for the watch, actually. But the first two look WAI. , Eric Tune wrote:
Oh, duh.
Working on which I suspect may fix this.
Okay, looking at a new test failure I see that there are expected failures, and the test just times out due to backoff by the kubelet. The following things happen: runs and fails (1) runs and fails (2) backs off for 10s runs and fails (3rd failure) backoff 20s runs and fails (4th failure) backoff 40s runs and fails (5th failure) backoff 1m20s runs and fails (6th failure) backoff 2m40s runs and fails (6th failure) kubelet backoff 5m runs and fails (6th failure) backoff 5m timeout now that at least 15m10s has elapsed in backoffs. Failures have a probability of 50%. Chance of getting 8 fails in a row is 1 in 256. Four pods per test run is 4 in 256 or 1 in 64. I see failures about 1 in 50, so that seems about right. So, I need to rewrite the test so that it writes to the emptydir after the first failure, and succeeds after that.
Ugh. My fix for local breaks non-local restarts.
Happened again.


Question: sudo bash -c "echo "${KUBE_PROXY_TOKEN},system:kube-proxy,uid:kube_proxy" >> /etc/srv/kubernetes/known_tokens.csv" &&  sudo bash -c "echo ${password},admin,admin > /etc/srv/kubernetes/basic_auth.csv"" <del> gcloud compute copy-files --zone="${ZONE}" --project="${PROJECT}"  </del> <ins> run-gcloud-compute-with-retries copy-files --zone="${ZONE}" --project="${PROJECT}"  </ins> "${SERVER_BINARY_TAR}"  "${RESOURCE_DIRECTORY}/kubemark-master-env.sh"  "${RESOURCE_DIRECTORY}/start-kubemark-master.sh"

https://k8s- Failed: Kubemark {} Previous issues for this test:
Builds: ci-kubernetes-kubemark-5-gce Failed: Kubemark {}
There are few things here: 3443 run - can you please invstigate it? 3480 run - gcloud copy-files should be retried 3550 run - we should better handle "already exists" errors (and not retry them). I think was looking into it at some point, so he may have already did something
https://k8s- Failed: Kubemark {}
I was looking at 'already exists' problem in RouteController, not in Kubemark.
Builds: ci-kubernetes-kubemark-5-gce Failed: Kubemark {}
Here are the causes of the above tests to fail and fixes: Test Fix ------ ------ ci-kubernetes-kubemark-5-gce/3443 Ignore (as its super flaky) or change create-certs/generate-certs to use a different directory ci-kubernetes-kubemark-5-gce/3480 Run the copy-files with retries as it is idempotent ci-kubernetes-kubemark-5-gce/3550 Don't retry when resource already exists ci-kubernetes-kubemark-5-gce/3666 Not worry as it is flaky or add retrying logic to all gclouds in that script ci-kubernetes-kubemark-5-gce/3787 Run the ssh with retries as it is idempotent
This PR fixes issues with the 2nd, 3rd and 5th tests in the above table. Could you PTAL at the PR? Also, I think we can safely ignore the 1st test (which is super flaky) and what do you suggest about the 4th test (do we add retrying logic to just call inside detect-master function of or let it stay as it is because there are a lot of other such gcloud calls inside the file, all of which should be changed rightfully)?
Let's not fix everything - fix just the things that you need, and direct vicinity.
Ok. So then let's ignore the 1st and the 4th tests in the above table as they are not really important. 2nd, 3rd and 5th are solved by PR . And regarding failure of , it looks like it is a flaky issue as well.

The density test succeeded, but the load test failed. From the logs it appears to be an issue with the client caches returning old resource versions for pod watches, with many such log statements: Does this worry you or you'd deem it to be just some weird flake?
It might mean a number of things, but most likely something gets overloaded/congested. It might be hard to debug, as we're seeing this from time to time, but it doesn't look very serious so I think we never investigated it thoroughly.


Question: fs.DurationVar(&c.StreamingConnectionIdleTimeout.Duration, "streaming-connection-idle-timeout", c.StreamingConnectionIdleTimeout.Duration, "Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: '5m'") fs.DurationVar(&c.NodeStatusUpdateFrequency.Duration, "node-status-update-frequency", c.NodeStatusUpdateFrequency.Duration, "Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller.") fs.DurationVar(&c.ImageMinimumGCAge.Duration, "minimum-image-ttl-duration", c.ImageMinimumGCAge.Duration, "Minimum age for an unused image before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'.") <del> fs.Int32Var(&c.ImageGCHighThresholdPercent, "image-gc-high-threshold", c.ImageGCHighThresholdPercent, "The percent of disk usage after which image garbage collection is always run.") fs.Int32Var(&c.ImageGCLowThresholdPercent, "image-gc-low-threshold", c.ImageGCLowThresholdPercent, "The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to.") </del> <ins> fs.Int32Var(&c.ImageGCHighThresholdPercent, "image-gc-high-threshold", c.ImageGCHighThresholdPercent, "The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. ") fs.Int32Var(&c.ImageGCLowThresholdPercent, "image-gc-low-threshold", c.ImageGCLowThresholdPercent, "The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold.") </ins> fs.DurationVar(&c.VolumeStatsAggPeriod.Duration, "volume-stats-agg-period", c.VolumeStatsAggPeriod.Duration, "Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0.") fs.Var(flag.NewMapStringBool(&c.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/experimental features. "+ "Options are:n"+strings.Join(utilfeature.DefaultFeatureGate.KnownFeatures(), "n"))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The kubelet will GC the images on the node to free space when the used space reaching the percent ot the high-threhold(90% as default), but when the containers are not just managed by kubernetes(such as managed by swarm together) and not wished to be GCed by kubelet, I found we can not close the GC function by the flag of image-gc-high-threshold. What you expected to happen: add one switch to close the function of imageGC in specific senario: when the flag image-gc-high-threshold is set as 0, we should not GC the images on the node, just record one warning event to kube-apiserver Or add another switch flag to close the whole function of imageGC How to reproduce it (as minimally and precisely as possible): run kubernetes with several nodes, and test with making the inode all consumed of the nodes then the kubelet can triggle the GC to delete the oldest images on the node regardless the kind of them. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind feature
/sig node
What happens when --image-gc-low-threshold and --image-gc-high-threshold are both set to 100 ?
when both set to 100, image GC will still delete one image to release space as it always runs into the logic and delete image in logic of func
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" <ins> imageutils "k8s.io/kubernetes/test/utils/image" </ins> ) const (

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: ) var ( <del> serverVersion = flag.Bool("server_version", false, "Print the server's version number.") </del> <ins> serverVersion = verflag.Version("server_version", verflag.VersionFalse, "Print the server's version information and quit") </ins> preventSkew   = flag.Bool("expect_version_match", false, "Fail if server's version doesn't match own version.") httpServer    = flag.String("h", "", "The host to connect to.") config        = flag.String("c", "", "Path to the config file.")

It's hard to tell if they match right now.
who has been changing this stuff. :)
Ah yes... I was thinking of adding a flag to print the %r of the version.Info... Do you think that would be an appropriate flag name? I'll start working on the PR since it's probably time to refactor the code to take the other bits from git I have been pushing through ldflags...
I'd change the current behavior rather than add a new flag. I don't really care which way it ends up looking as long as both flags print the same format :)
Fixed by


Question: } waitForQuota(t, quota, clientset) <del> if _, err := clientset.Core().Pods(ns.Name).Create(pod); err != nil { </del> <ins> // attempt to create a new pod once the quota is propagated err = wait.PollImmediate(5*time.Second, time.Minute, func() (bool, error) { // retry until we succeed (to allow time for all changes to propagate) if _, err := clientset.Core().Pods(ns.Name).Create(pod); err == nil { return true, nil } return false, nil }) if err != nil { </ins> t.Fatalf("unexpected error: %v", err) } }

the timing is suspicious. Your pull merged, right?
it did, i will take a look.


Question: "github.com/GoogleCloudPlatform/k8s-cloud-provider/pkg/cloud" compute "google.golang.org/api/compute/v1" <ins> option "google.golang.org/api/option" </ins> "k8s.io/client-go/tools/cache" )

What happened: Unit test failed by command : What you expected to happen: How to reproduce it (as minimally and precisely as possible): Just run command . Anything else we need to know?: Happened on . I first met this issue at v1.15. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/remove-kind bug /kind failing-test /sig cloud-provider
/area provider/gcp
I would be curious to know how is passing then
The reason is that is required here: I guess all CI machinery has a . This will fail in all environments without a .
ah, fixed in
Great!


Question: } proxier = proxierUserspace serviceEventHandler = proxierUserspace <del> klog.V(0).Info("Tearing down pure-winkernel proxy rules.") winkernel.CleanupLeftovers() </del> } return &ProxyServer{

What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

What happened: When restarting kube-proxy in iptables mode, there will be several seconds of timeouts because it flushes the KUBE-SERVICES nat chain and several others. What you expected to happen: Restarting kube-proxy should not impact traffic in any way. How to reproduce it (as minimally and precisely as possible): an HTTP server behind a ClusterIP service an http client in a loop making requests to the HTTP server -15 the kube-proxy process on the client node timeout for a few seconds until kube-proxy re-syncs the iptables rules Anything else we need to know?: Running with makes it pretty clear what's happening: it says it's cleaning up inactive rules, but these are crucial to the iptables proxier. You can trivially reproduce by just running: Environment: Kubernetes version (use ): 1.10.11 Cloud provider or hardware configuration: amazon OS (e.g: ): custom Kernel (e.g. ): 4.14.77-hs623.el6.x86_64 Install tools: custom Others: iptables 1.6.2 We're running kube-proxy 1.10.11, but I've confirmed this issue with kube-proxy 1.13.4 too
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Possibly a bug in the proxy. It flushes iptabes if the proxier is able to use IPVS which seems like the wrong behavior to me if there are shared tables/chains between the IPVS and iptables proxier. One workaround I can think of right now is to unload the ipvs kernel module to skip this check.
/assign
oh, that's an interesting workaround, I'll give that a shot, thanks!
np, I'll try to work on the bug fix in the meantime!
/triage unresolved
note that unloading the ipvs module is not actually enough of a workaround because kube-proxy will load the kernel module if it sees that it is available. If I unload the ipvs module and then rename to something else in then it does the right thing. Dug slightly further: the way it probes for modules is by actually running modprobe.

So the simplest hack that allows us to keep ipvs loaded for other things on the system is to put a modprobe script on its PATH that just always exits 1 for the ip_vs modules. (We can stop mounting the modules dir into the kube-proxy container in our kubernetes clusters, but we also run kube-proxy on non-kubernetes nodes via the init system, so the PATH hack works well enough there). This should hold us over fine until we get to a version with your fix in it.
Good to know, thanks for sharing!
/assign
Quick update on this issue from today's SIG Network call: we're going to try to get rid of the automatic proxy clean up altogether for v1.14.1 since this is considered a bug. is working on the KEP & implementation.


Question: p.ResourceVersion = "" p.Generation = 0 p.Status = v1.PodStatus{} <ins> p.ManagedFields = nil p.Finalizers = nil </ins> return p } return !reflect.DeepEqual(strip(oldPod), strip(newPod))

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: if [[ -z "${KUBE_MASTER_IP-}" ]]; then local master_address_name="${MASTER_NAME}-ip" echo "Looking for address '${master_address_name}'" >&2 <del> KUBE_MASTER_IP=$(gcloud compute addresses describe "${master_address_name}"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') fi if [[ -z "${KUBE_MASTER_IP-}" ]]; then echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" >&2 exit 1 </del> <ins> if ! KUBE_MASTER_IP=$(gcloud compute addresses describe "${master_address_name}"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') ||  [[ -z "${KUBE_MASTER_IP-}" ]]; then echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" >&2 exit 1 fi </ins> fi echo "Using master: $KUBE_MASTER (external IP: $KUBE_MASTER_IP)" >&2 }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: Attempting to run e2e test on GCE without cluster started fails with stack trace What you expected to happen: A useful error message to be printed How to reproduce it (as minimally and precisely as possible): Without a cluster started: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing /assign
This happens because gcloud returns an error when a resource isn't found and ginkgo- sets errexit


Question: // Interface is an abstract interface for testability.  It abstracts the interface to cAdvisor. type Interface interface { <ins> Start() error </ins> DockerContainer(name string, req *cadvisorApi.ContainerInfoRequest) (cadvisorApi.ContainerInfo, error) ContainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (*cadvisorApi.ContainerInfo, error) SubcontainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (map[string]*cadvisorApi.ContainerInfo, error)

This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?

I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.


Question: securityRuleIDTemplate      = "/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Network/networkSecurityGroups/%s/securityRules/%s" ) <ins> var providerIDRE = regexp.MustCompile(`^` + CloudProviderName + `://(.+)$`) </ins> // returns the full identifier of a machine func (az *Cloud) getMachineID(machineName string) string { return fmt.Sprintf(

This is required to make Cloud Controller Manager operational on the various clouds that Kubernetes runs on. We have already PRs for the following clouds: GCE - AWS - Azure - OpenStack - The following clouds still need to be addresses: CloudStack Mesos OVirt Photon Rackspace vSphere If anyone is interested in contributing to this effort, please ping me directly (on kubernetes slack or through email)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle
cc:
I will take care of the vSphere and Photon.
I will take care of the Rackspace.
Merging for OpenStack
Reposting I am trying to use my custom out-of-tree cloud provider with Kubernetes. But I noticed that kubelet calls apis like with empty for cloud provider. Digging in the code for 1.7 branch, I see that is null for cloud-providers. As a result returns empty string. As a result, ByID() methods don't work for cloud providers since is not set.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // Assume that migration is always possible, rely on migrateCoreDNSCorefile() to fail if not. canMigrateCorefile := true <del> if corefileMigrationRequired { </del> <ins> if corefile == "" || migration.Default("", corefile) { // If the Corefile is empty or default, the latest default Corefile will be applied if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil { return err } } else if corefileMigrationRequired { // If migration is required, try and migrate the Corefile </ins> if err := migrateCoreDNSCorefile(client, coreDNSConfigMap, corefile, currentInstalledCoreDNSVersion); err != nil { // Errors in Corefile Migration is verified during preflight checks. This part will be executed when a user has chosen // to ignore preflight check errors.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: does not retaining coredns configmap using v1.19.4 makes no difference What you expected to happen: should follow documentation and retain coredns configmap How to reproduce it (as minimally and precisely as possible): upgrading from v1.18.6 to v1.19.3 Anything else we need to know?: works like a charm Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): CentOS 7 Kernel (e.g. ): 4.20.13- Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
seems to be a regression ?
/assign
Would you check you dns configmap for ? As , federation is removed from coredns. If you have federation enabled in your coredns configmap, the check may fail in
We don't have any parameter in our coredns configmap, or any kube-dns parameters in kubeadm config
Looking closely at the dry run messages above, this may not be a regression. The apparent reason it works in 1.18.12 is that support for CoreDNS 1.7.0 was not until 1.19. i.e. 1.18 sees an unsupported version of CoreDNS and therefore skips the corefile migration. Whereas in 1.19, CoreDNS 1.7.0 is supported, and the migration takes place. What does your complete Corefile look like before the migration?
If it's possible, can you also attach the entire dry run logs?
Sure.
Thanks.

I think the issue may be in the detection of the "default" configuration. That is, in the upgrade process, if it detects that you are using the "default" coredns corefile, it simply installs the new default over top of it, rather than migrating the existing one. I think since the default config stanza is in there in its original form, the whole Corefile is incorrectly being detected as a default. If i'm right, the fix could either go in the package (should at least eventually be fixed there), and/or kubeadm could just always migrate the existing config instead of deploying the default over top of the existing one.
I just tested against the package manually - and it correctly identifies the config as "non-default", so that's not it... And I also tested a migration on the Corefile, and the result included the extra stanzas as expected. So, the issue doesn't appear to lie in the migration package.
After testing with the code, I find the issue in , I fixed it in . A workaround for you is to create an empty kube-dns configmap. ———————————————————— edited new comments here I find still fix a problem , when kube-dns configmap is using federations.
Thanks, I'll try it
empty kube-dns configmap doesn't helped.
sorry for my mistake. According to my test, the create or update will trigger is no migrating. Even your corefile will meet some dep that need to be migrated. max_concurrent is new default. However, there is a typo in . Below is a workaround(not good one), add upstream to your corefile in kubernetes. ! It works for me.
if your current coredns is 1.6.7, upstream is deprecated but not removed. In new 1.7.0 coredns, upstream is removed.
thanks for the discussion and PRs this one is mergable for 1.21 as non critical: what do you think about this one and should we backport it as critical (also, to which k8s versions)?
/area kubeadm
It's not obvious to me how this relates to . The kubeadm dry run log appears to be replacing the existing config with the default. It should only be doing this if the existing config is semantically the same as a default Corefile, i.e.

if is true.
just explains that the migration is not triggered at all. if is not a fix for this issue, please remove this line from the PR description.
I think issue here is in/around , and how the result is interpreted... The code correctly assumes that if a Corefile is a default Corefile, then no migration is required. However later it incorrectly assumes the reciprocal: If migration is not required, then the corefile must be a default. Hence, it ends up replacing any Corefile that does not require migration with a default… even if it wasn't a default to begin with.
yes. The need be fixed as well, I will learn to fix it later if no one is assigned. I changed releaseNote to None.


Question: g := mounter.(*gitRepoVolumeMounter) g.exec = fake <del> g.SetUp(volume.MounterArgs{}) </del> <ins> err := g.SetUp(volume.MounterArgs{}) if err != nil { allErrs = append(allErrs, err) } </ins> if fake.CommandCalls != len(expecteds) { allErrs = append(allErrs,

Apply this patch: Then run: make test WHAT=./pkg/volume/git_repo The test fails: Test should pass. See above. The full loop is: The first problem is that fakeOutputs is capturing , so all the funcs on that list use the from the final iteration, not the one from the iteration with the [1] == "clone". But then this problem appears to be covering up a different problem in the test scenarios themselves. I looked at it for a little bit but I haven't puzzled out what the test is trying to do exactly. Hopefully someone with more knowledge of the test can take a look. It seems like a buggy test, accidentally passing due to this other loopvar sharing bug. HEAD N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon /cc Thanks for the report and the detailed explanation
/assign
I wonder if it could be beneficial to invest some time and effort to identify a maximum of loops that contain the loopvar sharing bug maybe with comment and a todo ?


Question: minion_index = n+1 minion_ip = $minion_ips[n] <del> minion.vm.provision "shell", run: "always", path: "#{ENV['KUBE_TEMP']}/minion-start-#{n}.sh" </del> <ins> if ENV['KUBE_TEMP'] then script = "#{ENV['KUBE_TEMP']}/minion-start-#{n}.sh" minion.vm.provision "shell", run: "always", path: script end </ins> minion.vm.network "private_network", ip: "#{minion_ip}" minion.vm.hostname = "#{ENV['INSTANCE_PREFIX']}-minion-#{minion_index}" end end end

The Vagrantfile requires that ENV['KUBE_TEMP'] is set. If it is not, any vagrant command will fail as the file cannot be parsed. The error received is: This is contrary to the suggested method of interaction in the documentation found at which says:
Looking at the scripts, this temp directory is set on script run and cleaned up on EXIT by a trap. I've wrapped a conditional around the provision statement which allows the other commands to work, but outputs a warning that the vm will not have a provisioner set. Pull req incoming.
Cc me on your PR and I will review. I have been swamped on other topics so it's possible something regressed.


Question: "k8s.io/apimachinery/pkg/runtime/schema" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/cli-runtime/pkg/genericclioptions" <ins> discovery "k8s.io/client-go/discovery" </ins> authorizationv1client "k8s.io/client-go/kubernetes/typed/authorization/v1" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" describeutil "k8s.io/kubernetes/pkg/kubectl/describe/versioned"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: // Dump debug information if the error was nil. if err == nil { // Collect pod logs when we see a failure. <del> logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, fmt.Sprintf("%s-container", podClient.Name)) </del> <ins> logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, "client") </ins> if logErr != nil { framework.Failf("Error getting container logs: %s", logErr) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In tests, always fails when the tests fail, with the error: . This is the message indicating that APIServer responds 400 code. What you expected to happen: Container logs should be collected correctly if tests fail. How to reproduce it (as minimally and precisely as possible): a NetworkPolicy test to make it fail and run it, or run NetworkPolicy tests with a CNI that doesn't implement NetworkPolicy. the tests fail, all attempts to get container logs will fail like the following. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: p.cpusToReuse[string(pod.UID)] = p.cpusToReuse[string(pod.UID)].Difference(cset) } <del> func (p *staticPolicy) Allocate(s state.State, pod *v1.Pod, container *v1.Container) error { if numCPUs := p.guaranteedCPUs(pod, container); numCPUs != 0 { klog.InfoS("Static policy: Allocate", "pod", klog.KObj(pod), "containerName", container.Name) // container belongs in an exclusively allocated pool if p.options.FullPhysicalCPUsOnly && ((numCPUs % p.topology.CPUsPerCore()) != 0) { // Since CPU Manager has been enabled requesting strict SMT alignment, it means a guaranteed pod can only be admitted // if the CPU requested is a multiple of the number of virtual cpus per physical cores. // In case CPU request is not a multiple of the number of virtual cpus per physical cores the Pod will be put // in Failed state, with SMTAlignmentError as reason. Since the allocation happens in terms of physical cores // and the scheduler is responsible for ensuring that the workload goes to a node that has enough CPUs, // the pod would be placed on a node where there are enough physical cores available to be allocated. // Just like the behaviour in case of static policy, takeByTopology will try to first allocate CPUs from the same socket // and only in case the request cannot be sattisfied on a single socket, CPU allocation is done for a workload to occupy all // CPUs on a physical core. Allocation of individual threads would never have to occur. return SMTAlignmentError{ RequestedCPUs: numCPUs, CpusPerCore:   p.topology.CPUsPerCore(), } } if cpuset, ok := s.GetCPUSet(string(pod.UID), container.Name); ok { p.updateCPUsToReuse(pod, container, cpuset) klog.InfoS("Static policy: container already present in state, skipping", "pod", klog.KObj(pod), "containerName", container.Name) return nil } // Call Topology Manager to get the aligned socket affinity across all hint providers. hint := p.affinity.GetAffinity(string(pod.UID), container.Name) klog.InfoS("Topology Affinity", "pod", klog.KObj(pod), "containerName", container.Name, "affinity", hint) </del> <ins> func (p *staticPolicy) Allocate(s state.State, pod *v1.Pod, container *v1.Container) (rerr error) { numCPUs := p.guaranteedCPUs(pod, container) if numCPUs == 0 { // container belongs in the shared pool (nothing to do; use default cpuset) return nil } </ins> <del> // Allocate CPUs according to the NUMA affinity contained in the hint. cpuset, err := p.allocateCPUs(s, numCPUs, hint.NUMANodeAffinity, p.cpusToReuse[string(pod.UID)]) if err != nil { klog.ErrorS(err, "Unable to allocate CPUs", "pod", klog.KObj(pod), "containerName", container.Name, "numCPUs", numCPUs) return err </del> <ins> klog.InfoS("Static policy: Allocate", "pod", klog.KObj(pod), "containerName", container.Name) // container belongs in an exclusively allocated pool metrics.CPUManagerPinningRequestsTotal.Inc() defer func() { if rerr != nil { metrics.CPUManagerPinningErrorsTotal.Inc() } }() if p.options.FullPhysicalCPUsOnly && ((numCPUs % p.topology.CPUsPerCore()) != 0) { // Since CPU Manager has been enabled requesting strict SMT alignment, it means a guaranteed pod can only be admitted // if the CPU requested is a multiple of the number of virtual cpus per physical cores. // In case CPU request is not a multiple of the number of virtual cpus per physical cores the Pod will be put // in Failed state, with SMTAlignmentError as reason. Since the allocation happens in terms of physical cores // and the scheduler is responsible for ensuring that the workload goes to a node that has enough CPUs, // the pod would be placed on a node where there are enough physical cores available to be allocated. // Just like the behaviour in case of static policy, takeByTopology will try to first allocate CPUs from the same socket // and only in case the request cannot be sattisfied on a single socket, CPU allocation is done for a workload to occupy all // CPUs on a physical core. Allocation of individual threads would never have to occur. return SMTAlignmentError{ RequestedCPUs: numCPUs, CpusPerCore:   p.topology.CPUsPerCore(), </ins> } <del> s.SetCPUSet(string(pod.UID), container.Name, cpuset) </del> <ins> } if cpuset, ok := s.GetCPUSet(string(pod.UID), container.Name); ok { </ins> p.updateCPUsToReuse(pod, container, cpuset) <ins> klog.InfoS("Static policy: container already present in state, skipping", "pod", klog.KObj(pod), "containerName", container.Name) return nil } // Call Topology Manager to get the aligned socket affinity across all hint providers. hint := p.affinity.GetAffinity(string(pod.UID), container.Name) klog.InfoS("Topology Affinity", "pod", klog.KObj(pod), "containerName", container.Name, "affinity", hint) // Allocate CPUs according to the NUMA affinity contained in the hint. cpuset, err := p.allocateCPUs(s, numCPUs, hint.NUMANodeAffinity, p.cpusToReuse[string(pod.UID)]) if err != nil { klog.ErrorS(err, "Unable to allocate CPUs", "pod", klog.KObj(pod), "containerName", container.Name, "numCPUs", numCPUs) return err </ins> } <del> // container belongs in the shared pool (nothing to do; use default cpuset) </del> <ins> s.SetCPUSet(string(pod.UID), container.Name, cpuset) p.updateCPUsToReuse(pod, container, cpuset) </ins> return nil }

Add to enable observability of the cpumanager. This is required to but it's also useful in general. To the best of my knowledge, it's very hard to observe the behavior of cpumanager without peeking into the pod and/or use the podresources interface (kubelet = 1.24). But even for the latter, no well known components expose metrics consuming this information. I believe anyway kubelet some expose some basic metrics about cpumanager behaviour, while deferring more fine grained metrics to external components (NFD, node_exporter, is fine)
/sig node /assign
/triage accepted /priority important-soon (as this is important for graduation for CPU Manager to GA)


Question: numVolumes++ allErrs = append(allErrs, validateAzureFile(source.AzureFile, fldPath.Child("azureFile"))...) } <ins> if source.VsphereVolume != nil { if numVolumes > 0 { allErrs = append(allErrs, field.Forbidden(fldPath.Child("vsphereVolume"), "may not specify more than 1 volume type")) } else { numVolumes++ allErrs = append(allErrs, validateVsphereVolumeSource(source.VsphereVolume, fldPath.Child("vsphereVolume"))...) } } </ins> if numVolumes == 0 { allErrs = append(allErrs, field.Required(fldPath, "must specify a volume type")) }

vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR


Question: serviceHealthServer := healthcheck.NewServiceHealthServer(hostname, recorder, nodePortAddresses, healthzServer) <del> var nftablesFamily knftables.Family if ipFamily == v1.IPv4Protocol { nftablesFamily = knftables.IPv4Family } else { nftablesFamily = knftables.IPv6Family } nft, err := knftables.New(nftablesFamily, kubeProxyTable) if err != nil { return nil, err } </del> proxier := &Proxier{ ipFamily:            ipFamily, svcPortMap:          make(proxy.ServicePortMap),

continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8.

If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?

build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?)

Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)

OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.


Question: // Each bind address below is a separate test case bindAddresses := []string{ "0.0.0.0", <del> "2001:db8::1", </del> <ins> "::", </ins> } for _, addr := range bindAddresses { options, err := NewOptions()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: MarkVolumeAsDetached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) // Marks desire to detach the specified volume (remove the volume from the node's <del> // volumesToReportedAsAttached list) </del> <ins> // volumesToReportAsAttached list) </ins> RemoveVolumeFromReportAsAttached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) error // Unmarks the desire to detach for the specified volume (add the volume back to <del> // the node's volumesToReportedAsAttached list) </del> <ins> // the node's volumesToReportAsAttached list) </ins> AddVolumeToReportAsAttached(volumeName v1.UniqueVolumeName, nodeName types.NodeName) }

kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that


Question: definitions MUST be published for custom resource definitions. release: v1.16 file: test/e2e/apimachinery/crd_publish_openapi.go <ins> - testname: Discovery, confirm the PreferredVersion for each api group codename: '[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]' description: Ensure that a list of apis is retrieved. Each api group found MUST return a valid PreferredVersion unless the group suffix is example.com. release: v1.19 file: test/e2e/apimachinery/discovery.go </ins> - testname: Event, delete a collection codename: '[sig-api-machinery] Events should delete a collection of events [Conformance]' description: A set of events is created with a label selector which MUST be found

] 20 July 2020 [x] Two weeks soak end date: 3 August 2020 According to this APIsnoop query, there are still an endpoint which is untested. Get a list of all group /apis Iterate through list Query each endpoint List the PreferredVersion for the endpoint Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 15 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/retitle Write checkAPIGroupPreferredVersion Test +Promote - +16 Endpoint coverage
/reopen Merged /assign Ready for creation of Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged /assign Ready for creation of Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Waits for specified PD to to detach from specified hostName func waitForPDDetach(diskName, hostName string) error { if framework.TestContext.Provider == "gce" || framework.TestContext.Provider == "gke" { <ins> framework.Logf("Waiting for GCE PD %q to detach from node %q.", diskName, hostName) </ins> gceCloud, err := getGCECloud() if err != nil { return err

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: {UpdateSetStatusFailure, simpleSetFn}, {PodRecreateDeleteFailure, simpleSetFn}, {NewRevisionDeletePodFailure, simpleSetFn}, <ins> {RecreatesPVCForPendingPod, simpleSetFn}, </ins> } for _, testCase := range testCases {

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: NODE_INSTANCE_PREFIX="${INSTANCE_PREFIX}-minion" NODE_TAGS="${NODE_TAG}" <ins> NODE_NETWORK="${NETWORK}" </ins> ALLOCATE_NODE_CIDRS=true PREEXISTING_NETWORK=false

The "Services should be able to create an internal type load balancer [Slow]" e2e test is failing on our 2k-node gce clusters - https://k8s- with the following error: We're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them. cc
Seems similar to Assigning to you as you seem to have the context already :) Also, is this expected to be fixed in 1.8 timeframe?
doesn't set the necessary variables for the GCE cloud provider. Will throw you a PR.
Cool - thanks a lot!
Ref
I see the last test on 09/09 failed, but I can't see the build log due to exceeding memory. Any other way I can see the log, /reopen
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
nevermind - went directly to raw log.
Friendly ping. This e2e is still failing (https://k8s-) and it's part of release-blocking suite.
/close


Question: <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/PR-SVG-20010719/DTD/svg10.dtd"> <del> <svg width="68cm" height="53cm" viewBox="-55 -75 1348 1045" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <g> <g> <rect style="fill: #ffffff" x="662" y="587.938" width="630" height="381"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="662" y="587.938" width="630" height="381"/> </g> <g> <rect style="fill: #ffffff" x="689" y="651.938" width="580" height="292"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="689" y="651.938" width="580" height="292"/> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="687" y="619.938"> <tspan x="687" y="619.938">Minion</tspan> </text> <g> <rect style="fill: #ffffff" x="721.2" y="698.938" width="69.6" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="721.2" y="698.938" width="69.6" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="756" y="721.838"> <tspan x="756" y="721.838">kubelet</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="821.676" y="698.938" width="76.65" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="821.676" y="698.938" width="76.65" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="860.001" y="721.838"> <tspan x="860.001" y="721.838">cAdvisor</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="711" y="675.938"> <tspan x="711" y="675.938">docker</tspan> </text> <g> <g> <rect style="fill: #ffffff" x="720.2" y="764.038" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="720.2" y="764.038" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="760.55" y="834.038" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="760.55" y="834.038" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="805.2" y="856.938"> <tspan x="805.2" y="856.938">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="749.8" y="824.138" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="749.8" y="824.138" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.45" y="847.038"> <tspan x="794.45" y="847.038">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="739.4" y="814.238" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="739.4" y="814.238" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="784.05" y="837.138"> <tspan x="784.05" y="837.138">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.2" y="830.538"> <tspan x="794.2" y="830.538"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="742.2" y="790.538"> <tspan x="742.2" y="790.538">Pod</tspan> </text> </g> <g> <g> <rect style="fill: #ffffff" x="1085.6" y="764.038" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1085.6" y="764.038" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="1125.95" y="834.038" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1125.95" y="834.038" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1170.6" y="856.938"> <tspan x="1170.6" y="856.938">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="1115.2" y="824.138" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1115.2" y="824.138" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1159.85" y="847.038"> <tspan x="1159.85" y="847.038">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="1104.8" y="814.238" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1104.8" y="814.238" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1149.45" y="837.138"> <tspan x="1149.45" y="837.138">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1159.6" y="830.538"> <tspan x="1159.6" y="830.538"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1107.6" y="790.538"> <tspan x="1107.6" y="790.538">Pod</tspan> </text> </g> <g> <g> <rect style="fill: #ffffff" x="902.9" y="764.038" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="902.9" y="764.038" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="943.25" y="834.038" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="943.25" y="834.038" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="987.9" y="856.938"> <tspan x="987.9" y="856.938">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="932.5" y="824.138" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="932.5" y="824.138" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="977.15" y="847.038"> <tspan x="977.15" y="847.038">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="922.1" y="814.238" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="922.1" y="814.238" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="966.75" y="837.138"> <tspan x="966.75" y="837.138">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="976.9" y="830.538"> <tspan x="976.9" y="830.538"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="924.9" y="790.538"> <tspan x="924.9" y="790.538">Pod</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="924.748" y="698.938" width="57.1" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="924.748" y="698.938" width="57.1" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="953.298" y="721.838"> <tspan x="953.298" y="721.838">Proxy</tspan> </text> </g> </g> </del> <ins> <svg width="68cm" height="56cm" viewBox="-55 -75 1348 1117" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> </ins> <g> <rect style="fill: #ffffff" x="662" y="192" width="630" height="381"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="662" y="192" width="630" height="381"/> </g> <g> <del> <rect style="fill: #ffffff" x="689" y="256" width="580" height="292"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="689" y="256" width="580" height="292"/> </del> <ins> <rect style="fill: #ffffff" x="688" y="321" width="580" height="227"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="688" y="321" width="580" height="227"/> </ins> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="687" y="224"> <del> <tspan x="687" y="224">Minion</tspan> </del> <ins> <tspan x="687" y="224">Node</tspan> </ins> </text> <g> <del> <rect style="fill: #ffffff" x="721.2" y="303" width="69.6" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="721.2" y="303" width="69.6" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="756" y="325.9"> <tspan x="756" y="325.9">kubelet</tspan> </del> <ins> <rect style="fill: #ffffff" x="723.2" y="235" width="69.6" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="723.2" y="235" width="69.6" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="758" y="257.9"> <tspan x="758" y="257.9">kubelet</tspan> </ins> </text> </g> <g> <del> <rect style="fill: #ffffff" x="821.676" y="303" width="76.65" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="821.676" y="303" width="76.65" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="860.001" y="325.9"> <tspan x="860.001" y="325.9">cAdvisor</tspan> </del> <ins> <rect style="fill: #ffffff" x="720.2" y="368.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="720.2" y="368.1" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="760.55" y="438.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="760.55" y="438.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="805.2" y="461"> <tspan x="805.2" y="461">container</tspan> </ins> </text> </g> <del> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="711" y="280"> <tspan x="711" y="280">docker</tspan> </text> </del> <g> <del> <g> <rect style="fill: #ffffff" x="720.2" y="368.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="720.2" y="368.1" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="760.55" y="438.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="760.55" y="438.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="805.2" y="461"> <tspan x="805.2" y="461">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="749.8" y="428.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="749.8" y="428.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.45" y="451.1"> <tspan x="794.45" y="451.1">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="739.4" y="418.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="739.4" y="418.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="784.05" y="441.2"> <tspan x="784.05" y="441.2">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.2" y="434.6"> <tspan x="794.2" y="434.6"></tspan> </del> <ins> <rect style="fill: #ffffff" x="749.8" y="428.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="749.8" y="428.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.45" y="451.1"> <tspan x="794.45" y="451.1">container</tspan> </ins> </text> <del> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="742.2" y="394.6"> <tspan x="742.2" y="394.6">Pod</tspan> </del> <ins> </g> <g> <rect style="fill: #ffffff" x="739.4" y="418.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="739.4" y="418.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="784.05" y="441.2"> <tspan x="784.05" y="441.2">cAdvisor</tspan> </ins> </text> </g> <ins> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="794.2" y="434.6"> <tspan x="794.2" y="434.6"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="742.2" y="394.6"> <tspan x="742.2" y="394.6">Pod</tspan> </text> </ins> <g> <g> <del> <rect style="fill: #ffffff" x="1085.6" y="368.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1085.6" y="368.1" width="148" height="133"/> </del> <ins> <rect style="fill: #ffffff" x="1108.6" y="368.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1108.6" y="368.1" width="148" height="133"/> </ins> </g> <g> <del> <rect style="fill: #ffffff" x="1125.95" y="438.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1125.95" y="438.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1170.6" y="461"> <tspan x="1170.6" y="461">container</tspan> </del> <ins> <rect style="fill: #ffffff" x="1148.95" y="438.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1148.95" y="438.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1193.6" y="461"> <tspan x="1193.6" y="461">container</tspan> </ins> </text> </g> <g> <del> <rect style="fill: #ffffff" x="1115.2" y="428.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1115.2" y="428.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1159.85" y="451.1"> <tspan x="1159.85" y="451.1">container</tspan> </del> <ins> <rect style="fill: #ffffff" x="1138.2" y="428.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1138.2" y="428.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1182.85" y="451.1"> <tspan x="1182.85" y="451.1">container</tspan> </ins> </text> </g> <g> <del> <rect style="fill: #ffffff" x="1104.8" y="418.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1104.8" y="418.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1149.45" y="441.2"> <tspan x="1149.45" y="441.2">container</tspan> </del> <ins> <rect style="fill: #ffffff" x="1127.8" y="418.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1127.8" y="418.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1172.45" y="441.2"> <tspan x="1172.45" y="441.2">container</tspan> </ins> </text> </g> <del> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1159.6" y="434.6"> <tspan x="1159.6" y="434.6"></tspan> </del> <ins> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1182.6" y="434.6"> <tspan x="1182.6" y="434.6"></tspan> </ins> </text> <del> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1107.6" y="394.6"> <tspan x="1107.6" y="394.6">Pod</tspan> </del> <ins> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1130.6" y="394.6"> <tspan x="1130.6" y="394.6">Pod</tspan> </ins> </text> </g> <g>

CA Advisor runs inside docker containers, but I don't believe it's correct to show the Kubelet and Proxy processes as inside the docker box on the minion. Also, shouldn't the internet-firewall-really load balance across minion proxies instead of be connected to just one SPOF minion, treating that connection more like a logical connection across all minion proxies?
This is the desired state, which is admittedly not the current state. Yeah, this is a good point. k8s currently relies on its environment for load balancing. Maybe or can speak to the desired state here.
We should just review the diagram and make it correctly reflect the state of 1.0.


Question: imageGCPolicy ImageGCPolicy, diskSpacePolicy DiskSpacePolicy, cloud cloudprovider.Interface, <ins> autoDetectCloudProvider bool, </ins> nodeLabels map[string]string, nodeStatusUpdateFrequency time.Duration, osInterface kubecontainer.OSInterface,

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: // rejectForwardingRedirects is whether to allow to forward redirect response rejectForwardingRedirects bool <ins> // tracerProvider is used to wrap the proxy transport and handler with tracing tracerProvider tracing.TracerProvider </ins> } // Complete fills in any fields not set that are required to have valid data. It's mutating the receiver.

Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted