Question: verifyResult(cs, podsNeededForSaturation, 1, ns) }) <del> // This test verifies we don't allow scheduling of pods in a way that sum of limits of pods is greater than machines capacity. // It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. </del> <ins> // This test verifies we don't allow scheduling of pods in a way that sum of // limits of pods is greater than machines capacity. // It assumes that cluster add-on pods stay stable and cannot be run in parallel // with any other test that touches Nodes or Pods. </ins> // It is so because we need to have precise control on what's running in the cluster. <ins> // Test scenario: // 1. Find the amount CPU resources on each node. // 2. Create one pod with affinity to each node that uses 70% of the node CPU. // 3. Wait for the pods to be scheduled. // 4. Create another pod with no affinity to any node that need 50% of the largest node CPU. // 5. Make sure this additional pod is not scheduled. </ins> It("validates resource limits of pods that are allowed to run [Conformance]", func() { <ins> framework.WaitForStableCluster(cs, masterNodes) </ins> nodeMaxAllocatable := int64(0) nodeToAllocatableMap := make(map[string]int64) for _, node := range nodeList.Items { <ins> nodeReady := false for _, condition := range node.Status.Conditions { if condition.Type == v1.NodeReady && condition.Status == v1.ConditionTrue { nodeReady = true break } } if !nodeReady { continue } // Apply node label to each node framework.AddOrUpdateLabelOnNode(cs, node.Name, "node", node.Name) framework.ExpectNodeHasLabel(cs, node.Name, "node", node.Name) // Find allocatable amount of CPU. </ins> allocatable, found := node.Status.Allocatable[v1.ResourceCPU] Expect(found).To(Equal(true)) nodeToAllocatableMap[node.Name] = allocatable.MilliValue()

cc cc Seeing a failure on our downgrade test. Can someone please look at this? https://k8s- Example failure https://k8s- Seeing timeouts on the downgrade test
Since we're so late in the release process and this is the first time we're seeing this fail, going to mark as non-release-blocking for now.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scheduling: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
It fails in more places than just the downgrade job, but when the downgrade job fails that test, that timeout appears to be the consistent error that it fails with
/assign I looked at the logs. The test creates a few pods to saturate the existing nodes and then creates an additional pod and ensures that this last pod stays pending. The problem we observe is that those saturating pods are often not scheduled, causing a test timeout. Those pods use all the resources on the nodes. So, any small changes to the amount of available node resources or even rounding errors could cause those saturating pods to not schedule. I will test this theory by cutting down the amount of resources that the saturating pods use, but I would like to check if the problem is fixed before submitting the PR. Where can I get the steps these cluster downgrade tests take?
The test setups are in the test-infra repo. E.g.
see my analysis in .


Question: annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: <ins> dnsPolicy: Default </ins> containers: - name: fluentd-gcp image: gcr.io/google_containers/fluentd-gcp:1.38

Currently, fluentd on master is unable to ingest logs from master node on GCE. After some investigation it turned out that DNS (10.0.0.10) is unavailable from inside the fluentd container: no host can be resolved, doesn't depend on the fluentd image version. Fluentd container on the minion node works, no problems whatsoever. Steps to reproduce: Spin up a cluster on GCE Check fluentd logs on the master node If cluster is built from 1.5 branch, fluentd on master node works, logs are being ingested. CC
cc
What is the commit that your cluster is built with?
From HEAD today, can't say the exact commit But the problem actually is there for quite some time, for two weeks at least.
There are two issues at work here: -proxy is not run on the master node, which means service VIPs will not be accessible from the master. pod that runs on the master uses ClusterFirst, which means the points to the service VIP. One of those needs to change for DNS to work.
OK, thanks! I will try to change the dnsPolicy tomorrow to see if it helps Could you please explain what have changed since 1.5? For what I know, fluentd was moved to DaemonSet, is this the reason?
-- this looks like your change:
As far as I know, the master has never run kube-proxy
The migration to DS should be unrelated here. We used to have fluentd as manifest pod before.
Using didn't help Is it possible to have the same for fluentd on master as the rest of the pods?
Define "doesn't work" after this? should show whatever the host machine has.
Sorry, my bad, was testing with the wrong setup. Using did indeed help for fluentd with Stackdriver, thanks a lot!


Question: } func (l *requestFIFO) Walk(f walkFunc) { <del> for current := l.Front(); current != nil; current = current.Next() { </del> <ins> var next *list.Element for current := l.Front(); current != nil; current = next { next = current.Next() // f is allowed to remove current </ins> if r, ok := current.Value.(*request); ok { if !f(r) { return

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned .

Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to .

hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .


Question: "encoding/json" "fmt" "sort" <del> "time" </del> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

What happened: On a resource with managedFields populated, a no-op patch modified the serialization order of managedFields. What you expected to happen: No-op patches do not result in a write to etcd. How to reproduce it (as minimally and precisely as possible): contains: the initial ADDED event from kubectl MODIFIED events from kube-controller-manager one MODIFIED event corresponding to the kubectl patch (from resourceVersion 1704 in 1.txt to resourceVersion 1718 in 2.txt) shows the kubectl and kube-controller-manager managed field sets swapped order: second patch was a no-op ( is empty) Anything else we need to know?: This was easiest to reproduce with server-side apply enabled at 100% /assign
/sig api-machinery
/wg apply
Can you print the before/after fieldmanager? I can't quite figure out what the actual values are. The code that sorts these lists is located
updated to unified diff and attached:
If you explicitly set the field manager to something unique, does it happen every time?
hm, no, it's definitely an ordering swap
is not sorted properly, since should come before .
We seem to also sort it in a second place:
Theory: the first time it goes through, the time is accurate to the nanosecond; the second time, it's accurate to the second
Yeah OK I'm pretty sure that's it, both times are the same second. Serialization drops the fractional part.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image"

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 14 March 2022 [X] test promotion pr : According to this APIsnoop query, there is still one remaining PodTemplate endpoint which is untested. Using a number of existing e2e test practices a new has been created for pod templates. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ClearUDPConntrackForIP uses the conntrack tool to delete the conntrack entries // for the UDP connections specified by the given service IP func ClearUDPConntrackForIP(execer exec.Interface, ip string) error { <del> parameters := parametersWithFamily(isIPv6(ip), "-D", "--orig-dst", ip, "-p", "udp") </del> <ins> parameters := parametersWithFamily(IsIPv6String(ip), "-D", "--orig-dst", ip, "-p", "udp") </ins> err := ExecConntrackTool(execer, parameters...) if err != nil && !strings.Contains(err.Error(), noConnectionToDelete) { // TODO: Better handling for deletion failure. When failure occur, stale udp connection may not get flushed.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question:
disruptedPods

DisruptedPods contains information about pods whose eviction was processed by the API server eviction subresource handler but has not yet been observed by the PodDisruptionBudget controller. A pod will be in this map from the time when the API server processed the eviction request to the time when the pod is seen by PDB controller as having been marked for deletion (or after a timeout). The key in the map is the name of the pod and the value is the time when the API server processed the eviction request. If the deletion didn’t occur and a pod is still there it will be removed from the list automatically by PodDisruptionBudget controller after some time. If everything goes smooth this map should be empty for the most of the time. Large number of entries in the map may indicate problems with pod deletions.

true

false

object

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PodDisruptionBudget provides inappropriate message and cannot be updated as is required filed. What you expected to happen: works fine for editing labels under . Also, produce correct error message when edit field. How to reproduce it (as minimally and precisely as possible): Then, we will get following error after step-2. When edit , it should provide correct message as .When edit especially labels under metadata, they should be updated without error. Anything else we need to know?: According to the code comment, is valid. Environment: Kubernetes version (use ): build on
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale there is a PR open


Question: expected: true, }, { <ins> name:     "QOSClass set as BestEffort. single-numa-node Policy. No Hints.", qosClass: v1.PodQOSBestEffort, policy:   NewRestrictedPolicy(), hp: []HintProvider{ &mockHintProvider{}, }, expected: true, }, { name:     "QOSClass set as BestEffort. Restricted Policy. No Hints.", qosClass: v1.PodQOSBestEffort, policy:   NewRestrictedPolicy(), hp: []HintProvider{ &mockHintProvider{}, }, expected: true, }, { </ins> name:     "QOSClass set as Guaranteed. BestEffort Policy. Preferred Affinity.", qosClass: v1.PodQOSGuaranteed, policy:   NewBestEffortPolicy(),

What happened: With the latest , which aims to resolve the issue of deploying pods with single-numa-node policy, it seems to have resulted in failure to deploy pods if the pod spec does not have resources and limits constraints. This is resulting in Kube-dns and other pods which do not have resource constraints mentioned fail to deploy due to Topology Affinity Error. kube-system kube-dns--22dks 0/3 Topology Affinity Error 0 What you expected to happen: Ideally, if there are no resource constraints mentioned we will still need the pod to be deployed. We cannot have Kube-dns failing to deploy due to Topology Affinity issue. How to reproduce it (as minimally and precisely as possible): With the latest fix, enabling single-numa-node policy, Kube-dns errors out on the node. One can just try to deploy simple pod without resource constraints in pod spec. Anything else we need to know?: This happens only when policy is single-numa-node. Pods do not fail to deploy for best-case policy. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.0-alpha.1.338+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2019-10-11T10:39:19Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Baremetal x86 server with 2 NUMA nodes and 32 cpus. OS (e.g: ): Fedora 28 Kernel (e.g. ): 5.0.13-100.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
thanks for alerting us to the issue.

Can you try the PR linked?
My guess is that this issue was introduced after landed. Before that, best-effort pods weren't even being considered by the TopologyManager, and the logic for admitting vs. not admitting was skipped. Which means that we need this fix for this on master, but do not need it on the 1.16 branch.
can you verify if it works as expected in the release-1.16 branch. This would verify my claim that it broke with the introduction of not the fix from . The patch for landed just a few hours after , so I could see how its hard to know which one caused the new issue.
PTAL
, That's right. This fix is needed for the master branch and not 1.16. 1.16 works well without .
/cc for visibility. This issue appeared as a result of
Thanks for the ping. So is the proposed fix to always admit best-effort pods in the topology manager?
This only affects the single-numa-node policy because of the "hack" we currently have in place for it: The right long term fix is to address this as part of: However, if we want to put a short term fix in place to "unbreak" master, has a PR linked above that will fix the immediate problem.


Question: if !reflect.DeepEqual(originalPod.Spec.Containers[0].Env, pod.Spec.Containers[0].Env) { framework.Failf("env of pod container does not match the env of the original pod: expected %#v, got: %#v", originalPod.Spec.Containers[0].Env, pod.Spec.Containers[0].Env) } <ins> if !reflect.DeepEqual(originalPod.Spec.InitContainers[0].Env, pod.Spec.InitContainers[0].Env) { framework.Failf("env of pod init container does not match the env of the original pod: expected %#v, got: %#v", originalPod.Spec.InitContainers[0].Env, pod.Spec.InitContainers[0].Env) } </ins> }) })

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /sig cluster-lifecyle What happened: I created a Pod Preset to inject an environment variable. The Pod container gets properly injected, but the init container does not. A small sample is here: What you expected to happen: The init container would get the pod preset injected How to reproduce it (as minimally and precisely as possible): Run kubectl create -f on the following gist: Reproduced here: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-09T07:26:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: minikube 0.23.0 OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Started with:
/sig cluster-lifecyle /sig service-catalog
My use case: I have an a git init container that checkout out configuration. This needs to run before the main container (hence why it is an init container). I'd like to be able to simply annotate the pod with a label and have various environment variables (GIT_REPO, etc) supplied to the init container
Update from Slack discussion: Supporting Init containers is in the Beta proposal for PodPreset. PodPreset resource is going to be moved from core to service-catalog. We have an implementation based on Initializers, but looks like Initializers are not going to Beta soon and we might have to reimplement them using Webhooks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
It'd be nice if this worked, I suspect it would help the folks over at Knative Build implement support for passing HTTP_PROXY env vars and CA cert secrets to build steps.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a customer interested in seeing this fixed - if we submit a PR, are there objections to merging it?
cc
This is also affecting me - I have an init container that loads data from AWS Secrets and stores them in a volume that is accessed by the main container. The init container needs credentials that are passed via a podpreset
any ETA when support for initcontainers will be provided by PodPreset ?
This was fixed in 1.14.


Question: return nil, nil } <ins> func (fVMC *fakeAzureVirtualMachinesClient) Update(ctx context.Context, resourceGroupName string, VMName string, parameters compute.VirtualMachineUpdate, source string) (resp *http.Response, err error) { fVMC.mutex.Lock() defer fVMC.mutex.Unlock() if _, ok := fVMC.FakeStore[resourceGroupName]; !ok { fVMC.FakeStore[resourceGroupName] = make(map[string]compute.VirtualMachine) } return nil, nil } </ins> func (fVMC *fakeAzureVirtualMachinesClient) Get(ctx context.Context, resourceGroupName string, VMName string, expand compute.InstanceViewTypes) (result compute.VirtualMachine, err error) { fVMC.mutex.Lock() defer fVMC.mutex.Unlock()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: return nil, fmt.Errorf("no ports provided to openstack load balancer") } <ins> floatingPool := getStringFromServiceAnnotation(apiService, ServiceAnnotationLoadBalancerFloatingNetworkId, lbaas.opts.FloatingNetworkId) glog.V(4).Infof("EnsureLoadBalancer using floatingPool: %v", floatingPool) </ins> // Check for TCP protocol on each port // TODO: Convert all error messages to use an event recorder for _, port := range ports {

Is this a BUG REPORT or FEATURE REQUEST?: Feature request /kind feature What happened: Currently we have kubernetes cluster installed, in node machines we have one subnet and two routers attached. One router for external network and one router for internal network. However, when we create type: loadbalancer services in kubernetes, we have to hardcode floatingnetwork id to only one of those (external or internal). What you expected to happen: I except that I could define from which floatingnetwork loadbalancer is taking its floating ip address in each service. One possible solution could be that if floatingnetworkid is defined like currently - that networkid is default one. Then we could have possibility to override that in each service configuration. Like if we do not want use default floating-pool we could change it: I do not know does this even work in openstack, it might need some manual route settings in lbaas. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): centos cc
Exposing first to external network so the external floatingip is working. Then update floating-networkid in kubernetes cloud-config in all nodes+masters and restart all machines. Exposing to internal network However, this last curl did not work from outside openstack. There is some route issue with it, if I try that outside openstack the result is: tcpdump from node the external one which works 192.168.1.12 is lbaas fixed ip: tcpdump from node the internal one which does not work 192.168.1.13 is lbaas fixed ip: This might mean that lbaas is using default router (which is external) to answer back to client, so the route in lbaas should be modified. No idea is that possible.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig network
I have defined host route to subnet, which is 10.222.0.0/16,192.168.1.4 this row basically means that all connections toward 10.222.0.0/16 should use 192.168.1.4 router interface, which is internal router. However, openstack lbaas is ignoring these host routes so before that is fixed there is no sense to code this feature to kubernetes. edit. looks like it is fixed that fix is not to our openstack yet. After that fix this is this my idea should work. Now the question is how we should implement this multiple floatingnetworkids to kubernetes. I can make PR but first I would like to discuss how it should be done.
Sounds good. cc
the problem what I am thinking currently is that I would like to have possibility to specify floating-pool in each service. Some service people may want expose service to external network and some service to internal network. However, the service api spec is quite hardcoded or can I add variable there? I am planning to add variable like there.


Question: SRCS=loader ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


What would you like to be : Add documentation/example showing the usage of dynamic package in the client-go repo. /sig api-machinery


Question: if [[ "${s3_bucket_location}" == "None" ]]; then # "US Classic" does not follow the pattern s3_url_base=https://s3.amazonaws.com <ins> s3_bucket_location=us-east-1 </ins> fi local -r staging_path="devel"

When I run: I get the following error: Waiting a bit and running got past this point because the bucket was ready the second time around, but the getting-started process should be more robust.
I experienced the same issue
We do try to query for the bucket existing, but perhaps S3 is even more eventually consistent than I thought! I did observe that we were very trusting about the user providing the correct S3 region, and have tried to fix this in . I don't think that was the problem here though (if you make_bucket, you can't have a region mismatch...) I guess we could retry the sync a few times; might help with e.g. a network interruption during the sync also.
I believe this to have been fixed by . I'm going to close, but if it happens again please do reopen.
FYI I got this with the latest k8s as of 10/13/15: I believe this is k8s 1.0.6 but I just tore down my AWS cluster and apparently kubectl needs to talk to the now-terminated master instance. I ran to get k8s. Waiting and running again doesn't show the issue since the bucket is fully created and accessible. I just provisioned a Vagrant cluster in hopes that would help show what version I'm using. Here's the output of :


Question: # --output-base    because this script should also be able to run inside the vendor dir of #                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir #                  instead of the $GOPATH directly. For normal projects this can be dropped. <del> "${CODEGEN_PKG}/generate-groups.sh" all  </del> <ins> bash "${CODEGEN_PKG}/generate-groups.sh" all  </ins> k8s.io/sample-apiserver/pkg/generated k8s.io/sample-apiserver/pkg/apis  "wardle:v1alpha1,v1beta1"  --output-base "$(dirname "${BASH_SOURCE[0]}")/../../.."  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt <del> "${CODEGEN_PKG}/generate-internal-groups.sh" "deepcopy,defaulter,conversion"  </del> <ins> bash "${CODEGEN_PKG}/generate-internal-groups.sh" "deepcopy,defaulter,conversion"  </ins> k8s.io/sample-apiserver/pkg/generated k8s.io/sample-apiserver/pkg/apis k8s.io/sample-apiserver/pkg/apis  "wardle:v1alpha1,v1beta1"  --output-base "$(dirname "${BASH_SOURCE[0]}")/../../.."

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I wanted to test the latest version of the sample-apiserver. I still did not find instructions or a script for building it. I found the script, but that relies on the directory produced when building k/k. What you expected to happen: I expected this sample to have standalone build instructions or scripting. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I created a script of my own to build it:
Also: since the file has a fixed image reference, only someone who pushes to that image can use that file unmodified. The README should say something about this.
Also, the existing is not designed to be invoked by a developer who has done but rather is designed to work only in the context of a full build of k/k.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Being addressed by


Question: load( "@io_bazel_rules_go//go:def.bzl", "go_library", <del> "go_test", ) go_test( name = "go_default_test", srcs = ["converter_test.go"], library = ":go_default_library", deps = [ "//vendor/k8s.io/apimachinery/pkg/util/diff:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/json:go_default_library", ], </del> ) go_library(

Kubernetes version (use ): master. What happened: Unstructured converter from does not roundtrip . What you expected to happen: When I call on a which is backed by it should basically make a copy of the content (can be optimized btw using deep copy ) and not a json object with key. How to reproduce it (as minimally and precisely as possible): See the test in my branch Test output: Anything else we need to know: For some reason changed the argument types from to . I guess the intent is to make the converter a more generic tool. But based on the previous signature I assume I'm doing the right thing. Actually, my code works with and I don't know if the actual object behind the interface is an or not - I just pass it to the converter and it works for usual API types, but not for unstructured. Also it would be useful to have a more type-safe version of the converter (i.e. the previous signature) that just delegates to a lower-level converter that does not care about types.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/kind bug
/assign
Looked into this issue just now. It is caused by the pointer vs non-pointer receiver of the method - similar to . In detail: is passed as to when it calls passing as . if passed value implements (has method ) does not implement this interface, does. Because on it . So perhaps the fix is to do the dereferencing inside of the ? Or maybe the same way as - check if the value is addressable, get a pointer to the value and then see if it matches the interface (I dunno if it will work that way)? In any case, the check whether a value implements the interface should be done on both the value itself () and the pointer to the value (; if the value is addressable) to cover both valid cases. This is what stdlib package does. Right now it is broken - it dereferences the pointer and then checks if the thing implements the interface. Does it make sense?

Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
does this helps? Look like we need to do MarshalJSON on non-pointers for it to work. I didn't get into the details but if that is the case then making Unstructured's MarshalJSON() receiver, a object instead of pointer should fix your problem, right?
Only if we don't fix the actual cause of the bug which I described above. From the stack overflow question that is mentioned in the link you provided: This code is buggy - to work it should be passing into and into . Yes, json package tries to take the address of the passed (so a pointer receiver should satisfy the interface check) but it cannot take an address of a function argument ( will return false). So, pointer receiver will be invoked if either is a field in a struct (so that an address can be taken via reflection) and a struct is passed (either by value or via a pointer) to / OR is passed. Please take a look at - both issues have the same cause.
I'm happy to take a stab at it and submit a PR but I don't want to spend my time if it is not going to get merged - if we could merge first to make sure people who can approve/lgtm are happy with the approach taken. I strongly believe it is the right thing to do - that is how standard library does it and, well, it is just logical and makes sense :)


Question: resp, err := rt.RoundTrip(req) if err != nil { <del> message := fmt.Sprintf("Error: '%s'nTrying to reach: '%v'", err.Error(), req.URL.String()) </del> <ins> message := fmt.Sprintf("Error trying to reach service: '%v'", err.Error()) </ins> resp = &http.Response{ <ins> Header:     http.Header{}, </ins> StatusCode: http.StatusServiceUnavailable, Body:       ioutil.NopCloser(strings.NewReader(message)), } <ins> resp.Header.Set("Content-Type", "text/plain; charset=utf-8") resp.Header.Set("X-Content-Type-Options", "nosniff") </ins> return resp, nil }

What happened: The name of an arbitrarily supplied URL parameter is copied into the HTML document as plain text between tags. The payload cfk88<scriptalert(1)</scripty9rri was submitted in the name of an arbitrarily supplied URL parameter. This input was echoed unmodified in the application's response. This proof-of-concept attack demonstrates that it is possible to inject arbitrary JavaScript into the response. The response does not state that the content type is HTML. The issue is only directly exploitable if a browser can be made to interpret the response as HTML. What you expected to happen: the response should not contain any valid HTML content. Avoid to nject arbitrary JavaScript. How to reproduce it (as minimally and precisely as possible): Just make the backend service unreachable. For example, https backend service is only supported. Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.2 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /priority important-soon
Similar issue kube-apiserver did not know how the response will be handled. But if it was interpreted by browser it's a security issue. Not sure we can transform the error message to something which cannot be interpreted by browser before return the response. cc
responses from kube-apiserver seems correctly setting header, either json or protobuf.
the json encoding explicitly escapes to to avoid the browser interpreting it as HTML /close
Closing this issue. <detailsIn response to : the json encoding explicitly escapes to to avoid the browser interpreting it as HTML Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can you check the origin issue Sorry to bring issue and get you confused. As you see the html element , was not encoded.
ah, you're right, I thought was the issue description /reopen
Reopened this issue. <detailsIn response to : ah, you're right, I thought was the issue description Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
Is there a PR/work associated with this issue yet?
there is not. a PR would be welcome
Hi, is there a PR to fix this issue?
opened to resolve this
Is this fix mapped with the latest versions? Looking to the v0.19.3 code I don't see these changes and scanning my docker image (XRAY scanning) that is integrating the version v0.19.3 I can see this vulnerability that seems fixed in a version older than the one I am using. Am I wrong?
Yes, it is present. The issue was fixed in That commit () is present in the repo in v0.16.4+ as


Question: <del> // Package diskcache provides an implementation of httpcache.Cache that uses the diskv package // to supplement an in-memory map with persistent storage // package diskcache import ( "bytes" "crypto/md5" "encoding/hex" "github.com/peterbourgon/diskv" "io" ) // Cache is an implementation of httpcache.Cache that supplements the in-memory map with persistent storage type Cache struct { d *diskv.Diskv } // Get returns the response corresponding to key if present func (c *Cache) Get(key string) (resp []byte, ok bool) { key = keyToFilename(key) resp, err := c.d.Read(key) if err != nil { return []byte{}, false } return resp, true } // Set saves a response to the cache as key func (c *Cache) Set(key string, resp []byte) { key = keyToFilename(key) c.d.WriteStream(key, bytes.NewReader(resp), true) } // Delete removes the response with key from the cache func (c *Cache) Delete(key string) { key = keyToFilename(key) c.d.Erase(key) } func keyToFilename(key string) string { h := md5.New() io.WriteString(h, key) return hex.EncodeToString(h.Sum(nil)) } // New returns a new Cache that will store files in basePath func New(basePath string) *Cache { return &Cache{ d: diskv.New(diskv.Options{ BasePath:     basePath, CacheSizeMax: 100 * 1024 * 1024, // 100MB }), } } // NewWithDiskv returns a new Cache using the provided Diskv as underlying // storage. func NewWithDiskv(d *diskv.Diskv) *Cache { return &Cache{d} } </del>

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: - replaceCoreV1PersistentVolume - replaceCoreV1PersistentVolumeStatus - replaceStorageV1CSINode <del> - replaceStorageV1StorageClass </del> - replaceStorageV1VolumeAttachment - replaceStorageV1VolumeAttachmentStatus

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: errors := []error{} if s.BindPort < 0 || s.BindPort > 65535 { <del> errors = append(errors, fmt.Errorf("--insecure-port %v must be between 0 and 65535, inclusive. 0 for turning off insecure (HTTP) port.", s.BindPort)) </del> <ins> errors = append(errors, fmt.Errorf("--insecure-port %v must be between 0 and 65535, inclusive. 0 for turning off insecure (HTTP) port", s.BindPort)) </ins> } return errors

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: }, }).Validate, }, <ins> { name:                   "JobControllerOptions ConcurrentJobSyncs equal 0", expectErrors:           true, expectedErrorSubString: "concurrent-job-syncs must be greater than 0", validate: (&JobControllerOptions{ &jobconfig.JobControllerConfiguration{ ConcurrentJobSyncs: 0, }, }).Validate, }, </ins> /* empty errs */ { name:         "CronJobControllerOptions",

What would you like to be : The ability to set for the kube-controller-manager at runtime. It looks like the option exists in the code, but a mechanism is not available to set it via kops. If I understand the code correctly, a flag needs to be in order to be able to set the value using kops. Why is this needed: We want to shift a lot of our batch processing jobs from running directly on our CI servers to running as jobs on Kubernetes. Unfortunately if too many jobs are launched on kubernetes, they begin to back up. We've noticed up to a 10 minute delay between when the logs indicate that a job has finished and when the kubernetes API actually reflects the same on our small ~30 node cluster.
/sig scalability
I think scalability is the best fit since this is impacting our ability to fully utilize and scale how we use our kubernetes cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Is there anything I can do to help get landed?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-sig api-machinery /sig apps
/triage accepted /remove-lifecycle rotten
can you please describe your use case in a more detail? how How this affects you, and how this flag would help to achieve your goals?
-- I left a comment describing my use case in the PR:


Question: } userInfo, _ := request.UserFrom(ctx) <del> staticAdmissionAttributes := admission.NewAttributesRecord(nil, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Update, userInfo) </del> var transformers []rest.TransformFunc <del> if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Update) { </del> <ins> if mutatingAdmission, ok := admit.(admission.MutationInterface); ok { </ins> transformers = append(transformers, func(ctx context.Context, newObj, oldObj runtime.Object) (runtime.Object, error) { <del> return newObj, mutatingAdmission.Admit(admission.NewAttributesRecord(newObj, oldObj, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Update, userInfo)) </del> <ins> isNotZeroObject, err := hasUID(oldObj) if err != nil { return nil, fmt.Errorf("unexpected error when extracting UID from oldObj: %v", err.Error()) } else if !isNotZeroObject { if mutatingAdmission.Handles(admission.Create) { return newObj, mutatingAdmission.Admit(admission.NewAttributesRecord(newObj, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Create, userInfo)) } } else { if mutatingAdmission.Handles(admission.Update) { return newObj, mutatingAdmission.Admit(admission.NewAttributesRecord(newObj, oldObj, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Update, userInfo)) } } return newObj, nil </ins> }) } createAuthorizerAttributes := authorizer.AttributesRecord{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: "k8s.io/client-go/util/workqueue/parallelizer.go", "k8s.io/client-go/util/workqueue/queue.go", "k8s.io/client-go/util/workqueue/rate_limitting_queue.go", <del> "k8s.io/client-go/util/workqueue/timed_queue.go", </del> ], tags = ["automanaged"], deps = [

The TimedWorkQueue object appears to lack a critical feature of the original WorkOueue, namely that similar items are aggregated together. By adding the time member to to the key type, it ensures that (time, item) pairs are unique-- you can add the same item multiple times as long as the time differs. This removes the guarantee that two goroutines wouldn't process the same item at the same time.
Iirc, the timed queue feeds a regular queue, so the guarantee still holds
Sorry, I was thinking of the delaying queue, which does key correctly
I'm going to remove timed workqueue. It's used to gather metrics for queues used in the garbage collector. This feature is supported by a regular workqueue now.


Question: runtimeApi "k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/runtime" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" <ins> "k8s.io/kubernetes/pkg/kubelet/dockertools" </ins> "k8s.io/kubernetes/pkg/kubelet/qos" "k8s.io/kubernetes/pkg/kubelet/types" )

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: QOSReserved featuregate.Feature = "QOSReserved" // owner: @chrishenzie <ins> // kep: https://kep.k8s.io/2485 </ins> // alpha: v1.22 <ins> // beta: v1.27 </ins> // // Enables usage of the ReadWriteOncePod PersistentVolume access mode. ReadWriteOncePod featuregate.Feature = "ReadWriteOncePod"

For 1.27 we should graduate the ReadWriteOncePod feature to beta. This is needed to rollout the feature to GA. In beta this feature will support preemption of pods using PVCs. See:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig scheduling


Question: } if in.HostIP != "" { <del> out.Status.Addresses = append(out.Status.Addresses, </del> <ins> newer.AddToNodeAddresses(&out.Status.Addresses, </ins> newer.NodeAddress{Type: newer.NodeLegacyHostIP, Address: in.HostIP}) } out.Spec.PodCIDR = in.PodCIDR

Since we append hostips to the list on every node controller run invocation which happens at 10s intervals, this list grows pretty large pretty soon: Maybe: would still satisfy the requirements?
Fix is here I think we generally use list instead of map
This could have been P1, but I just merged a fix so I'll leave at P2. Remaining work to do: Add to validation something that limits the number of IP addresses. (and maybe checks dups?)
Yeah, I'll add validation later.


Question: // using host based configurations. func NewDenyEscalatingExec() *DenyExec { return &DenyExec{ <del> Handler:    admission.NewHandler(admission.Connect), hostIPC:    true, hostPID:    true, privileged: true, </del> <ins> Handler:     admission.NewHandler(admission.Connect), hostNetwork: true, hostIPC:     true, hostPID:     true, privileged:  true, </ins> } } // NewDenyExecOnPrivileged creates a new admission controller that is only checking the privileged <del> // option.  This is for legacy support of the DenyExecOnPrivileged admission controller.  Most // of the time NewDenyEscalatingExec should be preferred. </del> <ins> // option. This is for legacy support of the DenyExecOnPrivileged admission controller. // Most of the time NewDenyEscalatingExec should be preferred. </ins> func NewDenyExecOnPrivileged() *DenyExec { return &DenyExec{ <del> Handler:    admission.NewHandler(admission.Connect), hostIPC:    false, hostPID:    false, privileged: true, </del> <ins> Handler:     admission.NewHandler(admission.Connect), hostNetwork: false, hostIPC:     false, hostPID:     false, privileged:  true, </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: currently only are checked in admission controller What you expected to happen: hostNetwork should also be checked to deny exec /attach How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug /sig auth
cc


Question: import ( "fmt" <del> "reflect" </del> <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: DigitalOcean OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others: What happened: when I do: it will run rolling update ONLY if there is a change. In case I do: Each time I run it - it will execute rolling update, which is misleading. What you expected to happen: I would like to have consistent behavior for Deployment and DaemonSet How to reproduce it (as minimally and precisely as possible): Create any DaemonSet with following:
Hmm, it shouldn't happen. TemplateGeneration is updated only when something changed. I will debug it.
I can reproduce this, and the DaemonSet is updated after every , even when nothing is changed. In , old/new DaemonSet template are seen as different with . I tried to change this part to use instead (needed to update code in validation as well), and the DaemonSet templates were then seen as the same, and thus the issue was gone. We used everywhere in the controllers. Is the right way to compare stuffs?
It's very strange. I tried to use pretty.Diff to print the differences but it returns empty slice. apiequality.Semantic.DeepEqual uses forked version of reflect.DeepEqual and it's little different from version in 1.7. will you prepare the patch?
The key of the reported DaemonSet config is its . Note that in the config file, (empty list); however, in the live DaemonSet object (created from the config file), (null list). That's why thinks they're different. If we use in the config file instead, won't change. From the field description, empty list and null list are different. Feel it's api machinery related
protobuf storage isn't able to store , so the object has that field set to when read back from storage. that is causing problems for that field in particular (where and have different meanings) see
cc for behavior of namespaces field
so it's not a DaemonSet bug
Well, still not sure whether Semantic.DeepEqual should be used or not. I'm still concerned with the behavior of the controller for other fields that can be populated with non-null empty lists by users, but serialize back out to null (or set omitempty)
deployments are using Maybe we should also switch?

If empty lists should be treated differently from null list, we should replace with in controller code.
that distinction is going away in all the fields I know of... I'm not super happy with Semantic.DeepEqual masking the difference between those two things, but that's probably the right thing for you to be using
Special meaning of [] was removed in I think a switch to Semantic.DeepEqual is still needed here
Filed


Question: 1.  command line arguments - `server`, `api-version`, `certificate-authority`, and `insecure-skip-tls-verify` 1.  If cluster info is present and a value for the attribute is present, use it. 1.  If you don't have a server location, error. <del> 1.  User is build using the same rules as cluster info, EXCEPT that you can only have one authentication  technique per user. The command line flags are: `auth-path`, `client-certificate`, `client-key`, and `token`.  If there are two conflicting techniques, fail. </del> <ins> 1.  Determine the actual user info to use. User is built using the same rules as cluster info, EXCEPT that you can only have one authentication technique per user. 1. Load precedence is 1) command line flag, 2) user fields from .kubeconfig, 3) kubernetes_auth file fields (if user has a `auth-path` or the `--auth-path` was provided) 1. The command line flags are: `auth-path`, `client-certificate`, `client-key`, `username`, `password`, and `token`. 1. If there are two conflicting techniques, fail. </ins> 1.  For any information still missing, use default values and potentially prompt for authentication information ## Manipulation of .kubeconfig via `kubectl config <subcommand>`

After installing nginx on the kubernetes master (currently non-k8s docker container) and forwarding external http requests to the kubernetes-ro service through nginx with simple htpasswd auth, I tested it (to use within CI tool), but it failed to pass nginx. Is it because we are required to use https when querying the apiserver using kubectl? Or because the -a option is used for a different auth ? How would you solve this issue ? Thank you in advance!
I tried to run kubectl proxy in one shell and query kubectl get in the other, without success: Shell 1 Shell 2
Looks like the isn't forwarding authorization as expected? Can you take a look please?
takes a full qualified URL. If you pass the plain ip string it will do a get over http, which the apiserver correctly rejects. Replace with and it should work as expected. Note that if you set up a kubeconfig file you can avoid some of this pain. Run for help/commands to set up a kubeconfig.
Please reopen if the above does not work for you.
I understood that I cannot use kubectl with remote server through http, so I modified my nginx to support ssl. I still have a problem with the or parameter. Is the kubernetes_auth supposed to have a different format maybe? I do not have the option to reopen the ticket. thanks for the kubectl config tip. But it's only a v0.9+ feature, so I'll try it when I upgrade later from v0.8.1
after testing , I came up with that kubeconfig: Still getting the same error:
That config looks fine. message is usually a json unmarshalling error. I wonder if there might be something wrong with your .kubernetes_auth file? It should look like
Sorry, forgot to come back here after solving the issue. Correct, the problem was with the kubernetesauth file. The keys must be capitalized, and I was trying This naming convention should be better documented, or more useful error should be raised. Neither the Json standard (ECMA) nor the JSON-RPC specs are requiring capitalized keys. Renaming this issue to "kubernetesauth missing documentation"
Thanks Adrien. I think the right thing to do here is raise an error if the kubernetes_auth file exists but doesn't have the required keys.
JSON RPC states that identifiers must be case-sensitive.


Question: kubeclientset "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5" fake_kubeclientset "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5/fake" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> "github.com/stretchr/testify/assert" )

https://k8s- Failed: TestNamespaceController {namespace} Previous issues for this test:
GetObjectFromChan() is timing out (20s) and returning nil causing the panic. Happens in TestSecretController (and probably TestIngressController) controller as well. https://k8s-
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
I have a fix for this in
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
/cc OWNERS of :
I think this is also related to Issue .
fix is pending


Question: return "ip" } <ins> // IPPortVar allows IP or IP:port formats. type IPPortVar struct { Val *string } func (v IPPortVar) Set(s string) error { if len(s) == 0 { v.Val = nil return nil } if v.Val == nil { // it's okay to panic here since this is programmer error panic("the string pointer passed into IPPortVar should not be nil") } // Both IP and IP:port are valid. // Attempt to parse into IP first. if net.ParseIP(s) != nil { *v.Val = s return nil } // Can not parse into IP, now assume IP:port. host, port, err := net.SplitHostPort(s) if err != nil { return fmt.Errorf("%q is not in a valid format (ip or ip:port): %v", s, err) } if net.ParseIP(host) == nil { return fmt.Errorf("%q is not a valid IP address", host) } if _, err := strconv.Atoi(port); err != nil { return fmt.Errorf("%q is not a valid number", port) } *v.Val = s return nil } func (v IPPortVar) String() string { if v.Val == nil { return "" } return *v.Val } func (v IPPortVar) Type() string { return "ipport" } </ins> type PortRangeVar struct { Val *string }

Whenever I try to change kube proxy metrics bind using flag , it gives me: Then, I the port , the error changes. I expect the metrics server to listen to 0.0.0.0:. Do I miss something? Environment: Kubernetes version: v1.7.8 Cloud provider or hardware configuration**: on-premise OS (e.g. from /etc/os-release): debian/stretch Kernel (e.g. ): Linux R11U2-7U20N-1 4.9.0-3-amd64 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy
The is being and does NOT allow specifying a port. Have you tried using a config file? The field allows you to specify a port (an example ).
/assign
Turned out we use a wrong type when validating and . Sent for fixing this. Will try to cherrypick the fix as well.
New finding, is broken on HEAD in a different way: Seems to be introduced by
Humm..We have some default logic for : Seems like for some reason it does not auto append the default port number OR the validation happens before defaulting.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
I got the same issue 0.0.0.0 0.0.0.0: both not work with kube-proxy 1.9.3 amd64
It is a BUG. Conflict between componentconfig.IPVar.Set(s string) and validation.validateHostPort.

Could you elaborate the issue a bit more? Which k8s version are you on?
I also meet the same issue with kubernetes 1.8. It's impossibe for Prometheus to fetch metrics from 127.0.0.1: in my case, any workaround? I think we'd better reopen this issue.
metrics binds to by default, are you trying to modify port as well? Thanks for reporting, I confirm the issue is remaining. (or any ip:port) does not work as intended even after the fix. Reopening this and filed a new issue for conflict kubeproxyconfig validation. /reopen
Thanks for reopening the issue. I want to bind metrics to 0.0.0.0: as well. With address 127.0.0.1:, Prometheus cannot pull metrics from this address unless they are on the same node.
is fixing this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale (I just got bitten by this on kube-proxy v1.9.10)
should fix this.
Closing this as is merged. /close
Closing this issue. <detailsIn response to : Closing this as is merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: srcs = [ ":package-srcs", "//cmd/kube-apiserver/app/options:all-srcs", <ins> "//cmd/kube-apiserver/app/preflight:all-srcs", </ins> ], tags = ["automanaged"], )

See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).


Question: if f.DynamicConfigDir.Provided() && !utilfeature.DefaultFeatureGate.Enabled(features.DynamicKubeletConfig) { return fmt.Errorf("the DynamicKubeletConfig feature gate must be enabled in order to use the --dynamic-config-dir flag") } <del> // ensure that nobody sets InitConfigDir if the dynamic config feature gate is turned off </del> <ins> // ensure that nobody sets InitConfigDir if the KubeletConfigFile feature gate is turned off </ins> if f.InitConfigDir.Provided() && !utilfeature.DefaultFeatureGate.Enabled(features.KubeletConfigFile) { return fmt.Errorf("the KubeletConfigFile feature gate must be enabled in order to use the --init-config-dir flag") }

See:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
For some reason GitHub didn't close this automatically when the referencing PR merged.


Question: return nil } <ins> func (f *fakeIPTables) ChainExists(tableName utiliptables.Table, chainName utiliptables.Chain) (bool, error) { _, _, err := f.getChain(tableName, chainName) if err != nil { return false, err } return true, nil } </ins> // Returns index of rule in array; < 0 if rule is not found func findRule(chain *fakeChain, rule string) int { for i, candidate := range chain.rules {

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: // Watch makes a matcher for the given label and field, and calls // WatchPredicate. If possible, you should customize PredicateFunc to produre a <del> // matcher that matches by key. </del> <ins> // matcher that matches by key. generic.SelectionPredicate does this for you // automatically. </ins> func (e *Etcd) Watch(ctx api.Context, label labels.Selector, field fields.Selector, resourceVersion string) (watch.Interface, error) { return e.WatchPredicate(ctx, e.PredicateFunc(label, field), resourceVersion) }

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.