Question: store.BeginCreate = genericStore.beginCreate store.BeginUpdate = genericStore.beginUpdate <ins> // users can patch the status to remove the finalizer, // hence statusStore must participate on the AfterDelete // hook to release the allocated resources statusStore.AfterDelete = genericStore.afterDelete </ins> return genericStore, &StatusREST{store: &statusStore}, &svcreg.ProxyREST{Redirector: genericStore, ProxyTransport: proxyTransport}, nil }

Why PatchService method is used for deleting finalizers in cloud-controller-manager, This method updates with service/status, It seems to go unnoticed by the API server, resulting in the API server being unaware when the finalizer field on the service is deleted, leading to a delay in the release of service-related NodePorts. service nodeport be released promptly when service is deleted. create a loadblancer service with nodeport seted,then delete it and recreate. cloud-controller-manager add a finalizer on service. delete this service and then cloud-controller remove finalizer. but nodeport is not released when service deleted. ref i change like this, it seems like releasing nodeport when service is deleted : No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
Are you sure? that method updates the status and remove the finalizer ... otherwise , who removes the finalizer?
I apologize for any confusion. What I meant was that the update of the status and the removal of the finalizer are performed through updating service/status. Therefore, removing the finalizer does not release resources.It seems like removal of the finalizer by service/status, store.AfterDelete is nil. i think remove the finalizer should by patch servcie and updates the status should by patch service/status.Now we patch both by servcie/status.
no need to apologize, let me see if I can reproduce this locally first, Services api allocation is complex , it overrides the standard AfterDelete and release all allocated resources there /assign
/sig network ack the problem
will look for a possibly-related past cloud provider issue.
this will be fixed by , it is a problem in the apiserver logic
Cool! I forgot that you can patch finalizers thru /status. I repro'ed it on both ClusterIP and NodePort.

Saw this in the logs: So the repair loop, yet again, saves our butt but also hides a bug.
we need metrics here about to have visibility of the repairs
I also found this problem in versions 1.23 and 1.27
yep, I started the backports , but we can only backport to the supported versions, 1.23 and 1.24 are EOL


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package architecture import ( "time" "github.com/onsi/ginkgo" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" ) var _ = SIGDescribe("Conformance Tests", func() { f := framework.NewDefaultFramework("conformance-tests") /* Release: v1.23 Testname: Conformance tests minimum number of nodes. Description: Conformance tests requires at least two untainted nodes where pods can be scheduled. */ framework.ConformanceIt("should have at least two untainted nodes", func() { ginkgo.By("Getting node addresses") framework.ExpectNoError(framework.WaitForAllNodesSchedulable(f.ClientSet, 10*time.Minute)) nodeList, err := e2enode.GetReadySchedulableNodes(f.ClientSet) framework.ExpectNoError(err) if len(nodeList.Items) < 2 { framework.Failf("Conformance requires at least two nodes") } }) }) </ins>

I can create a cluster with one node and pass conformance, ~there are some tests that require at least 2 nodes to run~ ~~ ~This is because the tests Skip if they are not able to run~ ~~ ~As consequence, the list of tests for conformance depends on the number of nodes, a single cluster node (that is not a cluster by definition of cluster :)) will run less tests than a multi cluster node and still be Conformant~ EDIT I have to correct myself, it seems that the conformance tests doesn't skip, the problem is: there are no conformance tests that exercise anything that requires more than one node, hence, single node cluster can pass conformance. I expect a cluster (a group of similar things or people positioned or occurring closely together) and a "distributed system" to not be able to be single-node. In addition, lots of most important Kubernetes features can not be exercised on one node, lowering the quality bar. local- pass conformance Related discussion <details</details<details</details<details</details<details</details<details</details<details</details
/assign Maybe this has changed, but last time I've checked the minimum for conformance was 1 control-plane and 2 workers, if that didn't change I will submit a PR to enforce it in the suite Regarding sig-network, with one node you can not verify the networking model, where pod to pod in different nodes should be able to communicate without NAT, that is a strong requirement ... I have to review closely what tests do we have in Conformance
/area conformance /sig network
that seems pretty intentional... changing the definition of conformance to require multiple nodes probably deserves more feedback than an issue/PR
I'm not changing the definition, just asserting it
if we are testing features like replicaset, daemonset, services, ..., in one node only we are missing a lot of testing on those features ... a podman/docker single node can be a kubernetes conformance cluster ... do I need a scheduler or network for anything, I'm just deploying always in the same node?
No I don't think so. Where are your results and configuration? You need two schedulable nodes to pass the conformance suite. That's not correct, we test rolling pod behavior with replicasets. This test requires multiple nodes. There is at least one.

That job is not passing conformance, from the configuration: (see: ) Conformance requires not setting --, you must run all tests without skipping. We don't, at the very least replicaset is being tested multi-node, just not in this job that is not running the full conformance suite. When conformance results are submitted they are now validated to ensure that all tests were run.


Question: for _, pv := range item.pvs { client.CoreV1().PersistentVolumes().Create(ctx, pv, metav1.CreateOptions{}) } <del> caches := informerFactory.WaitForCacheSync(ctx.Done()) for _, synced := range caches { if !synced { t.Errorf("error waiting for informer cache sync") } } // Verify </del> <ins> t.Log("Start informer factory after initialization") informerFactory.Start(ctx.Done()) t.Log("Wait for all started informers' cache were synced") informerFactory.WaitForCacheSync(ctx.Done()) t.Log("Verify") </ins> p := pl.(*VolumeBinding) nodeInfo := framework.NewNodeInfo() nodeInfo.SetNode(item.node) state := framework.NewCycleState() <del> t.Logf("call PreFilter and check status") </del> <ins> t.Logf("Verify: call PreFilter and check status") </ins> gotPreFilterStatus := p.PreFilter(ctx, state, item.pod) if !reflect.DeepEqual(gotPreFilterStatus, item.wantPreFilterStatus) { t.Errorf("filter prefilter status does not match: %v, want: %v", gotPreFilterStatus, item.wantPreFilterStatus)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: The unit test Which test(s) are flaking: Testgrid link: Reason for failure: In volume scheduling, we maintain internal caches that watch for PV/PVC events. However, we don't have a method to wait for them to be synced in UT. When the cache in the informer factory is synced but the execution of registered handlers may not finish. <!-- In production, this is not a problem. Even if the scheduler marks the pod uncacheable due to the stale cache in PVAssumeCache/PVCAssumeCache, the pod will be moved to the active queue later as the scheduler . related: --Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig scheduling /sig storage
Even if I check the PV/PVC assume cache and wait for PV/PVC objects to be into the cache, the PV/PVC objects may not be . It seems that the shared informer with the fake client has a bug that registered event handlers may not be called (or handlers are not registered successfully).
The share informer with the fake client may miss events of objects which are created before caches are synced. it's better to create testing objects before starting the informers
Although the event handlers may not finish when the caches were synced in theory, however, it seems it's not necessary to wait for PV/PVC assume cache to be synced practically. I've run a thousand times (parallel: 10) with locally and no failures.


Question: expected: &dockercontainer.HostConfig{ NetworkMode: dockercontainer.NetworkMode(sandboxNSMode), IpcMode:     dockercontainer.IpcMode(sandboxNSMode), <ins> PidMode:     namespaceModeHost, </ins> }, }, }

Kubernetes version (use ): 1, 1.6.0 (both run into this issue) Environment: Cloud provider or hardware configuration: VirtualBox 5.1.18 VM OS: 16.04.2 LTS (Xenial Xerus) Kernel: Install tools: kubeadm What happened: A pod scheduled using hostPID wasn't using the host's PID namespace. I've investigated this issue together with What you expected to happen: I expected in the Docker container's config. Instead, the scheduled pod's container had How to reproduce it: Use this config to get a pod running a Docker container: Use on the running container and check whether PidMode is set to host or not. Anything else we need to know*: This was working properly on Kubernetes 1.5.3.
doesn't look like you've specified Docker version, it may be crucial here.
I've tested this with Docker 17.03.1-ce and can confirm that it's the case for pods.
It was Docker 1.12.6 from what I know. I'll check again to be sure it was 1.12.6 and update the post.
Is this with the CRI? If you disable the CRI do you get the desired outcome?
I don't know if that's enabled by default. I've noticed that hostPID doesn't work as it should with Docker. I'm not familiar with CRI.
I'll test with .
With CRI disabled it doesn't work on Docker 17.03, simply because it doesn't like the new versioning scheme. it'd be really great if you can try setting in your , e.g. by modifying (will need to do , and after that you either reinitialise the cluster or kill all the pods).
Seems a bug of kuberuntime, will file a PR to fix this.
Filed for this.
Reoppened this issue to ensure we cherrypick the fix to the 1.6 branch
has been merged to the release-1.6 branch. From what has been discussed so far, this will make it to Kubernetes 1.6.2. I'll close this issue now.
Can anyone help how this issue is fixed? I tried with pidMode: host but it is failing. I am using i.11 version.
I found it --hostPID: true


Question: "fmt" "math" "reflect" <del> "sync" </del> "testing" "github.com/google/go-cmp/cmp" dto "github.com/prometheus/client_model/go" "k8s.io/component-base/metrics" <del> "k8s.io/component-base/metrics/legacyregistry" </del> "k8s.io/utils/pointer" )

TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in


Question: "github.com/spf13/pflag" "k8s.io/apimachinery/pkg/util/duration" <ins> "k8s.io/klog/v2" </ins> kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmscheme "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme"

When is set to a value other than the default value (/etc/kubernetes/pki/), and run on the working kubernetes controller node, the cert is not renewed as expected. This impacts the as well. or work as expected. Here is an example which show cert renew doesn't work, create a cert,and then renew the cert,check if the cert is renewed, It actually renewed the cert in the default path(/etc/kubernetes/pki/), The issue can be revealed by the test as well, NOTE This issue only exist with a working cluster since the code will fall back to use the cluster configuration when config file is not specified. And if the cluster cannot be connected with the , the will be honored, this is why the test can pass with the CI jobs. Check the log printed by this test pr: No response <details</details<detailslocal </details<details</details<details</details<details</details<details</details
/assign
/sig cluster-lifecycle
/area kubeadm
/triage accepted


Question: # Disable prompts from apt. ENV DEBIAN_FRONTEND noninteractive <del> # Install prerequisites. RUN apt-get update &&  apt-get install -y -q curl make g++ &&  apt-get clean &&  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # Install Fluentd. RUN /usr/bin/curl -L https://td-toolbelt.herokuapp.com/sh/install-ubuntu-trusty-td-agent2.sh | sh # Change the default user and group to root. # Needed to allow access to /var/log/docker/... files. RUN sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent # Install the Elasticsearch Fluentd plug-in. RUN td-agent-gem install fluent-plugin-kubernetes_metadata_filter fluent-plugin-elasticsearch </del> # Copy the Fluentd configuration file. COPY td-agent.conf /etc/td-agent/td-agent.conf <ins> COPY build.sh /tmp/build.sh RUN /tmp/build.sh </ins> # Run the Fluentd service. ENTRYPOINT ["td-agent"]

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: continue fi <del> echo "Deleting security group: ${sg_id}" $AWS_CMD delete-security-group --group-id ${sg_id} > $LOG </del> <ins> delete_security_group ${sg_id} </ins> done subnet_ids=$($AWS_CMD describe-subnets

See example log below. I'm pretty sure the SG in question was from an ELB, but we should just auto-retry anyway.


Question: namespace := nsObj.(*api.Namespace) <ins> // Ensure we have a UID precondition if options == nil { options = api.NewDeleteOptions(0) } if options.Preconditions == nil { options.Preconditions = &api.Preconditions{} } if options.Preconditions.UID == nil { options.Preconditions.UID = &namespace.UID } else if *options.Preconditions.UID != namespace.UID { err = apierrors.NewConflict( api.Resource("namespaces"), name, fmt.Errorf("Precondition failed: UID in precondition: %v, UID in object meta: %v", *options.Preconditions.UID, namespace.UID), ) return nil, err } </ins> // upon first request to delete, we switch the phase to start namespace termination <ins> // TODO: enhance graceful deletion's calls to DeleteStrategy to allow phase change and finalizer patterns </ins> if namespace.DeletionTimestamp.IsZero() { <del> now := unversioned.Now() namespace.DeletionTimestamp = &now namespace.Status.Phase = api.NamespaceTerminating result, _, err := r.status.Update(ctx, namespace) return result, err </del> <ins> key, err := r.Store.KeyFunc(ctx, name) if err != nil { return nil, err } preconditions := storage.Preconditions{UID: options.Preconditions.UID} out := r.Store.NewFunc() err = r.Store.Storage.GuaranteedUpdate( ctx, key, out, false, &preconditions, storage.SimpleUpdate(func(existing runtime.Object) (runtime.Object, error) { existingNamespace, ok := existing.(*api.Namespace) if !ok { // wrong type return nil, fmt.Errorf("expected *api.Namespace, got %v", existing) } // Set the deletion timestamp if needed if existingNamespace.DeletionTimestamp.IsZero() { now := unversioned.Now() existingNamespace.DeletionTimestamp = &now } // Set the namespace phase to terminating, if needed if existingNamespace.Status.Phase != api.NamespaceTerminating { existingNamespace.Status.Phase = api.NamespaceTerminating } return existingNamespace, nil }), ) if err != nil { err = storageerr.InterpretGetError(err, api.Resource("namespaces"), name) err = storageerr.InterpretUpdateError(err, api.Resource("namespaces"), name) if _, ok := err.(*apierrors.StatusError); !ok { err = apierrors.NewInternalError(err) } return nil, err } return out, nil </ins> } // prior to final deletion, we must ensure that finalizers is empty

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: } s.AddUnversionedTypes(gv, &InternalSimple{}) <ins> s.AddUnversionedTypes(gv, &InternalSimple{}) </ins> if len(s.KnownTypes(gv)) != 1 { t.Errorf("expected only one %v type after double registration with custom name", gv) }

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: }) /* <del> Testname: var-expansion-subpath-with-absolute-path Description: Make sure a container's subpath can not be set using an expansion of environment variables when absolute path is supplied. </del> <ins> Release : v1.19 Testname: VolumeSubpathEnvExpansion, subpath with absolute path Description: Make sure a container's subpath can not be set using an expansion of environment variables when absolute path is supplied. </ins> */ <del> ginkgo.It("should fail substituting values in a volume subpath with absolute path [sig-storage][Slow]", func() { </del> <ins> framework.ConformanceIt("should fail substituting values in a volume subpath with absolute path [sig-storage][Slow]", func() { </ins> podName := "var-expansion-" + string(uuid.NewUUID()) pod := &v1.Pod{

Is this a BUG REPORT or FEATURE REQUEST?: /kind api-change /sig storage /sig node /sig testing Description: Promote e2e tests for VolumeSubpathEnvExpansion to Conformance as it is GA approved. Related Issues: GA Upgrade: Feature Request: KEP:
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } func (detacher *fcDetacher) UnmountDevice(deviceMountPath string) error { <del> return volumeutil.UnmountPath(deviceMountPath, detacher.mounter) </del> <ins> // Specify device name for DetachDisk later devName, _, err := mount.GetDeviceNameFromMount(detacher.mounter, deviceMountPath) if err != nil { glog.Errorf("fc: failed to get device from mnt: %snError: %v", deviceMountPath, err) return err } // Unmount for deviceMountPath(=globalPDPath) err = volumeutil.UnmountPath(deviceMountPath, detacher.mounter) if err != nil { return fmt.Errorf("fc: failed to unmount: %snError: %v", deviceMountPath, err) } unMounter := volumeSpecToUnmounter(detacher.mounter) err = detacher.manager.DetachDisk(*unMounter, devName) if err != nil { return fmt.Errorf("fc: failed to detach disk: %snError: %v", devName, err) } glog.V(4).Infof("fc: successfully detached disk: %s", devName) return nil </ins> } func volumeSpecToMounter(spec *volume.Spec, host volume.VolumeHost) (*fcDiskMounter, error) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When user uses fc volume plugin, device file on operating system such as /dev/sdX remains after deleting a Pod. After deleting the Pod, PVC and PV, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up. The photon volume plugin already has this functionality in the removeFromScsiSubsystem(). Also the gce volume plugin has similar functionality in the udevadmChangeToDrive() too. photon: echo 1 /sys/block/<dev name/device/delete gce: udevadm trigger --action=change --property-match=DEVNAME=%s Also os-brick library in the OpenStack Cinder component uses same way with photon driver. What you expected to happen: The device file on operating system such as /dev/sdX will be cleaned up after deleting PVs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: KVM OS (e.g. from /etc/os-release): Fedora 25 Kernel (e.g. ):Install tools: local-up- Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
I'll work on this.
Reiterating the mentions to trigger a notification: <detailsIn response to : I'll work on this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: dbus = utildbus.New() iptInterface = utiliptables.New(execer, dbus, protocol) <del> ipvsInterface = utilipvs.New(execer) </del> kernelHandler = ipvs.NewLinuxKernelHandler() ipsetInterface = utilipset.New(execer) <ins> if canUse, _ := ipvs.CanUseIPVSProxier(kernelHandler, ipsetInterface); canUse { ipvsInterface = utilipvs.New(execer) } </ins> // We omit creation of pretty much everything if we run in cleanup mode if cleanupAndExit {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube proxy report following error even if ipvs not enabled. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Install kubernetes and start proxy. Anything else we need to know?: Environment: Kubernetes version (use ): We are using 1.9.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /cc comments?
/sig network
It's a noise, but it's thrown by libnetwork/ipvs util. Is that a big deal?
please take a look when you have a chance :)
Yes, it always bring noise for production when customer got warning messages. Why printing some ipvs message even it was not enabled?
/area ipvs


Question: ) func newTestDockerSevice() (*dockerService, *dockertools.FakeDockerClient, *clock.FakeClock) { <del> c := dockertools.NewFakeDockerClient() </del> fakeClock := clock.NewFakeClock(time.Time{}) <ins> c := dockertools.NewFakeDockerClientWithClock(fakeClock) </ins> return &dockerService{client: c}, c, fakeClock }

Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:


Question: <ins> package(default_visibility = ["//visibility:public"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", ) go_library( name = "go_default_library", srcs = ["metrics.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/server/metrics", deps = [ "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: type netlinkHandle struct { netlink.Handle <ins> isIPv6 bool </ins> } // NewNetLinkHandle will crate a new NetLinkHandle <del> func NewNetLinkHandle() NetLinkHandle { return &netlinkHandle{netlink.Handle{}} </del> <ins> func NewNetLinkHandle(isIPv6 bool) NetLinkHandle { return &netlinkHandle{netlink.Handle{}, isIPv6} </ins> } // EnsureAddressBind checks if address is bound to the interface and, if not, binds it. If the address is already bound, return true.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area ipv6 /area ipvs /sig network What happened: In an ipv6-only cluster with proxy-mode=ipvs the entries for NodePort in ipvs are created for ipv4 addresses. The target's (if there are any) consists of ipv6 addresses (wrongly) interpreted as ipv4; Attempts to use the NodePort with ipv6 addresses are black-holed. What you expected to happen: Ipv6 addresses to the nodes should be used for NotePort and the LB-targets shall be the ipv6 address of the pods. Traffic to NodePorts shall be distributed among the (ipv6) endpoints. How to reproduce it (as minimally and precisely as possible): Setup an ipv6-only cluster Create a service with type with endpoints Try to connect to the NodePort Do Anything else we need to know?: The nodes have both ipv4 and ipv6 addresses and it is possible that if the nodes only had ipv6 addresses it would work (I havn't tested). But even in an ipv6-only k8s cluster the nodes almost always need ipv4 addresses for other purposes. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm VMs on Ubuntu 18.04.01 LTS OS (e.g. from /etc/os-release): Own BusyBox based system Kernel (e.g. ):Install tools: None Others:
The NodeIPs() function seem to return only ipv4 addresses.
I can't see why (yet) since is used in the GetLocalAddresses function
/cc Please give a hint before I dive into the netlink code.
This issue is used as an example here;
The functions filters on Src != nil . But ipv6 addresses seems to have Src == nil always. As it is this function will never return any ipv6 addresses.


Question: createTagFactor       = 2.0 createTagSteps        = 9 <del> // encryptedCheck* is configuration of poll for created volume to check // it has not been silently removed by AWS. </del> <ins> // volumeCreate* is configuration of exponential backoff for created volume. </ins> // On a random AWS account (shared among several developers) it took 4s on <del> // average. encryptedCheckInterval = 1 * time.Second encryptedCheckTimeout  = 30 * time.Second </del> <ins> // average, 8s max. volumeCreateInitialDelay  = 5 * time.Second volumeCreateBackoffFactor = 1.2 volumeCreateBackoffSteps  = 10 </ins> // Number of node names that can be added to a filter. The AWS limit is 200 // but we are using a lower limit on purpose

What happened: When creating an EBS volume, we're currently waiting for the volume be in the correct state before proceeding: However, we're using the interval defined in (e.g., 1 second), which was meant to be used for another reason and might be too soon for this specific case. Also, this might contribute to the API quota being exceeded. CC /sig storage /sig aws
Yeah I think, this should be considered a regression. Many of our AWS clusters run on very thin margin of API quota and this would tip them over the edge.


Question: } // New creates a cAdvisor and exports its API on the specified port if port > 0. <del> func New(port uint, runtime string, rootPath string) (Interface, error) { </del> <ins> func New(address string, port uint, runtime string, rootPath string) (Interface, error) { </ins> sysFs := sysfs.NewRealSysFs() // Create and start the cAdvisor container manager.

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: // given a wwn and lun, find the device and associated devicemapper parent func findDisk(wwn, lun string, io ioHandler, deviceUtil volumeutil.DeviceUtil) (string, string) { <del> fcPath := "-fc-0x" + wwn + "-lun-" + lun </del> <ins> fcPathExp := "^(pci-.*-fc|fc)-0x" + wwn + "-lun-" + lun r := regexp.MustCompile(fcPathExp) </ins> devPath := byPath if dirs, err := io.ReadDir(devPath); err == nil { for _, f := range dirs { name := f.Name() <del> if strings.Contains(name, fcPath) { </del> <ins> if r.MatchString(name) { </ins> if disk, err1 := io.EvalSymlinks(devPath + name); err1 == nil { dm := deviceUtil.FindMultipathDeviceForDevice(disk) klog.Infof("fc: find disk: %v, dm: %v", disk, dm)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently, we have logic here to look for the disk names with but this fails when we have devices which are starting with e.g: on ppc64le architecture, NPIV based fc disks are discovered as follows: and here is the udev information for one of the disk: Because of the existing logic k8s fails to detect the disk because it does not start with instead starts with , so pod creation fails with the following error: What you expected to happen: FC disk should get attached properly to the pod How to reproduce it (as minimally and precisely as possible): sample pod yaml on a machine with path names starts with Anything else we need to know?: Environment:
/sig storage
/triage accepted


Question: BackendIPConfigurations: &[]network.InterfaceIPConfiguration{ { Name: to.StringPtr("ip-1"), <del> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg1/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/networkInterfaces/nic"), </del> <ins> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg1/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-0/virtualMachines/0/networkInterfaces/nic"), </ins> }, }, }, }, } <del> expectedVMSS := buildTestVMSS([]string{testLBBackendpoolID0, testLBBackendpoolID1}) expectedVMSSVMs, _, _ := buildTestVirtualMachineEnv(ss.cloud, testVMSSName, "", 0, []string{"vmss-vm-000000"}, "succeeded") vmssVMNetworkConfigs := expectedVMSSVMs[0].NetworkProfileConfiguration vmssVMIPConfigs := (*vmssVMNetworkConfigs.NetworkInterfaceConfigurations)[0].VirtualMachineScaleSetNetworkConfigurationProperties.IPConfigurations lbBackendpools := (*vmssVMIPConfigs)[0].LoadBalancerBackendAddressPools *lbBackendpools = append(*lbBackendpools, compute.SubResource{ID: to.StringPtr(testLBBackendpoolID1)}) </del> <ins> vmss0 := buildTestVMSS("vmss-0", []string{testLBBackendpoolID0, testLBBackendpoolID1}) vmss1 := buildTestVMSS("vmss-1", []string{testLBBackendpoolID0, testLBBackendpoolID1}) expectedVMSSVMsOfVMSS0, _, _ := buildTestVirtualMachineEnv(ss.cloud, "vmss-0", "", 0, []string{"vmss-vm-000000"}, "succeeded") expectedVMSSVMsOfVMSS1, _, _ := buildTestVirtualMachineEnv(ss.cloud, "vmss-1", "", 0, []string{"vmss-vm-000001"}, "succeeded") for _, expectedVMSSVMs := range [][]compute.VirtualMachineScaleSetVM{expectedVMSSVMsOfVMSS0, expectedVMSSVMsOfVMSS1} { vmssVMNetworkConfigs := expectedVMSSVMs[0].NetworkProfileConfiguration vmssVMIPConfigs := (*vmssVMNetworkConfigs.NetworkInterfaceConfigurations)[0].VirtualMachineScaleSetNetworkConfigurationProperties.IPConfigurations lbBackendpools := (*vmssVMIPConfigs)[0].LoadBalancerBackendAddressPools *lbBackendpools = append(*lbBackendpools, compute.SubResource{ID: to.StringPtr(testLBBackendpoolID1)}) } </ins> mockVMSSClient := ss.cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface) <del> mockVMSSClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachineScaleSet{expectedVMSS}, nil).AnyTimes() mockVMSSClient.EXPECT().Get(gomock.Any(), ss.ResourceGroup, testVMSSName).Return(expectedVMSS, nil).MaxTimes(2) mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any()).Return(nil).Times(2) </del> <ins> mockVMSSClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachineScaleSet{vmss0, vmss1}, nil).AnyTimes() mockVMSSClient.EXPECT().Get(gomock.Any(), ss.ResourceGroup, "vmss-0").Return(vmss0, nil).MaxTimes(2) mockVMSSClient.EXPECT().Get(gomock.Any(), ss.ResourceGroup, "vmss-1").Return(vmss1, nil).MaxTimes(2) mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), ss.ResourceGroup, gomock.Any(), gomock.Any()).Return(nil).Times(2) </ins> mockVMSSVMClient := ss.cloud.VirtualMachineScaleSetVMsClient.(*mockvmssvmclient.MockInterface) <del> mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() mockVMSSVMClient.EXPECT().UpdateVMs(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any(), gomock.Any()).Return(nil).Times(2) </del> <ins> mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, "vmss-0", gomock.Any()).Return(expectedVMSSVMsOfVMSS0, nil).AnyTimes() mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, "vmss-1", gomock.Any()).Return(expectedVMSSVMsOfVMSS1, nil).AnyTimes() mockVMSSVMClient.EXPECT().UpdateVMs(gomock.Any(), ss.ResourceGroup, gomock.Any(), gomock.Any(), gomock.Any()).Return(nil).Times(2) </ins> backendpoolAddressIDs := []string{testLBBackendpoolID0, testLBBackendpoolID1, testLBBackendpoolID2} <ins> testVMSSNames := []string{"vmss-0", "vmss-1", "vmss-0"} </ins> testFunc := make([]func() error, 0) <del> for _, id := range backendpoolAddressIDs { </del> <ins> for i, id := range backendpoolAddressIDs { i := i </ins> id := id testFunc = append(testFunc, func() error { <del> return ss.EnsureBackendPoolDeleted(&v1.Service{}, id, testVMSSName, backendAddressPools) </del> <ins> return ss.EnsureBackendPoolDeleted(&v1.Service{}, id, testVMSSNames[i], backendAddressPools) </ins> }) } errs := utilerrors.AggregateGoroutines(testFunc...)

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: app: federated-cluster module: federation-controller-manager spec: <ins> volumes: - name: ssl-certs hostPath: path: /etc/ssl/certs </ins> containers: - name: controller-manager <ins> volumeMounts: - name: ssl-certs readOnly: true mountPath: /etc/ssl/certs </ins> image: {{.FEDERATION_CONTROLLER_MANAGER_IMAGE_REPO}}:{{.FEDERATION_CONTROLLER_MANAGER_IMAGE_TAG}} command: - /usr/local/bin/federation-controller-manager

Current the federation-controller-manager docker images does not include a root ca cert bundle which causes syncing with DNS providers to fail: To fix this issue I created my own container image using the following Dockerfile: This resulted in a working image with a root cert bundle. Syncing with the GCE DNS provider now works.
cc: FYI...
I'm willing to grab this tomorrow.
Cool, thanks . Re-assigning.


Question: t.Errorf("The code did panic") } }() n.OnNodeAdd(node) }) } } func TestNodePodCIDRHandlerUpdate(t *testing.T) { <ins> oldKlogOsExit := klog.OsExit defer func() { klog.OsExit = oldKlogOsExit }() klog.OsExit = customExit </ins> tests := []struct { name            string oldNodePodCIDRs []string

There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. As suggested by in , we can replace by /sig network /help /kind cleanup
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. /help /kind cleanup </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to raise a PR to fix this.


Question: if containerRequest, ok := c.Resources.Requests[resource]; ok { podSum += containerRequest.MilliValue() } else { <del> return nil, fmt.Errorf("missing request for %s", resource) </del> <ins> return nil, fmt.Errorf("missing request for %s in container %s of Pod %s", resource, c.Name, pod.ObjectMeta.Name) </ins> } } }

What happened: HPA always has a target of and events that say: There is a single container in the pods and it has resource requests and limits set. The metrics-server is running All pods have metrics show in All pods have metrics in Here's the HPA in YAML: What you expected to happen: No in HPA target How to reproduce it (as minimally and precisely as possible): I can't be sure. It's only a single HPA in our cluster. 10 other HPAs are working OK. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.6 Cloud provider or hardware configuration: EKS
/sig autoscaling
could you please share the yaml content of the pod? It seems the pod's cpu request is not set.
Sure. Here's from the deployment (): This shows there's only a single container in these pods. Here's a list of pods: And resources from all pods (): That just repeats 14 times, once for each pod. And then the 3 completed pods show as:
Ahhh, I deleted those pods and suddenly the HPA is back in action:
But these pods are not from the deployment that is specified in the HPA, they are created from a . Sure they don't have set but they should be ignored by the HPA, right? Here's the pod JSON from one: I will test creating more pods WITHOUT set and see if the issue returns. And then test creating more pods WITH and see if it's OK.
It is a little weird. How could I reproduce it?
HPA use only selector to filter pods without checking ownership, I think this is a horrible mistake!
It might be the cause of this issue. Would you like to send a PR for it? If not, I could try to fix it. Could you share the yaml of and , especially the labels part? cc
I am not working on this. :)
Here the job: And here's the deployment: I deleted all the pods and the HPA started working again. And then when new pods are present, I see these events:
Maybe worth noting that even though I got these events and the HPA is saying it still didn't transition to after 10 minutes. But if I create the HPA while there is these pods then it stays in state.

Any update
... I am not working on this issue.... are you working on this? Hope you dit not get an impression that I am working on this.....
Ah sorry, my mistake!
Ah I missed that :( I'll try to fix it.
Experiencing this same issue
I checked the code, but not sure whether it is intended to get pods list just by labels and not check the owner reference. Could you please help confirm it? Thanks! If it is not intended, I could send a PR to fix it.
I get the same issue: Here is my HPA spec: Wonder why this only happens to ONE StatefulSet but not the others?
Hi, Unfortunately, I am also facing the same: Warning FailedGetResourceMetric 3m59s (x35656 over 6d5h) horizontal-pod-autoscaler missing request for cpu. Any idea how to fix this?

I submitted a tentative PR
Configured HPA as and Deployment CPU resources as Cluster details
Experiencing the same issue: apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: dontdeletedcshpa namespace: docu spec: maxReplicas: 5 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: StatefulSet name: dontdeletedcs-pg targetCPUUtilizationPercentage: 80 kubectl describe hpa dontdeletedcshpa -n docu Name: dontdeletedcshpa Namespace: docu Labels: <noneAnnotations: <noneCreationTimestamp: Tue, 07 Jan 2020 12:14:55 +0530 Reference: StatefulSet/dontdeletedcs-pg Metrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown/ 80% Min replicas: 1 Max replicas: 5 StatefulSet pods: 3 current / 0 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedGetResourceMetric 8s horizontal-pod-autoscaler missing request for cpu Warning FailedComputeMetricsReplicas 8s horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu
I ran into this as well and this fixed it for me: I am running pods with more than one container. In my case, the other container is a linkerd sidecar. I was setting the resource requests and limits for my deployment but did not set resources for linkerd proxy. You must set resources for all containers within a pod otherwise you will get the error "failed to get cpu utilization". Maybe this error message could be updated? Hope this helps!
This worked for me. I had to add resource requests/limits to a sidecar container and remove "Completed" jobs from the namespace.
I had the same case with sidecars, setting request/limits for all containers fixed the problem. Thanks!
That's great but it's still a bug for people who only have a single container in their pods.

I've asked for reviews on
Encountered the same issue and was able to resolve it by making sure that the cronjob pod should be different from the deployment . In case deployment pod matchlabels cronjob pod labels
It worked. Thanks!
Using the new beta API seems to solve it
Experiencing the same issue with one container. any suggestion? didn't help
I have the same issue: Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: did not receive metrics for any ready pods I use aks with windows nodes. aks 1.15.5 only one id three pod is active. warnings: Warning FailedGetResourceMetric horizontalpodautoscaler/kcglob1-hpa did not receive metrics for any ready pods 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kcglob1-hpa Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: did not receive metrics for any ready pods
Running AKS 1.15.7 HPA on stable/nginx-ingress is unable to get cpu or mem metrics. This deployment has only one container and all resources are defined I tried with both and had the same result and Example HPA:
I removed the cluster, and rebuild it from scratch. The problem doesn't appear anymore.
Interesting. I was able to create an HPA for a simple sample deployment but Nginx is still unable to retrieve metrics. Looks like this is related to how HPA uses labels
same issue:the HPA was unable to compute the replica count: missing request for cpu. single container in a pod..
work form me !! thanks
I had a proxy container injected into the pods with no nor defined. Once I defined them and reinitiated the deployments hpa was happily working.
Thank you!! work form me! (I tried in a single container)
I have this same problem for while it works perfectly for my , i.e: As it turns out target is unknown for and exact reason for this according to is: *Environment: Kubernetes version (use kubectl version): 1.18.1 Cloud provider or hardware configuration: AKS
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
for the you can either set the same limits on the defaultBackend or probably use 's idea and set probably and/or specifically if the requests/limits are much higher on the controller pods and it would be a waste of resources
I suspect the reason most of us are here is our beloved nginx-ingress-controller fails to autoscale leading to many tears and much frustration. It seems the chart is giving same labels to the defaultbackend that overlap with the controller deployment causing this issue. Set this in your values yaml to get around it: Feel free to change the labels to whatever you want if you want to be more creative. :D Other option is to set: This is cleaner option, but you cannot set it to this without a maintenance if you are currently deployed because you need to redeploy the controller in this instance because the label is immutable.
see comments above, two methods seems useful, setting request/limits and delete completed status pod but i try both of them, not working i meet this issue for istio in the process of installing kubeflow [root ~]# kubectl get hpa -n istio-system NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE cluster-local-gateway Deployment/cluster-local-gateway unknown/80% 1 5 1 35m istio-ingressgateway Deployment/istio-ingressgateway unknown/80% 1 5 1 35m istio-pilot Deployment/istio-pilot unknown/80% 1 5 1 35m istio-policy Deployment/istio-policy unknown/80% 1 5 1 35m istio-telemetry Deployment/istio-telemetry unknown/80% 1 5 1 35m really need help
Setting explicit resource requests for each container in the pod worked for me.
Hi I want to do same can you help me how i can set resources for the linkerd proxy containers
I did the injection on the Namespaces level: Alternatively you could apply on the Deployment level:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I found the same issue and in my case the reason because the pod or pods are failing with the metrics is because the POD is not 100% ready... Check the healtchecks , security groups, etc. Here more info:
In my case, I had to add the request resource to the deployment to fix the issue
It works for me thanks!!
I`ve got this issue when all my pods were under one selector an I had to explicity fill resoures blocks for each pod affected this selector
Had same issue with a deployment that could not scale because of the error that the HPA of the deployment was showing. Finally got it fixed now. Here the reasons & background: My deployment consist of a Job that run at the beginning regular POD with three containers - two "sidecar" containers and one with the main app The "main app" container had "resources" set. Both "sidecar" containers had not. So first problem were the missing "resources" specs on both sidecar containers. Such behavior with multiple containers in the POD is described in The second problem was that the "Job" that ran before the actual app deployment has ALSO to have "resources" defined. And THAT was really unexpected. That is something where also stumbled upon & what i've tested then. - thanks for the hint :beer: So, TIL: enable "resources" on ALL containers/jobs in the POD
Also run into this. Had some old pods without requests which were in state (because of preemptible nodes). Deleting these pods got the HPA working.
Thank you so much <3 in minikube case just run "minikube start"
Encountered the same issue when attempting HPA based on CPU utilisation. Fixed the issue by setting CPU resource request at the Deployment level.
it worked for me. The cause is I used Helm to deploy the app along with post-install-job without resource limit. I confirm the Pod's containers must be set relevant resource request
Faced this same issue. In my case my pod container had limits set, but my Dapr injected sidecar container didn't.

I think (substitutes , proposed by approach that checks the GroupVersionKind of pod metrics is suboptimal. E.g. one may have two deployments that use the same selector and then HPA would get metrics for pods of both deployments, checking the kind would not work to tell them apart. See also - basically HPA, although it targets deployment by name, uses deployment selector labels to get pod's metrics.
I talked with and the underlying issue is that you can only query metrics-server by label. So our workaround is the way to go. ref:
My application is running on a pod with a as sidecar. I have solved this adding resource limits to the sidecar definition on the deployment but now I'm confused about how I should read the metrics. Should I read the value as an average of my containers? Also I have trying set this on my HPA definition but I have two issues: Documentation is not clear about how to refer the container in the pod It seems like there is a bug on this How can I set the metric based on my main container ignoring the sidecar?
Wondering if this issue should remain open, it seems what another user pointed out above could be the cause, and was true in my case:
I'm getting same issue, I have enable metric-server in minikube, when I create hpa its always says my deployment able to get scale, but not scale down even after hours, --------Edited------ I have tried my same deployment with kind cluster and its working fine, there is some issue with minikube
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten we keep seeing this with PSS on AKS, EKS. I suspect there is something wrong here. I didn't seen any evidence here tht this is being fixed/worked on? I might be wrong
Reading the discussion it seems to me the error message can have multiple causes, which adds to the confusion. IMO a good action item would be to make the message more detailed, e.g. pointing to which pod and container didn't have the requests set.
/assign


Question: // It is more extensive than IsLikelyNotMountPoint // and it detects bind mounts in linux func IsNotMountPoint(mounter Interface, file string) (bool, error) { <ins> // Resolve any symlinks in file, kernel would do the same and use the resolved path in /proc/mounts resolvedFile, err := mounter.EvalHostSymlinks(file) if err != nil { return true, err } </ins> // IsLikelyNotMountPoint provides a quick check // to determine whether file IS A mountpoint <del> notMnt, notMntErr := mounter.IsLikelyNotMountPoint(file) </del> <ins> notMnt, notMntErr := mounter.IsLikelyNotMountPoint(resolvedFile) </ins> if notMntErr != nil && os.IsPermission(notMntErr) { // We were not allowed to do the simple stat() check, e.g. on NFS with // root_squash. Fall back to /proc/mounts check below.

This is related to and , but these issues contain several different bugs, so I am creating a new one. Steps to reproduce: must be a symlink somewhere else. If you don't have a symlink, then it's a different bug. a pod with ConfigMap volume with SubPath: (the pod should run just fine) -The pod is Terminating forever From the logs: You can see there is no "Unmounting /var/lib/kubelet/pods/-ba83-11e8-983c-/volume-subpaths/vol/busybox/0", so kubelet did not even try to unmount it. Environment: Kubernetes version 1.9.x (and today's master). /kind bug /sig storage another subpath bug :-)
Posted , kubelet gets confused by /var/lib/kubelet being a symlinks and it looks for /var/lib/kubelet/pods/... in /proc/mounts instead of target of the symlink.


Question: <ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail run_convert_tests() { set -o nounset set -o errexit ### Convert deployment YAML file locally without affecting the live deployment # Pre-condition: no deployments exist kube::test::get_object_assert deployment "{{range.items}}{{${id_field:?}}}:{{end}}" '' # Command # Create a deployment (revision 1) kubectl create -f hack/testdata/deployment-revision1.yaml "${kube_flags[@]:?}" kube::test::get_object_assert deployment "{{range.items}}{{${id_field:?}}}:{{end}}" 'nginx:' kube::test::get_object_assert deployment "{{range.items}}{{${image_field0:?}}}:{{end}}" "${IMAGE_DEPLOYMENT_R1}:" # Command output_message=$(kubectl convert --local -f hack/testdata/deployment-revision1.yaml --output-version=apps/v1beta1 -o yaml "${kube_flags[@]:?}") # Post-condition: apiVersion is still apps/v1 in the live deployment, but command output is the new value kube::test::get_object_assert 'deployment nginx' "{{ .apiVersion }}" 'apps/v1' kube::test::if_has_string "${output_message}" "apps/v1beta1" # Clean up kubectl delete deployment nginx "${kube_flags[@]:?}" ## Convert multiple busybox PODs recursively from directory of YAML files # Command output_message=$(! kubectl-convert -f hack/testdata/recursive/pod --recursive 2>&1 "${kube_flags[@]:?}") # Post-condition: busybox0 & busybox1 PODs are converted, and since busybox2 is malformed, it should error kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" # check that convert command supports --template output output_message=$(kubectl-convert "${kube_flags[@]:?}" -f hack/testdata/deployment-revision1.yaml --output-version=apps/v1beta2 --template="{{ .metadata.name }}:") kube::test::if_has_string "${output_message}" 'nginx:' set +o nounset set +o errexit } </ins>

was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release


Question: }, []string{"group", "version", "kind"}, ) <del> // Because of volatality of the base metric this is pre-aggregated one. Instead of reporing current usage all the time </del> <ins> // Because of volatility of the base metric this is pre-aggregated one. Instead of reporting current usage all the time </ins> // it reports maximal usage during the last second. currentInflightRequests = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: import ( "fmt" "reflect" <ins> "strconv" </ins> "strings" "testing" "time"

Kubernetes version Code from master. What happened: does not return and fields after they are set to non-nil values using . How to reproduce it (as minimally and precisely as possible): See PR
/sig api-machinery
This is a good find and I don't think I saw the notification come through :(


Question: // SkipUnlessSSHKeyPresent skips if no SSH key is found. func SkipUnlessSSHKeyPresent() { <del> if _, err := e2essh.GetSigner(TestContext.Provider); err != nil { skipInternalf(1, "No SSH Key for provider %s: '%v'", TestContext.Provider, err) </del> <ins> if _, err := e2essh.GetSigner(framework.TestContext.Provider); err != nil { skipInternalf(1, "No SSH Key for provider %s: '%v'", framework.TestContext.Provider, err) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. They are skipped because they require SSH access to drop the Flexvolume plugin onto the host, but these test clusters do not have SSH access. We should change all Flexvolume tests to use the DaemonSet installation mechanism instead which does not require SSH access. See and Flexvolume tests I'm aware of are below, but there could be more: Looking for help with this bug! /sig storage /priority important-soon /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. /priority important-soon /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Can I work on this issue? I think i know how to handle it.
are you still working on this? I'm interested in helping out as well.
/assign


Question: return "", err } <ins> bearerToken, err := framework.GenerateSecureToken(16) if err != nil { return "", err } </ins> // Launch kubelet. <del> if err := launchKubelet(host, workspace, results, testArgs); err != nil { </del> <ins> if err := launchKubelet(host, workspace, results, testArgs, bearerToken); err != nil { </ins> return "", err } // Stop kubelet.

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: serviceParam: &v1.Service{ObjectMeta: metav1.ObjectMeta{Name: "svc1", Namespace: "ns1"}}, slicesParam:  []*discovery.EndpointSlice{epSlice1NewerGen}, expectNewer:  false, <ins> }, { name: "slice in params is expected to be deleted", tracker: &endpointSliceTracker{ generationsByService: map[types.NamespacedName]generationsBySlice{ {Name: "svc1", Namespace: "ns1"}: { epSlice1.UID: deletionExpected, }, }, }, serviceParam: &v1.Service{ObjectMeta: metav1.ObjectMeta{Name: "svc1", Namespace: "ns1"}}, slicesParam:  []*discovery.EndpointSlice{epSlice1}, expectNewer:  true, }, { name: "slice in tracker but not in params", tracker: &endpointSliceTracker{ generationsByService: map[types.NamespacedName]generationsBySlice{ {Name: "svc1", Namespace: "ns1"}: { epSlice1.UID: epSlice1.Generation, }, }, }, serviceParam: &v1.Service{ObjectMeta: metav1.ObjectMeta{Name: "svc1", Namespace: "ns1"}}, slicesParam:  []*discovery.EndpointSlice{}, expectNewer:  true, </ins> }} for _, tc := range testCases {

The controllers can create a second EndpointSlice after an immediate sync if the first EndpointSlice they created is not in the cache yet. Here's what can happen: gets event that triggers sync for Service syncs and creates EndpointSlice gets event that triggers sync for same Service syncs and creates EndpointSlice (the slice created in 2 has not made it to the cache yet) A single EndpointSlice to be created. Run integration tests hundreds of times that involve 2 consecutive events that will trigger a sync. Example: /assign /sig network
/triage accepted
/cc
When this bug is fixed, we'll need to rollback
just amend it in the PR that fix the bug, so we merge all together


Question: # Prepare a RBD volume "foo" (only with layering feature, the others may # require newer clients). # NOTE: we need Ceph kernel modules on the host that runs the client! <ins> # As the default pool `rbd` might not be created on arm64 platform for ceph deployment, # should create it if it does not exist. arch=$(uname -m) if [[ ${arch} = 'aarch64' || ${arch} = 'arm64' ]]; then if [[ $(ceph osd lspools) = "" ]]; then ceph osd pool create rbd 8 rbd pool init rbd fi fi </ins> rbd import --image-feature layering block foo # Prepare a cephfs volume

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I run some e2e storage testcases locally, below testcases as an example, I consistently saw the below error message, ceph should be deployed successfully and testcase should pass. Run the testcases mentioned above. e.g. Kubernetes version (use ): ALL Cloud provider or hardware configuration: ARM64 OS (e.g: ): ubuntu18 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /sig storage
/assign
is this happening in CI? please provide links to occurences?
Hey, Ben, I update the issue description a little bit, this won't be an issue on X86 which is currently the default arch CI is running, but tons of testcases will fail on ARM64 due to this issue. Eventually, we hope the k8s will have a ARM CI along with the more adoption of arch beside the X86, and thus, those e2e testcases should workable on ARM64 firstly. I will explain why it failed on ARM64 in the PR desc, pls help to review, thanks!
/triage accepted


Question: podutil.DropDisabledPodFields(pod, nil) <del> applyWaitingForSchedulingGatesCondition(pod) </del> <ins> applySchedulingGatedCondition(pod) </ins> mutatePodAffinity(pod) }

<img width="844" alt="image" src="" <img width="942" alt="image" src="" Here are some incorrect comments and function name about SchedulingGated Comments and function names need to be changed to SchedulingGated nothing No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
/remove-kind bug /kind cleanup


Question: // TODO: get rid of these tests or port them to secure serving genericConfig.SecureServing = &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}} <ins> // if using endpoint reconciler the service subnet IP family must match the Public address genericConfig.PublicAddress = netutils.ParseIPSloppy("10.1.1.1") </ins> err = etcdOptions.ApplyWithStorageFactoryTo(storageFactory, genericConfig) if err != nil { panic(err)

Clusters created with kubeadm using DualStack are not able to work, because CoreDNS pods never get ready Seeing one failed job logs It seems that the pod is "waiting for the kubernetes service"?? coredns version seems the same as in the working jobs CoreDNS pod should be ready Instal a Dual-stack cluster using master with kubeadm (or kind ) ! It started to fail on Oct12 <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc I didn't notice it was failing for more than 1 month :facewithhead_bandage: I didn't have time to dig into it yet, but this PR landed just that day how is that the pods running has the version 1.8.4 but the PR seems to update kubeadm to 1.8.6?
Looking into this today. The error from CoreDNS should only happen when the kubernetes plugin fails to sync with the Kubernetes API, either because it can’t connect, or it can’t sync the lists of services/endpointslices. There were changes and fixes to the API connection code, so it could be a regression.
Actually no. The changes I was thinking of were already in 1.8.4.
How can we reproduce it : Instal a Dual-stack cluster using master with kubeadm (or kind ) I was unable to reproduce this issue in a kind dual stack cluster.

After upgrading coredns to 1.8.6 (from 1.8.0 installed by kind), and adding list/watch permission for endpointslice, the pods were ready. edit: It also works for coredns 1.8.4 - I'm unclear as to which version I should be checking.
let me check if is a regression in kind :thinking:
working with 1.8.6 and yesterdays kubernetes podIPs
This is weird, I hit a similar issue in openshift, endpoints doesn't match the IP family of the kubernetes.default service apiserver service is single stack, at least until this KEP fixes it What I don't understand is why the order of the IPs in kind is different let me try to fix both things /assign
my fault, sorry for the noise :sweat_smile:


Question: testCannotConnect(f, f.Namespace, "client-a", service, allowedPort) }) <ins> ginkgo.It("should work with Ingress,Egress specified together [Feature:NetworkPolicy]", func() { const allowedPort = 80 const notAllowedPort = 81 protocolUDP := v1.ProtocolUDP nsBName := f.BaseName + "-b" nsB, err := f.CreateNamespace(nsBName, map[string]string{ "ns-name": nsBName, }) framework.ExpectNoError(err, "Error occurred while creating namespace-b.") podB, serviceB := createServerPodAndService(f, nsB, "pod-b", []int{allowedPort, notAllowedPort}) defer cleanupServerPodAndService(f, podB, serviceB) // Wait for Server with Service in NS-B to be ready framework.Logf("Waiting for servers to be ready.") err = e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, podB.Name, nsB.Name, framework.PodStartTimeout) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Ready.") ginkgo.By("Create a network policy for the server which denies both Ingress and Egress traffic.") policy := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "ingress-egress-rule", }, Spec: networkingv1.NetworkPolicySpec{ PolicyTypes: []networkingv1.PolicyType{networkingv1.PolicyTypeIngress, networkingv1.PolicyTypeEgress}, Ingress: []networkingv1.NetworkPolicyIngressRule{{ From: []networkingv1.NetworkPolicyPeer{{ NamespaceSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{ "ns-name": nsBName, }, }, }}, Ports: []networkingv1.NetworkPolicyPort{{ Port: &intstr.IntOrString{IntVal: allowedPort}, }}, }}, Egress: []networkingv1.NetworkPolicyEgressRule{ { Ports: []networkingv1.NetworkPolicyPort{ // Allow DNS look-ups { Protocol: &protocolUDP, Port:     &intstr.IntOrString{Type: intstr.Int, IntVal: 53}, }, }, }, { To: []networkingv1.NetworkPolicyPeer{ { NamespaceSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{ "ns-name": nsBName, }, }, }, }, Ports: []networkingv1.NetworkPolicyPort{{ Port: &intstr.IntOrString{IntVal: allowedPort}, }}, }, }, }, } policy, err = f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policy, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error creating Network Policy %v: %v", policy.ObjectMeta.Name, err) defer cleanupNetworkPolicy(f, policy) ginkgo.By("client-a should be able to communicate with server port 80 in namespace-b", func() { testCanConnect(f, f.Namespace, "client-a", serviceB, allowedPort) }) ginkgo.By("client-b should be able to communicate with server port 80 in namespace-a", func() { testCanConnect(f, nsB, "client-b", service, allowedPort) }) ginkgo.By("client-a should not be able to communicate with server port 81 in namespace-b", func() { testCannotConnect(f, f.Namespace, "client-a", serviceB, notAllowedPort) }) ginkgo.By("client-b should not be able to communicate with server port 81 in namespace-a", func() { testCannotConnect(f, nsB, "client-b", service, notAllowedPort) }) }) </ins> ginkgo.It("should enforce egress policy allowing traffic to a server in a different namespace based on PodSelector and NamespaceSelector [Feature:NetworkPolicy]", func() { var nsBserviceA, nsBserviceB *v1.Service var nsBpodServerA, nsBpodServerB *v1.Pod

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We are missing a test which specifies Ingress,Egress (both) for network policys (in ). Why is this needed: So that our network policy test coverage matches the semantics of what is commented and supported in the API.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign
/assign I'd like to do this if you don't mind
Sure go ahead. I hadn't gotten to it for a while. Appreciate you picking this up


Question: fs.StringVar(&s.KeystoneURL, "experimental-keystone-url", s.KeystoneURL, "If passed, activates the keystone authentication plugin.") <ins> fs.StringVar(&s.KeystoneCAFile, "experimental-keystone-ca-file", s.KeystoneCAFile, ""+ "If set, the Keystone server's certificate will be verified by one of the authorities "+ "in the experimental-keystone-ca-file, otherwise the host's root CA set will be used.") </ins> // See #14282 for details on how to test/try this option out. // TODO: remove this comment once this option is tested in CI. fs.IntVar(&s.KubernetesServiceNodePort, "kubernetes-service-node-port", s.KubernetesServiceNodePort, ""+

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: return &bo, nil } <ins> // sleepOp defines an op that can be used to sleep for a specified amount of time. // This is useful in simulating workloads that require some sort of time-based synchronisation. type sleepOp struct { // Must be "sleep". Opcode string // duration of sleep. Duration time.Duration } func (so *sleepOp) UnmarshalJSON(data []byte) (err error) { var tmp struct { Opcode   string Duration string } if err = json.Unmarshal(data, &tmp); err != nil { return err } so.Opcode = tmp.Opcode so.Duration, err = time.ParseDuration(tmp.Duration) return err } func (so *sleepOp) isValid(_ bool) error { if so.Opcode != sleepOpcode { return fmt.Errorf("invalid opcode %q; expected %q", so.Opcode, sleepOpcode) } return nil } func (so *sleepOp) collectsMetrics() bool { return false } func (so sleepOp) patchParams(_ *workload) (realOp, error) { return &so, nil } </ins> func BenchmarkPerfScheduling(b *testing.B) { testCases, err := getTestCases(configFile) if err != nil {

Note: this issue should be viewed in the context of . What would you like to be : Currently we support three types of ops: , , . This issue proposes adding a op that would simulate user action by causing the main thread to sleep. Note: I don't have a use-case for this in mind, but one can come up with trivial and contrived examples easily. Why is this needed: Not clear at this time. This is just an open discussion where others can chime in with whether this will be useful. /sig scheduling /sig testing (Not assigning myself until we can figure out if/how we want this.)
when you get a chance, can you group all enhancements issues relate to the new benchmark framework under one issue with a task list.
/assign
see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/unassign
I'd like to take this one /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is still an active pull request for this issue. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : There is still an active pull request for this issue. /reopen </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "context" "testing" <del> "github.com/google/go-cmp/cmp" </del> "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/validation/field" <del> "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" </del> "k8s.io/utils/pointer" )

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc


Question: path      string success   bool }{ <del> "basic":            {pod, &api.Container{Name: "foo"}, "spec.containers[0]", true}, "basic2":           {pod, &api.Container{Name: "baz"}, "spec.containers[2]", true}, "basicSamePointer": {pod, &pod.Spec.Containers[0], "spec.containers[0]", true}, </del> <ins> "basic":            {pod, &api.Container{Name: "foo"}, "spec.containers{foo}", true}, "basic2":           {pod, &api.Container{Name: "baz"}, "spec.containers{baz}", true}, "emptyName":        {pod, &api.Container{Name: ""}, "spec.containers[2]", true}, "basicSamePointer": {pod, &pod.Spec.Containers[0], "spec.containers{foo}", true}, </ins> "missing":          {pod, &api.Container{Name: "qux"}, "", false}, }

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004


Question: } func (c *BlobDiskController) getAllStorageAccounts() (map[string]*storageAccountState, error) { <del> accountListResult, err := c.common.cloud.StorageAccountClient.List() </del> <ins> accountListResult, err := c.common.cloud.StorageAccountClient.ListByResourceGroup(c.common.resourceGroup) </ins> if err != nil { return nil, err }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Deploy a cluster on azure with managed identity. The controller manager won't start: The following code tries to list all storage account in subscription, while it only has permission for the containing resource group. This issue would also apply for service principals with permission for the cluster resource group only. What you expected to happen: The cluster works correctly. How to reproduce it (as minimally and precisely as possible): Refer to: Anything else we need to know?: Environment: Updated: The client would only list the resources it has permission to, so the issue's cause here is not RBAC permission. Please see below comments for detail.
/sig azure
Checked the inner cause of the error log, the response returned by local msi endpoint is: There are 2 issues need to be fixed here: now needs metadata header Pending on integration with: storage account in current resource group master branch already contains that fix in 'go-autorest' , the issue is for v1.8 only
I have the same issue and expect fix for 1.8. Thanks
I don't think there would be a cherry-pick of azure SDK upgrade for a preview feature.