Question: // Setup expectations. const ( <del> kb = 1000 mb = 1000 * kb gb = 1000 * mb tb = 1000 * gb </del> <ins> kb int64 = 1000 mb int64 = 1000 * kb gb int64 = 1000 * mb tb int64 = 1000 * gb </ins> maxStartAge = time.Hour * 24 * 365 // 1 year maxStatsAge = time.Minute

Bug report. Kubernetes version (use ): Build of master branch commit Environment: Ubuntu 16.04, 24GB memory, 8 cores (parallel compilation used as 11GB memory) Linux gbhc1 4.4.0-38-generic -Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: Following the build-from-source directions, I cloned kubernetes; cd <dir>; make release This fails with : +++ [0929 09:16:04] linux/amd64: go build started +++ [0929 09:17:42] linux/amd64: go build finished +++ [0929 09:16:04] linux/arm: go build started # constant overflows int constant overflows int constant overflows int constant overflows int constant overflows int constant overflows int constant overflows int constant overflows int +++ [0929 09:16:04] linux/arm64: go build started +++ [0929 09:18:06] linux/arm64: go build finished Makefile:79: recipe for target 'all' failed make[1]:  [all] Error 1 Makefile:255: recipe for target 'cross' failed make:  [cross] Error 1 Makefile:239: recipe for target 'release' failed make:  [release] Error 1 What you expected to happen: A successful build How to reproduce it (as minimally and precisely as possible): make release Anything else do we need to know: gavin awk '(NR==57 NR==183) {print}' < fsCapacityBounds := bounded(100mb, 100gb) "AvailableBytes": bounded(100mb, 100gb), "UsageBytes": bounded(10mb, 10gb), "WorkingSetBytes": bounded(10mb, 10gb), "RxBytes": bounded(1mb, 100gb), "TxBytes": bounded(10kb, 10gb), "UsedBytes": bounded(kb, 10gb),
can you look into this build failure please?
oh, this was broken by a PR that went in yesterday, .


Question: FEATURE_GATES: $(yaml-quote ${FEATURE_GATES}) EOF fi <ins> if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then cat >>$file <<EOF VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/home/kubernetes/flexvolume}) EOF fi </ins> if [ -n "${PROVIDER_VARS:-}" ]; then local var_name

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: Feature Default FlexVolume driver directory is , which is not writable on COS images. Should have a different default directory to enable FlexVolume on COS. /kind feature /assign
/sig storage
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Question: ) (string, error) { klog.V(4).Infof(log("blockMapper.publishVolumeForBlock called")) <del> publishVolumeInfo := attachment.Status.AttachmentMetadata </del> <ins> publishVolumeInfo := map[string]string{} if attachment != nil { publishVolumeInfo = attachment.Status.AttachmentMetadata } </ins> nodePublishSecrets := map[string]string{} var err error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: During I develop a driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. PS: I use cluster-driver register and make requiredAttach as false. kubectl get csidriver csi-xxx -o yaml ! What you expected to happen: Mount a raw block device successfully. How to reproduce it (as minimally and precisely as possible): driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. cluster-driver register and make requiredAttach as false. a pod with raw block devices Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: BaiduCloud OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /assign
I would send a PR later


Question: return expectNodes[i].ID() < expectNodes[j].ID() }) startNodes := c.start.graph.Nodes() <del> sort.Slice(expectNodes, func(i, j int) bool { </del> <ins> sort.Slice(startNodes, func(i, j int) bool { </ins> return startNodes[i].ID() < startNodes[j].ID() }) assert.Equal(t, expectNodes, startNodes)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pulled master and ran: $ make test I get test failure for TestDeleteEdges_locked in What you expected to happen: I expected all tests to run successfully How to reproduce it (as minimally and precisely as possible): Run $ make test, using the most recent version of master (head at commit ), observe that test fails. Anything else we need to know?: This is my first issue, so please let me know if I need to provide any other information! Environment: Kubernetes version (use ): upstream/master (head at commit ) OS (e.g. from /etc/os-release): macOs 10.13.3
/sig testing
/assign
/priority failing-test
This looks like a flaky test. I wonder if the ordering is no longer guaranteed stable in go1.10?
nope, typo in the sorting the test did. fix incoming
/assign


Question: APIListChunking: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.32 <del> APIPriorityAndFairness: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 </del> APIResponseCompression: {Default: true, PreRelease: featuregate.Beta}, APIServerIdentity: {Default: true, PreRelease: featuregate.Beta},

Priority and Fairness is going GA in 1.29, the feature has been marked stable and the feature gate has been locked to default. See In 1.31, we need to remove the feature gate.
/kind bug /sig api-machinery /cc
/assign
/triage accepted


Question: Email    string } <ins> const ( readURLTimeout = time.Second * 20 ) </ins> var ( preferredPathLock sync.Mutex preferredPath     = ""

http://pr-
Something holding the port already?
The previous masterless case just barely passed with a 12.5 second timeout (25 tries at 0.5 second interval) I think we just need to increase the timeout.
I randomly sampled 5 green builds and they all have this test (starting kubelet in masterful/less mode) finished in 3s or less. Judging for the successful masterless mode startup before the failure, kubelet spent the most time trying to register credential providers (a 10s gap in the log). The same roughly applies to the failed "masterful mode", based on where the log message stopped. From , the registering process basically tries to access the metadata url to see if the credential provider is enabled. I think this may just be the slow response from the metadata server.
I am wondering even if this issue might be related to a slow response from the metadata server, why it does not show "Registering credential provider: .dockercfg" always in failed logs?
If there is a possibility that ReadUrl in may take longer than expected, would it be preferred to use a Timeout (0) with the http client to not get stuck longer than expected?
I am unsure why it didn't show that, but even in the passing round we see a 10 second gap, which almost maxed out the timeout. I think adding a timeout is reasonable, but I am not sure what value is reasonable given that the previous request took 10s. do you have any idea?
Even I am not sure exactly what should be a reasonable value. My guess would be that if in normal cases, average response time is x seconds (perhaps a guestimate could be computed based on "normal" successful tests), then x could be a starting point, or otherwise another option is choosing something randomly (like 5s). And I think that if at some point, the server is responding in 10s, and even though the test passes, its not normal and something is going wrong.
uses a 10 second timeout, but the request will be retried. Given that we don't retry this request, a short timeout is worrisome. Maybe we should add both retry and timeout in this case.
SGTM.
I reopen this one since pr was reverted.

Noticed a similar hang in: The logs are a bit different from those observed in
if kubelet couldn't start up properly, the test itself should've timed out and logged "Timed out waiting for kubelet to answer at http://127.0.0.1:/healthz; tried 25 waiting 0.5 between each". I am not sure if this failure is related.
failure is not relevant to this issue. The most recent report of such flake was over a week ago. I think the cause is the slow metadata url response in GCE, and that should've been fixed by the recent rollout. I am going to close this issue. Feel free to reopen if there is another occurrence.


Question: testCases := []struct { name string <ins> isIPv6Primary          bool </ins> enableDualStack        bool specExpctPrimary       bool clusterIPExpectPrimary bool

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: SupportedFsType: sets.NewString( "", // Default fsType ), <del> SupportedMountOption: sets.NewString("proto=tcp", "nosuid"), </del> <ins> SupportedMountOption: sets.NewString("proto=tcp", "relatime"), </ins> RequiredMountOption:  sets.NewString("vers=4.1"), IsPersistent:         true, IsFsGroupSupported:   false,

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O job in our CI using kubernetes master tests Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options Since when has it been failing: since this PR has been merged: Testgrid link: link to our job is here: https://openshift-gce- https://openshift-gce- https://openshift-gce- Reason for failure: we don't know the reason of the failure but it doesn't seem related to CRI-O. It's constantly reproducible also. Anything else we need to know: /cc <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig node /sig storage
/cc
Curiously, it does not seem to have flaked in core kubernetes CI
I'm still looking at it on our side
This test is newly in this commit. It tests that nfs external dynamic provisioner can provision a volume with specific mount options and the volume is properly mounted in the pod. The mount options specified in this test are "vers=4.1", "proto=tcp", and "nosuid". () The test seems to fail to create a pod with the volume and other tests that don't specify these mount options don't fail. So, I guess that this is caused because this environment doesn't support either of the option above. Is it possible to try testing it manually by changing these options?
yes, I have a box where I can change them and re-test, do you have suggestions on what to use to test?
(sorry fat fingers)
If it was an unsupported mount option, then the test should fail every time, not just flake?
yes, it's failing every time for us
Are there kubelet node logs we can look at?
yes, we have artifacts for each job at https://gcsweb-
As a starting point, I would recommend deleting only "proto=tcp", and check if it still fails, first. Then moving back "proto=tcp" and deleting only "nosuid", and check what happens. And continue testing with combinations of options.

Or, if we can find harmless mount options that can be specified for nfs, we will be able to replace the options with certainly harmless ones. Note that tests for AWS and GCE uses "debug" and "nouid32", which don't seem to be valid. for nfs
The container is failing with exit-code 1 with: I'll try to see which one is the bad one
do you think we could make the test easier to debug and figure out which mount option doesn't work?
yes, I'm trying to add something to that command to at least have some logs if that fails cause right now, if you do kubectl get logs on the failing pod you don't get anything
looks like removing makes the container happy and test as well. I'm not sure why it fails on rhel and fedora though, I'll dig into that, but yeah, we could use something different if possible
confirmed that if I remove nosuid the test works fine, I"m modifying the test code to output something useful in case it fails
Thank you for your quick investigation and improvement!
I'm going to open a PR for the test output to have something useful () and understand why it fails with
sorry, I am the one who wrote that hideous bash one-liner. TY for any patches
so, isn't shown: is that required for the test? can we have another option checked instead?
maybe we can use ? who is setting and where?
PTAL at
do you guys have any insight on this?
As for this e2e test, we will be able to use any options which are valid for nfs e2e test, because the purpose of the test is to check that the volume mount options work well for provisioning and the options are properly reflected to the mount options inside the pod. (Actually, I chose for nfs test just because I found it first.) As for issue for nfs dynamic provisioner with this environment, it might be still worth looking into, if this option can be specified by users. However, this could be out of the scope of this issue.
exactly, thanks, I'm going to investigate it not as part of this PR
Since this test failure is non-blocking for release, setting priority approprately. /priority important-longterm


Question: framework.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test-3") } <del> // NOTE: we cannot guarantee our output showed up in the container logs before stdin was closed, so we have // to loop test. err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, 1*time.Second) { framework.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test-3") } logOutput := framework.RunKubectlOrDie(ns, "logs", runTestPod.Name) gomega.Expect(logOutput).ToNot(gomega.ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) framework.ExpectNoError(err) </del> gomega.Expect(c.CoreV1().Pods(ns).Delete(context.TODO(), "run-test-3", metav1.DeleteOptions{})).To(gomega.BeNil()) })

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: maxReadLength      = 10 * 1 << 20 // 10MB ) <del> var containerRegistryUrls = []string{"*.azurecr.io", "*.azurecr.cn", "*.azurecr.de", "*.azurecr.us"} </del> <ins> var ( containerRegistryUrls = []string{"*.azurecr.io", "*.azurecr.cn", "*.azurecr.de", "*.azurecr.us"} acrRE                 = regexp.MustCompile(`.*.azurecr.io|.*.azurecr.cn|.*.azurecr.de|.*.azurecr.us`) ) </ins> // init registers the various means by which credentials may // be resolved on Azure.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, Spec: storage.VolumeAttachmentSpec{ NodeName: node, <del> Attacher: csiPluginName, </del> <ins> Attacher: csiSource.Driver, </ins> Source: storage.VolumeAttachmentSource{ PersistentVolumeName: &pvName, },

Internal volume plugin for CSI creates VolumeAttachment with (= the plugin name). It should be CSI driver name instead, as the external attacher must know which CSI driver to call. /sig storage /kind bug please prioritize PTAL. It should be one-line change in
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ACK. In progress


Question: }, "bound-pvc,pv-not-exists": { podPVCs:    []*v1.PersistentVolumeClaim{boundPVC}, <del> shouldFail: true, </del> <ins> shouldFail: false, reasons:    ConflictReasons{ErrReasonPVNotExist}, </ins> }, "prebound-pvc": { podPVCs:    []*v1.PersistentVolumeClaim{preboundPVC},

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: var ( file               = flag.String("config", "", "Path to the config file") <del> etcd_servers       = flag.String("etcd_servers", "", "Url of etcd servers in the cluster") </del> <ins> etcdServers        = flag.String("etcd_servers", "", "Url of etcd servers in the cluster") </ins> syncFrequency      = flag.Duration("sync_frequency", 10*time.Second, "Max period between synchronizing running containers and config") fileCheckFrequency = flag.Duration("file_check_frequency", 20*time.Second, "Duration between checking file for new data") httpCheckFrequency = flag.Duration("http_check_frequency", 20*time.Second, "Duration between checking http for new data") <del> manifest_url       = flag.String("manifest_url", "", "URL for accessing the container manifest") </del> <ins> manifestUrl        = flag.String("manifest_url", "", "URL for accessing the container manifest") </ins> address            = flag.String("address", "127.0.0.1", "The address for the info server to serve on") port               = flag.Uint("port", 10250, "The port for the info server to serve on") <ins> hostnameOverride   = flag.String("hostname_override", "", "If non-empty, will use this string as identification instead of the actual hostname.") </ins> ) const dockerBinary = "/usr/bin/docker"

It would be great to have an easy way to run a kubernetes master/minion locally. Currently, it's not that easy to set up. We should have a convenient script to make it work.
The integration test binary is pretty close. It'd be pretty simple to glue in the kubelet to that binary. Brendan On Jun 10, 2014 2:53 PM, "Daniel Smith" wrote:
I'll take a look.
+1 for this -- especially in script form rather than a go executable.
what exactly do you mean by "script form"? Were you thinking of something like a version of cloudcfg that can be run without needing to have started up a daemon first?
Basically, what I had in mind, is a shell script that looks something like
Ah, ok. That's very similar to local-; the only difference is that everything is packaged into a single go binary. We probably will switch to using individual binaries for this when we start running our components inside docker containers.
The script just makes things more obvious how to run things for those who are looking to integrate them into non-GCE environments.
That's a good point. I'll look into this some more later.
Create a pad as follows: Name: non-persistent-redis Container image: redis Named-volum with name: cache-control Mount path: /data/redis It should launch in the pre-prod namespace and the volume MUST NOT be persistent. apiVersion: v1 kind: Pod metadata: name: non-persistent-redis namespace: pre-prod spec: containers: name: non-persistent-redis image: redis volumeMounts: name: cache-control mountPath: "/data/redis" volumes: name: cache-control emptyDir: {}

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.

When I'm accessing persistentVolumes and persistentVolumeClaims, I'm getting nil instead of empty list of items, as happening in other entities when there are no items. { "kind": "PersistentVolumeList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/persistentvolumes", "resourceVersion": "" } }
Yes, this seems like a bug. I see an empty list when I access pods: And items is nil when I access persistent volumes:
Items shouldn't be tagged with omitempty:
cc
PersistentVolumeList and PersistentVolumeClaimList appear to be the only resource types with this problem.
Please see for fix.


Question: rcs:      []*v1.ReplicationController{{Spec: v1.ReplicationControllerSpec{Selector: map[string]string{"foo": "bar"}}}}, services: []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"baz": "blah"}}}}, // "baz=blah" matches both labels1 and labels2, and "foo=bar" matches only labels 1. This means that we assume that we want to <del> // do spreading between all pods. The result should be exactly as above. expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, </del> <ins> // do spreading pod2 and pod3 and not pod1. expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 0}}, </ins> name:         "service with partial pod label matches with service and replication controller", }, {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: ), }, expected: &DeviceRunContainerOptions{ <del> Annotations: []kubecontainer.Annotation{ {Name: "cdi.k8s.io/devicemanager_pod-container", Value: "vendor1.com/class1=device1,vendor2.com/class2=device2,vendor3.com/class3=device3"}, }, </del> CDIDevices: []kubecontainer.CDIDevice{ {Name: "vendor1.com/class1=device1"}, {Name: "vendor2.com/class2=device2"},

Kubelet should stop using annotations to pass CDI device IDs to CRI runtimes. CDIDevices CRI field should be used for this purpose. This should be done when Kubelet that doesn't support CRI field reaches EOL and two major CRI runtimes support CRI field. Here is a PR comment that gives more details about this: Kubelet started using CDIDevices CRI field in . Previous version 1.27 will reach EOL .CRI-O started using it in . Previous version 1.27 will reach EOL together with Kubernetes 1.27.Containerd started using it in . Version 1.7 will reach EOL in a 6 months after Containerd 2.0 release Previous version 1.6 doesn't support CDI injections and should not be considered. So, safer option would be to wait until Containerd 1.7 reaches EOL. However it's relatively safe to do it earlier when 1.7.0 and 1.7.1 have little chances to be in real use. Containerd v1.7.0 and v1.7.1 is not used by the Linux distributions from , so it's safe to remove annotations support from the Kubelet in 1.31 in my opinion.
/sig node
/triage accepted /priority important-longterm
/assign


Question: Namespace:        foo Address: Ingress Class:    test <del> Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>) </del> <ins> Default backend:  <default> </ins> Rules: Host         Path  Backends ----         ----  --------

I ran on a fresh minikube with no other services in it and saw this output. Which suggests there is a service in the cluster called (there isn't), and further suggests there is something wrong (there isn't) because there are no endpoints found. I expected to provide me with an accurate description of my Ingress object. This behaviour appears to be hardcoded into the kubectl describe command. Which contains a comment suggesting that this is default behaviour, but no logic to check if the service actually exists before attempting to default to it. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
See also do you know the history here?
not beyond what shows: Looks like an (incorrect) assumption in the describer that the only implementation would behave in this way
/assign
work in progress...
when you get the PR up please link this issue and in it and ping me directly on slack to get a fast review. I agree with what was stated here and in the other issue, that this seems like an unnecessary assumption that should've not been there, we probably should just leave it empty or per print as the value.


Question: } } <ins> func TestRetryInternalError(t *testing.T) { testCases := []struct { name                string maxInternalDuration time.Duration rewindTime          int wantRetries         int }{ { name:                "retries off", maxInternalDuration: time.Duration(0), wantRetries:         0, }, { name:                "retries on, all calls fail", maxInternalDuration: time.Second * 30, wantRetries:         31, }, { name:                "retries on, one call successful", maxInternalDuration: time.Second * 30, rewindTime:          10, wantRetries:         40, }, } for _, tc := range testCases { err := apierrors.NewInternalError(fmt.Errorf("etcdserver: no leader")) fakeClock := testingclock.NewFakeClock(time.Now()) bm := &fakeBackoff{clock: fakeClock} counter := 0 lw := &testLW{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "1"}}, nil }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { counter = counter + 1 t.Logf("Counter: %v", counter) if counter == tc.rewindTime { t.Logf("Rewinding") fakeClock.Step(time.Minute) } fakeClock.Step(time.Second) w := watch.NewFakeWithChanSize(1, false) status := err.Status() w.Error(&status) return w, nil }, } r := &Reflector{ name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), } r.MaxInternalErrorRetryDuration = tc.maxInternalDuration stopCh := make(chan struct{}) r.ListAndWatch(stopCh) close(stopCh) if counter-1 != tc.wantRetries { t.Errorf("%v unexpected number of retries: %d", tc, counter-1) } } } </ins> func TestReflectorResync(t *testing.T) { iteration := 0 stopCh := make(chan struct{})

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: --disk name="${MASTER_NAME}-pd" device-name=master-pd mode=rw boot=no auto-delete=no } <ins> # TODO(mbforbes): Make $1 required. # TODO(mbforbes): Document required vars (for this and call chain). # $1 version </ins> function create-node-instance-template { <del> create-node-template "${NODE_INSTANCE_PREFIX}-template" "${scope_flags[*]}"  </del> <ins> local suffix="" if [[ -n ${1:-} ]]; then suffix="-${1}" fi create-node-template "${NODE_INSTANCE_PREFIX}-template${suffix}" "${scope_flags[*]}"  </ins> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh"  "kube-env=${KUBE_TEMP}/node-kube-env.yaml" }

See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.


Question: return nil, nil // success } <del> // Delete the PVC and wait for the PV to become Available again. // Validate that the PV has recycled (assumption here about reclaimPolicy). </del> <ins> // Delete the PVC and wait for the PV to become Available again. Validate that // the PV has recycled (assumption here about reclaimPolicy). Return the pv and // pvc to reflect that these resources have been retrieved again (Get). If the // delete is successful the returned pvc should be nil and the pv non-nil. // Note: the pv and pvc are returned back to the It() caller so that the //   AfterEach func can delete these objects if they are not nil. </ins> func deletePVCandValidatePV(c *client.Client, ns string, pvc *api.PersistentVolumeClaim, pv *api.PersistentVolume) (*api.PersistentVolume, *api.PersistentVolumeClaim, error) { <del> By("Deleting PersistentVolumeClaim to trigger PV Recycling") </del> framework.Logf("Deleting PersistentVolumeClaim %v to trigger PV Recycling", pvc.Name) err := c.PersistentVolumeClaims(ns).Delete(pvc.Name) if err != nil {

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err := gce.deleteForwardingRule(loadBalancerName, gce.region); err != nil { return nil, fmt.Errorf("failed to delete existing forwarding rule %s for load balancer update: %v", loadBalancerName, err) } <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): deleted forwarding rule", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): deleted forwarding rule", loadBalancerName, serviceName) </ins> } if tpExists && tpNeedsUpdate { if err := gce.deleteTargetPool(loadBalancerName, gce.region); err != nil { return nil, fmt.Errorf("failed to delete existing target pool %s for load balancer update: %v", loadBalancerName, err) } <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): deleted target pool", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): deleted target pool", loadBalancerName, serviceName) </ins> } // Once we've deleted the resources (if necessary), build them back up (or for

Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick


Question: else bazel-release: bazel build //build/release-tars <ins> bazel shutdown </ins> endif

Which jobs are flaking: pull-kubernetes-e2e-gce pull-kubernetes-e2e-gce-rbe Which test(s) are flaking: Stage Testgrid link: Reason for failure: This isn't technically a test that's failing, it's part of the build-stage-up-test cycle used by our e2e jobs. There are two main clusters of failures that I can tell. One is timeouts. The other looks like: Example -gce jobs: () () () () Example -gce-rbe jobs: (source gist ) Anything else we need to know: Triage link for the job that actually reports to PRs: ends up running Some googling for "bazel error code 14" leads me to believe that bazel (or its jvm) is getting killed due to OOM. which is a value that's been cargo culted around as far as I can tell. I was unable to find the decision that led to this value. I'm wondering if a bump to 8Gi of memory would help reduce occurrences of this flake. I'm referencing the -rbe jobs to point out that this happens even if we use RBE, but they're not blocking
/sig testing Top flake according to
With merged we'll wait and see if the Stage flakes on pull-kubernetes-e2e-gce go down. If they do, there are other presubmit jobs that could benefit from that. If they don't, we should revert.
/priority important-soon
/cc
Still seeing flakes, so I don't think just bumping memory request helped. ref: my theory is that making this non-privileged broke our ability to free memory This happened via around 2019-04-11 - flakes were still occurring prior to this merge - but not prior to 2019-04-01 - and very infrequently prior 2019-03-15 It could be that bazel's memory usage grew when we bumped it to 0.23.2 on 2019-03-29 (or maybe on 2019-04-02 is what actually deployed the change) Either way, I still think it looks like bazel is eating more memory than the container is allowed, and is getting oomkilled.

Two ideas: Instead of allowing the bazel server to exist as a long lived process, have kubetest call in between the Build and Stage steps, in addition to trying to flush memory Constrain bazel's host JVM to a hardcoded memory limit via in bazelrc
Idea 3: switch build modes and see how expensive / flaky that is. Including: use RBE use the dockerized build instead
I opened to try and constrain the JVM's memory. Jobs that use RBE behave the same way, so try doing the same to them for comparison's sake.
this is still being seen extremely frequently (seen multiple times in a row on the same PR at )
I opened for the last idea I have for working around bazel's behavior, beyond this I am out of my depth on how to replace bazel for the release/push- script
... but I'm willing to try with a canary job and see what happens
bumping priority, this looks like 99/122 flakes according to /priority critical-urgent
Another occurrence:
I'd like to see if we can root cause what changed around 1/27-1/29 that greatly increased this
kubernetes/kubernetes changes: 1/26, ~6am UTC: 1/28: ~6pm UTC: test-infra changes: 1/25, ~3pm UTC: 1/28: ~8pm UTC: in that delta kubekins changed from v20200123- to v20200127- nothing jumps out at me in those diffs, but wanted to link to them in case it helped anyone else see something
It doesn't have to be one, the CI is run overcommitted with no limits, something else could be upping memory pressure..? On Mon, Feb 10, 2020, 08:38 Jordan Liggitt <:
ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ /close
Closing this issue. <detailsIn response to : ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # See the License for the specific language governing permissions and # limitations under the License. <ins> # This script builds protoc-gen-gogo binary in runtime and genertates # `*/api.pb.go` from the protobuf file `*/api.proto`. </ins> set -o errexit set -o nounset set -o pipefail

What would you like to be : There are many useful scripts in "". For example, I often use and for creating simple k8s cluster quickly. However, there are only few explanation or nothing how to use it. I think it becomes more useful and helpful if some instruction or guidance is explained in each script. Why is this needed: It aims and expects to increase new k8s developers by describing how to use each script. My proposal Add the following description under the copyright for each script file. Overview of the script Simple usage Supplementary explanation(This is option) Other concern I would like to know the purpose of use about hack directory. (e.g. Where to put tools for developers(therefore there is no guidance?).)
I don't know who is in charge of each scripts of hack, but if he agrees the idea of this feature request , I am going to do this task myself!!
/sig testing
I believe there is a script generating swagger/openapi specfilters under hack, but I am still struggling with it. It could be of great help if there is some how-to doc for each scripts.
this basis, why not create a file under the hack folder, because a lot of kk CIs are derived from multiple script commands under the hack. There are already 268+ files under the hack file. For contributors unfamiliar with these, these comments and instructions are helpful to them, and it is also helpful for new contributors to handle pr ci error reports.
/cc What do you think
I think it's good idea to create (It may be easy to understand if the script's explanation is described by category or use in ) Apart from that, it is helpful that the script file itself has a brief description.
When we create , it may tough work if creating it including all scripts explanation at one time. (There are many scripts and I understand only some scripts now.) Regarding , current my idea is to add a description to each script little by little and update the at the same time. This means that will be completed little by little in conjunction with adding the description of each script. In the future, if someone create new script in hack, he also may have to add description of his script into
Self-documenting commands sound useful. A high level overview also sounds useful.

/cc /cc Regarding adding help information, I suggest that we can submit pr in one or several files as a unit, so that we can track and statistics. For the documentation, I think it should be concise and not too long. How about this advice? We can claim tasks through , the pr title yes , i agree with you. [x] kubernetes/hack/benchmark- [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] kubernetes/hack/boilerplate/BUILD [ ] kubernetes/hack/boilerplate/test/BUILD [ ] [ ] [ ] [ ] [ ] kubernetes/hack/BUILD [x] kubernetes/hack/build- [x] kubernetes/hack/build- [x] [ ] kubernetes/hack/conformance/BUILD [ ] [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-push- [ ] kubernetes/hack/e2e-internal/BUILD [ ] kubernetes/hack/e2e-internal/e2e-cluster- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e-grow- [ ] kubernetes/hack/e2e-internal/e2e-shrink- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e- [x] kubernetes/hack/e2e-node- [ ] kubernetes/hack/generate- [x] kubernetes/hack/generate- [ ] kubernetes/hack/gen-swagger-doc/gen-swagger- [x] kubernetes/hack/get- [x] kubernetes/hack/ginkgo- [x] kubernetes/hack/grab- [x] kubernetes/hack/install- [ ] kubernetes/hack/jenkins/benchmark- [ ] kubernetes/hack/jenkins/OWNERS [ ] [ ] kubernetes/hack/jenkins/test- [ ] kubernetes/hack/jenkins/verify- [ ] kubernetes/hack/lib/BUILD [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [x] kubernetes/hack/lint- [x] kubernetes/hack/list-feature- [x] kubernetes/hack/local-up- [ ]

kubernetes/hack/make-rules/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/subdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ]

kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-without-gofiles/README [ ] kubernetes/hack/make-rules/Makefile.manifest [ ] kubernetes/hack/make-rules/make- [ ] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make-rules/test-e2e- [ ] kubernetes/hack/make-rules/test-e2e- [x] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/OWNERS [x] kubernetes/hack/pin- [x] kubernetes/hack/print-workspace- [x] kubernetes/hack/run-in- [ ] kubernetes/hack/testdata/auth- [x] [x] [x] kubernetes/hack/testdata/CRD/foo-- [x] kubernetes/hack/testdata/CRD/foo-deleted- [x] kubernetes/hack/testdata/CRD/foo-updated- [x] [x] kubernetes/hack/testdata/CRD/multi-crd-list-- [x] kubernetes/hack/testdata/CRD/multi-crd-list-deleted- [x] kubernetes/hack/testdata/CRD/multi-crd-list-updated- [x] kubernetes/hack/testdata/CRD/multi-crd- [x] [x] [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-multicontainer- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment-with- [x] [x] [x] [x] [x] [x] kubernetes/hack/testdata/filter/pod-apply- [x] kubernetes/hack/testdata/filter/pod-dont- [x]

kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/invalid-rc-with-empty- [x] kubernetes/hack/testdata/kubernetes- [x] [x] [x] [x] [x] [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-json- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-list- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-rclist- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-svclist- [x] kubernetes/hack/testdata/multi-resource-yaml- [x] kubernetes/hack/testdata/multi-resource- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/OWNERS [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod-with-api- [x] kubernetes/hack/testdata/pod-with-large- [x] kubernetes/hack/testdata/pod-with- [x] [x] [x] [x] [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/recursive/deployment/deployment/nginx- [x] [x] [x] [x] [x] kubernetes/hack/testdata/recursive/pod/pod/busybox- [x] [x] [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod-modify/pod/busybox- [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod- [x] [x] kubernetes/hack/testdata/recursive/rc/rc/busybox- [x] [x] [x] kubernetes/hack/testdata/redis-slave- [x]

kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/rollingupdate-daemonset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/rollingupdate-statefulset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/scale-deploy-1.yaml [x] kubernetes/hack/testdata/scale-deploy-2.yaml [x] kubernetes/hack/testdata/scale-deploy-3.yaml [x] [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [ ] [ ] [x] kubernetes/hack/test- [ ] kubernetes/hack/test- [x] kubernetes/hack/update- [ ] kubernetes/hack/update- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-generated-api-compatibility- [x] kubernetes/hack/update-generated-device-plugin- [ ] kubernetes/hack/update-generated-device- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-kms- [ ] kubernetes/hack/update-generated- [x] kubernetes/hack/update-generated-kubelet-plugin-registration- [ ] kubernetes/hack/update-generated-kubelet-plugin- [x] kubernetes/hack/update-generated-pod-resources- [ ] kubernetes/hack/update-generated-pod- [ ] kubernetes/hack/update-generated-protobuf- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-runtime- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-swagger- [x] kubernetes/hack/update- [x] kubernetes/hack/update-import- [ ] kubernetes/hack/update-openapi- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-vendor- [ ] kubernetes/hack/update- [ ] kubernetes/hack/update-workspace- [x] kubernetes/hack/verify-

[x] kubernetes/hack/verify-api- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-cli- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-conformance- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-external-dependencies- [ ] kubernetes/hack/verify-flags/excluded- [x] kubernetes/hack/verify-flags/OWNERS [x] kubernetes/hack/OWNERS [ ] kubernetes/hack/verify-flags- [x] kubernetes/hack/verify-generated-device- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-files- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-kubelet-plugin- [x] kubernetes/hack/verify-generated-pod- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-swagger- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-no-vendor- [x] kubernetes/hack/verify-openapi- [x] kubernetes/hack/verify-pkg- [x] kubernetes/hack/verify-publishing- [x] kubernetes/hack/verify-readonly- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-staging-meta- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-typecheck- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-vendor- [x]

kubernetes/hack/verify- [x]
i will pick up , i will do it step by step.
I agree with your idea, and thanks the checkbox to check progress. there are many scripts, so let's work together!
/assign /assign
I will pick up hack/verify-
I will pick up hack/verify-
I will pick up hack/verify- (all of verify scripts). But there are large number of scripts(over 40), I want to split it into two or three PRs.
I will pick up hack/update-
I will pick up these scripts. kubernetes/hack/benchmark- kubernetes/hack/build- kubernetes/hack/build- kubernetes/hack/dev-build-and- kubernetes/hack/dev-build-and- kubernetes/hack/dev-push- kubernetes/hack/e2e-node- kubernetes/hack/generate-
I will pick up these scripts get- ginkgo- grab- install- lint- list-feature- local-up- pin- print-workspace- run-in- test- test-
How about this now? /help Marking this tag so that others can easily access it.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : How about this now? /help Marking this tag so that others can easily access it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we've finished adding simple explanations to the almost scripts. It seems that several scripts of update- does not complete yet. I think we may able to close after that work.
I will pick up these scripts:
/assign


Question: assert.Equal(t, testCase.region, result) } } <ins> func TestCloud_sortELBSecurityGroupList(t *testing.T) { type args struct { securityGroupIDs []string annotations      map[string]string } tests := []struct { name                 string args                 args wantSecurityGroupIDs []string }{ { name: "with no annotation", args: args{ securityGroupIDs: []string{"sg-1"}, annotations:      map[string]string{}, }, wantSecurityGroupIDs: []string{"sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-4", "sg-3", "sg-2", "sg-1"}, }, { name: "with both annotation", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4", "sg-5", "sg-6"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups":       "sg-3,sg-2,sg-1", "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-6,sg-5", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1", "sg-4", "sg-6", "sg-5"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := &Cloud{} c.sortELBSecurityGroupList(tt.args.securityGroupIDs, tt.args.annotations) assert.Equal(t, tt.wantSecurityGroupIDs, tt.args.securityGroupIDs) }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When multiple securityGroup are specified for AWS ELB type service, a random one will be chosen to grant inbound access to worker node. What you expected to happen: Always first LoadBalancer securityGroup be chosen to access worker nodes. How to reproduce it (as minimally and precisely as possible): use or on aws elb. Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.

What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups.

We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster.

Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently.

I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).


Question: github.com/polyfloyd/go-errorlint v1.1.0 h1:VKoEFg5yxSgJ2yFPVhxW7oGz+f8/OVcuMeNvcPIi6Eg= github.com/polyfloyd/go-errorlint v1.1.0/go.mod h1:Uss7Bc/izYG0leCMRx3WVlrpqWedSZk7V/FUQW6VJ6U= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= <ins> github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g= </ins> github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=

<!-- Please only use this template for submitting enhancement requests --What would you like to be : should detect the appropriate GOMAXPROCS based off of the cpu limit supplied to the pod. Why is this needed: If not, test failures can occur (e.g. ) where GOMAXPROCS defaults to the number of cpus on the machine, while the cpu limit is actually far less. Too many compile/test processes launch, and any timing tests can become flaky. The end result is that running the test jobs on a node with more CPUs can make it flaky when running the test on a node with fewer CPUs cleanly passes. It's possible to set GOMAXPROCS=cpu limits in test jobs to avoid this, but it would be nicer if would autodiscover and set GOMAXPROCS appropriately.
/sig testing /sig k8s-infra
/transfer kubernetes
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign dims
xref:
This is already done, which is great, but for some more info I did some benchmarking of setting GOMAXPROCs vs not: Pretty substantially better in some cases


Question: // It is designed so that http.Server satisfies this interface, type httpServer interface { Serve(listener net.Listener) error <ins> Close() error </ins> } // Implement listener in terms of net.Listen.

When a LoadBalancer Service with Local ExternalTrafficPolicy was deleted normally, an error log "use of closed network connection" was always generated in kube-proxy log, which confused people that there was something wrong: Do not generate error log when it's expected. a LoadBalancer Service with Local ExternalTrafficPolicy it No response <detailsv1.26.0 </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package scheduler import ( "testing" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/kubernetes/pkg/features" ) func TestNodeResourceLimits(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ResourceLimitsPriorityFunction, true)() context := initTest(t, "node-resource-limits") defer cleanupTest(t, context) // Add one node expectedNode, err := createNode(context.clientSet, "test-node1", &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(2000, resource.DecimalSI), }) if err != nil { t.Fatalf("Cannot create node: %v", err) } // Add another node with less resource _, err = createNode(context.clientSet, "test-node2", &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(1000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(1000, resource.DecimalSI), }) if err != nil { t.Fatalf("Cannot create node: %v", err) } podName := "pod-with-resource-limits" pod, err := runPausePod(context.clientSet, initPausePod(context.clientSet, &pausePodConfig{ Name:      podName, Namespace: context.ns.Name, Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(500, resource.DecimalSI)}, }, })) if err != nil { t.Fatalf("Error running pause pod: %v", err) } if pod.Spec.NodeName != expectedNode.Name { t.Errorf("pod %v got scheduled on an unexpected node: %v. Expected node: %v.", podName, pod.Spec.NodeName, expectedNode.Name) } else { t.Logf("pod %v got successfully scheduled on node %v.", podName, pod.Spec.NodeName) } } </ins>

What would you like to be : An integration test for NodeResourceLimits plugin. Why is this needed: No end-to-end test coverage current exist for this alpha feature. /sig scheduling /help
/assign /remove-help


Question: "manager":     "Manager is an identifier of the workflow managing these fields.", "operation":   "Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'.", "apiVersion":  "APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted.", <del> "time":        "Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'", </del> <ins> "time":        "Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over.", </ins> "fieldsType":  "FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1"", "fieldsV1":    "FieldsV1 holds the first JSON version format as described in the "FieldsV1" type.", "subresource": "Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource.",

Godoc says that Time should be empty on mangedFields entry created by "Apply" operation, but kubernetes populates time on apply. Godoc consistent with kubernetes behaviour Godoc: Kubernetes: No response <details</details<details</details<details</details<details</details<details</details<details</details
/wg api-expression /sig api-machinery
/cc
/cc /triage accepted
Yeah, good catch. The comment matches the behavior from alpha, which we changed later on. Thanks for letting us now :-) I think we could do a better job describing how this value is changed now.


Question: metricsBindAddress string }{ { <del> name:               "iptables mode, IPv4 config", </del> <ins> name:               "iptables mode, IPv4 all-zeros bind address", mode:               "iptables", bindAddress:        "0.0.0.0", clusterCIDR:        "1.2.3.0/24", healthzBindAddress: "1.2.3.4:12345", metricsBindAddress: "2.3.4.5:23456", }, { name:               "iptables mode, non-zeros IPv4 config", </ins> mode:               "iptables", bindAddress:        "9.8.7.6", clusterCIDR:        "1.2.3.0/24",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: volumePluginMgr               *volumepkg.VolumePluginMgr skippedDuringReconstruction   map[v1.UniqueVolumeName]*globalVolumeInfo kubeletPodsDir                string <del> timeOfLastSync                time.Time volumesFailedReconstruction   []podVolume volumesNeedDevicePath         []v1.UniqueVolumeName volumesNeedReportedInUse      []v1.UniqueVolumeName </del> <ins> // lock protects timeOfLastSync for updating and checking timeOfLastSyncLock          sync.Mutex timeOfLastSync              time.Time volumesFailedReconstruction []podVolume volumesNeedDevicePath       []v1.UniqueVolumeName volumesNeedReportedInUse    []v1.UniqueVolumeName </ins> } func (rc *reconciler) Run(stopCh <-chan struct{}) {

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: <ins> reviewers: - vendor-reviewers approvers: - vendor-approvers </ins>

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: // Returns the object key that denotes the controller object to which vmdk is attached. func getVirtualDiskControllerKey(volPath string, vmDevices object.VirtualDeviceList, dc *object.Datacenter, client *govmomi.Client) (int32, error) { <ins> volPath = removeClusterFromVDiskPath(volPath) </ins> volumeUUID, err := getVirtualDiskUUIDByPath(volPath, dc, client) if err != nil {

With version 1.4.7 I could use a datastore that is in a cluster as a vsphere volume as . When I upgraded to 1.6.0, this same exact path does not work and throws a error. I'm struggling to find what changed, but am guessing it's difference in how the datastore is being looked up in vcenter. Has anyone had luck with this?
We have verified creating pod using static PV and dynamic provisioning, both works fine with Can you post here? cc:
Did that include using a datastore that was in a datastore cluster? That seemed to be working before because the datastore previously (before 1.5) was read out of the global config, which supported datastore path with cluster name. But post 1.5 datastore is parsed from the entry, which if you include the path to the cluster it gets you past the datastore lookup, but then when you go to add the disk it fails because that is not valid to vCenter. It's kind of this weird mixed mode, because to look up the datastore you need the path including the cluster, but to create the file you need the path without the cluster. I presume the PV is working because they use the cloud config values? I was not using a PV but putting a volumePath directly in my deployment yaml. Should I be using a PV to do this? If the deployment is deleted and re-created how do I guarantee that my pod would get the same PV claim? That was why I did not go down that path originally. I'm pretty sure I have narrowed down where in the provider this is happening, but didn't see a good path to get it to work. I have, admittantly, not tried PV/PVCs yet, due to the concern raised above, but could try those to see. The pertinent part of the deployment yaml: The cloud config:
Just confirmed I have the same issue using a PV and PVC....
Here it needs it in format: (which it gets on line 791 by parsing . But here it needs it in true format, without the datastorecluster: Therefore if you do it gets you past line 796 but fails on 802. But if you do then you don't get past line 796.
Thanks for detailed look up. This is the commit which caused this regression. We will work on the fix.

cc:
Ah yes, I see that now. What is the lifecycle of a dynamic disk? Does that disk live on after the PVC is released? After the PV is deleted? It seems like I may just not fully be exercising the power of the volume plugins as well. I would still have the issue with existing volumes as they have data that I need to maintain, but would be interesting for new volumes to look into these more.
RECLAIMPOLICY for PV created using Dynamic Provisioning is Delete PV will get deleted if you delete PVC. Disk also gets deleted from datastore.


Question: metadata: name: %s finalizers: <del> - test-finalizer </del> <ins> - test/finalizer </ins> spec: cronSpec: "* * * * */5" ports:

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: fakeRuntime.AssertKilledPods([]string{"12345678"}) } <ins> func TestKillPodFollwedByIsPodPendingTermination(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() defer testKubelet.kubelet.podKiller.Close() go testKubelet.kubelet.podKiller.PerformPodKillingWork() pod := &kubecontainer.Pod{ ID:        "12345678", Name:      "foo", Namespace: "new", Containers: []*kubecontainer.Container{ {Name: "bar"}, }, } fakeRuntime := testKubelet.fakeRuntime fakeContainerManager := testKubelet.fakeContainerManager fakeContainerManager.PodContainerManager.AddPodFromCgroups(pod) // add pod to mock cgroup fakeRuntime.PodList = []*containertest.FakePod{ {Pod: pod}, } kl := testKubelet.kubelet kl.podKiller.KillPod(&kubecontainer.PodPair{ APIPod:     nil, RunningPod: pod, }) if !(kl.podKiller.IsPodPendingTerminationByUID(pod.ID) || fakeRuntime.AssertKilledPods([]string{"12345678"}) == nil) { t.Fatal("Race condition: When KillPod is complete, the pod should be pending termination or be killed") } } </ins> type testNodeLister struct { nodes []*v1.Node }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: // Then rollback the deployment to revision 10 (doesn't exist in history) should fail. // Finally, rollback current deployment (revision 4) to revision 4 should be no-op. func testRollbackDeployment(f *framework.Framework) { <ins> // extensions/v1beta1 is deprecated, only testing if the server serves this api framework.SkipIfMissingResource(f.DynamicClient, DeploymentGroupVersionResource, f.Namespace.Name) </ins> ns := f.Namespace.Name c := f.ClientSet podName := "nginx"

Which jobs are failing: ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel Which test(s) are failing: [sig-apps] Deployment deployment should support rollback Since when has it been failing: 06-22 - 17:13 PDT Testgrid link: Reason for failure: Anything else we need to know: /priority critical-urgent /kind failing-test /sig apps
I will look in test together with
/milestone v1.16
stopped serving those APIs in 1.16. The test in the release-1.15 branch should skip if the server it is testing against does not serve the deprecated extensions/v1beta1 API
closing
and did not completely resolve the failing tests here or in /reopen
Reopened this issue. <detailsIn response to : and did not completely resolve the failing tests here or in /reopen </details
this job is running 1.14-level tests against master (1.16). the test should be updated to run 1.15 against master
Possible issue here is that we may have misconfigured upgrade/downgrade jobs (upgrading/downgrading to incorrect k8s versions). For full discussion see
/assign
Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close
Closing this issue. <detailsIn response to : Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details