Question: t.Run("PatchCRDSchemaless", func(t *testing.T) { testFieldValidationPatchCRDSchemaless(t, rest, schemalessGVK, schemalessGVR) }) t.Run("ApplyCreateCRDSchemaless", func(t *testing.T) { testFieldValidationApplyCreateCRDSchemaless(t, rest, schemalessGVK, schemalessGVR) }) t.Run("ApplyUpdateCRDSchemaless", func(t *testing.T) { testFieldValidationApplyUpdateCRDSchemaless(t, rest, schemalessGVK, schemalessGVR) }) <ins> t.Run("testFinalizerValidationApplyCreateCRD", func(t *testing.T) { testFinalizerValidationApplyCreateAndUpdateCRD(t, rest, schemalessGVK, schemalessGVR) }) </ins> } // testFieldValidationPost tests POST requests containing unknown fields with

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: done`, testCmd) } <ins> loaderAffinity := &api.Affinity{ PodAffinity: &api.PodAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []api.PodAffinityTerm{{ Namespaces: []string{f.Namespace.Name}, LabelSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{loaderLabelKey: loaderLabelValue}, }, TopologyKey: "kubernetes.io/hostname", }}, }, } </ins> pod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "test-apparmor-",

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         false, </ins> } res2 := TestResource{ resourceName:     "domain2.com/resource2", resourceQuantity: *resource.NewQuantity(int64(1), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev3", "dev4"}}, <ins> topology:         true, </ins> } testResources := make([]TestResource, 2) testResources = append(testResources, res1)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: framework.ExpectNoError(err) } } <ins> ginkgo.DeferCleanup(func(ctx context.Context) { ginkgo.By("Clean all PVs") for nodeName, localVolumes := range allLocalVolumes { ginkgo.By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(ctx, config, localVolumes) } }) </ins> ginkgo.By("Start a goroutine to recycle unbound PVs") backgroundCtx, cancel := context.WithCancel(context.Background()) var wg sync.WaitGroup

master-informing gce-cos-master-serial Kubernetes e2e suite.[It] [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes 2023-02-15 15:12:38 +0000 UTC { failed [FAILED] persistentvolumes "local-pvt2rz2" not found In [AfterEach] at: test/e2e/storage/persistent_volumes- @ 02/21/23 18:15:38.461 } Failed multiple times: /sig storage
I think the AfterEach is racing with the test's AfterEach cleanup: Test's PV recycling go routine: We're supposed to before proceeding with cleanup. do you know if this is still a valid assumption?
/triage accepted
The recent Ginkgo changes didn't change anything regarding Go routines. If a test relies on a Go routine to have stopped, it needs to wait for that explicitly. Note that the created for an node automatically gets cancelled when the callback returns. However, that does not necessarily mean that Go routines terminate immediately.
x-post: This has been analyzed as a test issue and not a product issue, so it shouldn't block tomorrow's v1.27.0-alpha.3 release
Just bumping that this test with this same error as OP is still a common flake as of today. <img width="2768" alt="Screenshot 2023-03-20 at 3 22 23 PM" src="https://user-"
FYI, these are still flaking. Here are some failures over the last few days: !
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command.

<detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
! Just bumping that this is still flaky and failed thrice in a row recently.
The test seems to add a DeferCleanup to stop the goroutine mentioned in From the docs Is guaranteed the order and the DeferCleanup will always run before the existing AfterEach block?
I believe the order is , then (first-in-first-out), (first-in-last-out).
yes is correct. for this test that goroutine certainly adds some complexity that makes things a bit harder to reason about. assuming it's necessary and/or if you'd prefer to avoid making too many changes i'd recommend breaking the up into some s I'd be happy to submit a PR that does just that.
then the order is wrong, because the goroutine should be stopped before AfterEach, that may explain the races please be my guest, tag me and/or patrick for review
Certainly we don't need to fix our tests for us?
I know, I assumed he wanted to do it just for fun or something ... is the problem of text, you can't read between lines :)
lol! i'm happy to do it and contribute.
LOL Thanks a ton for all that you do !
PR is here: - I'm not familiar with all the k8s backslash verbs and nouns but let me know if you need me to do anything on the PR to help move it along. (and/or teach me how this works by telling me what I should do next)


Question: no trailing newline. The first `tr` command will remove the newline if your editor added one. <ins> **Note:** if your cluster enforces **_selinux_** and you will be using [Host Path](#host-path) for storage, then please follow this [extra step](#selinux). </ins> ```shell tr --delete 'n' <password.txt >.strippedpassword.txt && mv .strippedpassword.txt password.txt kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/mysql-wordpress-pd/local-volumes.yaml

Version: Running this example on RHEL 7.2 (assume same for centos), and using hostPath, I get a permissions error when the mysql container does a . If I set selinux to permissive on the host it works. If I manually run the container with it works. I was not able to get it to work when adding to the deployment. My fix was to locate the PVs in by changing the local- file, and doing . I this comment to local- If this issue is valid I will create a quick PR to doc this selinux solution.
See pr


Question: framework.ExpectNoError(err, "failed to list PodTemplate") framework.ExpectEqual(len(podTemplateList.Items), 0, "PodTemplate list returned items, failed to delete PodTemplate") }) <ins> ginkgo.It("should delete a collection of pod templates", func() { podTemplateNames := []string{"test-podtemplate-1", "test-podtemplate-2", "test-podtemplate-3"} ginkgo.By("Create set of pod templates") // create a set of pod templates in test namespace for _, podTemplateName := range podTemplateNames { _, err := f.ClientSet.CoreV1().PodTemplates(f.Namespace.Name).Create(context.TODO(), &v1.PodTemplate{ ObjectMeta: metav1.ObjectMeta{ Name:   podTemplateName, Labels: map[string]string{"podtemplate-set": "true"}, }, Template: v1.PodTemplateSpec{ Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "token-test", Image: imageutils.GetE2EImage(imageutils.Agnhost)}, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod template") framework.Logf("created %v", podTemplateName) } ginkgo.By("get a list of pod templates with a label in the current namespace") // get a list of pod templates podTemplateList, err := f.ClientSet.CoreV1().PodTemplates(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{ LabelSelector: "podtemplate-set=true", }) framework.ExpectNoError(err, "failed to get a list of pod templates") framework.ExpectEqual(len(podTemplateList.Items), len(podTemplateNames), "looking for expected number of pod templates") ginkgo.By("delete collection of pod templates") // delete collection framework.Logf("requesting DeleteCollection of pod templates") err = f.ClientSet.CoreV1().PodTemplates(f.Namespace.Name).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{ LabelSelector: "podtemplate-set=true"}) framework.ExpectNoError(err, "failed to delete all pod templates") ginkgo.By("check that the list of pod templates matches the requested quantity") err = wait.PollImmediate(podTemplateRetryPeriod, podTemplateRetryTimeout, checkPodTemplateListQuantity(f, "podtemplate-set=true", 0)) framework.ExpectNoError(err, "failed to count required pod templates") }) </ins> }) <ins> func checkPodTemplateListQuantity(f *framework.Framework, label string, quantity int) func() (bool, error) { return func() (bool, error) { var err error framework.Logf("requesting list of pod templates to confirm quantity") list, err := f.ClientSet.CoreV1().PodTemplates(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{ LabelSelector: label}) if err != nil { return false, err } if len(list.Items) != quantity { return false, err } return true, nil } } </ins>

[x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "bytes" <ins> "crypto/sha256" </ins> "io/ioutil" "net/http" "net/url"

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: ContainerRuntimeOptions: *NewContainerRuntimeOptions(), CertDirectory:           "/var/lib/kubelet/pki", RootDirectory:           v1alpha1.DefaultRootDir, <del> // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations // with cloud providers instead of in the core repo. // More details here: https://github.com/kubernetes/kubernetes/issues/50986 CloudProvider:      v1alpha1.AutoDetectCloudProvider, RotateCertificates: false, </del> <ins> RotateCertificates:      false, </ins> } }

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") // .status.num should be 999 expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num")    // .spec.num should remain 10 <del> rv, found, err := unstructured.NestedString(patchedNoxuInstance.UnstructuredContent(), "metadata", "resourceVersion") if err != nil { t.Fatal(err) } if !found { t.Fatalf("metadata.resourceVersion not found") } // this call waits for the resourceVersion to be reached in the cache before returning. // We need to do this because the patch gets its initial object from the storage, and the cache serves that. // If it is out of date, then our initial patch is applied to an old resource version, which conflicts // and then the updated object shows a conflicting diff, which permanently fails the patch. // This gives expected stability in the patch without retrying on an known number of conflicts below in the test. // See https://issue.k8s.io/42644 _, err = noxuResourceClient.Get(context.TODO(), "foo", metav1.GetOptions{ResourceVersion: patchedNoxuInstance.GetResourceVersion()}) if err != nil { t.Fatalf("unexpected error: %v", err) } </del> // no-op patch <del> t.Logf("Patching .status.num again to 999") patchedNoxuInstance, err = noxuResourceClient.Patch(context.TODO(), "foo", types.MergePatchType, patch, metav1.PatchOptions{}, "status") if err != nil { t.Fatalf("unexpected error: %v", err) </del> <ins> rv := "" found := false // TODO: remove this retry once http://issue.k8s.io/75564 is resolved, and expect the resourceVersion to remain unchanged 100% of the time. // server-side-apply incorrectly considers spec fields in patches submitted to /status when updating managedFields timestamps, so this patch is racy: // if it spans a 1-second boundary from the last write, server-side-apply updates the managedField timestamp and increments resourceVersion. for i := 0; i < 10; i++ { rv, found, err = unstructured.NestedString(patchedNoxuInstance.UnstructuredContent(), "metadata", "resourceVersion") if err != nil { t.Fatal(err) } if !found { t.Fatalf("metadata.resourceVersion not found") } t.Logf("Patching .status.num again to 999") patchedNoxuInstance, err = noxuResourceClient.Patch(context.TODO(), "foo", types.MergePatchType, patch, metav1.PatchOptions{}, "status") if err != nil { t.Fatalf("unexpected error: %v", err) } // make sure no-op patch does not increment resourceVersion expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num") if patchedNoxuInstance.GetResourceVersion() == rv { break } t.Logf("resource version changed - retrying") </ins> } <del> // make sure no-op patch does not increment resourceVersion expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num") </del> expectString(t, patchedNoxuInstance.UnstructuredContent(), rv, "metadata", "resourceVersion") // empty patch

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestSubresourcePatch Testgrid link: Reason for failure: Unknown. A diff with the original object would be helpful. Anything else we need to know: /sig api-machinery
OK Trying to reproduce, see what's going on.
Clearly this is because SSA is updating the tracked entry's time. It happens when the two request don't happen in the same second. Having a between the two requests makes this into a definitive error.
/wg apply
This seems to be because of . A repeated patch like will always trigger a new managedFields entry to be created, since it's always going to change the object (status or spec will be wiped depending on the endpoint), so the patch will always cause the object to change, which will always cause the entry to be created.
/assign
/assign
/assign
Still seeing this, bumped into it over in
deflake in while the underlying issue is being worked on in


Question: framework.ExpectEqual(labelPod.Spec.NodeName, nodeName) }) <del> /* Release : v1.18 Testname: Scheduler, taints-tolerations matching Description: Find an available node and taint it with a key with effect NoSchedule. Schedule a pod with a corresponding nodeLabel and toleration spec such that it should only be able to run on the selected node. Ensure that the pod is scheduled and running on the node. */ framework.ConformanceIt("validates that taints-tolerations is respected if matching [Disruptive]", func() { </del> <ins> // 1. Run a pod to get an available node, then delete the pod // 2. Taint the node with a random taint // 3. Try to relaunch the pod with tolerations tolerate the taints on node, // and the pod's nodeName specified to the name of node found in step 1 ginkgo.It("validates that taints-tolerations is respected if matching", func() { </ins> nodeName := getNodeThatCanRunPodWithoutToleration(f) ginkgo.By("Trying to apply a random taint on the found node.")

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: kind serial conformance jobs Which test(s) are failing: The conformance test SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching [Disruptive] [Conformance] Since when has it been failing: The test was promoted in kubernetes/kubernetes passing the kind-ipv6 job but it keeps failing after that in both ipv4 and ipv6 conformance jobs Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: Investigating Anything else we need to know: Seems the job in presubmit is not testing Serial conformance :/
/cc
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
EDIT: posting GCE results because it is failing there too
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/retitle Failing Test: SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching [Disruptive] [Conformance]
/remove-sig network /sig scheduling /area conformance
I've opened to revert


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package testing import ( gotest "testing" "k8s.io/component-base/featuregate" ) func TestSpecialGates(t *gotest.T) { gate := featuregate.NewFeatureGate() gate.Add(map[featuregate.Feature]featuregate.FeatureSpec{ "alpha_default_on":         {PreRelease: featuregate.Alpha, Default: true}, "alpha_default_on_set_off": {PreRelease: featuregate.Alpha, Default: true}, "alpha_default_off":        {PreRelease: featuregate.Alpha, Default: false}, "alpha_default_off_set_on": {PreRelease: featuregate.Alpha, Default: false}, "beta_default_on":         {PreRelease: featuregate.Beta, Default: true}, "beta_default_on_set_off": {PreRelease: featuregate.Beta, Default: true}, "beta_default_off":        {PreRelease: featuregate.Beta, Default: false}, "beta_default_off_set_on": {PreRelease: featuregate.Beta, Default: false}, "stable_default_on":         {PreRelease: featuregate.GA, Default: true}, "stable_default_on_set_off": {PreRelease: featuregate.GA, Default: true}, "stable_default_off":        {PreRelease: featuregate.GA, Default: false}, "stable_default_off_set_on": {PreRelease: featuregate.GA, Default: false}, }) gate.Set("alpha_default_on_set_off=false") gate.Set("beta_default_on_set_off=false") gate.Set("stable_default_on_set_off=false") gate.Set("alpha_default_off_set_on=true") gate.Set("beta_default_off_set_on=true") gate.Set("stable_default_off_set_on=true") before := map[featuregate.Feature]bool{ "AllAlpha": false, "AllBeta":  false, "alpha_default_on":         true, "alpha_default_on_set_off": false, "alpha_default_off":        false, "alpha_default_off_set_on": true, "beta_default_on":         true, "beta_default_on_set_off": false, "beta_default_off":        false, "beta_default_off_set_on": true, "stable_default_on":         true, "stable_default_on_set_off": false, "stable_default_off":        false, "stable_default_off_set_on": true, } expect(t, gate, before) cleanupAlpha := SetFeatureGateDuringTest(t, gate, "AllAlpha", true) expect(t, gate, map[featuregate.Feature]bool{ "AllAlpha": true, "AllBeta":  false, "alpha_default_on":         true, "alpha_default_on_set_off": true, "alpha_default_off":        true, "alpha_default_off_set_on": true, "beta_default_on":         true, "beta_default_on_set_off": false, "beta_default_off":        false, "beta_default_off_set_on": true, "stable_default_on":         true, "stable_default_on_set_off": false, "stable_default_off":        false, "stable_default_off_set_on": true, }) cleanupBeta := SetFeatureGateDuringTest(t, gate, "AllBeta", true) expect(t, gate, map[featuregate.Feature]bool{ "AllAlpha": true, "AllBeta":  true, "alpha_default_on":         true, "alpha_default_on_set_off": true, "alpha_default_off":        true, "alpha_default_off_set_on": true, "beta_default_on":         true, "beta_default_on_set_off": true, "beta_default_off":        true, "beta_default_off_set_on": true, "stable_default_on":         true, "stable_default_on_set_off": false, "stable_default_off":        false, "stable_default_off_set_on": true, }) // run cleanups in reverse order like defer would cleanupBeta() cleanupAlpha() expect(t, gate, before) } func expect(t *gotest.T, gate featuregate.FeatureGate, expect map[featuregate.Feature]bool) { t.Helper() for k, v := range expect { if gate.Enabled(k) != v { t.Errorf("Expected %v=%v, got %v", k, v, gate.Enabled(k)) } } } </ins>

PR was reverted by because it (erroneously) relied on OpenAPIV3 feature to be enabled, but would non-deterministically leave the feature disabled after running. I tracked it down to the fact that this test disables all or features in a loop which iterates over a , which are known to have unstable iteration order: When key is processed before key , then would disable before s default value is to be saved. Normally, ignores features that have been explicitly set, so it only affects those which fall back on the value. But this issue does not just affect but any non-GA feature. I suggest the loop in be made to skip the special feature keys such as . /cc /cc The value before every feature gate should be restored after running . This test always fails on my machine (just pasted it into ): No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
Wow, good catch, sorry to make you track down what turned out to be a bug in our test
/sig auth
/area test
/triage accepted


Question: } // BackoffManager manages backoff with a particular scheme based on its underlying implementation. It provides <del> // an interface to return a timer for backoff, and caller shall backoff until Timer.C returns. If the second Backoff() // is called before the timer from the first Backoff() call finishes, the first timer will NOT be drained. </del> <ins> // an interface to return a timer for backoff, and caller shall backoff until Timer.C() drains. If the second Backoff() // is called before the timer from the first Backoff() call finishes, the first timer will NOT be drained and result in // undetermined behavior. </ins> // The BackoffManager is supposed to be called in a single-threaded environment. type BackoffManager interface { Backoff() clock.Timer

I noticed that this code: was printing which should not happen. Also occurs in wait.Forever. I suspect that in regressed this as that was the most recent major update and I only started seeing this on a recent apimachinery bump.
/sig api-machinery /assign as the reviewer
GitHub didn't allow me to assign the following users: zhan849. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery /assign as the reviewer Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
This is a correctness regression because the wait methods expect not to be run run more often than the period and may guard expensive behavior. Suggest a revert followed by a more rigorous test and review on merge.
thanks for reporting this, and I apologize about the regression I made. I did some quick test, and I am able to reproduce it occasionally, which indicate a race somewhere: Test: Failed output: Successful output? In the original code I checked in, I initialized the timer in the backoff manager construction time: However, after I changed to initializing the timer when is called, I'm not able to reproduce the race. The results indicated that the timer initialization is likely to be the place that caused the regression , and seems to be able to get fixed by change the way the timer gets initialized. However I'm unable to identify why it could cause such race the first time. Are you able to provide any insights? /cc
why are we initializing a timer with a zero duration? that looks related to this issue:
I see us resetting a timer we have no guarantee is stopped/expired:
dropped the information needed to accurately stop/reset the timer as documented
Hi thanks for providing the information! seems like the correct place to initialize the timer is in the "Backoff" method, since zero duration is problematic.

When I was putting up the interface, I documented that the timer will not be drained, and the reason is that returns a timer explicitly for user to block on that, so it might be confusing to block in itself Shall we change the impl to always drain the channel in backoff instead or relying on user to use it correctly? with this being said, will be changed to , which means that the will still rely on user's providing information about whether the channel is drained or not.
/cc thoughts? ^^
- sorry for that - we hit the bug with resetting timer not the first time, I should have caught that (I remember at least two other bugs with that). I'm not sure revert is a good option, because there were PRs that were chained on that: - this would have to be a very careful manual revert which I'm not sure it justifies the risk, given that 1.18 is already out. Currently the behavior is unpredictable, because if getNextbackoof() called for the first time is non-zero, it may or may not be ready immediately depending on the race. That is obviously bad. Initializing with infinity may solve this particular problem, but I need to think more about it to ensure that we don't have any problems in the later phase too...
thanks ! and sry again I did not read the timer's doc carefully enough but simply followed the Clock interface :( To fix, we can also initialize the timer to nil and create a fresh one when is first called, it's safe as this component is designed to run in single-threaded environment. But to ensure timer is drained before reset, we need to have user explicitly pass in "if timer is drained" into the to accurately decide if needs to drain. Or, is it ok to document the behavior and assume user will drain on before calling next ?
Very quick (I might have missed something) glance from the code suggests, that the only place we would need to change is actually BackoffUntil method. So passing the "if timer is drained" bit seems less error prone imho.
We should avoid changing the interface in a patch release.

I think initializing the timer in Backoff() and documenting the returned timer must be expired and drained before calling Backoff() again is the most compatible thing we can do:
Thanks and I'm also more inclined to Jordan's proposal, besides avoid interface change, if user are able to accurately pass "if timer is drained" info, it's fair to let user finish draining. do you have a PR already? of if not, I can make one, and add more tests if possible about this failure case.
I do not have a PR open


Question: func (oe *operationExecutor) MountVolume( waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, <del> actualStateOfWorld ActualStateOfWorldMounterUpdater) error { </del> <ins> actualStateOfWorld ActualStateOfWorldMounterUpdater, isRemount bool) error { </ins> mountFunc, err := oe.operationGenerator.GenerateMountVolumeFunc( <del> waitForAttachTimeout, volumeToMount, actualStateOfWorld) </del> <ins> waitForAttachTimeout, volumeToMount, actualStateOfWorld, isRemount) </ins> if err != nil { return err }

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: "k8s.io/utils/ptr" ) <ins> // generateRandomIP is copied from pkg/apis/networking/fuzzer/fuzzer.go func generateRandomIP(is6 bool, c fuzz.Continue) string { n := 4 if is6 { n = 16 } bytes := make([]byte, n) for i := 0; i < n; i++ { bytes[i] = uint8(c.Rand.Intn(255)) } ip, ok := netip.AddrFromSlice(bytes) if ok { return ip.String() } // this should not happen panic(fmt.Sprintf("invalid IP %v", bytes)) } // generateRandomCIDR is copied from pkg/apis/networking/fuzzer/fuzzer.go func generateRandomCIDR(is6 bool, c fuzz.Continue) string { ip, err := netip.ParseAddr(generateRandomIP(is6, c)) if err != nil { // generateRandomIP already panics if returns a not valid ip panic(err) } n := 32 if is6 { n = 128 } bits := c.Rand.Intn(n) prefix := netip.PrefixFrom(ip, bits) return prefix.Masked().String() } // getRandomDualStackCIDR returns a random dual-stack CIDR. func getRandomDualStackCIDR(c fuzz.Continue) []string { cidrIPv4 := generateRandomCIDR(false, c) cidrIPv6 := generateRandomCIDR(true, c) cidrs := []string{cidrIPv4, cidrIPv6} if c.RandBool() { cidrs = []string{cidrIPv6, cidrIPv4} } return cidrs[:1+c.Intn(2)] } </ins> // Funcs returns the fuzzer functions for the kube-proxy apis. func Funcs(codecs runtimeserializer.CodecFactory) []interface{} { return []interface{}{

pull-kubernetes-unit TestRoundTripFuzzing Only noticed the failure once in No response No response No response
/sig network may be caused by
This flakes in /cc
/assign so the problem is that sometimes the fuzzer generates a random string value that contains a comma, and when that gets converted back from v1alpha1, the comma is treated as punctuation rather than as part of the value. we should probably manually fuzz ClusterCIDRs to contain 0-2 actual CIDR values
/triage accepted
Saw here:


Question: case m.isFor("GET", "/namespaces/default/pods/bar"): return &http.Response{StatusCode: 404, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, &corev1.Pod{})}, nil case m.isFor("GET", "/pods"): <ins> if test.failUponEvictionOrDeletion && atomic.LoadInt32(&evictions) > 0 || atomic.LoadInt32(&deletions) > 0 { return nil, errors.New("request failed") } </ins> values, err := url.ParseQuery(req.URL.RawQuery) if err != nil { t.Fatalf("%s: unexpected error: %v", test.description, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: return "", imagePullResult.err.Error(), ErrImagePull } <del> m.logIt(ref, v1.EventTypeNormal, events.PulledImage, logPrefix, fmt.Sprintf("Successfully pulled image %q", container.Image), klog.Info) </del> <ins> m.logIt(ref, v1.EventTypeNormal, events.PulledImage, logPrefix, fmt.Sprintf("Successfully pulled image %q in %v", container.Image, time.Since(startTime)), klog.Info) </ins> m.backOff.GC() return imagePullResult.imageRef, "", nil }

What would you like to be : The 'Successfully pulled image' event should contain the duration for image pulling. Why is this needed: Currently the success event is in this form: User has to trace back to the corresponding event for the beginning of image pulling and calculate the duration him/herself:
/sig node /assign


Question: } } <ins> func TestReconcileExtendedResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.kubeClient = nil // ensure only the heartbeat client is used extendedResourceName1 := v1.ResourceName("test.com/resource1") extendedResourceName2 := v1.ResourceName("test.com/resource2") cases := []struct { name         string existingNode *v1.Node expectedNode *v1.Node needsUpdate  bool }{ { name: "no update needed without extended resource", existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), }, }, }, needsUpdate: false, }, { name: "extended resource capacity is zeroed", existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(2), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(10), resource.DecimalSI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(2), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(10), resource.DecimalSI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(0), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(0), resource.DecimalSI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), extendedResourceName1:       *resource.NewQuantity(int64(0), resource.DecimalSI), extendedResourceName2:       *resource.NewQuantity(int64(0), resource.DecimalSI), }, }, }, needsUpdate: true, }, } for _, tc := range cases { defer testKubelet.Cleanup() kubelet := testKubelet.kubelet initialNode := &v1.Node{} needsUpdate := kubelet.reconcileExtendedResource(initialNode, tc.existingNode) assert.Equal(t, tc.needsUpdate, needsUpdate, tc.name) assert.Equal(t, tc.expectedNode, tc.existingNode, tc.name) } } </ins> func TestValidateNodeIPParam(t *testing.T) { type test struct { nodeIP   string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In 1.11, removed externalID from Node.Spec which used to be used by kubelet to detect node recreation in certain environments during node upgrade. This changed node upgrade behavior in those environments because in tryRegisterWithAPIServer(node *v1.Node), without externalID checking, we now consider the recreated node after upgrade as a previously registered node and thus will get old node state from API server instead of regenerating them based on node local state. For extended resource reported by a device plugin, this causes problem because after node upgrade/recreation, the device plugin DaemonSet pod needs to be restarted by Kubelet, finishes certain setup, and then report to kubelet that its resource is available at the node. However, the node status capacity and allocatable from the API server still contains the old state before the node upgrade. After kubelet syncs up with the API server, the node status field gets overwritten and previously reported extended resources will appear on the node status capacity/allocatable even though the node is not ready for pods to consume such resources. As the result, Kubelet may start a pod requesting such resource without proper container runtime setup it needs to learn from device plugin. Note to cope with the externalID removal change, has been updated () to explicitly uncordon the node after checking it is restarted and becomes ready. This however doesn't help device plugin case because node readiness doesn't depend on individual extended resource readiness. This issue is to explore how we can fix this problem on head and 1.11. In particular, I wonder whether we may switch to the model that we consider Kubelet as the only source to update extended resource capacity/allocatable in node status. I.e., don't support manually updating node status capacity/allocatable fields as documented in This would allow us to re-generate node status capacity/allocatable every time in node status update, which I think is much simpler and more robust. As I heard, some folks have been using this mechanism to do simple resource exporting/accounting through a central controller. I wonder whether people are open to switch to the device plugin model in those cases.

Even though those extended resources may not require special container setup, it seems a more secure model to have Kubelet totally own its node status. FYI, here is the related OSS issue: where we first introduced extended resource concept. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Running the gpu upgrade test from with --upgrade-target set to any 1.11+ version would hit the issue. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
/cc
Having extended resources be a concept encapsulating both device plugin resources and user set resources (opaque int resources) has been a hassle to deal with (as it introduced unexpected behavior in the device plugin lifecycle). We should also consider a few alternatives : separating them from a code perspective (i.e: different fields in node status) . separating them from a name (/dns) perspective.
Agree with If we do need to support the use case of exporting a resource in node status capacity/allocatable not through Kubelet, seems better to introduce special naming convention for such resources.
We had a discussion in another issue about using the check point to keep the node in not ready until all expected device plugins have re-registered. Is that option still on the table? I’m hesitant to suggest changing the semantics of extended resources again, at least until we are sure it’s a valid long term solution.
We did implement the mechanism to use check point data to determine previously registered device plugin resource and keep the resource not ready till the expected device plugin re-register. It helps normal node reboot case but not for node upgrade/recreation case, because boot disk is recreated in the later case and the check point data is also gone.
Kubelet's device manager should recover gracefully from a deleted checkpoint. It should be able to identify which resources in the node object are extended resources and it should remove them if the checkpoint disappears. Checkpoints can disappear for any number of reasons and kubelet needs to react appropriately. cc since this is fallout from . Device manager was relying on node object deletion/recreation to function after GCE upgrades.

+1 Node API object deletion/recreation was only for the case where you have a completely different instance (and deletion was only done because the external ID was made immutable). Deletion/recreation in response to an upgrade or machine restart is a form of data loss (admin- taints, labels, etc, are unrecoverable). Additionally, depending on Node object deletion on upgrade as previously seen in GCE is highly environment-dependent and will not be consistent between cloud-providers, environments, etc.
how about during node registration reconciliation, we reconcile extended resources by setting their capacity to zero? That means any external controller that exports extended resources by directly patching node status capacity/allocatable needs to re-do that after any kubelet restart, which I guess is already what they are doing to handle node upgrade. The required change should be small:
what are the implications of doing that on every kubelet process restart? does available resource flutter impact already-running workloads on a node (does the kubelet evaluation of PodFitsResources evict running workloads? does any controller notice workloads running on a node are using resources the kubelet says it doesn't have, and evict them?)
For extended resources exported through device plugin, kubelet device manager will handle their restarts, currently based on checkpoint data and in the future, we will explore whether we may store this information in some form of API. If the checkpoint data is gone, pods will fail kubelet admission handling and the pod's controller should recreate a new pod. For extended resources exported by an external controller through direct node status capacity patching, the controller should watch for kubelet readiness change, and repatch the field after kubelet becomes ready again.
Having an external controller watching the nodes status update to repatch the nodes sounds like a bad "user" experience. Maybe looking at it from a user's perspective and then figuring out what we'll do is a better way to look at this?

The behavior I'd expect to see is the following: For non device plugin extended resources A user patches the node status with the extended resource If the node restarts I don't expect the Allocatable / Capacity fields to change If the node is upgraded I don't expect the Allocatable / Capacity fields to change If the node is recreated I don't expect the Allocatable / Capacity fields to change For device plugin extended resources Kubelet advertises the extended resources when the device plugin registers itself against kubelet If the node restarts I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is upgraded I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is recreated I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices As for the case where the checkpoint data is gone, should we expect the pod's controller to recreate a new pod? That seems like a surprising behavior are we doing this somewhere else? Does the general behavior outlined here seem sane to the people in this thread?
I think in 1.12 time frame, we should start to scope the work on improving extended resource life cycle. Right now, as we are approaching 1.11 code freeze timeline, I am trying to figure out a solution that is least surprise to any existing workloads. Till 1.11, if there is indeed a real use case where an external component directly patches node status capacity to export a resource, the solution already needs to handle the capacity loss after node upgrade by repatching the field. That is the least surprising solution I can think of now. If folks have concerns on the breaking behavior this may introduce, we should seriously consider reverting part of in 1.11.
+1 for considering reverting for v1.11 timeframe. What would be the side-effect of that ? Ownership of fields in the Node object has been a long standing issue. For example node labels suffered from the same issue. I do agree that kubelet has to be resilient and has to recover from control plane data loss with minimal impact on running applications. Having to expose arbitrary extended resources as device plugin resources is also not ideal.

For instance, if one wants to expose inodes or network bandwidth as a resource, device plugins would not be an ideal interface (we can consider fixing this). k8s node upgrades should ideally involve cordoning (or tainting) the node during the duration of the upgrade. During this upgrade phase the node is primarily expected to keep existing pods running (if the upgrades are in place) and the system (scheduler) is not expected to place additional workloads. Since the node object is a shared resource, every component that relies on the node object must ensure eventual consistency. Arbitrary users are not expected to update/patch the node object since it is core to k8s functionality. does node feature discovery continually update the state on the node object to reflect the current state on the node? Until we have a better resource representation format, kubelet will not be able to tag ownership for individual extended resources attached to the node object. I think this limitation is acceptable and the users who introduce additional extended compute resources are expected to be sophisticated enough that they can run long running processes to ensure their additional compute resources are accurately reflected in the node object. As part of Resource Classes, and v2 compute resource APIs, we should consider incorporating a owner reference into concrete compute resource objects which can help with the ownership issues. Node upgrades also need non-trivial automation today to ensure that all node dependencies (runtime, kernel, OS, network, storage plugins, etc.) are logically ready prior to adding a node back to the scheduling pool. Assuming patches to node compute resources are continually updated and nodes are cordoned during the upgrade phase, node upgrade automation can ensure that all the expected extended compute resources on certain nodes are updated to a non-zero capacity prior to adding a node back to the scheduling pool (uncordon) after an upgrade. In addition, additional automation can be employed to auto-taint nodes that do not expose non-zero capacity for expected extended resources. For the near future, kubelet can on-startup can only try to restore state on a best effort basis. If checkpoints are not around, kubelet may override existing patched fields. Device plugins and other extended resource plugins will have to ideally update capacity/allocatable to reflect the state on the local node prior to marking an upgrade as complete.

it would delay removal of the self-deletion permission for yet another release (already pushed to 1.12 because of version-skew), meaning node taints would not be usable for isolating compromised nodes until at least 1.14 (because the associated kubelet could simply delete/re-register their Node object). I also don't think it would actually solve this issue. According to , even were we to revert and restore kubelet self-deletion, changes made in stabilized the computation of external id in GCE such that it no longer changes as part of node upgrades in 1.11+. Undoing that, destabilizing the GCE computed external id, and reintroducing self-deletion cases to solve the stale resource issue in one particular cloud environment doesn't seem like a good approach.
The feature that would be slipping is which is particularly critical for and because it's required to run a cluster with dedicated nodes securely. We would be slipping another release on something that is already painfully slow due to backward compatibility considerations. The PR () that changed the behavior device manager was depending on merged two months ago. It's really unfortunate that this report came in so late. I'm pretty convinced that this is a device manager issue that is only mitigated by clearing the node object. Is there anything I can do to help fix this? I'm happy to argue that changes here should be exempt from the codefreeze deadline.
The node feature discovery project only deals with node labels, and not extended resources. NFD fairly recently the ability to run as a daemon, but also retained the option to publish labels once and exit.
So far, it sounds is perhaps the most realistic solution for 1.11 time frame. Although we want to support good user experience, I do feel we need to balance the complexity we need to add into the system v.s. the use cases we want to optimize for. I feel the ask for the external component to repatch node status capacity after kubelet restart seems reasonable and perhaps is already implemented by any serious use cases.
The change that was backported to 1.10 (and maybe all versions?) for this seems to have had the unexpected consequence of zeroing out all of our extended resources upon kubelet restart and causing a large number of our pods to get evicted in our QA cluster since they all use extended resources.

I filed an issue here: if you have any advice on intentions/workarounds. For now, we reverted that commit in our internal build as a workaround. Thanks!
How is your extended resource exported on a node? Are they being exported by a device plugin or do you directly update your node objects? If it is the later case, do you have a central controller that may watch for node updates and re-export your resources?
When the host is first configured, our configuration management system running on the node starts the kubelet service, waits for kubelet to register itself in the API by polling and then patches the node capacity directly. We do not have a central controller to rexport the resources. It also sounds like watching and rexporting is not sufficient, since the value would still hit zero for some brief period of time before the central controller could react, at which point kubelet will have already begun hard evicting all the pods. The only central solution I think would work is doing it from an admission controller that ignores the kubelet's attempt to set the extended resources in our namespace to zero.
Can the configuration management system re-patch node capacity after each Kubelet registration?
The kubelet service can restart at any time, whether the config management system is currently configuring the system or not. Ex: kubelet could get OOM-killed, crash, fail healthchecks, etc. If every kubelet restart is a kubelet registration, then no, unless we made the kubelet service also launch a script every time it restarts. So lets say that we hypothetically did configure a script to re-patch the node capacity every time kubelet restarts: there is still an unsolvable race condition where the node temporarily has its capacity set to zero. Consider the situation: is up and healthy. The value for . kubelet service restarts reconciliation logic sets . kubelet immediately begins hard-evicting every pod that requested a . script patches the node capacity back to , but it's too late, every pod has already been killed non-gracefully by kubelet due to the resource having been reset back to 0 at step 3.
Sigh, this is the deficiency we documented in Unfortunately, I couldn't think of a clean solution without API change. One thing we may consider is to instead of zeroing out extended resource capacity, we delete them.

Because we are currently taking an open policy on extended resource admission, pods already assigned on the node wouldn't be affected, and your configuration manager can re-patch node later. Another possible workaround you may consider is to implement device plugin API in your configuration manager. Because Kubelet maintains device plugin checkpoint, it would allow pods to pass admission based on the checkpointed state. cc for thoughts on this issue.
Note for the first mentioned workaround, i.e., instead of zeroing out extended resource capacity, we delete them, I think we will rely on to re-populate device plugin resource runtime states during container restart retries. This is not ideal but we already need this to handle the edge cases that node API object may get deleted but pods are still assigned to node, or pods can be admitted before node re-register with API server.
Unfortunately, I couldn't think of a clean solution without API change. The effect of this change as it stands is that kubernetes users run the risk of having 100% of pods in their cluster completely wiped out with no respect for PDB, terminationGracePeriodSeconds, and preStop hooks just by restarting the kubelet processes. This really feels like a critical bug that has much worse consequences than the bugs the change sought to solve. If this is going to continue being the behavior, there should be a really loud disclaimer on which currently makes it appear that extended resources are trivial to manage. I had a team of 5 people restoring 350 stateful database clusters to a clean state for 3+ hours on Monday night. Restarting kubelets was never a destructive action before, so we weren't restarting them as slowly and carefully as if we were rolling out a config change. If this had happened in our production cluster instead, it could have brought the company down for hours. So you're saying that simply removing the entry from the capacity map instead of zeroing it would allow pods using those resources to continue running? When you re-patch the capacity, does that break the accounting at all? (Ex if 3 pods were consuming an extended resource, and the resource is deleted and re-, does the scheduler realize that 3 of that resource are still consumed by pods, or would it reset to thinking zero pods are using it or would it count them correctly?) I'm willing to do that if it's really necessary.

That said, I couldn't find the code that does that during kubelet registration in 1.10, what happens if the device plugin is down during a kubelet restart? Does kubelet then zero out the resource? Does the device plugin need to be running for a container using the resource to successfully start? As a possible suggestion that doesn't require an (http) API change or the cluster admins to run a device plugin daemon, has it been considered to implement similar to ? on kubelet. That way the admin would have a trivial way to explicitly tell kubelet to always register with a static extended resource capacity value, making the reconciliation process much more straightforward.
Paul Furtado <: state. It would see three pods are already assigned to the node and wouldn't assign more. being scheduled on the node till the resource is ready and device plugin re-registers. However, in Kubelet device plugin manager, we use check-pointed state to bypass predicate admission handler so running pods wouldn't be affected. The function that does this is at: Jiaying
Could we add a whitelist configration to skip reconcile specified extended resource? Or skip a specific prefix
That would require changing kubelet flags, which I am hoping to avoid.
but we have no way to distinguish the extended resource is reported by device plugin or patched by users directly. Seems we can't use non device plugin extended resources anymore. b/c we can't guarantee to repatch resources before pods evicted.
It is indeed a question whether we want to support the use case of directly patching node object to export a resource. Capacity/Allocatable are node status fields. By definition, they should be observable from the node. I.e., even when Kubelet is isolated from the API server, it should be able to reconstruct its status fields. I think even without zeroing out extended resource capacity during API registration, there is perhaps already a race between pod admission and API registration. They are performed by separate go routines on Kubelet and there is no guaranteed ordering between these two processes. This wan't a problem in early versions of 1.10 because we changed extended resource admission into open policies. It remains a question whether we want to keep this open admission policies or not.

Adding an explicit Kubelet flag to specify what extended resources (and perhaps also their capacities) it should export may be a better way than directly patching node API object, because it would make that information truly observable. Whether we have enough use cases to justify this flag is perhaps a question we should discuss in sig-node.
Could we have a "generic device plugin" / "extended resource device plugin"? e.g: You deploy this device plugin and start interacting with a CRD rather than patching the node object in the spec?
For this case, shall we take the following ways to avoid pod restarts when kubelet restart? (1) we can use the default namespace prefix ,"" to avoid being zero out (2) remove extended resource before kubelet start, and patch add again when kubelet started Both of the two ways can make pod with extended resource not evicted then kubelet restart


Question: All Docker names are suffixed with a hash derived from the file path (to allow concurrent usage on things like CI machines) and a version number.  When the version number changes all state is cleared and clean build is started.  This allows the build infrastructure to be changed and signal to CI systems that old artifacts need to be deleted. <ins> ## Build artifacts The build system output all its products to a top level directory in the source repository named `_output`. These include the binary compiled packages (e.g. kubectl, kube-scheduler etc.) and archived Docker images. If you intend to run a component with a docker image you will need to import it from this directory with the appropriate command (e.g. `docker import _output/release-images/amd64/kube-scheduler.tar k8s.io/kube-scheduler:$(git describe)`). </ins> ## Releasing The [`build/release.sh`](release.sh) script will build a release.  It will build binaries, run tests, (optionally) build runtime Docker images.

I spent quite a while figuring out how to build the docker images for k8s, when I finally succeeded, the build process told me the images are deleted. I was quite puzzled: After I built the images again. only to receive the same message, I started digging, and I found out that the images are exported with . I think a notification where the images are found would be useful. If you do it a couple of times you remember this, but for newcomers or the occasional contributor this can be very frustrating. A log message I'm happy to submit a PR, if you think this is useful.
FWIW all output goes under , I think we should ensure this is sufficiently documented at least.
It took me a while to find this. Also, I tried using the scripts directly without much success. The makefile is much more helpful. It might be worth mentioning that in the linked document.
i had the same problem
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /help /triage accepted /kind documentation /retitle document _output directory though also a log message might be good? thing is where to stick it, we have a lot of variations on building that won't pass through the same codepaths ... IMO this absolutely ought to be documented at the very least
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help /triage accepted /kind documentation /retitle document _output directory though also a log message might be good? thing is where to stick it, we have a lot of variations on building that won't pass through the same codepaths ... IMO this absolutely ought to be documented at the very least Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
to begin with, I only a note about the location of the build artifacts.

The is referenced already in many places in the developer guide. Maybe it is better to completely remove the mentioning of manual invocation of the individual scripts and instead recommend the specific make target? IMHO, this is more consistent with the rest of the documentation. However, it is definitely out of scope here. If you or other dev's think it is a good idea, I can submit another PR for that.
completely remove the mentioning of manual invocation of the individual scripts and instead recommend the specific make target? Generally I think this is true, but there are some specific scripts in the repo where it can make sense to just invoke them directly and a few that are not in the makefile at all (like ) so it doesn't necessarily apply to everything, but I'd agree it generally applies to building at least. And I think that it would be good to make this more consistent
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Ping. Can this be merged?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var matched string scanner := bufio.NewScanner(strings.NewReader(s)) for scanner.Scan() { <ins> // losetup output has symlinks expanded if strings.HasSuffix(scanner.Text(), "("+realPath+")") { matched = scanner.Text() break } // Just in case losetup changes, check for the original path too </ins> if strings.HasSuffix(scanner.Text(), "("+path+")") { matched = scanner.Text() break

How to reproduce it (as minimally and precisely as possible): /var/lib/kubelet a symlink kubelet a block PVC + a pod that uses it. the pod. -The pod is forever. In kubelet logs: The reason is that shows: kubelet can't find there, because losetup shows real path . Environment: Kubernetes version (use ): 1.18 /sig storage /priority important-soon
/triage accepted


Question: AWS            | Saltstack    | Ubuntu | OVS         | [docs](../../docs/getting-started-guides/aws.md)       | Community (@justinsb)        | Uses K8s version 0.5.0 Vmware         | CoreOS       | CoreOS | flannel     | [docs](../../docs/getting-started-guides/coreos.md)    | Community (@kelseyhightower) | Uses K8s version 0.15.0 Azure          | Saltstack    | Ubuntu | OpenVPN     | [docs](../../docs/getting-started-guides/azure.md)     | Community                    | <del> Bare-metal     | custom       | Ubuntu | flannel     | [docs](../../docs/getting-started-guides/ubuntu.md)    | Community (@resouer @WIZARD-CXY)       | use k8s version 0.15.0 </del> <ins> Bare-metal     | custom       | Ubuntu | flannel     | [docs](../../docs/getting-started-guides/ubuntu.md)    | Community (@resouer @WIZARD-CXY)       | use k8s version 0.18.0 </ins> Docker Single Node        | custom       | N/A    | local       | [docs](docker.md) | Project (@brendandburns) | Tested @ 0.14.1 | Docker Multi Node        | Flannel| N/A    | local       | [docs](docker-multinode.md) | Project (@brendandburns) | Tested @ 0.14.1 | Local          |              |        | _none_      | [docs](../../docs/getting-started-guides/locally.md)   | Community (@preillyme)                     |

This flag is deprecated and no longer has any effect.
I wonder which version does this include.Now ubuntu approach is using ubuntu 0.15.0.
This change was checked in this week, so it will be true for clusters at 0.18.0+ (once the next release is cut next week). So the next time the ubuntu documentation is updated, this change should be reflected.
Ohit means that minion node can be automatically discovered by the master node when then kubelet set ? No other configuration is needed?
Yes, the kubelet, when passed the location of the apiserver, will register itself and then post node update messages. The only part that may depend on your deployment environment is the set of addresses that the kubelet passes to the apiserver. They will be based on the name of the node, reverse resolving the name of the node, or checking for addresses from the cloud provider.
Got it.I will include this when I updates the doc and scripts. Besides, I have a little questions here. Eric told me the addon services do not have service-account now.Will 0.18.0+ have it?
Service accounts will be included as part of 0.18.0 as well.
got it
- makes sense


Question: "k8s.io/kubernetes/pkg/kubelet/dockershim/network/kubenet" "k8s.io/kubernetes/pkg/kubelet/server/streaming" "k8s.io/kubernetes/pkg/kubelet/util/cache" <del> utilstore "k8s.io/kubernetes/pkg/kubelet/util/store" </del> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" "k8s.io/kubernetes/pkg/kubelet/dockershim/metrics"

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: plugin := plugins[0] fakeMounter := iscsiDiskMounter{ iscsiDisk: &iscsiDisk{ <ins> Iface:  "192.168.1.10:pv0001", </ins> plugin: plugin.(*iscsiPlugin)}, exec: fakeExec, } <del> newIface := "192.168.1.10:pv0001" cloneIface(fakeMounter, newIface) </del> <ins> cloneIface(fakeMounter) </ins> if cmdCount != 1 { t.Errorf("expected 1 CombinedOutput() calls, got %d", cmdCount) }

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign