Question: "id": "v1beta1.PodDisruptionBudgetStatus", "description": "PodDisruptionBudgetStatus represents information about the status of a PodDisruptionBudget. Status may trail the actual state of a system.", "required": [ <del> "disruptedPods", </del> "disruptionsAllowed", "currentHealthy", "desiredHealthy",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PodDisruptionBudget provides inappropriate message and cannot be updated as is required filed. What you expected to happen: works fine for editing labels under . Also, produce correct error message when edit field. How to reproduce it (as minimally and precisely as possible): Then, we will get following error after step-2. When edit , it should provide correct message as .When edit especially labels under metadata, they should be updated without error. Anything else we need to know?: According to the code comment, is valid. Environment: Kubernetes version (use ): build on
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale there is a PR open


Question: protocol: protocol, } if proxier.portsMap[lp] != nil { <del> newLocalPorts[lp] = proxier.portsMap[lp] </del> <ins> glog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] </ins> } else { socket, err := openLocalPort(&lp) if err != nil { glog.Errorf("can't open %s, skipping this externalIP: %v", lp.String(), err) continue } <del> newLocalPorts[lp] = socket </del> <ins> replacementPortsMap[lp] = socket </ins> } } // We're holding the port, so it's OK to install iptables rules. args := []string{

The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.

Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:


Question: EOF } <ins> function install_cni_if_needed { echo "Checking CNI Installation at /opt/cni/bin" if ! command -v /opt/cni/bin/loopback &> /dev/null ; then echo "CNI Installation not found at /opt/cni/bin" install_cni fi } </ins> # If we are running in the CI, we need a few more things before we can start if [[ "${KUBETEST_IN_DOCKER:-}" == "true" ]]; then echo "Preparing to test ..."

Since we switched from dockershim to containerd, my cluster seems to come up okay but pods get stuck in ContainerCreating. When I investigated further, I found Most dependencies of launching a cluster are installed automatically. In this case, it seems we don't get the CNI plugins installed and set up as needed. /cc /sig node network Pod should have been created as expected. Try to create pods on a system with local-up-cluster without CNI plugins already installed. No response <details</details<detailsN/A </details<details</details<detailshack/local-up- </details<details</details<details</details
maybe also /sig testing ?
cc patron saint of local-up-
/assign
I think it's as simple as checking if exists (or wherever it's looking for the CNI plugins) and if not, downloading the latest Linux version for the appropriate arch of
already working on installing cni plugin for the local-up ci job but if the script is running by user, doesn't we make an assumption that user will install cni plugins along with container runtime?
/assign the local e2e job is now working, have to refine the script we are anyways installing cni for the ci job, will add a provision so that users can also install.
/triage accepted
similar bug:


Question: return &TestScorePlugin{scorePlugin1, inj}, nil } <ins> func newScorePlugin2(_ context.Context, injArgs runtime.Object, f framework.Handle) (framework.Plugin, error) { var inj injectedResult if err := DecodeInto(injArgs, &inj); err != nil { return nil, err } return &TestScorePlugin{scorePlugin2, inj}, nil } </ins> func newPluginNotImplementingScore(_ context.Context, _ runtime.Object, _ framework.Handle) (framework.Plugin, error) { return &PluginNotImplementingScore{}, nil }

Scheduler's default plugin returns an invalid score that is not in the range of [0, 100], resulting in no Pods being scheduled: After investigation, I found that there was a bug in the code that built the framework when the scheduler started. When using the plugins declared in the regular extension point to override the plugins declared in the MultiPoint, the traversed variable will be modified in the following loop: This may cause a same scoring plugin (e.g. ) to be loaded multiple times: Further, this results in a node being processed multiple times by a plugin's (e.g. 's) method, resulting in an invalid score that is not in the range of [0, 100]: If the scheduler's MultiPoint feature is used (either through the default plugins or through manual configuration), and the configuration (e.g. the scoring weight) of a certain plugin is overridden in a regular extension point, the plugin will only be loaded once in this extension point, and no illegal scores will be returned. An example KubeSchedulerConfiguration: No response K8s v1.24.15 or higher None <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Can I work on this?
this is being worked on. see
I see, Is there any other task that you are aware of and I can work on? Thanks.


Question: endpoint:             "/var/log/kubelet/plugins/myplugin/csi.sock", versions:             []string{"v1.2.3", "boo", "v0.3.0", "2.0.1"}, foundInDeprecatedDir: true, <del> shouldFail:           true, </del> <ins> shouldFail:           false, </ins> }, { pluginName:           "test.plugin",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: CSI allows 0.x drivers only in the deprecated kubelet plugin dir. CSI allows 1.x drivers only in the new kubelet plugin dir. If a driver implements both 0.x and 1.x (which is unlikely since the side car containers do not support this), kubelet requires the driver live in new kubelet plugin dir. So CSI drivers that implement both 0.x and 1.x if they want a seamless transition between k8s 1.12 to 1.13 would need to deploy in both old and new dir. What you expected to happen: argues that this is adding churn for CSI authors. He recommends allowing any driver that implements 0.x (even if it implements 1.x) to use either new or old (not just new). /milestone v1.13 If release team approves, PR is small and I can get prepare it pretty quickly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
discussed in the burndown meeting, please go ahead and prepare the PR
/sig storage /priority critical-urgent /milestone v1.13
we need to wait for a fix for this before lifting code freeze


Question: # This option is for running docker manifest command export DOCKER_CLI_EXPERIMENTAL := enabled # golang version should match the golang version of the official build from https://github.com/etcd-io/etcd/releases. <del> GOLANG_VERSION?=1.12.17 </del> <ins> GOLANG_VERSION?=1.16.7 </ins> GOARM?=7 TEMP_DIR:=$(shell mktemp -d)

Right now the golang version in the Makefile is set to GOLANGVERSION?=1.12.17. Build fails with " undefined: errors.Is", presumably because errors.Is didn't exist in golang 1.12.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can take a look.
/triage accepted
According to , we should use 1.16. /kind bug /area release-eng
I send a fix. It would be good to make sure this doesn't break in the future. Question to How we can include building cloud/image as part of presubmits?
Somehow I thought maybe we'd switched to an ectd image built from the etcd project and needed to eventually drop this one? If not this is rather concerning. We don't have a great presubmit to shove this in, I would suggest a runifchanged presubmit to let reviewers of this image know if it builds w/o making it a required job on all PRs (since also changes should be scoped to a directory and infrequent).
For example the conformance image works this way


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" clientscheme "k8s.io/client-go/kubernetes/scheme"

[X] Test approval issue: [X] Test PR: 15 Oct 2022 [X] Two weeks soak end date: 28 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still one remaining Namespace endpoint which is untested. replaceCoreV1NamespaceFinalize Note: The test also removes the fake finalize from before calling the finalize action on the test namespace a second time. This will allow the test framework to clean up all namespaces when the e2e finishes, else the test namespace will be stuck in the state. Using a number of existing e2e test practices a new has been created for one Namespace endpoint. The e2e logs for this test are listed below. This query shows that the Finalize endpoint is hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/util/tail:go_default_library", "//pkg/util/version:go_default_library", "//vendor/github.com/armon/circbuf:go_default_library", <del> "//vendor/github.com/docker/docker/pkg/jsonlog:go_default_library", "//vendor/github.com/fsnotify/fsnotify:go_default_library", </del> "//vendor/github.com/golang/glog:go_default_library", "//vendor/github.com/google/cadvisor/info/v1:go_default_library", "//vendor/google.golang.org/grpc:go_default_library",

We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok


Question: execHandler: execHandler, }, containerManager:  cm.NewContainerManager(cgroupsName, client), <del> checkpointHandler: NewPersistentCheckpointHandler(), </del> <ins> checkpointHandler: checkpointHandler, </ins> } if streamingConfig != nil { var err error

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: // RunController creates a new replication controller named 'name' which creates 'replicas' pods running 'image'. func RunController(image, name string, replicas int, client client.Interface, portSpec string, servicePort int) error { <ins> if servicePort > 0 && !util.IsDNSLabel(name) { return fmt.Errorf("Service creation requested, but an invalid name for a service was provided (%s). Service names must be valid DNS labels.", name) } </ins> controller := &api.ReplicationController{ JSONBase: api.JSONBase{ ID: name,

-s=118 -p 9376:9376 run thockin/hostname 3 hostnameController results in a validation error: Service.ID: invalid value 'hostnameController' "hostnameController" is accepted for the replication controller, but not for the service ID.


Question: importpath = "k8s.io/kubernetes/pkg/windows/service", deps = select({ "@io_bazel_rules_go//go/platform:windows": [ <ins> "//staging/src/k8s.io/apiserver/pkg/server:go_default_library", </ins> "//vendor/golang.org/x/sys/windows:go_default_library", "//vendor/golang.org/x/sys/windows/svc:go_default_library", "//vendor/k8s.io/klog:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I configured Kubelet to run as a service on Windows, with a setting to restart on failure (). I noticed that it restarts even when I explicitly stop the service with , even though the documentation states the following (I left it at , the default): What you expected to happen: I expected the Kubelet to only restart when killed unexpectedly, per above docs. Anything else we need to know?: I think the problem might be due to this part of our service control integration, but I'm not really sure: It looks like we send and then , but the official golang example sends and just returns without exiting. (). Is this difference meaningful? Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Windows /cc /sig node /sig windows
The same should apply to kube-proxy as well.
i believe you implemented this functionality. can you please take a look? cc:


Question: }) assert.NoError(err) } <ins> func TestSumDiskCache(t *testing.T) { assert := assert.New(t) // Ensure that we'll return a cache miss if the backing file doesn't exist. t.Run("NoSuchKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that we'll return a cache miss if the backing file is empty. t.Run("EmptyFile", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" f, err := os.Create(filepath.Join(cacheDir, sanitize(key))) if err != nil { t.Fatal(err) } f.Close() got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that we'll return a cache miss if the backing has an invalid // checksum. t.Run("InvalidChecksum", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("testing") mismatchedValue := []byte("testink") sum := sha256.Sum256(value) // Create a file with the sum of 'value' followed by the bytes of // 'mismatchedValue'. f, err := os.Create(filepath.Join(cacheDir, sanitize(key))) if err != nil { t.Fatal(err) } f.Write(sum[:]) f.Write(mismatchedValue) f.Close() // The mismatched checksum should result in a cache miss. got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that our disk cache will happily cache over the top of an existing // value. We depend on this behaviour to recover from corrupted cache // entries. When Get detects a bad checksum it will return a cache miss. // This should cause httpcache to fall back to its underlying transport and // to subsequently cache the new value, overwriting the corrupt one. t.Run("OverwriteExistingKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("cool value!") // Write a value. c.Set(key, value) got, ok := c.Get(key) // Ensure we can read back what we wrote. assert.True(ok) assert.Equal(value, got) differentValue := []byte("I'm different!") // Write a different value. c.Set(key, differentValue) got, ok = c.Get(key) // Ensure we can read back the different value. assert.True(ok) assert.Equal(differentValue, got) }) // Ensure that deleting a key does in fact delete it. t.Run("DeleteKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("coolValue") c.Set(key, value) // Ensure we successfully set the value. got, ok := c.Get(key) assert.True(ok) assert.Equal(value, got) c.Delete(key) // Ensure the value is gone. got, ok = c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) // Ensure that deleting a non-existent value is a no-op. c.Delete(key) }) } </ins>

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: } <ins> func TestFindAndAddNewPods_WithDifferentConditions(t *testing.T) { tests := []struct { desc          string hasAddedPods  bool podState      string expectedFound bool // Found pod is added to DSW }{ { desc:          "HasAddedPods is false, ShouldPodRuntimeBeRemoved and ShouldPodContainerBeTerminating are both true", hasAddedPods:  false, podState:      Removed, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is false, ShouldPodRuntimeBeRemoved is false, ShouldPodContainerBeTerminating is true", hasAddedPods:  false, podState:      Terminating, expectedFound: true, // Pod should be added to DSW }, { desc:          "HasAddedPods is false, other condition", hasAddedPods:  false, podState:      Other, expectedFound: true, // Pod should be added to DSW }, { desc:          "HasAddedPods is true, ShouldPodRuntimeBeRemoved is false, ShouldPodContainerBeTerminating is true", hasAddedPods:  true, podState:      Terminating, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is true, ShouldPodRuntimeBeRemoved and ShouldPodContainerBeTerminating are both true", hasAddedPods:  true, podState:      Removed, expectedFound: false, // Pod should not be added to DSW }, { desc:          "HasAddedPods is true, other condition", hasAddedPods:  true, podState:      Other, expectedFound: true, // Pod should be added to DSW }, } for _, tc := range tests { t.Run(tc.desc, func(t *testing.T) { // create dswp dswp, fakePodManager, fakePodState := prepareDswpWithVolume(t) // create pod containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      "dswp-test-volume-name", MountPath: "/mnt", }, }, }, } pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) fakePodManager.AddPod(pod) switch tc.podState { case Removed: fakePodState.removed = map[kubetypes.UID]struct{}{pod.UID: {}} case Terminating: fakePodState.terminating = map[kubetypes.UID]struct{}{pod.UID: {}} case Other: break } dswp.hasAddedPods = tc.hasAddedPods // Action dswp.findAndAddNewPods() // Verify podsInDSW := dswp.desiredStateOfWorld.GetPods() found := false if podsInDSW[types.UniquePodName(pod.UID)] { found = true } if found != tc.expectedFound { t.Fatalf( "Pod with uid %v has expectedFound value %v in pods in DSW %v", pod.UID, tc.expectedFound, podsInDSW) } }) } } </ins> func TestFindAndAddNewPods_WithReprocessPodAndVolumeRetrievalError(t *testing.T) { // create dswp <del> dswp, fakePodManager := prepareDswpWithVolume(t) </del> <ins> dswp, fakePodManager, _ := prepareDswpWithVolume(t) </ins> // create pod containers := []v1.Container{

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: } func TestRegister(t *testing.T) { <ins> m1 := NewMockNoopPluginFactory() m2 := NewMockNoopPluginFactory() </ins> tests := []struct { name              string registry          Registry

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: }) fakeClient.AddReactor("update", "horizontalpodautoscalers", func(action core.Action) (handled bool, ret runtime.Object, err error) { <del> tc.Lock() defer tc.Unlock() obj := action.(core.UpdateAction).GetObject().(*autoscalingv1.HorizontalPodAutoscaler) assert.Equal(t, namespace, obj.Namespace, "the HPA namespace should be as expected") assert.Equal(t, hpaName, obj.Name, "the HPA name should be as expected") assert.Equal(t, tc.desiredReplicas, obj.Status.DesiredReplicas, "the desired replica count reported in the object status should be as expected") if tc.verifyCPUCurrent { if assert.NotNil(t, obj.Status.CurrentCPUUtilizationPercentage, "the reported CPU utilization percentage should be non-nil") { assert.Equal(t, tc.CPUCurrent, *obj.Status.CurrentCPUUtilizationPercentage, "the report CPU utilization percentage should be as expected") </del> <ins> obj := func() *autoscalingv1.HorizontalPodAutoscaler { tc.Lock() defer tc.Unlock() obj := action.(core.UpdateAction).GetObject().(*autoscalingv1.HorizontalPodAutoscaler) assert.Equal(t, namespace, obj.Namespace, "the HPA namespace should be as expected") assert.Equal(t, hpaName, obj.Name, "the HPA name should be as expected") assert.Equal(t, tc.desiredReplicas, obj.Status.DesiredReplicas, "the desired replica count reported in the object status should be as expected") if tc.verifyCPUCurrent { if assert.NotNil(t, obj.Status.CurrentCPUUtilizationPercentage, "the reported CPU utilization percentage should be non-nil") { assert.Equal(t, tc.CPUCurrent, *obj.Status.CurrentCPUUtilizationPercentage, "the report CPU utilization percentage should be as expected") } </ins> } <del> } tc.statusUpdated = true </del> <ins> tc.statusUpdated = true return obj }() </ins> // Every time we reconcile HPA object we are updating status. tc.processed <- obj.Name return true, obj, nil

What happened: The unit test for Horizontal Pod Autoscaler frequently deadlocked on the OpenShift CI system. What you expected to happen: The unit test should be robust and succeed without having to win a race condition to avoid deadlock. How to reproduce it (as minimally and precisely as possible): I'm not sure. Anything else we need to know?: Here are some logs of the problem happening: https://openshift-gce- The deadlocks seem to happen when the chan fills up while waiting for events. If that happens, there's a race where immediately after an event is read but before it can acquire the lock to verify the test, 2 event reactors are able to acquire the lock and write to the chan. The second will block forever since the chan is full and there's nobody to read from it. I've opened a PR with the fix. See
/sig autoscaling
/assign


Question: // To handle kubelet restarts, test pod admissibility using AllocatedResources values // (for cpu & memory) from checkpoint store. If found, that is the source of truth. podCopy := pod.DeepCopy() <del> for _, c := range podCopy.Spec.Containers { allocatedResources, found := kl.statusManager.GetContainerResourceAllocation(string(pod.UID), c.Name) if c.Resources.Requests != nil && found { c.Resources.Requests[v1.ResourceCPU] = allocatedResources[v1.ResourceCPU] c.Resources.Requests[v1.ResourceMemory] = allocatedResources[v1.ResourceMemory] } } </del> <ins> kl.updateContainerResourceAllocation(podCopy) </ins> // Check if we can admit the pod; if not, reject it. if ok, reason, message := kl.canAdmitPod(activePods, podCopy); !ok { kl.rejectPod(pod, reason, message)

After enabling In-Place Pod Vertical Scaling, if deploy a pod without setting memory (or cpu) request, kubelet will be failed at its second restart kubelet restart successfully kubelet with In-Place Pod Vertical Scaling enabled a pod without setting memory request, e.g: kubelet for the first time kubelet for the second time will fail with error: After deploying the pod, the following information is saved in the file : Then restart kubelet for the first time, the relevant content in the file becomes: When deploying the pod, as there are no records in the checkpoint (), kubelet will get resource config from directly and save it to the checkpoint (without memory request) When restarting kubelet for the first time, it will restore the previously saved data from the checkpoint. Since the memory request was not previously set, it will be set to an empty value and resaved to the checkpoint When restarting kubelet for the second time, it will process as a value of 0 (not empty), which resulting in the inconsistency between the calculated checksum during data recovery and the previous one <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
cc
/triage accepted
/assign
xlink: /cc
thanks for finding this issue and the fix. It looks good with suggestion. For beta, I wonder if we can do away with local checkpoint and rely on PodStatus as persistent store and source of truth for allocated resources. It will simplify the implementation.
Using resource.Quantity in checkoutPoint is a bad usage. I found except the case in this issue, whenever adding a static pod to the kubelet, then restart the kubelet, restore the resize status checkpoint will fail, casing kubelet panic. The reason is , before restart kubelet, the checkpoint struct in kubelet will looks like: and the the podstatusmanager_state file is looks like: and after restart kubelet and recover the checkoutpoint state, the checkpoint struct in kubelet will looks like: the difference is between "kube-scheduler:map[cpu:{i:{value:6 scale:-1} d:{Dec:<nil} s: Format:DecimalSI}" and "kube-scheduler:map[cpu:{i:{value:600 scale:-3} d:{Dec:<nil} s:600m Format:DecimalSI} ".

It's caused by json Marshal and Unmarshal of resource.Quantity. So, I think Using resource.Quantity in checkoutPoint is a bad case. We should consider using PodStatus in apiserver instead of checkoutPoint.
Is this problem solved? Now I can still reproduce this problem in the official version of 1.28. Thanks~
Could anyone please let me know what is the state of the art to solve this issue?
we have seen the similar problem after enabling InPlaceUpdate pod resources and could we reopen this ?
can we just do marshal&unmarshal before storing state checkpoint to avoid the issue of resource.Quantity?
I am +1 on doing away with the checkpointing code altogether and leveraging PodStatus. We discussed this in the Kubernetes Contributor Summit in Chicago. I just have not been able to find the cycles to work on it :(


Question: import ( "context" "os" <ins> "path/filepath" </ins> "testing" "time"

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: <del> v1.11.2-1 </del> <ins> v1.11.3-1 </ins>

The go team has released 1.11.3 with three security fixes. From their e-mail announcement: We should kick off upgrades of both the kubernetes build system, and our CI, as soon as possible. Do you have an ETA for the rules_go bump that will contain 1.11.3 toolchain support? cc:
I'm preparing releases that use Go 1.11.3 by default. Those will be on the rules_go 0.14, 0.15, and 0.16 branches. I expect to tag them tonight as soon as they get through CI.
Fantastic! Thank you
rules_go 0.14.7, 0.15.9, and 0.16.4 are now tagged and ready to go.
related commits and issue:
Reopening to track other branches, as well as the CI bump
go1.11.3 and go1.10.6 introduced a regression, and so go1.11.4 and go1.10.7 were released Friday afternoon with a fix. We'll need to bump everything again on master, and should hold off on cherry-picks for now.
This looks fixed by Is there anything else that needs to be done as well?
Still need to merge for k8s 1.13.
Closing as all branches are now up to date.


Question: func (o *ConfigMapOptions) Run() error { configMap, err := o.createConfigMap() if err != nil { <del> return nil </del> <ins> return err </ins> } if err := util.CreateOrUpdateAnnotation(o.CreateAnnotation, configMap, scheme.DefaultJSONEncoder()); err != nil { return err

prints no error message with zero exit code even with invalid command option. e.g. --from-file=<non-existing-pathprints error message with non-zero exit code. Please try following command. It does not print any error and exit code is . will fix this issue.
/sig cli
/triage accepted


Question: glog.V(6).Infof("Resizing is not enabled for this volume %s", volumeToMount.VolumeName) return nil } mounter := og.volumePluginMgr.Host.GetMounter(pluginName) // Get expander, if possible expandableVolumePlugin, _ :=

We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc


Question: } f.appendContainerTrace("Started", id) container, ok := f.ContainerMap[id] <ins> timestamp := f.Clock.Now() </ins> if !ok { <del> container = convertFakeContainer(&FakeContainer{ID: id, Name: id}) </del> <ins> container = convertFakeContainer(&FakeContainer{ID: id, Name: id, CreatedAt: timestamp}) </ins> } container.State.Running = true container.State.Pid = os.Getpid() <del> container.State.StartedAt = dockerTimestampToString(f.Clock.Now()) </del> <ins> container.State.StartedAt = dockerTimestampToString(timestamp) </ins> container.NetworkSettings.IPAddress = "2.3.4.5" f.ContainerMap[id] = container f.updateContainerStatus(id, statusRunningPrefix)

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apiserver/pkg/authentication/serviceaccount" "k8s.io/kubernetes/pkg/api/v1" rbacv1beta1 "k8s.io/kubernetes/pkg/apis/rbac/v1beta1"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} What happened: Dynamic provisioned volumes in different zones with master failed to deleted What you expected to happen: Since the volume can be dynamic provisioned when the zones differs from master, then it should be deleted after delete the PVC which created them. How to reproduce it (as minimally and precisely as possible): up an all-in-one cluster. The instance is created in zone us-central1-a. several storageClass. One in zone us-central1-a. Another in zone us-central1-b. several PVC. pvc1 using storageClass1(zone us-central1-a), pvc2 using storageClass2(zone us-central1-b). PV/volume were provisioned in proper size/mode/zone. PVCs. with volume in zone us-central1-b failed with error can not find the volume.

Anything else do we need to know: Name: pvc--70d1-11e6-b17e- Labels: failure- failure- Status: Failed Claim: default/pvc5 Reclaim Policy: Delete Access Modes: RWO Capacity: 4Gi Message: Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" Source: Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine) PDName: kubernetes-dynamic-pvc--70d1-11e6-b17e- FSType: Partition: 0 ReadOnly: false Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message 11m 11m 1 {persistentvolume-controller } Warning VolumeFailedDelete Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" creationTimestamp: '2016-09-01T22:49:26.439-07:00' description: '{"":"pvc--70d1-11e6-b17e-","":"pvc5","":"default"}' id: '' kind: compute#disk name: kubernetes-dynamic-pvc--70d1-11e6-b17e- selfLink: sizeGb: '4' status: READY type: zone:
Kubernetes can delete GCE PDs only in zones it manages. I.e. either in the same zone as the master or, when multizone is enabled in GCE config, in all zones in the same region. This should not be allowed in the first place. Kubernetes should not create disks outside zones it manages and report an error instead. So... please use following , it will be allow creation/deletion of disks in all zones in us-central1 region: And I'll fix Kubernetes to report an error when creating a disk in all other zones.


Question: out := e.NewFunc() if err := e.Storage.Create(ctx, key, obj, out, ttl, dryrun.IsDryRun(options.DryRun)); err != nil { err = storeerr.InterpretCreateError(err, qualifiedResource, name) <del> err = rest.CheckGeneratedNameError(e.CreateStrategy, err, obj) </del> <ins> err = rest.CheckGeneratedNameError(ctx, e.CreateStrategy, err, obj) </ins> if !apierrors.IsAlreadyExists(err) { return nil, err }

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: return synthesized } <del> // verifyRunAsNonRoot verifies RunAsNonRoot. func verifyRunAsNonRoot(pod *v1.Pod, container *v1.Container, uid *int64, username string) error { effectiveSc := securitycontext.DetermineEffectiveSecurityContext(pod, container) // If the option is not set, or if running as root is allowed, return nil. if effectiveSc == nil || effectiveSc.RunAsNonRoot == nil || !*effectiveSc.RunAsNonRoot { return nil } if effectiveSc.RunAsUser != nil { if *effectiveSc.RunAsUser == 0 { return fmt.Errorf("container's runAsUser breaks non-root policy") } return nil } switch { case uid != nil && *uid == 0: return fmt.Errorf("container has runAsNonRoot and image will run as root") case uid == nil && len(username) > 0: return fmt.Errorf("container has runAsNonRoot and image has non-numeric user (%s), cannot verify user is non-root", username) default: return nil } } </del> // convertToRuntimeSecurityContext converts v1.SecurityContext to runtimeapi.SecurityContext. func convertToRuntimeSecurityContext(securityContext *v1.SecurityContext) *runtimeapi.LinuxContainerSecurityContext { if securityContext == nil {

Hello, What happened: When we apply PodSecurityPolicy with rule: WIndows containers are still blocked instead we put: the pod is still blocked with logs: What you expected to happen: the Container should start How to reproduce it (as minimally and precisely as possible): Create Psp disableling Root Users: Create a pod Windows with runAsUserName: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux 7 (Core) / windows server 2019 version 1809 Kernel (e.g. ): 3.10.0-514.10.2.el7.x86_64 Others: /sig windows
/sig windows
/assign I will test it and then post the test results
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: # Pods <del> A _pod_ (as in a pod of whales or pea pod) is a relatively tightly coupled group of containers that are scheduled onto the same host. It models an application-specific "virtual host" in a containerized environment. Pods serve as units of scheduling, deployment, and horizontal scaling/replication, and share fate. </del> <ins> ## What is a _pod_? </ins> <del> Why doesn't Kubernetes just support an affinity mechanism for co-scheduling containers instead? While pods have a number of benefits (e.g., simplifying the scheduler), the primary motivation is resource sharing. </del> <ins> A _pod_ (as in a pod of whales or pea pod) models an application-specific "logical host" in a containerized environment. It may contain one or more containers which are relatively tightly coupled -- in a pre-container world, they would have executed on the same physical or virtual host. </ins> <del> In addition to defining the containers that run in the pod, the pod specifies a set of shared storage volumes. Pods facilitate data sharing and IPC among their constituents. In the future, they may share CPU and/or memory ([LPC2013](http://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdf)). </del> <ins> Like running containers, pods are considered to be relatively ephemeral rather than durable entities. As discussed in [life of a pod](pod-states.md), pods are scheduled to nodes and remain there until termination (according to restart policy) or deletion. When a node dies, the pods scheduled to that node are deleted. Specific pods are never rescheduled to new nodes; instead, they must be replaced (see [replication controller](replication-controller.md) for more details). (In the future, a higher-level API may support pod migration.) </ins> <del> The containers in the pod also all use the same network namespace/IP (and port space). The goal is for each pod to have an IP address in a flat shared networking namespace that has full communication with other physical computers and containers across the network. [More details on networking](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md). </del> <ins> ## Motivation for pods </ins> <del> While pods can be used to host vertically integrated application stacks, their primary motivation is to support co-located, co-managed helper programs, such as: </del> <ins> ### Resource sharing and communication Pods facilitate data sharing and communication among their constituents. The containers in the pod all use the same network namespace/IP and port space, and can find and communicate with each other using localhost. Each pod has an IP address in a flat shared networking namespace that has full communication with other physical computers and containers across the network. The hostname is set to the pod's Name for the containers within the pod. [More details on networking](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md). In addition to defining the containers that run in the pod, the pod specifies a set of shared storage volumes. Volumes enable data to survive container restarts and to be shared among the containers within the pod. In the future, pods will share IPC namespaces, CPU, and memory ([LPC2013](http://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdf)). ### Management Pods also simplify application deployment and management by providing a higher-level abstraction than the raw, low-level container interface. Pods serve as units of deployment and horizontal scaling/replication. Co-location, fate sharing, coordinated replication, resource sharing, and dependency management are handled automatically. ## Uses of pods Pods can be used to host vertically integrated application stacks, but their primary motivation is to support co-located, co-managed helper programs, such as: </ins> - content management systems, file and data loaders, local cache managers, etc. - log and checkpoint backup, compression, rotation, snapshotting, etc. - data change watchers, log tailers, logging and monitoring adapters, event publishers, etc.

Currently replication controllers define an exact value. Are there disadvantages to allowing the replication controller to define more of a policy - at least X, at most X, exactly X? Since replication controllers aren't really aware of overlapping sets, I guess you can't really use that to guide allocation between two pools because they would just end up fighting unproductively
What is the use case? This seems sort of like auto-scaling policy. I'd prefer to put those objects in the auto-scaler.
It sounds like you want a canary-controller that scales the old version down while scaling the new one up. Combine with a service that spreads traffic across both versions and this should Just Work (tm).
I'm thinking about this from the mechanics of doing a deployment by composing replication controllers. With controllers as they are today: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo,deployment=2 x0 alter controller 2 to be x1 (or up by the batch size) alter controller 1 to be x4 (or down by the batch size) ... repeat until done delete controller 1 The thought around atLeast was about explicitly representing the invariant you want during an available deployment: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo, atLeast=5 set controller to be atMost=5 delete a pod from the name=foo,deployment=1 query ... continue until 0 pods in that query delete controller 1 update controller 2 to change the label query to name=foo,deployment=2 I'm not sure that's better.... just the for loop for deletes seems more reasonable. Not a huge deal, just trying to work through how we'd model common deploy cases for end users.
Overlapping replicationController sets: I think we should validate that this isn't possible upon creation of a replicationController. We can do this by determining whether the label selector may overlap with any others, though to do this analytically without looking at the combinations of labels attached to pods it requires that all replicationController label selectors have at least one key in common, with a different value than all other replicationControllers. Policy: As mentioned, we intended the policy to go in a separate auto-scaling policy entity, in a higher-level layer / separate component.

Deployment: As pointed out, it is possible to create a new replicationController and gradually scale it up while scaling the original down. Or, one could do the Asgard trick of creating a new replicationController and then shift traffic to it. It's possible to add/remove instances from a service by tweaking their labels. For instance one could toggle the value of an in_service=true/false label. The update demo changes the template of a single replication controller and then kills instances one by one until they all get replaced. I don't like that due to the risk of unpredictable replacements and complexity involved with rollbacks. Splitting out the template ( ) would address the rollback issue, but not the unpredictability. It seems to me that the atMost approach suffers from the same risk. That said, I see what you mean that atLeast/atMost are not auto-scaling policies, but are invariants to maintain. I could imagine that flexibility could be useful. For example, a worker pool manager could use that to opportunistically request resources, where some pod requests may fail. Or, perhaps in the case that we need to temporarily create a new pod for some reason (CRIU-based migration?) but we don't want an old one to be killed. So, I like the idea, just not the use case. :-) FWIW, exactly X could be implemented by combining atLeast and atMost.
Agree that without a kill policy on the replication controller the uses cases described don't work. Is a kill policy something you've discussed for the replication controller (oldest, ones that don't match my template first, ones on most overloaded systems)?
I agree that kill policy would be useful and all of the policies you mention are reasonable in particular scenarios. However, I see this as a rich, unbounded policy space, much like scheduling. IIRC, Joe and/or Brendan proposed kill policies early on and I asked for them to be removed. Instead, I propose that we leave the default policy entirely up to the system, much like scheduling, so we can take as many factors into account as we like, and otherwise I'd like to enable the user to implement an arbitrary policy using event hooks . For instance, I could imagine POSTing to a URL whenever a pod needed to be killed.

The user's hook would have N seconds to respond (perhaps configurable), after which the replication controller would kill its default choice.
Thanks for the clarification - I think I'll close this for now until the API stabilizes, and then add some of this context to the doc.
I suggest re-opening this issue, but change the title to "add XXX context to doc YYY", so that we can reference in the PR for posterity.


Question: return deployments.Delete(name, gracePeriod) } <ins> type updateDeploymentFunc func(d *extensions.Deployment) func (reaper *DeploymentReaper) updateDeploymentWithRetries(namespace, name string, applyUpdate updateDeploymentFunc) (deployment *extensions.Deployment, err error) { deployments := reaper.Extensions().Deployments(namespace) err = wait.Poll(10*time.Millisecond, 1*time.Minute, func() (bool, error) { if deployment, err = deployments.Get(name); err != nil { return false, err } // Apply the update, then attempt to push it to the apiserver. applyUpdate(deployment) if deployment, err = deployments.Update(deployment); err == nil { return true, nil } return false, nil }) return deployment, err } </ins> func (reaper *PodReaper) Stop(namespace, name string, timeout time.Duration, gracePeriod *api.DeleteOptions) error { pods := reaper.Pods(namespace) _, err := pods.Get(name)

It looks that we lack retry in the test. cc
As can be seen by number of references, this seems to be happening often. Bumping to P0.
is waiting to be merged.

http://pr- cc
This can be reproduced by running cc
Update: It didn't always generate errors; when this happened, the error came from
The conflict happens because when we stop the deployment, we first update it: cc


Question: } instanceRunning = true } <ins> // If instance didn't reach running state in time, return with error now. if err != nil { return name, err } // Instance reached running state in time, make sure that cloud-init is complete if isCloudInitUsed(imageConfig.metadata) { cloudInitFinished := false for i := 0; i < 60 && !cloudInitFinished; i++ { if i > 0 { time.Sleep(time.Second * 20) } var finished string finished, err = remote.SSH(name, "ls", "/var/lib/cloud/instance/boot-finished") if err != nil { err = fmt.Errorf("instance %s has not finished cloud-init script: %s", name, finished) continue } cloudInitFinished = true } } </ins> return name, err } <ins> func isCloudInitUsed(metadata *compute.Metadata) bool { if metadata == nil { return false } for _, item := range metadata.Items { if item.Key == "user-data" && strings.HasPrefix(item.Value, "#cloud-config") { return true } } return false } </ins> func getExternalIp(instance *compute.Instance) string { for i := range instance.NetworkInterfaces { ni := instance.NetworkInterfaces[i]

e2e_node tests on cos-beta-59-9460-20-0 image use GPUs. To install the NVIDIA drivers, we have to disable loadpin which requires a restart. This results in the runner losing connection with the instance which results in the tests failing/erroring. One way to fix this is to make sure that cloud-init has finished before starting the tests. Something like: We need to make sure that we only do this for images which use cloud-init.
Example: https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
is working on a fix for node e2es. It's currently failing serial tests on COS.


Question: package main <del> import "syscall" </del> <ins> import ( "os" "os/signal" "syscall" ) </ins> func main() { <del> // Halts execution, waiting on signal. syscall.Pause() </del> <ins> c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt, os.Kill, syscall.SIGTERM) // Block until a signal is received. <-c </ins> }

Kubernetes use kubernetes/pause image to create network container. The problem is that whatever signals the pause container received, it will exit with code 2. Instead, with a normal exit, it should exit with code 0; while it exit with port binding issue, it should exit with code -1.
The port-binding issues will occur before the pause image gets invoked, but the pause image itself should exit 0.
I filed to docker to capture port-binding issue there. Port-binding is not my concern since we validate the port to avoid the conflict anyway. The concern I am having is PreStart we plan to introduce between docker container creation and start ().
There seems to be another issue - quick testing shows that if there is a conflict with a port allocated OUTSIDE of docker (e.g. the kube-proxy), Docker will accept the container and run it, but the host port doesn't work at all, and there's no way to tell except trawling logs. , Dawn Chen wrote:
Is there anything we could do to help go through, to eliminate the need for the network container?
Good catch. But following your steps above, my quick testing shows that docker container is running, but doesn't work at all. Also daemon logs doesn't have any error information.

Will file another issue to docker: 1) # netstat -lnt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN tcp6 0 0 :::4194 ::: LISTEN tcp6 0 0 ::: ::: LISTEN tcp6 0 0 :::8080 ::: LISTEN tcp6 0 0 :::22 :::* LISTEN 2) # docker run -p :80 -d dockerimages/apache 6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11 3) # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dockerimages/apache:latest "apache2 -DFOREGROUN 6 seconds ago Up 5 seconds 0.0.0.0:-80/tcp jollynewton 4) # docker logs AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.11. Set the 'ServerName' directive globally to suppress this message AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.11. Set the 'ServerName' directive globally to suppress this message [Tue Sep 30 00:03:16. 2014] [core:warn] [pid 1:tid ] AH00098: pid file overwritten -- Unclean shutdown of previous Apache run? [Tue Sep 30 00:03:16. 2014] [mpmevent:notice] [pid 1:tid ] AH00489: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations [Tue Sep 30 00:03:16. 2014] [core:notice] [pid 1:tid ] AH00094: Command line: 'apache2 -D FOREGROUND' 5) # cat | grep [] +job log(create, 6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11,

dockerimages/apache:latest) = OK (0) [info] GET /containers/6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11/json [] +job containerinspect(6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11) [] -job start(6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11) = OK (0) [] -job containerinspect(6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11) = OK (0) [info] GET /containers/6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11/json [] +job containerinspect(6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11) [] -job containerinspect(6f7309782be63f95dbc3f87bded69637e3299cd255668f1f171dc5eef28f5a11) = OK (0) [info] GET /v1.14/containers//json [] +job containerinspect() [] -job containerinspect() = OK (0) [info] GET /containers//logs?stderr=1&stdout=1&tail=all [] +job containerinspect() [] -job containerinspect() = OK (0) [] +job logs() [] -job logs() = OK (0)
I agreed we should help docker/docker go through, so that network container should be gone forever which makes our system, especially kubelet much simple. But that doesn't help with docker/docker and a new issue found by following.
have you tried this on docker master because it should have been fixed by , just want to prevent duplicate issues
I just rebuild docker from HEAD, and tested it with above steps I posted.

The problem is fixed: 37a039245cd9faba6ae900cc463313eae77c9c9dce589616ec992a7e8eecb458 2014/09/30 16:29:20 Error response from daemon: Cannot start container 37a039245cd9faba6ae900cc463313eae77c9c9dce589616ec992a7e8eecb458: Bind for 0.0.0.0:8080 failed: port is already allocated But another issue I filed (docker/docker) is still exist. Thanks!
great, just wanted to make sure! and yes I can see if want to change the existing functionality for docker/docker


Question: } for _, wantedRule := range svc.Spec.Ports { expectedRuleCount++ <del> wantedRuleName := az.getLoadBalancerRuleName(&svc, wantedRule, subnet(&svc)) </del> <ins> wantedRuleName := az.getLoadBalancerRuleName(&svc, wantedRule.Protocol, wantedRule.Port, subnet(&svc)) </ins> foundRule := false for _, actualRule := range *loadBalancer.LoadBalancingRules { if strings.EqualFold(*actualRule.Name, wantedRuleName) &&

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Mixed protocols LoadBalacer services are not supported by apiserver. It will decline those services with error "cannot create an external load balancer with non-TCP/UDP ports". Hence another annotation for this purpose. e.g. However, this is still not working. The UDP LB rule is still not created by the above service. What you expected to happen: Both TCP and UDP rules created for the service. How to reproduce it (as minimally and precisely as possible): Create a service with above annotation. Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.1 Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
I tried with the above exact service config, and it works well on my k8s v1.13.3 cluster created by aks-engine, how can I repro that error?
correction: there is no error, while UDP port rule is not created, I can repro that.


Question: local request=$2 local expected=$3 <del> res=$(kubectl get $object -o template -t "$request") </del> <ins> res=$(kubectl get "${kube_flags[@]}" $object -o template -t "$request") </ins> if [[ "$res" =~ ^$expected$ ]]; then echo -n ${green}

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: } for _, pod := range dswp.podManager.GetPods() { <del> if dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </del> <ins> // Keep consistency of adding pod during reconstruction if dswp.hasAddedPods && dswp.podStateProvider.ShouldPodContainersBeTerminating(pod.UID) { </ins> // Do not (re)add volumes for pods that can't also be starting containers continue } <ins> if !dswp.hasAddedPods && dswp.podStateProvider.ShouldPodRuntimeBeRemoved(pod.UID) { // When kubelet restarts, we need to add pods to dsw if there is a possibility // that the container may still be running continue } </ins> dswp.processPodVolumes(pod, mountedVolumesForPod) } }

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: <ins> // +build !windows /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( "bytes" "fmt" "io" "os/exec" "strings" "k8s.io/klog" ) func (r *streamingRuntime) portForward(podSandboxID string, port int32, stream io.ReadWriteCloser) error { container, err := r.client.InspectContainer(podSandboxID) if err != nil { return err } if !container.State.Running { return fmt.Errorf("container not running (%s)", container.ID) } containerPid := container.State.Pid socatPath, lookupErr := exec.LookPath("socat") if lookupErr != nil { return fmt.Errorf("unable to do port forwarding: socat not found.") } args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath, "-", fmt.Sprintf("TCP4:localhost:%d", port)} nsenterPath, lookupErr := exec.LookPath("nsenter") if lookupErr != nil { return fmt.Errorf("unable to do port forwarding: nsenter not found.") } commandString := fmt.Sprintf("%s %s", nsenterPath, strings.Join(args, " ")) klog.V(4).Infof("executing port forwarding command: %s", commandString) command := exec.Command(nsenterPath, args...) command.Stdout = stream stderr := new(bytes.Buffer) command.Stderr = stderr // If we use Stdin, command.Run() won't return until the goroutine that's copying // from stream finishes. Unfortunately, if you have a client like telnet connected // via port forwarding, as long as the user's telnet client is connected to the user's // local listener that port forwarding sets up, the telnet session never exits. This // means that even if socat has finished running, command.Run() won't ever return // (because the client still has the connection and stream open). // // The work around is to use StdinPipe(), as Wait() (called by Run()) closes the pipe // when the command (socat) exits. inPipe, err := command.StdinPipe() if err != nil { return fmt.Errorf("unable to do port forwarding: error creating stdin pipe: %v", err) } go func() { io.Copy(inPipe, stream) inPipe.Close() }() if err := command.Run(); err != nil { return fmt.Errorf("%v: %s", err, stderr.String()) } return nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.