Question: ":(glob)${root}"/'**/zz_generated.conversion.go'  | xargs -0 rm -f <del> local inputs=() </del> <ins> local input_args=() </ins> for arg in "${input_pkgs[@]}"; do <del> inputs+=("--input-dirs" "$arg") </del> <ins> input_args+=("--input-dirs" "$arg") done local extra_peer_args=() for arg in "${extra_peers[@]:+"${extra_peers[@]}"}"; do extra_peer_args+=("--extra-peer-dirs" "$arg") </ins> done "${gobin}/conversion-gen"  -v "${v}"  -O zz_generated.conversion  --go-header-file "${boilerplate}"  --output-base "${out_base}"  <del> "${inputs[@]}" </del> <ins> "${extra_peer_args[@]:+"${extra_peer_args[@]}"}"  "${input_args[@]}" </ins> fi }

Earlier it was possible to pass flag to while calling script. Now with the new script, is called under which doesnot support any other flags apart from , and . As it can be seen here there is no option to pass flag to conversion-gen Earlier in the script, it was considering all the inputs passed flag is supported by under the hood, but the function in script is not allowing the flag.
/cc
/sig api-machinery
/assign for code generation /triage accepted
/assign
Can you explain why you need this flag? What I found in k/k was that we don't actually need it at all.
You know what, my memory was faulty. We do use it (until I fix that, which is in a development branch, not merged). But we don't use it in any of the "submodule" configs.
A lot of the "magic" behind the conversion logic is very brittle. Can you help me understand where / how this impacts you?
I whipped up a quick patch but have not tested much:
We are using the flag several places in our script On removal of the flag, the autoConvert functions in. files has the error : Althoug I can see the function is already present at . But there is also a comment like Hence, I think there is a need for the in order to use this pkg.
I see. The component-base is something I didn't account for. If you apply the above patch, does the new argument to work for you?
Above patch throws an error for the cases when flag is not passed to function. Instead I followed this approach :
I'd rather not accumulate into a string - can you tell me what error it throws? Note, mine wants Also - is the argument here a dir (e.g. /home/me/src/foo/bar) or a pkg (e.g. ) ? I think it's actually a pkg, right?
You implementation throws error for the cases when we don't specify the option while calling , as shown below: For such cases I am getting following error
It seems to be an old-bash problem. Try this one?
Yes this is working fine.
OK. I'll prep a PR soon.
Thank you


Question: tpVal, ok := node.Labels[c.topologyKey] if !ok { klog.V(5).Infof("node '%s' doesn't have required label '%s'", node.Name, tpKey) <del> return migration.PredicateResultToFrameworkStatus([]predicates.PredicateFailureReason{predicates.ErrTopologySpreadConstraintsNotMatch}, nil) </del> <ins> return framework.NewStatus(framework.Unschedulable, ErrReasonConstraintsNotMatch) </ins> } selfMatchNum := int32(0)

Part of /priority important-soon See example /sig scheduling /kind cleanup
/help
/assign
/remove-help


Question: }}, allowed: true, }, <ins> { name: "new supported sysctls: net.ipv4.tcp_fin_timeout", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_fin_timeout", Value: "60"}}, }, }}, allowed: true, }, { name: "new supported sysctls: net.ipv4.tcp_keepalive_intvl", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_keepalive_intvl", Value: "75"}}, }, }}, allowed: true, }, { name: "new supported sysctls: net.ipv4.tcp_keepalive_probes", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_keepalive_probes", Value: "9"}}, }, }}, allowed: true, }, </ins> } for _, tc := range tests {

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: kube::log::status "Setup complete" } <ins> ######################################################## # Kubectl version (--short, --client, --output) # ######################################################## run_kubectl_version_tests() { kube::log::status "Testing kubectl version" TEMP="${KUBE_TEMP}" # create version files, one for the client, one for the server. # these are the files we will use to ensure that the remainder output is correct kube::test::version::object_to_file "Client" "" "${TEMP}/client_version_test" kube::test::version::object_to_file "Server" "" "${TEMP}/server_version_test" kube::log::status "Testing kubectl version: check client only output matches expected output" kube::test::version::object_to_file "Client" "--client" "${TEMP}/client_only_version_test" kube::test::version::object_to_file "Client" "--client" "${TEMP}/server_client_only_version_test" kube::test::version::diff_assert "${TEMP}/client_version_test" "eq" "${TEMP}/client_only_version_test" "the flag '--client' shows correct client info" kube::test::version::diff_assert "${TEMP}/server_version_test" "ne" "${TEMP}/server_client_only_version_test" "the flag '--client' correctly has no server version info" kube::log::status "Testing kubectl version: verify json output" kube::test::version::json_client_server_object_to_file "" "clientVersion" "${TEMP}/client_json_version_test" kube::test::version::json_client_server_object_to_file "" "serverVersion" "${TEMP}/server_json_version_test" kube::test::version::diff_assert "${TEMP}/client_version_test" "eq" "${TEMP}/client_json_version_test" "--output json has correct client info" kube::test::version::diff_assert "${TEMP}/server_version_test" "eq" "${TEMP}/server_json_version_test" "--output json has correct server info" kube::log::status "Testing kubectl version: verify json output using additional --client flag does not contain serverVersion" kube::test::version::json_client_server_object_to_file "--client" "clientVersion" "${TEMP}/client_only_json_version_test" kube::test::version::json_client_server_object_to_file "--client" "serverVersion" "${TEMP}/server_client_only_json_version_test" kube::test::version::diff_assert "${TEMP}/client_version_test" "eq" "${TEMP}/client_only_json_version_test" "--client --output json has correct client info" kube::test::version::diff_assert "${TEMP}/server_version_test" "ne" "${TEMP}/server_client_only_json_version_test" "--client --output json has no server info" kube::log::status "Testing kubectl version: compare json output using additional --short flag" kube::test::version::json_client_server_object_to_file "--short" "clientVersion" "${TEMP}/client_short_json_version_test" kube::test::version::json_client_server_object_to_file "--short" "serverVersion" "${TEMP}/server_short_json_version_test" kube::test::version::diff_assert "${TEMP}/client_version_test" "eq" "${TEMP}/client_short_json_version_test" "--short --output client json info is equal to non short result" kube::test::version::diff_assert "${TEMP}/server_version_test" "eq" "${TEMP}/server_short_json_version_test" "--short --output server json info is equal to non short result" kube::log::status "Testing kubectl version: compare json output with yaml output" kube::test::version::json_object_to_file "" "${TEMP}/client_server_json_version_test" kube::test::version::yaml_object_to_file "" "${TEMP}/client_server_yaml_version_test" kube::test::version::diff_assert "${TEMP}/client_server_json_version_test" "eq" "${TEMP}/client_server_yaml_version_test" "--output json/yaml has identical information" } </ins> # Runs all pod related tests. run_pod_tests() { kube::log::status "Testing kubectl(v1:pods)"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "kubectl version", "kubectl version" json Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request Kubernetes version (use ): $ kubectl version --client Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-12T05:22:15Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS 10.12.2 I find it difficult to parse the output of programmatically. I have to use a YAML parser to do it. I'd prefer to use JSON, or to be given just the string and asked to parse it with a regex. This is obviously a relatively low-priority request.
I wouldnt mind helping out on this, if there are no objections?
are you working on this? If not, I would like to work on it.
I plan to, but if i'll keep you posted if i get too delayed.
How can I direct message you?
didnt see this till now, felt bad i had not kept up on this so i decided to just push through it. If my email is not available in my profile, please let me know, otherwise AT gmail would be the best way to contact me.
Thanks!
Included below are some examples. Let me know what you think, my only concern is the the original output of included the key and then the object type which is a little strange. Note the output flag is optional so we don't break previous behavior and contract, which is now baked and part of e2e testing. here are the couple of samples:


Question: // don't wait due to limited number of clients, but backoff after the default number of steps return retry.RetryOnConflict(retry.DefaultRetry, func() error { set.Status = *status <del> _, updateErr := ssu.client.Apps().StatefulSets(set.Namespace).UpdateStatus(set) </del> <ins> _, updateErr := ssu.client.AppsV1beta1().StatefulSets(set.Namespace).UpdateStatus(set) </ins> if updateErr == nil { return nil }

The statefulset controller and controller history implementations should be migrated off of the Apps client (deprecated) and onto the AppsV1BetaClient. Currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/assign kow3ns
/assign kow3ns
/cc
/assign


Question: func (c *FakeEvents) PatchWithEventNamespace(event *v1.Event, data []byte) (*v1.Event, error) { // TODO: Should be configurable to support additional patch strategies. pt := types.StrategicMergePatchType <del> action := core.NewRootPatchAction(eventsResource, event.Name, pt, data) </del> <ins> var action core.PatchActionImpl </ins> if c.ns != "" { action = core.NewPatchAction(eventsResource, c.ns, event.Name, pt, data) <ins> } else { action = core.NewPatchAction(eventsResource, event.GetNamespace(), event.Name, pt, data) </ins> } obj, err := c.Fake.Invokes(action, event) if obj == nil {

Opening this issue based on what I ran into on with namespace as follows While unit testing the controller, we create a fake client and setup the controller to run using the client. This ends up creating a having the namespace set to When the call eventually reaches The action gets associated with a namespace only if the object has a namespace associated with it. Which in this will be Which will result in the error in to manage the event like does. In case if the namespace is set to Above is a test case that can easily simulate the issue in question. Referring back to the test under PR NA <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: } } <del> func TestTranslateAzureDiskInTreeStorageClassToCSI(t *testing.T) { </del> <ins> func TestTranslateAzureDiskInTreeInlineVolumeToCSI(t *testing.T) { </ins> sharedBlobDiskKind := v1.AzureDedicatedBlobDisk translator := NewAzureDiskCSITranslator()

This test fails on Kubernetes cluster in "westus" region (without any number) and with CSI migration enabled: Error: failed to provision volume with StorageClass "e2e-topology-84vk7sp": error generating accessibility requirements: topology map[] from selected node "ci-op-9j1nby77--9zlwv-worker-westus-ljw4g" is not in requisite: [map[]] StorageClass generated by the test: PVC created by the test: <details</detailsLabels of the selected node: The external-provisioner does not even call the CSI driver and reports this event: The volume should get provisioned. See above, "westus" region with no availability zones seems to be the key. It looks like the translation from in-tree StorageClass to CSI one translates topology requirements to , but the CSI one expects <details</details<details</details<detailsOpenShift 4.10 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon cc
/sig storage
per https://kubernetes-, is not a zone, it's actually a fault-domain num. valid zone format is , could you change the storage class with setting, and try again? Thanks.
o, if "westus" region does not support zone, then I think you should test setting in another zone, make sure you have created a zoned node first.
I have as SC that works without CSI migration, with CSI migration it does not work. That's a regression. It should not matter if the SC is somehow odd. The labels are selected by test code, some other production code might use these labels too.
Out of curiosity, how reliable are zone numbers in Azure? I can see .. in a region with zones. If is used only in regions without a zone, then the translation can translate to , which the CSI driver expects.
I would say that's a wrong usage for zone.

following config would create a pv that could only be scheduled to a non-zone node with fault domain , if there is no node supports zone, then should not be set.
But nodes report that they are in , therefore it make sense to accept the same input in StorageClass.
if we translation "0" to "", then the pv created by that sc could be scheduled to any non zone node with any fault domain. That's a corner case that cluster is not in zone, and user still wants to set in sc
I am not sure I understand. If I provision a CSI volume in the westus cluster, its PV gets anyway: I have the cluster only with nodes in "westus" region, so all of them are without any zone. What would the CSI driver do if there were nodes in "westus" and say "ukwest", which does not have zones either? I can't test that setup easily, but I think the driver would report for both of them anyway, hence translating to does not make things worse, I think.
/sig storage /sig cloud-provider /area provider/azure
can you please check my previous comment? In what cases would be translating "0" to "" worse than the translation code we have today?
it's not worse or better, a single num () is actually not a valid zone, such in sc is actually not valid.
I mean we could translate the single digit to to make it work while the usage is wrong.
/sig storage
/sig cloud-provider


Question: import ( "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/sets"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: }) func cleanTest(f *framework.Framework, block bool) { <ins> // delete the APIService first to avoid causing discovery errors aggrclient := f.AggregatorClient _ = aggrclient.ApiregistrationV1beta1().APIServices().Delete("v1alpha1.wardle.k8s.io", nil) </ins> namespace := "sample-system" client := f.ClientSet _ = client.ExtensionsV1beta1().Deployments(namespace).Delete("sample-apiserver", nil)

Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-


Question: NFSconfig = VolumeTestConfig{ namespace:   api.NamespaceDefault, prefix:      "nfs", <del> serverImage: "gcr.io/google_containers/volume-nfs:0.7", </del> <ins> serverImage: "gcr.io/google_containers/volume-nfs:0.6", </ins> serverPorts: []int{2049}, serverArgs:  []string{"-G", "777", "/exports"}, }

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= <del> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.3 h1:gpw/0Ku+6RgF3jsi7fnCLmlcikBHfKBCUcu1qgc16OU= github.com/vmware/govmomi v0.20.3/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg=

Expectation- We should be able to create PVCs via a storageclass with storagePolicyName defined while using a singed ca cert for secure communications with vCenter. Bug- The PVC created with a storageclass with storagePolicyName defined fails with a error when PVCs created with a storageclass without the storagPolicyName work against the same vCenter and datastore. Scenario: Utilizing the vSphere in-tree plugin on kubernetes 1.12.7. Utilizing secure communication with vSphere (signed ca cert imported into the cluster) We are able to create PVCs with the following storage class: However, if we include a storagePolicyName we receive a failure: Here is the yaml for our sc1 storage class: /sig vmware
I think we are having issue with creating PBM client using certificate set on the vCenter Client. We need to create a PBM client to get list of compatible datastore for given storage policy. I think call is failing at - Can you please provide controller-manager logs for further debugging. Make sure log level is set to 4.
Sorry for the delayed response. I'm not confident I got the log level correct, however this is what I find:
I believe I have log level 4 now set:
For comparison, here is a PVC created w/o storagePolicyName on the same datastore which succeeds:
thanks for posting logs. Now this is confirmed, issue is happening while creating PBM client when we use certificate based authentication. One more thing I want to confirm regarding the certificate, are you using solution user as mentioned here -
Can you also provide value of in the file? Is it set to true or false / 1 or 0 ? Also confirm version of vCenter you are using.
is unset, as for the solution user I'd have to leave that to to expand on.
I don't believe we used any solution user because we simply used "" as the default user during installation. please correct me if I'm wrong.
How is the certificate and private key retrieved for the user . I am successfully able to connect to the PBM using vpxd certificte and key. I ssh into the vCenter and executed following commands to retrieve certificate and private key for vpxd. with vpxd cert and private key, I do not see this issue. In addition I also set while making connection.

Can you try setting Insecure to false? That's the flag setting that triggers the cert error.
with , I see the exact same error - , while creating VC client.
Isn't that because it's self-signed? In the case of the issue in this ticket the cert is signed by a trusted CA /cc to confirm.
The cert is self-signed and its CA isn't trusted.
I used vpxd certificate and issuer is I require to set .
Is there any way to make it work if insecure=false?
- then I’d say that’s expected? If the root CA isn’t trusted, then it is insecure?
well, while that's expected, I don't know if there's a method to add the root CA to a trusted list.
Correct me if I’m wrong but isn’t the trusted list the root CA store of the K8S node’s OS in this case? I’d imagine whatever CA Root signed the cert would need to be to the OS of choice’s CA store.
you are correct, we need to download trusted root certificate and add them in the store on node vm. For now on my mac, I have downloaded trusted root certificate from the vCenter -and them in the keychain and flagged as "Always Trust". ! After this, I am able to connect to VC client with or without specifying this flag at all using VPXD certificate and private key. So certificate needs to be to the CA trust store on the master node VM. after adding trusted root certificate from vCenter to the trust store, you should be able to make a connection using .
After importing the root cert by placing it in /etc/ssl/certs on the master node, I'm able to provision a PVC while specifying a storage policy. Thanks for all the help.
Can you close this issue and work with KK to get this automated in the product?
It turns out that importing the root cert into the host was just a workaround, and the actual bug was fixed in That fix is included in govmomi v0.21.0.


Question: func (sip *serialImagePuller) processImagePullRequests() { for pullRequest := range sip.pullRequests { <ins> startTime := time.Now() </ins> imageRef, err := sip.imageService.PullImage(pullRequest.ctx, pullRequest.spec, pullRequest.pullSecrets, pullRequest.podSandboxConfig) pullRequest.pullChan <- pullResult{ <del> imageRef: imageRef, err:      err, </del> <ins> imageRef:     imageRef, err:          err, pullDuration: time.Since(startTime), </ins> } } }

While going over Kubernetes events I noticed that image pulls for some small images were taking several minutes according to the "Successfully pulled image" event. From investigating this is misleading as the actual pull time is much smaller but due to --serialize-image-pulls=true being the default in the observed cluster and other pods being initialized at the time, the reported pull time ends up accounting for time while waiting for other pods to finish pulling. I expected to see the time it takes to download the image itself from a registry. Customers seem to rely on this metric for this information and we have received some complaints as a result where customers noted slow pull times as a result of misunderstanding the current behavior of this metric. Start with a K8s cluster with --serialize-image-pulls=true (default) - For my check I used default minikube v1.26.1 pod-large where pod-large pulls any large image (the image cannot be locally cached) after step 1 (within seconds) create pod-small where pod-small pulls any small image Check the output for the event after running: Note that despite the small size, pull time for the image in pod-small is several minutes, we should instead see it take seconds (which is how long the pull actually takes). To note this is the case you can try pulling another small image like alpine that isnt cached locally and compare after the above are done. Here is the code that pulls an image in kubernetes and logs the event: As can be seen from it, it's not true pull time, just time from invoking the puller to receiving a result. The serial puller is using a Golang channel as a queue and will just wait until a previous image is done pulling as I noted before, thus it will end up recording the amount of time in the queue + pull time yielding a misleading metric. <details</details<detailsShould reproduce anywhere (tried it on minikube) but have personally observed this in Azure as well </details<details</details<details</details<details</details<details</details
/sig node
But means that kubelet can pull multi images at the same time. For kubelet, is by default true. is for promoting the use of parallel image pulls. The problem maybe with . I did some simple test with 3 ubuntu images as below. The time was collected from events.

-- --serialize-image-pulls=true -- -- image pulling serialize ubuntu:22.10 73.6 MiB 1m21.284220239s ubuntu:20.04 75.7 MiB 2m7.876426944s ubuntu:14.04 212.9 MiB 4m52.884660938s
You are right thanks for taking a look at this. It seems I got the two reversed in my head. The issue indeed happens when --serialize-image-pulls=true which is the default as you pointed out. I'll update the issue to reflect that
/triage accepted /priority important-longterm


Question: atomic.AddInt32(tpCount, int32(count)) } } <del> pl.parallelizer.Until(context.Background(), len(allNodes), processNode) </del> <ins> pl.parallelizer.Until(context.Background(), len(filteredNodes), processNode) </ins> // calculate min match for each topology pair for i := 0; i < len(constraints); i++ {

We know that nodeAffinity/nodeSelector is honored during the calculating of PodTopolgySpread, however, when some existing pods match the incoming pod's topologySpreadConstratints, while also excluded by the nodeAfffinity, things get a bit tricky. Raising this issue to discuss if we should read it as a bug. Here are the repro steps: you have a 3-node cluster, worker1 and worker2 belong to zoneA, and worker3 belongs to zoneB: make a deployment with 3 replicas. Suppose 2 land on worker2, and 1 land on worker. a new deployment with topologySpreadConstraints as well as nodeAffinity (not schedule to worker2): test pod will land on worker3. This behavior concerns me: it looks like the 2 pods on worker2 are taken into consideration, even though worker2 is excluded by nodeAffinity, so the pods on zoneA/zoneB distribute as 2:2. However, shouldn't we disregard the matching pods that are excluded by nodeAffinity? In that case, the test pod should land on worker1 to reach a pods distribution as 1:1. Please share your thoughts. /sig scheduling /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc/
That is not the issue, the 2 pods on work2 aren't actually being taken into consideration; the problem is similar to what describes. What happened is that zoneA is now not "visible" at all, and so it is as if we have a single zone. The solution should apply to both cases.
I think we should consider it as a bug. Imagine in this situation, all of these pods are labeled with , then we want to apply a like this: is excluded in scheduling. But now when we try to calculate the skew, we will still calculate the pods in . Since : is , then we will schedule the new pod to or , but since they are resource exhausted, then it will be in pending state.

But if we filtered out the , the pod will be scheduled to successfully. I think it's the right algorithm.
it's not the same as , even if worker2 becomes invisible, there are still 2 zones:
ah, I didn't notice existed. What part of the code is causing this behavior? I would have assumed that the spreading would happen between and . The following code shows that we should be skipping counting the pods on worker2
it's the that is passed to processNode(): We should pass on a filtered list of nodes to processNode(). included the fix in , and that bits should be extracted out to be a standalone PR, if we get to a consensus on this issue is a bug.
I still don't get where the bug is. I think iterating over all node is the right way and the line I pointed to above should filter out the nodes that don't match the affinity. Passing a filteredList (as in exclude nodes that were filtered by previous filters) is certainly not the right solution because we need to count pods on all nodes. Let alone the filteredlist is different based on how we order the filters, which should never be a factor in the semantics of the PodTopologySpread.
The bug lies in the preFilterState's calculation. For the above case, it's expected to be , but it ended up with . The code I posted above is in PodTopologySpread's PreFitler, not Filter.
ok, this is indeed a bug, we need to check the node affinity constraints again inside the second loop.
actually, doesn't seem efficient that we have two loops in the first place
It seems we are doing it right in the score function, excluding those nodes:
Feel the same. Let's fix the correctness issue first, and then improve the efficiency.
Do we know of a use case where this would be useful? It's kind of odd that there are matching pods in a node where we don't intend to schedule. Maybe during some sort of migration?
It's about clarifying/correcting the semantics of constraints (nodeAffinity, and the to-be- ones like Unschedulable taint) embedded in PodTopologySpread. So IMO we should fix the correctness issue, no matter there is a use case or not.
One thing to be aware of is whether a potential fix could render some pods unschedulable, thus be a breaking change.


Question: fs.StringVar(&s.HostnameOverride, "hostname-override", s.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") fs.BoolVar(&s.ForceUserspaceProxy, "legacy-userspace-proxy", true, "Use the legacy userspace proxy (instead of the pure iptables proxy).") fs.DurationVar(&s.SyncPeriod, "iptables-sync-period", 5*time.Second, "How often iptables rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.") <ins> fs.BoolVar(&s.MasqueradeAll, "masquerade-all", false, "If using the pure iptables proxy, SNAT everything") </ins> } // Run runs the specified ProxyServer.  This should never exit.

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: "io" "github.com/pkg/errors" <del> "sigs.k8s.io/yaml" </del> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/serializer" <ins> yamlserializer "k8s.io/apimachinery/pkg/runtime/serializer/yaml" </ins> errorsutil "k8s.io/apimachinery/pkg/util/errors" utilyaml "k8s.io/apimachinery/pkg/util/yaml" clientsetscheme "k8s.io/client-go/kubernetes/scheme"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: w.Write(LEVEL_0, "nSettings:n") w.Write(LEVEL_1, "Allow Privileged:t%tn", psp.Spec.Privileged) <ins> w.Write(LEVEL_1, "Allow Privilege Escalation:t%vn", psp.Spec.AllowPrivilegeEscalation) </ins> w.Write(LEVEL_1, "Default Add Capabilities:t%vn", capsToString(psp.Spec.DefaultAddCapabilities)) w.Write(LEVEL_1, "Required Drop Capabilities:t%sn", capsToString(psp.Spec.RequiredDropCapabilities)) w.Write(LEVEL_1, "Allowed Capabilities:t%sn", capsToString(psp.Spec.AllowedCapabilities))

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig auth What happened: doesn't describe option. What you expected to happen: value should be present. How to reproduce it (as minimally and precisely as possible): Environment:


Question: }, }, { <del> description: "pod specified a toleration for taint `unreachable:NoExecute`, expect add toleration for `notReady:NoExecute`", </del> <ins> description: "pod specified a toleration for taint `unreachable:NoExecute`, expect add toleration for `not-ready:NoExecute`", </ins> requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: return &unsupportedPodContainerManager{} } <del> func NewContainerManager(_ mount.Interface, _ cadvisor.Interface, _ NodeConfig) (ContainerManager, error) { </del> <ins> func NewContainerManager(_ mount.Interface, _ cadvisor.Interface, _ NodeConfig, failSwapOn bool) (ContainerManager, error) { </ins> return &unsupportedContainerManager{}, nil }

Failed: https://k8s- Run so broken it didn't make JUnit output!
Looks like .
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!

Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Should be fixed by
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!


Question: if _, err := iptClient.EnsureRule(utiliptables.Append, utiliptables.TableNAT, KubePostroutingChain, "-j", "MARK", "--xor-mark", masqueradeMark); err != nil { klog.Errorf("Failed to ensure unmarking rule for %v: %v", KubePostroutingChain, err) <del> return </del> <ins> return false </ins> } masqRule := []string{ "-m", "comment", "--comment", "kubernetes service traffic requiring SNAT",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: nonResourceURL: "/logs/,/healthz", resources:      "pods", expectedClusterRole: &rbac.ClusterRole{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "ClusterRole"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: clusterRoleName, },


Question: // TODO: Improve error messages by first testing if 'tar' is present in the container? if o.NoPreserve { <del> cmdArr = []string{"tar", "--no-same-permissions", "--no-same-owner", "-xf", "-"} </del> <ins> cmdArr = []string{"tar", "--no-same-permissions", "--no-same-owner", "-xmf", "-"} </ins> } else { <del> cmdArr = []string{"tar", "-xf", "-"} </del> <ins> cmdArr = []string{"tar", "-xmf", "-"} </ins> } destDir := path.Dir(dest.File) if len(destDir) > 0 {

What happened: Copying a folder to a pod, e.g. postgresql-16-1-m7wjw results in the following error: time stamp 2019-02-12 12:17:06 is 0. s in the future What you expected to happen: Successful file upload to a given pod. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: perhaps this might come useful Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig cli
Ran into this as well. That and it's annoying that cannot be used to create a directory structure when copying files into a container.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: }, }, { <del> name: "caKeyPresent true", </del> <ins> name: "caKeyPresent true for v1.12.0-beta.2", </ins> cfg: &kubeadmapi.InitConfiguration{ APIEndpoint: kubeadmapi.APIEndpoint{AdvertiseAddress: "1.2.3.4"}, ClusterConfiguration: kubeadmapi.ClusterConfiguration{ <del> KubernetesVersion: "v1.7.0", </del> <ins> KubernetesVersion: "v1.12.0-beta.2", </ins> Networking:        kubeadmapi.Networking{ServiceSubnet: "10.96.0.0/12", DNSDomain: "cluster.local"}, }, },

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: } for _, tc := range cases { <del> testNamespace := "test_projected_namespace" tc.secret.ObjectMeta = metav1.ObjectMeta{ Namespace: testNamespace, Name:      tc.name, } </del> <ins> t.Run(tc.name, func(t *testing.T) { testNamespace := "test_projected_namespace" tc.secret.ObjectMeta = metav1.ObjectMeta{ Namespace: testNamespace, Name:      tc.name, } </ins> <del> source := makeProjection(tc.name, tc.mode, "secret") source.Sources[0].Secret.Items = tc.mappings source.Sources[0].Secret.Optional = &tc.optional testPodUID := types.UID("test_pod_uid") pod := &v1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: testNamespace, UID: testPodUID}} client := fake.NewSimpleClientset(tc.secret) _, host := newTestHost(t, client) var myVolumeMounter = projectedVolumeMounter{ projectedVolume: &projectedVolume{ sources: source.Sources, podUID:  pod.UID, plugin: &projectedPlugin{ host:      host, getSecret: host.GetSecretFunc(), </del> <ins> source := makeProjection(tc.name, tc.mode, "secret") source.Sources[0].Secret.Items = tc.mappings source.Sources[0].Secret.Optional = &tc.optional testPodUID := types.UID("test_pod_uid") pod := &v1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: testNamespace, UID: testPodUID}} client := fake.NewSimpleClientset(tc.secret) tempDir, host := newTestHost(t, client) defer os.RemoveAll(tempDir) var myVolumeMounter = projectedVolumeMounter{ projectedVolume: &projectedVolume{ sources: source.Sources, podUID:  pod.UID, plugin: &projectedPlugin{ host:      host, getSecret: host.GetSecretFunc(), }, </ins> }, <del> }, source: *source, pod:    pod, } </del> <ins> source: *source, pod:    pod, } </ins> <del> actualPayload, err := myVolumeMounter.collectData() if err != nil && tc.success { t.Errorf("%v: unexpected failure making payload: %v", tc.name, err) continue } if err == nil && !tc.success { t.Errorf("%v: unexpected success making payload", tc.name) continue } if !tc.success { continue } if e, a := tc.payload, actualPayload; !reflect.DeepEqual(e, a) { t.Errorf("%v: expected and actual payload do not match", tc.name) } </del> <ins> actualPayload, err := myVolumeMounter.collectData() if err != nil && tc.success { t.Errorf("%v: unexpected failure making payload: %v", tc.name, err) return } if err == nil && !tc.success { t.Errorf("%v: unexpected success making payload", tc.name) return } if !tc.success { return } if e, a := tc.payload, actualPayload; !reflect.DeepEqual(e, a) { t.Errorf("%v: expected and actual payload do not match", tc.name) } }) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I run the unit tests for , but I found it would not clean the tempdir it created. ! What you expected to happen: automatically clean up when finishing unit tests. How to reproduce it (as minimally and precisely as possible): go test $ Anything else we need to know?: Environment: k8s 1.15 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /sig testing
/assign Assign the issue to you since you have fired a pull request.


Question: Name: podName, }, Spec: v1.PodSpec{ <del> RestartPolicy: v1.RestartPolicyNever, </del> <ins> PriorityClassName: "system-node-critical", RestartPolicy:     v1.RestartPolicyNever, </ins> Containers: []v1.Container{ createContainer(ctnName), },

ci-cgroupv1-containerd-node-arm64-e2e-serial-ec2-eks ci-cgroupv1-containerd-node-e2e-serial-ec2-eks It has been failing consistently for a long time. No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
There's another bug that was filed a long time ago. But there's no update on it and the same test in gce jobs seem to be green now.
cc


Question: } go func() { <ins> mux := http.NewServeMux() healthz.InstallHandler(mux) </ins> if s.EnableProfiling { <del> http.HandleFunc("/debug/pprof/", pprof.Index) http.HandleFunc("/debug/pprof/profile", pprof.Profile) http.HandleFunc("/debug/pprof/symbol", pprof.Symbol) </del> <ins> mux.HandleFunc("/debug/pprof/", pprof.Index) mux.HandleFunc("/debug/pprof/profile", pprof.Profile) mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol) </ins> } <del> http.Handle("/metrics", prometheus.Handler()) http.ListenAndServe(net.JoinHostPort(s.Address.String(), strconv.Itoa(s.Port)), nil) </del> <ins> mux.Handle("/metrics", prometheus.Handler()) server := &http.Server{ Addr:    net.JoinHostPort(s.Address.String(), strconv.Itoa(s.Port)), Handler: mux, } glog.Fatal(server.ListenAndServe()) </ins> }() configFactory := factory.NewConfigFactory(kubeClient)

Found this by calling the "/validate" end point on the master: root curl 127.0.0.1:8080/validate <snip{ "component": "scheduler", "health": "failure", "healthCode": 1, "msg": "404 page not foundn", "err": "unhealthy http status code: 404 (404 Not Found)" }, </snipSeems like any requests to it return a 404: root netstat -p -l | grep kube-scheduler tcp 0 0 localhost: : LISTEN 4105/kube-scheduler root curl 127.0.0.1:/healthz 404 page not found root curl 127.0.0.1: 404 page not found
I'm not seeing this on my 0.14.1 cluster: $ curl http://127.0.0.1:8080/validate { "component": "scheduler", "health": "success", "msg": "ok", "err": "nil" }, $ curl http://127.0.0.1:/healthz ok
This is happening on HEAD. seems to have caused this. /cc
Could we rollback this PR?
is this only on the scheduler?
Yes. The change basically broke the /healthz mapping in kube-scheduler, so the health pings are failing. The "validate-" doesn't hit it, but the master's "/validate" does. This is causing the internal cluster validation checks to fail.
GKE relies on the /validate endpoint on the master (since it can't run the validate- script) and this has broken our GKE-CI build since it was merged this morning.
I think we can revert the PR and let profiling break for now. what do you think?
SGTM, sorry for the breakage.
Ahh, you guys already revert the commit? The fix should be very simple. Let me send PR shortly to fix this.
Thanks for the fix and sorry for the breakage


Question: audit.AddAuditAnnotation(ctx, podsecurityadmissionapi.AuditAnnotationPrefix+k, v) } if !result.Allowed { <del> if result.Result != nil && len(result.Result.Message) > 0 { // TODO: use code/reason/etc from status return admission.NewForbidden(a, errors.New(result.Result.Message)) </del> <ins> // start with a generic forbidden error retval := admission.NewForbidden(a, errors.New("Not allowed by PodSecurity")).(*apierrors.StatusError) // use message/reason/details/code from admission library if populated if result.Result != nil { if len(result.Result.Message) > 0 { retval.ErrStatus.Message = result.Result.Message } if len(result.Result.Reason) > 0 { retval.ErrStatus.Reason = result.Result.Reason } if result.Result.Details != nil { retval.ErrStatus.Details = result.Result.Details } if result.Result.Code != 0 { retval.ErrStatus.Code = result.Result.Code } </ins> } <del> return admission.NewForbidden(a, errors.New("Not allowed by PodSecurity")) </del> <ins> return retval </ins> } return nil }

Address the TODO here:
I'm interested in knocking this out, but am not entirely sure where this would be propagated. Seems related to / ?


Question: // Build a slice of iptables args for a portal rule. func iptablesPortalArgs(destIP net.IP, destPort int, protocol api.Protocol, proxyIP net.IP, proxyPort int, service string) []string { <ins> // This list needs to include all fields as they are eventually spit out // by iptables-save.  This is because some systems do not support the // 'iptables -C' arg, and so fall back on parsing iptables-save output. // If this does not match, it will not pass the check.  For example: // adding the /32 on the destination IP arg is not strictly required, // but causes this list to not match the final iptables-save output. // This is fragile and I hope one day we can stop supporting such old // iptables versions. </ins> args := []string{ "-m", "comment", "--comment", service, "-p", strings.ToLower(string(protocol)), <del> "-d", destIP.String(), </del> <ins> "-m", strings.ToLower(string(protocol)), "-d", fmt.Sprintf("%s/32", destIP.String()), </ins> "--dport", fmt.Sprintf("%d", destPort), } // This is tricky.  If the proxy is bound (see Proxier.listenAddress)

I am running kubernetes 0.5.2 + and I've noticed that the number of entries in iptables increases every few seconds. I started with a clean install, no pods or replicationControllers, there is now two services by default (kubernetes-ro and kubernetes). I ran and got . Next, I turned up 7 services and 3 replicationControllers which resulted in 3 pods also being turned up. I started a watch... I saw "38", "43, "50, "64", "66", "68", "70", ..., "115" and on until I stopped . It appears the proxy continues to add REDIRECT's for the services, for example the lines that keep being look like: A similar line for each service gets every second or so, it appears to all be in the table. I'm thinking the newly function might be the culprit, since calls and in the log I see every few seconds (looks like does a sync every 5 seconds).
I noticed that the 0.5.3 and 0.5.4 releases don't include the , though it's in master. Perhaps the issue I've filed is already known and I missed it when searching the issues.
I have not heard this issue. Can you paste a listing of iptables-save -t nat when you are seeing this problem? Alternatively, I can cut you a branch with some extra debug instrumentation if you're willing to test it out today...
I'm not able to test it out today. As far as the iptables output, I'd have to type it in (can't copy/paste -- different physical network). However, the output has lines, like the one given in the issue, which are every 5 seconds to the "nat" table. It seems like the check to see if the rule already exists is perhaps not functioning properly. I am working on a fix for (because it uses the iptables -C flag which isn't supported in older iptables), in that fix I created a function to check if an existing rule already exists: Example call: Using may not be ideal, perhaps breaking up the output of the iptables command into a slice and checking each line would be better(?) sure yet.
Fix is pending - would be great if someone could verify it against an actually old iptables (I just faked the version string).

should be able to try it out friday, maybe monday
Feel free to rip off our checks with out echeck function :) On Nov 26, 2014 7:36 PM, "Craig Wickesser" wrote:
that it actually works. On Nov 26, 2014 7:40 PM, "Tim Hockin" wrote:
I fired up kuberenetes 0.5.2 (plus + your changes) and things looked good initially (when starting up kubernetes, no pods/services, etc). The number of entries iptables-save remained constant. I then turned up some services and watched the length of iptables-save increase. I turned down the services and the number of entries in iptables-save continues to increase. At the moment, every 5 seconds it's back to adding two entries (one for the kubernetes service and one for the kubernetes-ro service). With log level set to 0, I see "Opened iptables portal for service.." messages in the proxy log every 5 seconds ().
Are you sure you have the right proxy version running? I could repro the initial problem, but not this - it works just fine. To be clear when you say it increases after creating a service - do you mean it increases by 1 rule or that it increases continually? , Craig Wickesser wrote:
On Monday/Tuesday I'll try the latest from master, perhaps I messed something up. To clarify, the increase was continuous. When I initially setup kubernetes (with a fresh/clean etcd), things looked normal with regards to iptables. I then deployed some pods and services and kept watching iptables-save. The number of entries increased every 5 seconds. I deleted pods, replication controllers and services and the number of entries continued to increase, but only by two (for the kubernetes and kubernetes-ro services) every 5 seconds.
Did you paste the output of iptables-save? Its possible that it is slightly different from my version. This is why parsing text is no fun :( On Nov 28, 2014 3:42 PM, "Craig Wickesser" wrote:
I did not include the output of iptables-save. I'll see if I can get additional output besides the line included in the initial issue.
Just a couple lines so I can see the format.

On Nov 28, 2014 7:32 PM, "Craig Wickesser" wrote:
here's a few lines from : All of the lines follow the exact same format. Looks like the proxier (to fit with the output I provided) would need to look like this:
FWIW - I'm using CentOS 6.5 with iptables version 1.4.7.
I recompiled with changes to proxier as mentioned earlier (changed the order of args) and did not resolve the issue. I continue seeing (in the proxy log):
I've some additional logging to log the and the . I am logging them using these two lines: When I reduce the output between the and the (when it should return "true", that fields is a superset), the result looks like: Notice has quotes around it in the set, and the does not? I think perhaps that's the issue and that's why the check is false.
I just tried changing to: and that's working. Basically, strip out quotes from the line. There might be a better alternative.
Uggh, seriously? Your iptables add quotes around the comment, mine omits the quotes for single tokens. I guess I can force it to not be a single word, something like "service: " service name. What a hack. Any better ideas? , Craig Wickesser wrote:
Hmm, it seems worse than that. If I feed in a comment with quotes, I get ""foo"", so I can't do that to be compatible with your implementation. Can you try something? iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment "foo1" iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment '"foo2"' iptables-save | grep foo -A PREROUTING -d 1.2.3.4/32 -m comment --comment ""foo2"" -A PREROUTING -d 1.2.3.4/32 -m comment --comment foo1 , Tim Hockin wrote:
i'll try that in about 8 hours :)
here's what I got:
ugh, that is SO BROKEN. I'm going to have to do something really unpleasant.

, Craig Wickesser wrote:
I have a potential fix, but airport wifi prevents me from pushing git <shrug. will publish it later.
sounds good, I appreciate the effort.
PR posted - please test before we commit.
I've tried the PR and a comment to it.
got the same problem here using Kubernetes v0.6.0-12-gd927031aecdeb6 on redhat 6.5
It looks like 0.6 was cut before - - can we patch that in, or should we wait of 0.7? , polariss0i wrote:
got the same problem on kubernetes v1.0.3 and iptables-v1.4.7. Should I file a new issue?


Question: unboundVolumesSatisfied := true boundVolumesSatisfied := true sufficientStorage := true <ins> boundPVsFound := true </ins> defer func() { if err != nil { return

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: } } <ins> func TestGetPersistentVolumeClaims(t *testing.T) { // nil inherits statefulset labels pod := newPod() statefulSet := newStatefulSet(1) statefulSet.Spec.Selector.MatchLabels = nil claims := getPersistentVolumeClaims(statefulSet, pod) pvc := newPVC("datadir-foo-0") pvc.SetNamespace(v1.NamespaceDefault) resultClaims := map[string]v1.PersistentVolumeClaim{"datadir": pvc} if !reflect.DeepEqual(claims, resultClaims) { t.Fatalf("Unexpected pvc:n %+vn, desired pvc:n %+v", claims, resultClaims) } // nil inherits statefulset labels statefulSet.Spec.Selector.MatchLabels = map[string]string{"test": "test"} claims = getPersistentVolumeClaims(statefulSet, pod) pvc.SetLabels(map[string]string{"test": "test"}) resultClaims = map[string]v1.PersistentVolumeClaim{"datadir": pvc} if !reflect.DeepEqual(claims, resultClaims) { t.Fatalf("Unexpected pvc:n %+vn, desired pvc:n %+v", claims, resultClaims) } // non-nil with non-overlapping labels merge pvc and statefulset labels statefulSet.Spec.Selector.MatchLabels = map[string]string{"name": "foo"} statefulSet.Spec.VolumeClaimTemplates[0].ObjectMeta.Labels = map[string]string{"test": "test"} claims = getPersistentVolumeClaims(statefulSet, pod) pvc.SetLabels(map[string]string{"test": "test", "name": "foo"}) resultClaims = map[string]v1.PersistentVolumeClaim{"datadir": pvc} if !reflect.DeepEqual(claims, resultClaims) { t.Fatalf("Unexpected pvc:n %+vn, desired pvc:n %+v", claims, resultClaims) } // non-nil with overlapping labels merge pvc and statefulset labels and prefer statefulset labels statefulSet.Spec.Selector.MatchLabels = map[string]string{"test": "foo"} statefulSet.Spec.VolumeClaimTemplates[0].ObjectMeta.Labels = map[string]string{"test": "test"} claims = getPersistentVolumeClaims(statefulSet, pod) pvc.SetLabels(map[string]string{"test": "foo"}) resultClaims = map[string]v1.PersistentVolumeClaim{"datadir": pvc} if !reflect.DeepEqual(claims, resultClaims) { t.Fatalf("Unexpected pvc:n %+vn, desired pvc:n %+v", claims, resultClaims) } } func newPod() *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "foo-0", Namespace: v1.NamespaceDefault, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "nginx", Image: "nginx", }, }, }, } } </ins> func newPVC(name string) v1.PersistentVolumeClaim { return v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{

What happened: When creating a StatefulSet that uses volumeClaimTemplates, the resulting PersistentVolumeClaims have incorrect labels. Instead of using the labels defined in their metadata, they use the labels from the StatefulSet's selector. What you expected to happen: I expect the created PersistentVolumeClaims to have the labels defined in their metadata. How to reproduce it (as minimally and precisely as possible): Resulting PersistentVolumeClaim: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE Install tools: Others: . /sig apps
+1 just ran into this myself.
/assign
GitHub didn't allow me to assign the following users: cwdsuzhou. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
As in . Even thought the StatefulSet controller looks for the PVC by name (not by label) we generate labels for the PVC based on the StatefulSet's selector. Removing this would be a backward incompatible change to a v1 API contract (can't do that). Adding additional labels might be acceptable, but, If you want to change the behavior of the controller wrt to this, I'd like to see a (small) KEP for it so it can be discussed a bit more broadly in the SIG.
I very much think this would be a good discussion to have. If you could, please let me know what's required to start a KEP on this topic. As I see it, the current implementation is in violation of the API: the volumeClaimTemplates data type is listed as " array". A PersistentVolumeClaim's metadata is listed as a standard type. The fact that the PersistentVolumeClaim's metadata is being handled differently when defined in a StatefulSet means that the PersistentVolumeClaim API isn't being properly followed in that context. I'd also like to argue that the change wouldn't be backwards incompatible.

The StatefulSet's volumeClaimTemplates description even says that the matching is done on name: "Every claim in this list must have at least one matching (by name) volumeMount in one container in the template." Respecting the PersistentVolumeClaim's metadata wouldn't necessarily cause any incompatibility from a technical perspective (although it may affect some users' implementations that rely on it). In summary, the current implementation violates the principle of least surprise, makes it impossible to label individual volume claims separately, and it directly contradicts the API specification.
The KEP process is described . You can open a KEP via a PR against the SIG Apps repository. You can use me as your approver. As to the compatibility of the current PR, the current label generation is v1 and has worked the same way for several years now. Users that select the PVCs created by the controller have a reasonable expectation that this should continue to function in subsequent releases. If you add additional labels but generate labels that match the selector this would allow for additional labels while maintaining compatibility (imo).


Question: }, }, }, <del> expErr: true, </del> <ins> expectedToplogy: []v1.TopologySelectorTerm{ { MatchLabelExpressions: []v1.TopologySelectorLabelRequirement{ { Key:    "test", Values: []string{"foo", "bar"}, }, }, }, }, </ins> }, } for _, tc := range testCases { t.Logf("Running test: %v", tc.name) gotTop, err := translateAllowedTopologies(tc.topology, GCEPDTopologyKey) <del> if err != nil && !tc.expErr { t.Errorf("Did not expect an error, got: %v", err) } if err == nil && tc.expErr { t.Errorf("Expected an error but did not get one") </del> <ins> if err != nil { t.Errorf("Unexpected error: %w", err) </ins> } if !reflect.DeepEqual(gotTop, tc.expectedToplogy) {

Unknown allowedTopology labels cause csi translation to fail: However, on GKE other labels are sometimes used in PV node affinity, such as or It's not common (100s of clusters as of July 2021), but seems enough that it should be dealt with. The suggestion is to just pass unknown topologies through. If they are used in the storageclass labels they will cause failures, but that is the current gce-pd behavior anyway. This reasoning probably holds for other clouds as well, although I haven't tested it myself. /assign /sig storage
/triage accepted


Question: image = imageutils.GetE2EImage(imageutils.Pause) ) <del> var _ = SIGDescribe("Windows volume mounts ", func() { </del> <ins> var _ = SIGDescribe("[Feature:Windows] Windows volume mounts ", func() { </ins> f := framework.NewDefaultFramework("windows-volumes") var ( emptyDirSource = v1.VolumeSource{

similar to PR en route
/sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // OperationGenerator interface that extracts out the functions from operation_executor to make it dependency injectable type OperationGenerator interface { // Generates the MountVolume function needed to perform the mount of a volume plugin <del> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (volumetypes.GeneratedOperations, error) </del> <ins> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) volumetypes.GeneratedOperations </ins> // Generates the UnmountVolume function needed to perform the unmount of a volume plugin GenerateUnmountVolumeFunc(volumeToUnmount MountedVolume, actualStateOfWorld ActualStateOfWorldMounterUpdater, podsDir string) (volumetypes.GeneratedOperations, error)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.


Question: // newClaimInfoCache creates a new claim info cache object, pre-populated from a checkpoint (if present). func newClaimInfoCache(stateDir, checkpointName string) (*claimInfoCache, error) { <del> stateImpl, err := state.NewCheckpointState(stateDir, checkpointName) </del> <ins> checkpointer, err := state.NewCheckpointer(stateDir, checkpointName) </ins> if err != nil { return nil, fmt.Errorf("could not initialize checkpoint manager, please drain node and remove dra state file, err: %+v", err) } <del> curState, err := stateImpl.GetOrCreate() </del> <ins> checkpoint, err := checkpointer.GetOrCreate() </ins> if err != nil { return nil, fmt.Errorf("error calling GetOrCreate() on checkpoint state: %v", err) } cache := &claimInfoCache{ <del> state:     stateImpl, claimInfo: make(map[string]*ClaimInfo), </del> <ins> checkpointer: checkpointer, claimInfo:    make(map[string]*ClaimInfo), </ins> } <del> for _, entry := range curState { </del> <ins> entries, err := checkpoint.GetClaimInfoStateList() if err != nil { return nil, fmt.Errorf("error calling GetEntries() on checkpoint: %w", err) } for _, entry := range entries { </ins> info := newClaimInfoFromState(&entry) cache.claimInfo[info.Namespace+"/"+info.ClaimName] = info }

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err } <ins> if err := volumeutil.MakeNestedMountpoints(b.volName, dir, b.pod); err != nil { return err } </ins> optional := b.source.Optional != nil && *b.source.Optional configMap, err := b.getConfigMap(b.pod.Namespace, b.source.Name)

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: if err != nil && !apierrors.IsNotFound(err) { return err } <ins> if d.OnPodDeletionOrEvictionStarted != nil { d.OnPodDeletionOrEvictionStarted(&pod, false) } </ins> } ctx := d.getContext() params := waitForDeleteParams{

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: func expectPrefix(t *testing.T, prefix string, errs errors.ErrorList) { for i := range errs { <del> if !strings.HasPrefix(errs[i].(errors.ValidationError).Field, prefix) { t.Errorf("expected prefix '%s' for %v", errs[i]) </del> <ins> if f, p := errs[i].(errors.ValidationError).Field, prefix; !strings.HasPrefix(f, p) { t.Errorf("expected prefix '%s' for field '%s' (%v)", p, f, errs[i]) </ins> } } }

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: return v.validateHostErr } <del> loadedProfiles, err := v.getLoadedProfiles() if err != nil { return fmt.Errorf("could not read loaded profiles: %v", err) } </del> var retErr error podutil.VisitContainers(&pod.Spec, podutil.AllContainers, func(container *v1.Container, containerType podutil.ContainerType) bool { <del> retErr = validateProfile(GetProfileName(pod, container.Name), loadedProfiles) </del> <ins> retErr = ValidateProfileFormat(GetProfileName(pod, container.Name)) </ins> if retErr != nil { return false }

What happened: Pods with invalid container-level seccomp profiles allow its init containers to be executed. Pods with unloaded apparmor profiles blocks the entire pod. What you expected to happen: Consistency between both. In one hand AppArmor blocks all pod execution regardless of the containers having or not a profile attached to it. On the other, Seccomp allows for an init container to deploy its own profile - as currently done on . How to reproduce it (as minimally and precisely as possible): Apply the yaml below: Resulting in only the init container from the being executed: ! Anything else we need to know?: AppArmor is currently being target to GA on 1.21 (). Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-alpha.0.292+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-12T23:13:21Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /area security
/cc
/sig security /triage accepted
I think we can consider this as a bug, right? Then we could work on a fix for v1.21 interdependently of the AppArmor GA graduation.
Sounds like a good plan :+1:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign


Question: return nil } <del> // This is the point at which we used to add/remove the overlap annotation. // Now we always remove it if it exists, because it is obsolete as of 1.6. // Although the server no longer adds or looks at the annotation, // it's important to remove it from controllers created before the upgrade, // so that old clients (e.g. kubectl reaper) know they can no longer assume // the controller is blocked due to selector overlap and has no dependents. if _, ok := d.Annotations[util.OverlapAnnotation]; ok { delete(d.Annotations, util.OverlapAnnotation) d, err = dc.client.ExtensionsV1beta1().Deployments(d.Namespace).UpdateStatus(d) if err != nil { return fmt.Errorf("couldn't remove obsolete overlap annotation from deployment %v: %v", key, err) } } </del> // List ReplicaSets owned by this Deployment, while reconciling ControllerRef // through adoption/orphaning. rsList, err := dc.getReplicaSetsForDeployment(d)

The Deployment overlap annotation () is no longer populated nor used by 1.6+ on the server side. However, in we maintained knowledge of the annotation for 1.6 clients in case they talk to pre-1.6 servers. That PR also adds code to clean up the annotation whenever it's found in 1.6+ clusters, so that pre-1.6 clients don't see it and make the wrong assumptions. The code handling this annotation should be removed when all the following conditions are met: We no longer support a 1.5.x client talking to the latest server version. We no longer support upgrading the server directly from 1.5.x to the latest version. The soonest milestone that meets both conditions according to the is v1.8. This issue should be moved to that milestone once it exists.
TODOs: [x] deployment controller: [x] deployment util: [x] deployment reaper: [x] deployment describer:
I can help on this
/assign
GitHub didn't allow me to assign the following users: wanghaoran1988. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks, feel free to send patches


Question: --v="${LOG_LEVEL}"  --config="${TMP_DIR}"/kube-scheduler.yaml  --feature-gates="${FEATURE_GATES}"  <ins> --emulated-version="${EMULATED_VERSION}"  </ins> --authentication-kubeconfig "${CERT_DIR}"/scheduler.kubeconfig  --authorization-kubeconfig "${CERT_DIR}"/scheduler.kubeconfig  --master="https://${API_HOST}:${API_SECURE_PORT}" >"${SCHEDULER_LOG}" 2>&1 &

Currently the local-up- script does not have support for the flag. This flag was recently here as a part of For development, testing, and full plumbing through of the feature the flag should be to local-up- to allow for usage of the feature when setting up a cluster via local-up-
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig testing /triage accepted /assign


Question: }, Spec: storage.VolumeAttachmentSpec{ NodeName: nodeName, <del> Attacher: csiPluginName, </del> <ins> Attacher: "mock", </ins> Source: storage.VolumeAttachmentSource{ PersistentVolumeName: &pvName, },

Internal volume plugin for CSI creates VolumeAttachment with (= the plugin name). It should be CSI driver name instead, as the external attacher must know which CSI driver to call. /sig storage /kind bug please prioritize PTAL. It should be one-line change in
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ACK. In progress