Question: // We have seen one of these calls take just over 15 seconds, so putting this at 30. proxyHTTPCallTimeout = 30 * time.Second <ins> podRetryPeriod       = 1 * time.Second podRetryTimeout      = 1 * time.Minute </ins> ) <ins> type jsonResponse struct { Method string Body   string } </ins> var _ = SIGDescribe("Proxy", func() { version := "v1" ginkgo.Context("version "+version, func() {

[x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance Where/How does PATCH/HEAD/OPTIONS requests for the following endpoints line up with the API reference? connectCoreV1PatchNamespacedPodProxyWithPath connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath Pod v1 core: Proxy Operations the following endpoints are listed as valid endpoints; connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath These requests are getting logged as a verb. Line : api/openapi-spec Line : api/openapi-spec Looking at there doesn't look to be a mapping for the above methods. According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Create a service and link it to the pod Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/re-open
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a #   day). # * keep only 5 old (rotated) logs, and will discard older logs. <del> local logrotate_files=( "kube-scheduler" "kube-proxy" "kube-apiserver" "kube-controller-manager" "kube-addons" ) for file in "${logrotate_files[@]}" ; do cat > /etc/logrotate.d/${file} <<EOF /var/log/${file}.log { </del> <ins> cat > /etc/logrotate.d/allvarlogs <<EOF /var/log/*.log { </ins> rotate 5 copytruncate missingok

GCI log rotation configuration missed "" file. We should just configure log rotation to rotate all "*.log" files in /var/log.
I cherrypicked your change last night. Please do what you need to do to confirm this is resolved.
Thank you, will do.
Verified. Closing issue.


Question: // createTargetedProbeCommand returns a command line that performs a DNS lookup for a specific record type func createTargetedProbeCommand(nameToResolve string, lookup string, fileNamePrefix string) (string, string) { fileName := fmt.Sprintf("%s_udp@%s", fileNamePrefix, nameToResolve) <del> probeCmd := fmt.Sprintf("dig +short +tries=12 %s %s > /results/%s", nameToResolve, lookup, fileName) </del> <ins> probeCmd := fmt.Sprintf("for i in `seq 1 30`; do dig +short %s %s > /results/%s; sleep 1; done", nameToResolve, lookup, fileName) </ins> return probeCmd, fileName }

There aren't any tests that cover ExternalPort services from Windows pods today. Here's an example manual case that can best explain what needs to be automated. a Windows pod on a Windows node this yaml This shows that DNS is working for the ExternalName service entry. The actual HTTP request fails (by design) because curl is putting the service name in the header which is rejected by the http server itself. /sig windows /good-first-issue
Can explain why you think http server is rejecting service name in header?
curl was using the URL given to set the hostname in the HTTP request. To illustrate what this does - here's another way to get the same result. Ask to give you the page for It can't serve that content so it returns a 404. Therefore, if I had run that would have succeeded.
/milestone v1.14
We could use the test ` for this. This test does: creates an initial ExternalName to and validates the DNS resolution. renames the ExternalName to and validates the new DNS resolution. changes the Service Type to ClusterIP and validates the new DNS resolution.
Is anyone working on this? I'd like to jump on it
sure, go for it. /assign
GitHub didn't allow me to assign the following users: keremgocen. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : sure, go for it. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What kind of modifications are required to run e2e in [Windows]. TestName: I'm in process to promote DNS based e2e from into Conformance suite.
HI I am also looking into that test, as it is a bit flaky on Windows. My recommendation would be to increase the retry period / number of retries for the dig command.
Thanks for pointing its Flaky behavior on Windows. I'm in process to setup k8s cluster in Windows env and then troubleshoot.
Ok, looks like the right things are happening. is approved. Once merges it should show up in testgrid for Windows too.

If doesn't make it, this can move to 1.15. I did manual testing (see steps above) and it's ok. Since there's a common test case that should work, we should rely on that instead of adding a Windows-only test. /remove-good-first-issue
/remove-help-wanted
/remove help-wanted


Question: mockCadvisor. On("ContainerInfoV2", "/", options).Return(infos, nil) fakeRuntimeService.SetFakeSandboxes([]*critest.FakePodSandbox{ <del> sandbox0, sandbox1, sandbox2, sandbox3, sandbox4, sandbox5, </del> <ins> sandbox0, sandbox1, sandbox2, sandbox3, sandbox4, sandbox5, sandbox6, </ins> }) fakeRuntimeService.SetFakeContainers([]*critest.FakeContainer{ <del> container0, container1, container2, container3, container4, container5, container6, container7, container8, </del> <ins> container0, container1, container2, container3, container4, container5, container6, container7, container8, container9, </ins> }) fakeRuntimeService.SetFakeContainerStats([]*runtimeapi.ContainerStats{ <del> containerStats0, containerStats1, containerStats2, containerStats3, containerStats4, containerStats5, containerStats6, containerStats7, containerStats8, </del> <ins> containerStats0, containerStats1, containerStats2, containerStats3, containerStats4, containerStats5, containerStats6, containerStats7, containerStats8, containerStats9, </ins> }) ephemeralVolumes := makeFakeVolumeStats([]string{"ephVolume1, ephVolumes2"})

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: CPU.Time is set to cs.StartTime but ps.Memory.Time is set to timestamp when collecting the memoryusage. What you expected to happen: CPU.Time is set to cs.CPU.Time How to reproduce it (as minimally and precisely as possible): Check the code Anything else we need to know?: Environment: Kubernetes version (use ): master branch when creating this issue. Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig kubelet
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I am happy to take over this. /assign


Question: return nil, err } if len(addrs) > 0 { <del> // This interface should suffice. break </del> <ins> for _, addr := range addrs { if addrIP, _, err := net.ParseCIDR(addr.String()); err == nil { if addrIP.To4() != nil { ip = addrIP.To4() break } } } if ip != nil { // This interface should suffice. break } </ins> } } } <del> if i == len(intfs) { return nil, err </del> <ins> if ip == nil { return nil, fmt.Errorf("no acceptable interface from host") </ins> } glog.V(4).Infof("Choosing interface %s for from-host portals", intfs[i].Name) <del> addrs, err := intfs[i].Addrs() if err != nil { return nil, err } glog.V(4).Infof("Interface %s = %s", intfs[i].Name, addrs[0].String()) ip, _, err := net.ParseCIDR(addrs[0].String()) if err != nil { return nil, err } </del> return ip, nil }

The master sets its endpoints into the 'kubernetes' service, but endpoints can only be IPv4 today. So ChooseHostInterface() should either not return an IPv6 address, or we should allow endpoints to be IPv6.
I think that this is likely the cause of .
I thought that function explicitly checked for v4. got any bandwidth for review? On May 4, 2015 11:24 AM, "Robert Bailey" wrote:
It doesn't - and on my mac, it fails to find this address when I add that logic. Not sure if that's FlagPointToPoint or something: ----- Original Message -----
PEBKAC for the issue I mentioned above, is opened to fix the choice.
Yeah, I can take a look at later this afternoon, just as soon as I've unbroken ELB's :-) smarterclayton@ let me know if is more urgent than that.
was smaller than I thought, so committed priority inversion I reviewed it :-)


Question: func (r *Request) transformResponse(resp *http.Response, req *http.Request) Result { var body []byte if resp.Body != nil { <del> if data, err := ioutil.ReadAll(resp.Body); err == nil { </del> <ins> data, err := ioutil.ReadAll(resp.Body) switch err.(type) { case nil: </ins> body = data <ins> case http2.StreamError: // This is trying to catch the scenario that the server may close the connection when sending the // response body. This can be caused by server timeout due to a slow network connection. // TODO: Add test for this. Steps may be: // 1. client-go (or kubectl) sends a GET request. // 2. Apiserver sends back the headers and then part of the body // 3. Apiserver closes connection. // 4. client-go should catch this and return an error. glog.V(2).Infof("Stream error %#v when reading response body, may be caused by closed connection.", err) streamErr := fmt.Errorf("Stream error %#v when reading response body, may be caused by closed connection. Please retry.", err) return Result{ err: streamErr, } default: glog.Errorf("Unexpected error when reading response body: %#v", err) unexpectedErr := fmt.Errorf("Unexpected error %#v when reading response body. Please retry.", err) return Result{ err: unexpectedErr, } </ins> } }

While this issue happens without this complicated reproduction, here is a sure-fire way to get it. a long-running kubectl get. In this example, I have a cluster with about 2,000 configmaps each around 100KiB. Once you see these lines Press , then type , and then you'll see... What's happening is that the apiserver is hitting , and then it . As the code says, this should make it obvious to the client (kubectl) that it didn't get the entire body, because no 0-length EOF chunk gets sent. I can confirm from using tcpdump that no such chunk gets sent. However, kubectl doesn't notice this, and instead tries to decode a 0-length Response body, giving the cryptic message. What I expected to happen the very least, a less cryptic error message like "connected closed". , this gets detected by the code and we can internally retry, just as if we got, e.g.,
cc Possibly seen here, too: openshift/origin
/assign mengqiy
Created a PR to detect it. Further improvement may be retrying when detect connection closure
closing this since you filed a PR for it


Question: import ( "encoding/json" <ins> "hash/crc32" </ins> <del> "k8s.io/kubernetes/pkg/kubelet/checkpointmanager" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager/checksum" </del> <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager/errors" </ins> ) <del> var _ checkpointmanager.Checkpoint = &DRAManagerCheckpoint{} </del> <ins> const ( CheckpointAPIGroup   = "checkpoint.dra.kubelet.k8s.io" CheckpointKind       = "DRACheckpoint" CheckpointAPIVersion = CheckpointAPIGroup + "/v1" ) </ins> <del> const checkpointVersion = "v1" </del> <ins> // Checkpoint represents a structure to store DRA checkpoint data type Checkpoint struct { // Data is a JSON serialized checkpoint data Data string // Checksum is a checksum of Data Checksum uint32 } </ins> <del> // DRAManagerCheckpoint struct is used to store pod dynamic resources assignments in a checkpoint type DRAManagerCheckpoint struct { Version  string             `json:"version"` Entries  ClaimInfoStateList `json:"entries,omitempty"` Checksum checksum.Checksum  `json:"checksum"` </del> <ins> type CheckpointData struct { metav1.TypeMeta ClaimInfoStateList ClaimInfoStateList </ins> } <del> // List of claim info to store in checkpoint type ClaimInfoStateList []ClaimInfoState </del> <ins> // NewCheckpoint creates a new checkpoint from a list of claim info states func NewCheckpoint(data ClaimInfoStateList) (*Checkpoint, error) { cpData := &CheckpointData{ TypeMeta: metav1.TypeMeta{ Kind:       CheckpointKind, APIVersion: CheckpointAPIVersion, }, ClaimInfoStateList: data, } cpDataBytes, err := json.Marshal(cpData) if err != nil { return nil, err } </ins> <del> // NewDRAManagerCheckpoint returns an instance of Checkpoint func NewDRAManagerCheckpoint() *DRAManagerCheckpoint { return &DRAManagerCheckpoint{ Version: checkpointVersion, Entries: ClaimInfoStateList{}, </del> <ins> cp := &Checkpoint{ Data:     string(cpDataBytes), Checksum: crc32.ChecksumIEEE(cpDataBytes), </ins> } <ins> return cp, nil </ins> } <del> // MarshalCheckpoint returns marshalled checkpoint func (dc *DRAManagerCheckpoint) MarshalCheckpoint() ([]byte, error) { // make sure checksum wasn't set before so it doesn't affect output checksum dc.Checksum = 0 dc.Checksum = checksum.New(dc) return json.Marshal(*dc) </del> <ins> // MarshalCheckpoint marshals checkpoint to JSON func (cp *Checkpoint) MarshalCheckpoint() ([]byte, error) { return json.Marshal(cp) </ins> } <del> // UnmarshalCheckpoint tries to unmarshal passed bytes to checkpoint func (dc *DRAManagerCheckpoint) UnmarshalCheckpoint(blob []byte) error { return json.Unmarshal(blob, dc) </del> <ins> // UnmarshalCheckpoint unmarshals checkpoint from JSON // and verifies its data checksum func (cp *Checkpoint) UnmarshalCheckpoint(blob []byte) error { if err := json.Unmarshal(blob, cp); err != nil { return err } // verify checksum if err := cp.VerifyChecksum(); err != nil { return err } return nil </ins> } <del> // VerifyChecksum verifies that current checksum of checkpoint is valid func (dc *DRAManagerCheckpoint) VerifyChecksum() error { ck := dc.Checksum dc.Checksum = 0 err := ck.Verify(dc) dc.Checksum = ck return err </del> <ins> // VerifyChecksum verifies that current checksum // of checkpointed Data is valid func (cp *Checkpoint) VerifyChecksum() error { expectedCS := crc32.ChecksumIEEE([]byte(cp.Data)) if expectedCS != cp.Checksum { return &errors.CorruptCheckpointError{ActualCS: uint64(cp.Checksum), ExpectedCS: uint64(expectedCS)} } return nil } // GetClaimInfoStateList returns list of claim info states from checkpoint func (cp *Checkpoint) GetClaimInfoStateList() (ClaimInfoStateList, error) { var data CheckpointData if err := json.Unmarshal([]byte(cp.Data), &data); err != nil { return nil, err } return data.ClaimInfoStateList, nil </ins> }

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: return "", errors.Wrap(err, "invalid format to parse for proxy") } if len(proxyIP) == 0 { <del> break </del> <ins> continue </ins> } pStanza := map[string]interface{}{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The following unit test case is flakey What you expected to happen: Testcase should pass. How to reproduce it (as minimally and precisely as possible): Run the testcase for 20 times, you will see failures couple of times. Test failure history: https://k8s-
/sig cluster-lifecycle
/assign
This is happening because in when there is empty proxyIP we are not continuing the pool instead we are breaking which will exit from the loop and ignore the next element in the dictionary of stubDomainData dictionary where there is not guaranty of execution. To fix this issue should be replaced with continue


Question: oldJobSchedule: "30 * * * *", newJobSchedule: "*/1 * * * *", }, <del> deltaTimeForQueue:    1*time.Second + nextScheduleDelta, roundOffTimeDuration: 750 * time.Millisecond, </del> <ins> deltaTimeForQueue: 1*time.Second + nextScheduleDelta, </ins> }, // TODO: Add more test cases for updating scheduling. }

pull-kubernetes-unit TestController2updateCronJob/spec.schedulechanged TestController2_updateCronJob /sig apps /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is important to resolve before promotion to v1
looks like the cause was
This function should be related.
Another one:
When the flake happen, there are messages like When the system is overloaded, it would happen. is trying to replace the current queue in tests with a mock.
Yeah, I spoke with to make sure we use a mock of the queue so we only get the most important bit of information ignoring the remaining parts


Question: glog.Infof("[cpumanager] detected CPU topology: %v", topo) reservedCPUs, ok := nodeAllocatableReservation[v1.ResourceCPU] if !ok { <del> // The static policy cannot initialize without this information. Panic! panic("[cpumanager] unable to determine reserved CPU resources for static policy") </del> <ins> // The static policy cannot initialize without this information. return nil, fmt.Errorf("[cpumanager] unable to determine reserved CPU resources for static policy") </ins> } if reservedCPUs.IsZero() { <del> // Panic! // </del> // The static policy requires this to be nonzero. Zero CPU reservation // would allow the shared pool to be completely exhausted. At that point // either we would violate our guarantee of exclusivity or need to evict // any pod that has at least one container that requires zero CPUs. // See the comments in policy_static.go for more details. <del> panic("[cpumanager] the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero") </del> <ins> return nil, fmt.Errorf("[cpumanager] the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero") </ins> } // Take the ceiling of the reservation, since fractional CPUs cannot be

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?


Question: // this error since all subsequent creations will fail. return } <del> if errors.IsTimeout(err) { // Pod is created but its initialization has timed out. // If the initialization is successful eventually, the // controller will observe the creation via the informer. // If the initialization fails, or if the pod keeps // uninitialized for a long time, the informer will not // receive any update, and the controller will create a new // pod when the expectation expires. return } </del> } if err != nil { klog.V(2).Infof("Failed creation, decrementing expectations for set %q/%q", ds.Namespace, ds.Name)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In some case, controller manager failed to create pod as replicas number defined in ReplicaSet spec. Normally if replicaset controller fails to create a pod, it will try later. () But func in ignore timeout error for some reason(), then if a create pod request really timeout during accessing apiserver, this failed will not add to the queue again. Then the pod should be created will never be created again. () I saw this happened in a production environment. What you expected to happen: Controller manager will retry and create pod again. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7 Kernel (e.g. ): Install tools: Others: /cc
/sig api-machinery
/sig apps /remove-sig api-machinery
In : The controllers ignore the timeout error when creating the pods, depending on how the initialization progress: the initialization is successful in less than 5 mins, the controller will observe the creation via the informer. All is good. the initialization fails, server will delete the pod, but the controller won't receive any event. The controller will not create new pod until the Creation expectation expires in 5 min. the initialization takes too long (5 mins), the Creation expectation expires and the controller will create extra pods. Note 2,3 that if the creation request failed with timeout, the ReplicaSetController will resync: And the expectation creation has already been in if diff < 0:
I got the similar situation. After getting type: 'Warning' reason: 'FailedCreate' Error creating: Timeout: request did not complete within allowed duration, Pod is not created. Its ReplicaSet has .spec.replicas = 0, but .status.replicas stays 0 without any reason. k8s version is 1.11.0.
hello , , i am thinking the code in causes the inconsistency above.

it will only RE-enqueue the RS when ReadyReplicas == Replicas && AvailableReplicas != Replicas, while there are situations that ReadyReplicas != Replicas.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Also seeing this in 1.12 on GKE (1.12.8-gke.6).
Just got this now on AWS (v1.12.6-eks-): Using Istio 1.1.7, if that helps at all. I updated some nodes and switched to a new node type, and whilst it was supposed to be bringing these pods back up, it just never did. It's only in our staging environment right now at least!
Ooooh, we're also running Istio. (And I just saw the issue again in a different GKE cluster.) With the automatic sidecar-injection. Maybe an interplay problem with that admission controller.
Hitting the same issue on EKS 1.13.7, and we are using Istio as well (1.2.0). We ran into the issue while doing a rolling-upgrade of the cluster, with a lot of pod churn, so I assume that the above hypothesis around admission-controller / sidecar injection might be correct. :) Edit: we're trying to mitigate this by making the sidecar injector HA; by default it is running a single instance.
did scaling the sidecar injector work? I'm still hitting this issue with Istio 1.2.2, and it's really making me nervous about upgrading the production server. It's consistently not bringing pods back.

it seems to have mitigated the problem for us, but I'm not sure we've done another full rolling-update since my previous message. In any case, scaling up the injector can't hurt
Happens on my cluster two, but we're not using Istio. Message is following, with no additional clue : One thing to note is that it happened after pods were Evicted, and OutofEphemeral condition. Disk was full during the eviction.
I actually found the issue I've been having "with Istio". We're using CloudFormation to create the nodes in AWS (almost the default CloudFormation template given by AWS). The problem with this is that it doesn't drain the nodes before killing them, so even if you replace nodes in batches, you're still killing pods. If you kill the Istio pods, it can then prevent other pods from starting up (giving you the aforementioned error). From that point I think some of the fault lies with Kubernetes too, because there are times when a node has been terminated, and I can go look at a ReplicaSet and it'll think it's got the right amount of pods running, then I can go look at pods, and there'll be nothing there for that service. I think this prevents Istio from recovering as quickly as it could. Istio then seems to be very picky about the order that services start in, so it takes quite a bit of time for Istio to recovery even once the Istio pods start coming back. So what I've done now is labels to the nodes (e.g. or ), and then when I want to upgrade the nodes, I create an entire new node group with the opposite deployment label. Then using that label I taint and drain the old nodes. At this point you may need to scale up some services to allow nodes to be drained because of PodDisruptionBudgets. Once the nodes are drained, I destroy the CloudFormation stack for the old node (alternatively I suppose you could set the desired number of nodes to 0). Luckily this isn't something that really needs to happen very often.

Also, I'm pretty sure using would ease some of this, but we're using Terraform for our infrastructure, and I'd rather keep it all in once place... Next step for me is trying to figure out how to replace nodes in batches, because ideally I don't want twice as many nodes as I need running at one point. I might hit some limits in AWS for certain node types.
I think I ran into this situation as well. It was caused by a similar situation as described by . A change to a Terraform module resulted in ASG replacement. The change was supposed to be deployed by hand on each ASG individually, which though not as graceful as using taints to drain the nodes in the old ASG, would have probably allowed Kubernetes to reschedule the unexpectedly terminated pods without too much difficulty. Instead, the change was inadvertently applied to a large number of ASGs all at roughly the same time. This caused many ReplicaSets to report 0 current and 0 ready pods, and to have the "Error creating: the server was unable to return a response in the time allotted, but may still be processing the request (post pods)" event. Luckily, manually scaling each ReplicaSet to 0 and then back to the original scale gave Kubernetes the kick it needed to create the pods.
I think I am hitting this bug, too. My situation is that I have an validating webhook admission controller attached to the namespace. If the admission controller goes offline, the replica set for regular Deployments on that namespace will not be able to create pods as they can't get past the admission controller. They list 'Timeout: request did not complete within allowed duration' errors, and then never try again to create the pod. Once the admission controller is back online. These replica sets just sit there. The only way to get the Deployment to deploy pods again is to delete its replica set. Then, it will make a new replica set and that one will happily deploy the pods. In particular, this is problematic if you have some sort of moderately widespread worker node outage. If a bunch/all of workers go offline at once, you could lose your admission controllers temporarily and then get into a situation where nothing comes back due to stuck replica sets.
Got the same situation in EKS (we have a nice monitoring alert that matches events containing ), server version .

Fixed using the commands: Delete stuck replicasets, Kubernetes creates another instead and if you don't have any other problems (like resource quotas, for example), they will start new pods.
Observed on GKE 1.13.6-gke.13 or 1.13.7-gke.24, as it happened during an upgrade. Lots of FailedCreate events. We got scheduling back on track by scaling deployments to zero replicas and then back to the intended number.
Found the following in ReplicaSetController#manageReplicas :
I am hitting this bug, too. If ReplicaSetController create pod request really timeout during accessing apiserver, this failed key will not add to the queue again. Then the pod should be created will never be created again. in manageReplicas, if create pod timeout < 5 minute, the queue has no key, the controller never sync when no key , so it never create a new pod even through the expectation expires. rs status kubectl get rs --all-namespaces NAMESPACE NAME DESIRED CURRENT READY AGE kube-system kubenab- 3 2 2 5d9h kube-controller-manager log: I1204 04:13:23. 1 ] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kubenab-", UID:"-b142-4db2-a914-", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: the server was unable to return a response in the time allotted, but may still be processing the request (post pods) kube-apiserver log: W1204 04:12:53. 1 ] Failed calling webhook, failing open kubenab- failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:12:53. 1 ] failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:13:23. 1 ] apiserver was unable to write a JSON response: http: Handler timeout E1204 04:13:23. 1 ] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"} I1204 04:13:23.

1 ] Trace[]: "Create /api/v1/namespaces/kube-system/pods" (started: 2019-12-04 04:12:23. +0000 UTC m=+.) (total time: 1m0.011414107s): Trace[]: [30.010012786s] [30.00986498s] About to store object in database Trace[]: [1m0.011414107s] [30.001401321s] END W1204 04:13:23. 1 ] Failed calling webhook, failing open kubenab- failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers) E1204 04:13:23. 1 ] failed calling webhook "kubenab-": Post context deadline exceeded (Client.Timeout exceeded while awaiting headers)
/assign /assign /assign It appears that the timeout code here: may be out-of-date now that initializers have been removed. The solution may be to just remove the code swallowing the timeout.


Question: }, }, { <ins> name:               "downward api pod ips reverse order", ns:                 "downward-api", enableServiceLinks: &falseValue, container: &v1.Container{ Env: []v1.EnvVar{ { Name: "POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, { Name: "POD_IPS", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIPs", }, }, }, { Name: "HOST_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.hostIP", }, }, }, }, }, podIPs:          []string{"fd00::6", "1.2.3.4"}, masterServiceNs: "nothing", nilLister:       true, expectedEnvs: []kubecontainer.EnvVar{ {Name: "POD_IP", Value: "1.2.3.4"}, {Name: "POD_IPS", Value: "1.2.3.4,fd00::6"}, {Name: "HOST_IP", Value: testKubeletHostIP}, }, }, { name:               "downward api pod ips multiple ips", ns:                 "downward-api", enableServiceLinks: &falseValue, container: &v1.Container{ Env: []v1.EnvVar{ { Name: "POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, { Name: "POD_IPS", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIPs", }, }, }, { Name: "HOST_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.hostIP", }, }, }, }, }, podIPs:          []string{"1.2.3.4", "192.168.1.1.", "fd00::6"}, masterServiceNs: "nothing", nilLister:       true, expectedEnvs: []kubecontainer.EnvVar{ {Name: "POD_IP", Value: "1.2.3.4"}, {Name: "POD_IPS", Value: "1.2.3.4,fd00::6"}, {Name: "HOST_IP", Value: testKubeletHostIP}, }, }, { </ins> name:               "env expansion", ns:                 "test1", enableServiceLinks: &falseValue,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: # Golang - name: "golang: upstream version" <del> version: 1.22 </del> <ins> version: 1.22.0 </ins> refPaths: - path: .go-version - path: build/build-image/cross/VERSION

https://prow.ppc64le- Tests are not getting executed due to improper go version. It has been failing since past 4 days. No response No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to be failing due to improper version addition in dependency files where is instead of at some places. I will raise a PR for it.
/assign
fixing that


Question: func main() { command := newCommand() if err := command.Execute(); err != nil { <del> fmt.Fprintf(os.Stderr, "error: %vn", err) </del> os.Exit(1) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl version --client=yes command is giving the following error message. Error message is repeated at the start and the end and also containing programming jargon Error: invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax What you expected to happen: The error message can be improved with one of the two variants invalid argument "yes" for "--client" flag Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). (or) Invalid argument "yes" for "--client" flag How to reproduce it (as minimally and precisely as possible): Run the command kubectl version --client=yes Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): Amazon Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Hi Thanks for posting this issue! It turns out that this duplicate output has been in kubectl, and all the other cli cmds, since . This should be fixed by , and hopefully we can cherry pick it back to at least .


Question: // How long a pod is allowed to become "running" and "ready" after a node // restart before test is considered failed. RestartPodReadyAgainTimeout = 5 * time.Minute <ins> // Number of times we want to retry Updates in case of conflict UpdateRetries = 5 </ins> ) var (

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance] {Kubernetes e2e suite}


Question: } func getConformanceData(targetFrame frame) (*ConformanceData, error) { <del> // TODO: This is terribly hacky.  Why can't we rely on the --source flag in // all cases? https://issues.k8s.io/123567 // // Filenames may be in a couple special places, depending on if they were // built dockerized or with the host go.  We want to trim this prefix to // produce portable relative paths. containerGopath := "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/" </del> root := *k8sPath if !strings.HasSuffix(root, string(os.PathSeparator)) { root += string(os.PathSeparator) } trimmedFile := strings.TrimPrefix(targetFrame.File, root) <del> trimmedFile = strings.TrimPrefix(trimmedFile, containerGopath) </del> targetFrame.File = trimmedFile freader, err := os.Open(targetFrame.File)

has code which takes a flag called but it tries to hunt for the file in both "local" and "dockerized" places. This doesn't feel right. /cc /cc
/triage accepted


Question: ) func TestEstablishMasterConnection(t *testing.T) { <ins> srv := stubServer(t) defer srv.Close() tests := []struct { c      string e      string expect bool }{ { c:      "", e:      "", expect: false, }, { c:      "", e:      srv.URL, expect: true, }, { c:      "foo", e:      srv.URL, expect: true, }, } for _, rt := range tests { s := &kubeadmapi.TokenDiscovery{} c := &kubeadmapi.ClusterInfo{Endpoints: []string{rt.e}, CertificateAuthorities: []string{rt.c}} _, actual := EstablishMasterConnection(s, c) if (actual == nil) != rt.expect { t.Errorf( "failed EstablishMasterConnection:ntexpected: %tnt  actual: %t", rt.expect, (actual == nil), ) } } } func TestEstablishMasterConnectionWithMultipleEndpoints(t *testing.T) { // ref. https://github.com/kubernetes/kubernetes/issues/36988 srv := stubServer(t) defer srv.Close() s := &kubeadmapi.TokenDiscovery{} c := &kubeadmapi.ClusterInfo{Endpoints: []string{srv.URL, srv.URL}, CertificateAuthorities: []string{"foo"}} defer func() { if r := recover(); r != nil { t.Errorf("failed EstablishMasterConnectionWithMultipleEndpoints; got a panic.") } }() EstablishMasterConnection(s, c) } func stubServer(t *testing.T) *httptest.Server { </ins> expect := version.Info{ Major:     "foo", Minor:     "bar",

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubeadm join kubeadm join crash kubeadm close of closed channel Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} kubeadm version (use ): kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Kernel (e.g.

): 3.10.0-327.36.1.el7.x8664 SMP Sun Sep 18 13:04:29 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: yum Others: What happened: kubeadm crash when join mulitple API endpoints What you expected to happen: How to reproduce it (as minimally and precisely as possible): join mulitple API endpoints Anything else do we need to know: kubeadm output:
It is already fixed by
seems not fixed, still having this issue with master code
I'm also running into this with a clean build from master kubeadm version kubeadm output kubectl version Environment kernel versionOS version
seems not fixed. I met the same issue . kubeadm version: kernel version: os version: kubectl version: error output by "kubeadm join --token=. 10.47.217.91":
I can also repro this issue - I will work on contributing a fix.
I can also reproduce this issue, following the guide with packaged versions of all the components


Question: return pod } <del> // Patch client pod with a map value func updateNetworkClientPodLabel(f *framework.Framework, namespace *v1.Namespace, podName string, patchOperation string, patchPath string, patchValue map[string]string) *v1.Pod { </del> <ins> // Patch pod with a map value func updatePodLabel(f *framework.Framework, namespace *v1.Namespace, podName string, patchOperation string, patchPath string, patchValue map[string]string) *v1.Pod { </ins> type patchMapValue struct { Op    string            `json:"op"` Path  string            `json:"path"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The NetworkPolicy e2e test fails regardless of which CNI plugin is enabled. The test assumes the client Pod can recheck connectivity after updating its label, but the client won't restart after the first failure, so the second check will always fail. What you expected to happen: The test should succeed if a CNI plugin implements NetworkPolicy correctly. How to reproduce it (as minimally and precisely as possible): a CNI that supports NetworkPolicy. the e2e test by focusing "should allow ingress access from updated pod". test will always fail. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I see this in Calico 3.10 as well confirming that its definitely an issue.


Question: } } if !found { <del> log.Fatalf("%v dependency: file %v doesn't contain the expected version %v or matcher %v did change. check the %v file", dep.Name, refPath.Path, dep.Version, refPath.Match, externalDepsFilePath) </del> <ins> pathsToUpdate = append(pathsToUpdate, refPath.Path) </ins> } } <ins> if len(pathsToUpdate) > 0 { log.Fatalf(mismatchErrorMessage, externalDepsFilePath, dep.Name, dep.Version, strings.Join(pathsToUpdate, "n"), dep.Name, externalDepsFilePath) } </ins> } }

What would you like to be : The current error message is vague and could be improved. (credit to Why is this needed: Improve contributor experience, make it easier to identify change to be fixed. /cc
/assign


Question: "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" "k8s.io/klog/v2" <ins> "k8s.io/kubernetes/pkg/kubelet/kuberuntime" </ins> "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"

Also appears to be failing on: Failures as of 01/05. /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon Guessing since all tests with apparmor started failing - maybe app armor config was removed or some PR submitted recently.
/assign
ci-cri-containerd-node-e2e-features Last green run: First failed run:
This () is the only PR merged in kubernetes during the time between: Test started yesterday at 8:11 PM passed after 13m50s. (more info) Test started yesterday at 9:11 PM failed after 16m6s. (more info) And it happens to be related to AppArmor:
Working on a fix in


Question: // NewConverter creates a new Converter object. func NewConverter() *Converter { <del> return &Converter{ </del> <ins> c := &Converter{ </ins> conversionFuncs:    map[typePair]reflect.Value{}, defaultingFuncs:    map[reflect.Type]reflect.Value{}, nameFunc:           func(t reflect.Type) string { return t.Name() },

E.g., it calls Convert() for every byte of a slice of uint8. E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting []v1.NamedExtension E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting []uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23.

5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 ... thousands of these.
There's an easy fix for this :/


Question: fi } <ins> # outputs md5 hash of $1, works on macOS and Linux function kube::util::md5() { if which md5 >/dev/null 2>&1; then md5 -q "$1" else md5sum "$1" | awk '{ print $1 }' fi } </ins> # kube::util::read-array # Reads in stdin and adds it line by line to the array provided. This can be # used instead of "mapfile -t", and is bash 3 compatible.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: run on macOS, all md5 hashes will be stripped from the output and the script will not fail or warn. What you expected to happen: The script should work or detect this issue and fail. Currently it silently ignores not existing on the host. We have some warning logic, but solely based on bash version, if the user upgraded their bash installation but did not install md5sum we don't warn them. How to reproduce it (as minimally and precisely as possible): run the script Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig testing /priority important-longterm /assign
cc


Question: "github.com/stretchr/testify/require" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/klog" registerapi "k8s.io/kubernetes/pkg/kubelet/apis/pluginregistration/v1" )

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: return httptest.NewServer(mux), &fakeEndpointsHandler } <ins> // makeBlockingEndpointDeleteTestServer will signal the blockNextAction channel on endpoint "POST" & "DELETE" requests. All // block endpoint "DELETE" requestsi will wait on a blockDelete signal to delete endpoint. If controller is nil, a error will // be sent in the response. func makeBlockingEndpointDeleteTestServer(t *testing.T, controller *endpointController, endpoint *v1.Endpoints, blockDelete, blockNextAction chan struct{}, namespace string) *httptest.Server { handlerFunc := func(res http.ResponseWriter, req *http.Request) { if controller == nil { res.WriteHeader(http.StatusInternalServerError) res.Write([]byte("controller has not been set yet")) return } if req.Method == "POST" { controller.endpointsStore.Add(endpoint) blockNextAction <- struct{}{} } if req.Method == "DELETE" { go func() { // Delay the deletion of endoints to make endpoint cache out of sync <-blockDelete controller.endpointsStore.Delete(endpoint) controller.onEndpointsDelete(endpoint) }() blockNextAction <- struct{}{} } res.WriteHeader(http.StatusOK) res.Write([]byte(runtime.EncodeOrDie(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), &v1.Endpoints{}))) } mux := http.NewServeMux() mux.HandleFunc("/api/v1/namespaces/"+namespace+"/endpoints", handlerFunc) mux.HandleFunc("/api/v1/namespaces/"+namespace+"/endpoints/", handlerFunc) mux.HandleFunc("/api/v1/namespaces/"+namespace+"/events", func(res http.ResponseWriter, req *http.Request) {}) mux.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) { t.Errorf("unexpected request: %v", req.RequestURI) http.Error(res, "", http.StatusNotFound) }) return httptest.NewServer(mux) } </ins> type endpointController struct { *EndpointController podStore       cache.Store

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? no. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): service endpoints missing Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: manual Others: What happened: Service in namespace sees no endpoints that are available What you expected to happen: For it to see available endpoints How to reproduce it (as minimally and precisely as possible): Not sure I can. It appeared on the latest remove/replace of worker nodes, but was fine before Anything else we need to know: I have a traefik Ingress controller defined and a service for it: Selectors match, the pods are running just fine, yet the endpoints do not show up. I suspect if I delete the service and recreate it, it probably will pick it up, but I shouldn't have to, and am concerned it might occur again; rather actually debug it. How can I check what kube is doing to affiliate it? How often does it check? Where?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
The system is asking for a SIG, but it isn't clear which SIG is appropriate for this. Taking best guess, apologies if I am wrong.

/sig sig/apps /sig sig/scheduling
/sig network
can you provide the following output?
I wish I could... in the week since, the cluster has been deleted and recreated several times. AWS instances may be cheap, but still aren't free. Client didn't want them sticking around too long for the cost.
Should we close this issue pending more information or keep it open for someone to test out?
we are killing and recreating clusters with some regularity over these weeks. I am happy if you want to close it since I cannot recreate it yet. Just let me reopen it if I manage to recreate and get more info. Just tell me, please, if it does come up again, what additional data should I get? There isn't much data in the docs on how the service resolves what endpoints it should use and how to figure out why it is misbehaving when it does. Happy to take pointers.
Please reopen when it happens again. Please record related endpoints objects and pod objects.
I'm facing same issue with Nginx by trying to expose the nodePort for Nginx-ingress-controller pod
Try .
Your service is using while your pods are using .
Sorry, didn't notice that
OS: ubuntu 16.04 kubernetes version: v1.8.4 hi, I have the same problem now. $ kc get endpoints -njenkins This situation has been going on for a long time and no solution has been found. Help out. Thanks
Same the problem, svc lost the endpoint after a few time. If deleting the pod, then the svc endpoint is show up again. How to debug this problem?
Im having the same problem on Windows Nodes, after some hours all svc on one node loses their endpoints. Kubernetes V1.15
May be fixed by this?

same issue for a custom helm that I created: kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T10:52:43Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-03T04:12:37Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} kubectl describe service/vnfdb-ui Name: vnfdb-ui Namespace: test-helm Labels: Annotations: <noneSelector: , Type: LoadBalancer IP: 10.105.168.158 Port: http 80/TCP TargetPort: http/TCP NodePort: http /TCP Endpoints: <noneSession Affinity: None External Traffic Policy: Cluster Events: <none$ kubectl get pods -l , NAME READY STATUS RESTARTS AGE vnfdb-ui--ttrwm 1/1 Running 0 9m $ kubectl get endpoints vnfdb-ui NAME ENDPOINTS AGE vnfdb-ui <none9m33s $ kubectl get endpoints vnfdb-ui -o yaml apiVersion: v1 kind: Endpoints metadata: creationTimestamp: "2019-08-24T21:28:58Z" labels: vnfdb Tiller ui "1.0" ui-0.1.0 name: vnfdb-ui namespace: test-helm resourceVersion: "" selfLink: /api/v1/namespaces/test-helm/endpoints/vnfdb-ui uid: -c6b6-11e9-a0d5- Any clue what is wrong here and how can I fix it?
Got it, apparently realized that its not only the selector that has to match labels. The port names should be the same in the pod definition and services definition as well. The latter was the culprit in my case.
Same the problem.My service and pod is running. But the service do not has endpoint,and this situation lasts for 10 hours. It is OK after I restart controller manager.

My last operation was to delete the service and create the service again. I have not done any other operations since then. Kubernetes V1.11.5
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Good catch!
Happens to us once in a while: We have a process creating a service, then a test deleting that same service, which ends up being automatically recreated. Once in a while (every ~100 test), the re-created service never gets any endpoints, even after waiting for several hours. I'm fairly confident recreating the service again fixes it, but that's not a solution. It just happened on a GKE cluster in version , for which I don't have controller-manager logs available unfortunately. I'll try to reach GCP support, they may be able to troubleshoot more from the controller-manager logs. Edit: the endpoint magically appeared after 22 hours.
Thanks for the detailed bug report! I'll try to recreate this scenario with some new tests. /assign
A quick follow up to let you know we haven't forgotten about this issue. From what we can tell, this is being caused by an Endpoints informer cache that can potentially be out of date when syncService() runs. Essentially syncService() would be called twice - once for the delete and once for the create event.

There's a chance that when the second run of syncService() happens the Endpoints cache may not have been updated with the latest run. If that happens, syncService() will not make any changes because it sees an endpoints resource that has all the right ports and addresses set. is working on a fix here that will mirror what we've done for the EndpointSlice controller with the . This will involve watching Endpoints changes and calling syncService() if the change has not already been accounted for by the controller. As far as I can tell, this should be a relatively rare bug. It should only happen if the cache is out of date, the recreated Service is identical, and the Pods have not changed at all. Given the relative complexity of the fix here, I'm not sure how far we'll be able to backport this. At the very least, we're hoping to have a fix in for 1.19.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think is almost ready, sorry for not catching the lifecycle close earlier.
Dude, you saved my life.


Question: return url.String() } <del> // updateURLs checks and updates any of n's attributes that are listed in tagsToAttrs. // Any URLs found are, if they share the source host, updated with the necessary changes // to make a visit to that URL also go through the proxy. // sourceURL is the URL of the page which we're currently on. func (t *Transport) updateURLs(n *html.Node, sourceURL *url.URL) { if n.Type != html.ElementNode { return } attrs, ok := tagsToAttrs[n.Data] if !ok { return } for i, attr := range n.Attr { if !attrs.Has(attr.Key) { continue </del> <ins> // rewriteHTML scans the HTML for tags with url-valued attributes, and updates // those values with the urlRewriter function. The updated HTML is output to the // writer. func rewriteHTML(reader io.Reader, writer io.Writer, urlRewriter func(string) string) error { // Note: This assumes the content is UTF-8. tokenizer := html.NewTokenizer(reader) var err error for err == nil { tokenType := tokenizer.Next() switch tokenType { case html.ErrorToken: err = tokenizer.Err() case html.StartTagToken, html.SelfClosingTagToken: token := tokenizer.Token() if urlAttrs, ok := atomsToAttrs[token.DataAtom]; ok { for i, attr := range token.Attr { if urlAttrs.Has(attr.Key) { token.Attr[i].Val = urlRewriter(attr.Val) } } } _, err = writer.Write([]byte(token.String())) default: _, err = writer.Write(tokenizer.Raw()) </ins> } <del> n.Attr[i].Val = t.rewriteURL(attr.Val, sourceURL) </del> } <del> } // scan recursively calls f for every n and every subnode of n. func (t *Transport) scan(n *html.Node, f func(*html.Node)) { f(n) for c := n.FirstChild; c != nil; c = c.NextSibling { t.scan(c, f) </del> <ins> if err != io.EOF { return err </ins> } <ins> return nil </ins> } <del> // fixLinks modifies links in an HTML file such that they will be redirected through the proxy if needed. func (t *Transport) fixLinks(req *http.Request, resp *http.Response) (*http.Response, error) { </del> <ins> // rewriteResponse modifies an HTML response by updating absolute links refering // to the original host to instead refer to the proxy transport. func (t *Transport) rewriteResponse(req *http.Request, resp *http.Response) (*http.Response, error) { </ins> origBody := resp.Body defer origBody.Close()

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: - 1 - 2 - 3 <ins> - description: sort nested pointers of ints original: mergeItemPtr: - name: 2 mergingIntList: - 1 - 3 - 2 - name: 1 mergingIntList: - 2 - 1 sorted: mergeItemPtr: - name: 1 mergingIntList: - 1 - 2 - name: 2 mergingIntList: - 1 - 2 - 3 - description: sort one pointer of maps original: mergeItemPtr: - name: 1 - name: 3 - name: 2 sorted: mergeItemPtr: - name: 1 - name: 2 - name: 3 </ins> `) func TestStrategicMergePatch(t *testing.T) {

PATCH http://IP:8080/api/v1beta3/namespaces/dev--ea6f-49cb-b5bb-/replicationcontrollers/redis-master {"spec":{"replicas":0}} answer: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server responded with the status code 415 but did not return more information", "details": {}, "code": 415 } Before v0.15.0 it had worked well.
That error is returned by go-restful because it did not find any valid route with the same content type as specified in the request: Can you try setting the content-type in your request as is described here: cc
well, I've the same issue which is not fixed by setting the value, at least with the "strategic merge patch". Here's the request: Here's the response body: And here's the log trace of the request in the apiserver pod:
Come from a typo, purposed PR here:
Btw, I'm still having strange issues with sub-objects. Note that it appears only is the sub-object is already set. Here's the request: Here's the API log:
Looks like a bug in strategic merge -- it doesn't deal with pointers correctly:
cc
is it possible to have a milestone for this so I can tell if I should merger ? Thanks.


Question: }}, }}, RestartPolicy: v1.RestartPolicyNever, <del> }}, metav1.CreateOptions{}) </del> <ins> }} _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), pod, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err, "failed to create pod") <del> err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodStatus(f, "test=response")) framework.ExpectNoError(err, "Pod didn't start within time out period") </del> <ins> framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(f.ClientSet, pod), "Pod didn't start within time out period") </ins> framework.Logf("Creating service...") _, err = f.ClientSet.CoreV1().Services(ns).Create(context.TODO(), &v1.Service{

pull-kubernetes-e2e-kind-ipv6 - Not sure. Time out on waiting Pod to be started. No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//staging/src/k8s.io/apimachinery/pkg/util/diff:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/validation/field:go_default_library", "//vendor/github.com/davecgh/go-spew/spew:go_default_library", <ins> "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/github.com/stretchr/testify/require:go_default_library", </ins> ], )

/kind bug What happened: In , there are tests for {Default,}AllowPrivilegeEscalation fields, but those in actually test the AllowPrivilegeEscalation as this is set in the test instead of testing its suggested Default* form, thus being mostly redundant to tests in . On top of that, part of the tests in test a scenario which is not allowed, that being are wrong due to but since this is assigned directly, it was never validated and thus no errors were found. What you expected to happen: There are no redundant or incorrect tests in . Environment: Kubernetes version: master
/sig auth
Resolved by


Question: "reason":                   events.ContainerProbeWarning, }.AsSelector().String() framework.ExpectNoError(e2eevents.WaitTimeoutForEvent(  f.ClientSet, f.Namespace.Name, expectedEvent, "0.0.0.0", framework.PodEventTimeout))   f.ClientSet, f.Namespace.Name, expectedEvent, "Probe terminated redirects, Response body: Found.", framework.PodEventTimeout))  }) /*

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Used a HTTP probe with external redirect: Got warning: Kubelet logs show: Warning messages either show additional context or don't show . I would probably have expected or just Not a big deal, just a bit confusing message Apply yaml above Kubernetes version (use ): v1.22.0-beta.1 Cloud provider or hardware configuration: kind OS (e.g: ): linux Kernel (e.g. ): 5.10.40 Install tools: kind /sig node
According to See and
thanks! I am not concerned about the actual implementation behavior, but rather the event: . We should not have an event ending with as it denotes there is more information to follow, but there is not.
Got it. /assign
/triage accepted I send to fix this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
In 1.18 (because of EKS long term support I don't really want to get into), I get this: This is mildly aggravating, as it means I get fourteen lines of warning logs indicating that a service which exists solely to provide redirects is working as expected. Does this bug mean that I'll instead get an even more aggravating warning when I eventually upgrade? :) I found this bug while looking for a way to tell the probe that "getting a 308 or 302 or <insert one or a list of values heremeans success". It'd be ideal in my mind if the httpget probe allowed an "ok return values array" which accepted either single values or ranges, perhaps with a default value like .

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: ) } <ins> func TestFilterOut(t *testing.T) { type Test struct { name            string input           []string x               string expected        []string expectedChanged bool } for _, tt := range []Test{ {"nil", nil, "foo", nil, false}, {"empty", []string{}, "foo", []string{}, false}, {"foo", []string{"foo"}, "foo", nil, true}, {"aaa", []string{"a", "a", "a"}, "a", nil, true}, {"abc", []string{"a", "b", "c"}, "c", []string{"a", "b"}, true}, {"abbbcc", []string{"a", "b", "b", "b", "c", "c"}, "b", []string{"a", "c", "c"}, true}, } { t.Run(tt.name, func(t *testing.T) { got, gotChanged := filterOut(tt.input, tt.x) if !reflect.DeepEqual(tt.expected, got) { t.Errorf("expected slice %v, got %v", tt.expected, got) } if tt.expectedChanged != gotChanged { t.Errorf("expected changed %v, got %v", tt.expected, got) } }) } } </ins> func max(i, j int) int { if i > j { return i

When doing in an OpenShift cluster with a nullable field being required (compare ) and of value null, one gets:
cc
is there a way to also fix the faulty kubectl validation in 1.18.x?


Question: } return nil } <del> if ignoreFile(path, extensions) { </del> <ins> // Don't check extension if the filepath was passed explicitly if path != paths && ignoreFile(path, extensions) { </ins> return nil }

It works when I pipe but not with filename.
broke in this commit cc
Did you try to add an extension .json to the file? We filter for file extensions
ahhh ok. so working as intended. it used to autodetect :( ... thanks
That was unintended. The file extensions should only be used for deciding which files in a directory to process. For a single file, any extension should be allowed. We don't actually infer the content type from the extensions. For stdin and http sources, for instance, there are no extensions. cc
Fix is pretty straightforward (ignore file extensions when dealing with single file); I have it working with manual test, just adding unit test. Will send shortly.
Thanks, but this is v1.0-post, so no rush


Question: // claimInfoCache is a cache of processed resource claims keyed by namespace/claimname. type claimInfoCache struct { sync.RWMutex <del> state     state.CheckpointState claimInfo map[string]*ClaimInfo </del> <ins> checkpointer state.Checkpointer claimInfo    map[string]*ClaimInfo </ins> } // newClaimInfoFromClaim creates a new claim info from a resource claim.

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: start program = "/etc/init.d/kubelet start" stop program = "/etc/init.d/kubelet stop" if does not exist then restart <ins> if failed port 10250 protocol HTTP request "/healthz" with timeout 10 seconds then restart </ins>

We've run into issues like where the Kubelet was up but not healthy. The monit monitoring for Kubelet should ensure it is "healthy" rather than just alive. We need to define what healthy means in this context and monitor it based on that. An alternate option is for the Kubelet to exit when it gets into a situation it can't serve from. This may lead to crashloops so we'd need to be careful going this route. The answer is probably a bit of both.
Courtesy of healthchecking with monit:
Health check is under this umbrella issue:
Will send a PR shortly.


Question: return b.String() } <del> // RunTogether returns an error the lhs and rhs run together </del> <ins> // RunTogether returns an error if containers don't run together </ins> func (o containerOutputList) RunTogether(lhs, rhs string) error { <ins> if err := o.RunTogetherLhsFirst(lhs, rhs); err != nil { if err := o.RunTogetherLhsFirst(rhs, lhs); err != nil { return err } } return nil } // RunTogetherLhsFirst returns an error if containers don't run together or if rhs starts before lhs func (o containerOutputList) RunTogetherLhsFirst(lhs, rhs string) error { </ins> lhsStart := o.findIndex(lhs, "Started", 0) if lhsStart == -1 { return fmt.Errorf("couldn't find that %s ever started, gotn%v", lhs, o)

master-blocking: ci-node-e2e Recent failures: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is also similar. The fix for that was bumping the delay time.
Hi Vyom - would this be a good first issue because there is a previous similar solution?
I hit the same failure in my . This seems caused because expects the regular container to start before the postStart: I posted a PR() to add to the postStart. However, there are other approaches such as: Introduce as mentioned . Remove from this test case because this check does not look the primary purpose of this testcase.
The test mentioned by is similar to this flake and doesn't use . I can just remove it. I'm also going to add a comment on RunTogether to indicate it expects ordering to be enforced. That being said, shouldn't PostStart be enforced to run after the container is started? /assign
Hey PRs are more than welcome :)
I've submitted the PR, let me know if I didn't follow any best practices or anything else is required, thanks!
Skipped after was merged. Last CI run passed. /close
Closing this issue. <detailsIn response to : Skipped after was merged. Last CI run passed. </details
/reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry.
Reopened this issue. <detailsIn response to : /reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, expectedTrace: []*spanExpectation{ { <del> name: "KubernetesAPI", </del> <ins> name: "GET /api/v1/nodes", </ins> attributes: map[string]func(*commonv1.AnyValue) bool{ "http.user_agent": func(v *commonv1.AnyValue) bool { return strings.HasPrefix(v.GetStringValue(), "tracing.test")

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: func (cache *schedulerCache) NodeTree() *NodeTree { return cache.nodeTree } <ins> // GetNodeInfo returns cached data for the node name. func (cache *schedulerCache) GetNodeInfo(nodeName string) (*v1.Node, error) { cache.mu.RLock() defer cache.mu.RUnlock() n, ok := cache.nodes[nodeName] if !ok { return nil, fmt.Errorf("error retrieving node '%v' from cache", nodeName) } return n.info.Node(), nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution.

After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.